Compare commits

...
Sign in to create a new pull request.

87 commits

Author SHA1 Message Date
61bc5f423b docs(releases): the 10.4.11 note — hybrid filter-before-hydrate, one shutdown owner, a faster open
Some checks are pending
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
Gate: final tip 27759a1b vs a8c5fbf9 (10.4.10) control — collected
3,212, 0 new reds after two fix cycles (coverage-guard registration +
perf-lane classification; a real budget flake in the batch-size test
switched to unvectored items). shasum ffc33df95b2709dfcc8c67ac961991e3153f8883.
2026-09-02 14:03:28 -07:00
3835a0e702 ci(publish): allow manual dispatch — replay lane for dropped tag events
Some checks are pending
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
2026-09-02 22:51:59 +02:00
08758c254f docs(releases): the 10.4.10 note — a planner door, batched containment repair, a fixed near()
Some checks failed
CI / Node 22 (push) Successful in 12m32s
CI / Node 24 (push) Successful in 12m18s
CI / Bun (latest) (push) Successful in 12m28s
CI / Integration + conformance (Node 22) (push) Failing after 17m3s
Gate: 10.4.10 candidate (a8c5fbf9) vs 10.4.9 control (eec90bdd) —
collected 3,223/3,211, 0 new reds. shasum ffff79c5c4bcbc614545ad72e8d0138c039062e9.
2026-09-02 12:16:42 -07:00
3dadbec8f2 ci: superseded pushes cancel their own runs (concurrency per ref)
Some checks are pending
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
2026-09-02 20:56:24 +02:00
4f1e27c9a0 docs(releases): the 11.0.5 note — graph-first finds in production, bounded recovery
Some checks failed
CI / Node 22 (push) Successful in 12m14s
CI / Node 24 (push) Successful in 12m12s
CI / Integration + conformance (Node 22) (push) Failing after 17m0s
CI / Bun (latest) (push) Successful in 12m36s
2026-09-02 08:32:21 -07:00
297a3d7657 docs(releases): the 10.4.9 note — graph-first finds, honest verb arrays, bounded recovery
Some checks are pending
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
2026-09-02 08:30:01 -07:00
7ab670b525 docs(releases): the 11.0.4 note — millisecond closes, storm-free rebuilds
Some checks failed
CI / Node 22 (push) Successful in 12m21s
CI / Node 24 (push) Successful in 12m12s
CI / Integration + conformance (Node 22) (push) Failing after 17m1s
CI / Bun (latest) (push) Successful in 12m24s
2026-09-01 13:55:27 -07:00
f097cbf6f2 docs(releases): the 10.4.7 note — count ledgers can no longer race themselves
Some checks are pending
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
2026-09-01 13:49:26 -07:00
e64e2bc175 docs(releases): the release-notes door — owner-language notes for both engines, backfilled
Some checks failed
CI / Node 22 (push) Successful in 12m23s
CI / Node 24 (push) Successful in 12m20s
CI / Integration + conformance (Node 22) (push) Failing after 17m3s
CI / Bun (latest) (push) Successful in 12m20s
The fleet's releases wall reads one public URL per product. These files are
that door for Brainy and Open Brainy: newest first, honest history from the
changelog, one entry appended by every release from here on.
2026-09-01 12:04:29 -07:00
655aa13ea7 build(release): the docs-push step retires — this engine documents itself in its own repository
Some checks failed
CI / Node 22 (push) Successful in 12m22s
CI / Node 24 (push) Successful in 12m25s
CI / Integration + conformance (Node 22) (push) Failing after 16m55s
CI / Bun (latest) (push) Successful in 12m28s
The one-doc-set ruling (2026-08-31) gives soulcraft.com/docs to the paid
product alone; the site serves redirects for the slugs this rail used to
push. The push script stays in the tree as history; the rail stops calling
it.
2026-08-31 09:30:46 -07:00
39c71ecdac Merge remote-tracking branch 'origin/reclaim/packed-history-density' 2026-08-31 09:30:08 -07:00
0759c03a82 Merge remote-tracking branch 'origin/fix/torn-log-tail-terminal-verdict' 2026-08-31 09:30:08 -07:00
9a888c37e9 fix(generations): a sealed segment may only declare the generations it holds
Some checks failed
CI / Node 22 (push) Successful in 12m24s
CI / Node 24 (push) Successful in 12m21s
CI / Bun (latest) (push) Successful in 12m28s
CI / Integration + conformance (Node 22) (push) Failing after 16m55s
Diagnosis of the "packed history is damaged" narration that fires on every
run of the affected stores. It is a WRITER defect, and the reader's refusal
was the symptom rather than the cause.

A sealed segment declares one contiguous range [firstGeneration,
lastGeneration], and every reader treats that range as containment:
coveringSegment is an interval test, hasGeneration returns true for anything
inside it, and open() seeds committedRanges from it.

repackHistory handed fold() a SPARSE batch. Three filters punch holes in its
candidate list mid-run — a generation absent from committedRanges never
appears, one still in the pending buffer is skipped, one whose tx.json will
not read is skipped — and fold() then computed the range from the first and
last survivor, claiming every generation in between. The next open merged
that mis-declared range back into committedRanges, re-admitting the hole as
committed history, so the following auto-compaction pass asked the packed
tier for a frame that was never written and failed. Re-merged at every open,
which is why it repeated on every run.

Confirmed against a forensic fixture: generation directories 1..2503 present
except exactly one, 1416; and its fact-log segment already showed the tell —
seg-...1410.bfl declaring 1410..1940 (531 generations) while recording 530
facts.

Three changes:

  - repackHistory folds each contiguous RUN as its own segment
    (`contiguousRuns`), so ranges describe exactly what the segments contain.
  - fold() REFUSES a non-contiguous batch, naming the gap and its width. The
    density law is now mechanical, so no future caller can reintroduce it. A
    refusal loses nothing: the generations stay live and readable.
  - Stores already carrying the damage heal instead of wedging. A segment
    whose declared span exceeds its frame count is SPARSE; `actualRanges()`
    reads the real generation list from its sidecar so open() never re-admits
    the holes, and readFrame reports such a hole as unpacked with a narration
    naming the segment, rather than throwing. A DENSE segment missing a frame
    is still loud damage — that one means the manifest and sidecar disagree.

Pins: nine unit cases (refusal and its message, honest ranges for separately
folded runs, a reconstructed pre-fix sparse segment serving its real frames
while reporting holes as unpacked, holes excluded from actualRanges, and the
dense-segment damage path still throwing) plus an end-to-end case that
deletes a generation directory and drives the real sequence — ordinary
close()-time repacking folds over the hole, then reopen and compact must both
complete. Verified red without the fix: the segment declared an
11-generation span while holding 10 frames.
2026-08-31 09:13:42 -07:00
David Snelling
298cb6daca fix(recovery): a torn generation-log tail is a terminal verdict, never a wait
Some checks failed
CI / Node 22 (push) Successful in 12m22s
CI / Node 24 (push) Successful in 12m21s
CI / Integration + conformance (Node 22) (push) Failing after 16m58s
CI / Bun (latest) (push) Successful in 12m23s
Two halves of one defect, found by a seeded-SIGKILL crash lane.

THE FALSE POSITIVE. stampEntityTree() recorded generationStore.generation()
— the ALLOCATED counter, a number a write in flight has claimed and may
never commit — while the JSDoc beside it already said the source is the
committed generation. Every crash inside a write window therefore produced
a spurious verdict at the next open: either 'sourceGeneration N is ahead of
the log head N-1' (the allocated generation died with the process) or
'rollup invariant nounCount: stamped X, observed Y' (the recovery fold
folded facts the stamp's counts predate). Both told the operator to run
repairIndex() — a whole-store recount — for a store that was coherent.
Measured before this commit: 4 of 11 SIGKILL cycles on a healthy store
raised one of the two. The stamp and the open now both read
committedGeneration(), which is what every other open-time watermark in the
class already reasons about.

THE TERMINAL VERDICT. A stamp still ahead of committed truth after the
recovery fold witnesses a generation that is not in the log — the stamp's
fsync outlived the tail's, and there is nothing to arrive. That is its own
verdict state now ('torn'), never folded in with 'incoherent': the two have
opposite cures. A writer open demotes it — the unusable stamped surface is
re-derived at the committed generation from the live counters, O(1),
straight-line, no loop and no await on external progress, narrated with
both count sets, the stamp's path and its committedAt. A read-only open
cannot re-stamp, so it says so and names the cure instead of guessing, and
still serves. Neither branch waits, and neither locks an owner out of a
canonical tree the stamp only describes.

Pins: the verifier returns the torn verdict with both generations; a
fabricated head-behind-source store narrates precisely, demotes inside a
bounded open, serves its rows, and is quiet at the next open (the demotion
converges); a read-only open narrates the same verdict and leaves the bytes
untouched.
2026-08-31 09:07:18 -07:00
b8475cc86a fix(release): the release page posts to this repository — soulcraftlabs/open-brainy, never the engine's
Some checks failed
CI / Node 22 (push) Successful in 12m21s
CI / Node 24 (push) Successful in 12m12s
CI / Bun (latest) (push) Successful in 12m25s
CI / Integration + conformance (Node 22) (push) Failing after 17m2s
Step 11 POSTed to repos/soulcraft/brainy while printing the correct URL; dormant only because FORGEJO_RELEASE_TOKEN was unset. Found during the 10.4.4 cut verification.
2026-08-28 13:00:42 -07:00
ff39941b0a chore(release): 10.4.4
Some checks failed
Publish (The Source) / Publish to The Source registry (push) Successful in 12m27s
CI / Node 22 (push) Has been cancelled
CI / Integration + conformance (Node 22) (push) Has been cancelled
CI / Node 24 (push) Has been cancelled
CI / Bun (latest) (push) Has been cancelled
2026-08-28 12:39:55 -07:00
d49148e140 fix(vfs): the old-root sweep narrates only when it has something to say
The release gate's own output caught this: every test brain printed
"[VFS] old-root sweep complete in 1ms and recorded" — hundreds of lines — and
a consumer would get two of them on the first open of every store.

They were emitted on the always-visible channel, which a production log level
deliberately CANNOT silence. That channel exists so an operator can always
learn why a database is slow; a 0ms no-op on a fresh store is not that, and
announcing it there trains people to ignore the one channel built to be
impossible to ignore. It was also inconsistent with every other narration in
this work, all of which is silent under a threshold.

The sweep now speaks when it has something to say — duplicate roots removed, or
a wall over a second that a person watching a slow first open deserves
explained — and otherwise does its work, records its marker, and stays quiet.
cleanupOldRoots() reports what it removed so the decision rests on a fact
rather than on a guess.

Pin: a fresh store's sweep emits nothing on the channel and still records its
marker, so the silence can never be mistaken for the work being skipped.
2026-08-28 12:30:30 -07:00
42e2da259b fix(tests): the health-gate pin follows the verdict, and the VFS suite uses its own store
Two gate failures on main, one real and one long-hidden.

THE HEALTH-GATE PIN encoded the old law — "narrates once per generation, twice
across a generation bump" — which the content-keyed dedupe deliberately
replaced. A provider's `generation` bumps on every ledger mutation and every
rebuild boundary, so keying narration on it re-printed an unchanged health line
on every read that consulted a busy provider, and let a provider that never
bumped suppress a line whose reasons had genuinely changed. The pin now asserts
BOTH directions: an unchanged verdict stays silent however the counter moves,
and a changed verdict is always heard.

THE VFS HYBRID-SEARCH SUITE configured its store with `options.basePath`, an
alias removed at the 8.0 major that configures nothing. The suite was therefore
never using its temp directory — it opened the DEFAULT store, shared with every
other run on the machine, and accumulated tens of thousands of rows until it
failed on that shared store's graph adjacency instead of on anything it tests.
It now passes `storage.path`. The suite drops from 6.5s to 0.3s, which is the
measure of how much foreign data it had been opening.

Neither failure was caused by the release branch; the first is the branch's own
behaviour change meeting its outdated pin, the second predates it.
2026-08-28 12:27:40 -07:00
5ebd3b4061 Merge branch 'next/open-lazy-open-and-counts'
Correctness and observability for 10.4.4: the writer-lock clean-close record
and the always-run terminal releases, the self-healing count ledger and an
atomic counts.json, always-visible open and repair narration, the non-blocking
open for a provider rebuilding itself, an event-driven flush-request watch, and
the operator-visible operator set (three served, four refused by name).
2026-08-28 12:09:20 -07:00
a8c724a202 docs: the contract manifest stands alone; public docs describe this engine only
Some checks failed
CI / Node 22 (push) Successful in 12m18s
CI / Node 24 (push) Successful in 12m17s
CI / Integration + conformance (Node 22) (push) Failing after 17m7s
CI / Bun (latest) (push) Successful in 12m19s
The manifest's prose pointer named a document that answers a confidential
specification, and such a document does not belong in a public repository even
in summary. The pointer is dropped — the manifest is generated from this
engine's own surface and is self-describing — and the requirement marking it
deliberately omits is recorded with the contract's owner rather than here.

The standard is written down so this is not relitigated per document.
2026-08-28 12:08:19 -07:00
61a469270e docs(releases): 10.4.4 consumer notes — correctness and observability, with the performance line stated exactly 2026-08-28 12:05:46 -07:00
02c6163637 docs: measurements in public history carry numbers, not provenance
A release audit found hostnames, store identities and operational anecdotes in
this branch's commit messages — not trade secrets, but nothing a public
repository's permanent history should carry either. The messages were rewritten
to keep every number and drop every provenance; the rule is written down here
so the next measurement does not have to be caught by an audit.
2026-08-28 12:04:49 -07:00
2cf3801007 feat(open): name the two steps that hold the vfs-bootstrap phase
MEASURED on a 14,056-noun / 72,679-verb production-shaped store, measured
solo under an exclusive lock: the vfs-bootstrap
phase costs 37.8s on main and 38.0s on this branch — unchanged — and NO
"vfs.init" step line was emitted at all, meaning the VFS's own init fell under
the 2s narration threshold. The phase is therefore almost entirely NOT the VFS,
and the old-root sweep this branch moved to the background was never what made
it expensive.

What else lives in that span is now named: the log-authority artifact read, the
adoption ORACLE (which verifies the log against canonical before flipping a
brain to durable-at-ack), the legacy pending-embed sidecar bridge, and the
pending-embed recovery fold. One of those holds ~38 seconds of every open of
this store and the next measurement will say which, by name, instead of leaving
a phase label to be guessed at.
2026-08-28 11:56:26 -07:00
5c22f9500c fix(storage): a dead flush watch falls back to the 500ms poll, not the 30s sweep
The safety sweep is armed alongside the watch, and startFlushRequestPolling()
declines to arm over an existing interval — so when a watch died mid-life the
fallback did nothing and the store quietly answered flush requests on a 30s
cadence instead of the 500ms one the door promises. The sweep is cleared first.
A degrade nobody asked for is still a degrade.
2026-08-28 11:31:57 -07:00
16d2e1a97e fix(storage): the flush watcher cannot arm twice in its async window
Arming is asynchronous — the request directory is created before it can be
watched — so during that window neither the watcher nor the sweep interval
exists yet and the guard let a second call through, leaving two watchers and
two sweeps for the life of the store. The callback is the flag that covers the
window.
2026-08-28 11:30:00 -07:00
fb1da1c56d perf(idle): the flush-request watch is event-driven; the heartbeat is observability
Three idle-burn items from the steady-state audit, and one correction.

THE FLUSH-REQUEST WATCH (the strongest of them). It readdir'd the request
directory every 500 ms, per brain, for the life of every writer — armed on
every non-reader brain whether or not any inspector process existed. In a process holding many stores that is tens of directory reads per second
on a completely idle service, plus a stale-request GC on every one of them. It now
uses fs.watch, so the arrival itself wakes it and a request is seen SOONER
than the poll saw it. Two concessions ride along, both stated in the code: a
30s safety sweep (fs.watch drops events on some network and fuse filesystems,
and the GC needs a tick of its own — two orders of magnitude fewer reads than
the poll made), and a fall back to the original 500 ms poll, narrated, on a
filesystem that cannot watch at all, because an inspector whose request is
never seen waits forever.

THE WRITER HEARTBEAT goes 10s → 60s. It is observability ONLY — staleness is
decided by pid liveness and the fence compares pid + hostname, so no decision
anywhere reads the timestamp — and at 10s it was a lock-file write every ten
seconds per brain forever, for a value nothing computes with. An operator
still sees a heartbeat inside the minute.

THE HEALTH NARRATION dedupes by CONTENT, not by the provider's generation
counter. That counter bumps on every ledger mutation and rebuild boundary, so
a provider bumping it on routine work re-emitted the same unchanged line on
every read, while one that never bumped could suppress a line whose reasons
had genuinely changed. The generation is still reported; it no longer decides
whether the line is worth saying.

CORRECTION, and it is against my own earlier claim: the idle-flush commit read
a reported idle-CPU observation (many stores, no writes, a flush every ~35s,
over a core burned) as caused by
the flush path. That does not follow — this engine's cadence is write-driven
(every trigger runs through noteWriteForPersistence, which only a committed
write calls), so something was CALLING flush() on those brains and the caller
is still unidentified. The clean-flush gate makes such a call free; it does not
account for it. The code comments and the idle lane now say exactly that.

Pins: tests/integration/flush-watcher-event-driven.test.ts — an idle writer
makes at most one request-directory read in 8 seconds (the old poll made ~16),
and a dropped request is still acked well inside the safety sweep.
2026-08-28 11:25:47 -07:00
417ddb5143 perf(open): answer "are there any entities?" with one directory read
The 7.x-to-8.0 layout probe runs on the open path of every store that does not
yet carry its completion marker — a restore, a store built by an older release
— and asked whether the canonical tree holds anything by LISTING it: a
recursive walk of every file in every entity directory, to learn a boolean. It
now asks the one-level door added for generation discovery, falling back to the
listing on an adapter that lacks it.

Also files a defect found while ratifying the operator set: the VFS builds its
path-prefix filter as `$startsWith`, an operator no engine spelling accepts,
so vfs.searchFiles({ path }) throws INVALID_QUERY on every call that passes a
path. Pre-existing, unrelated to the operator work, and left as a filing —
a path-prefix search needs a design answer, not a spelling correction.
2026-08-28 11:13:24 -07:00
9dd399216b perf(generations): discover generations by directory name, not by walking the log
MEASURED on a production-shaped store (14,056 nouns / 72,679 verbs, an 11 GB
generation history), measured solo under an exclusive lock: the generation-store phase cost 55,538 ms of a WARM
REOPEN after a clean close — with the fold correctly skipped, so nothing in
that phase's name explained it.

This is what it was doing. Discovering which generations exist on disk called
listRawObjects('_generations'), which RECURSES the whole tree and returns
every file in every generation directory — to extract a set of integers that
the top-level directory NAMES already spell out. The cost scales with the
entire history, is paid on every open, warm or cold, and grows for the life of
the store.

A one-level door — listRawPrefixes(prefix), the immediate child directory
names — is added to the storage seam. The filesystem adapter answers it with a
single readdir; BaseStorage derives it from the recursive listing, so an
adapter without a cheap implementation is never wrong, only never faster; and
the generation store falls back to the old listing when the door is absent.

One behavioural difference, stated: an EMPTY generation directory is now
discovered where the file listing could not see it. Above the committed
watermark that is a crash scar, and recovery already has an explicit branch
for it ("indeterminate partial dir" — dropped, narrated). Below it, it becomes
a resolvable generation holding no records, which is what an empty generation
means.

Suites: the durability kill matrix (15), db-mvcc (30), history repacking (4),
rollback trapdoor (3), entity-tree stamp (4) and the full unit suite (2,105)
all green.
2026-08-28 11:09:05 -07:00
e4c27fbca8 fix(flush): clear() and repairIndex() set the dirty witness themselves
Both mutate durable state outside the two commit paths, so neither was seen by
the flush witness added with the idle-flush law. A clear() followed by a
flush() would have found the brain "clean" and skipped the entity-tree stamp,
leaving a stamp describing the population the clear had just removed — a false
divergence warning at the next open. Closing the gap where it is, rather than
widening the witness to guess.
2026-08-28 11:06:06 -07:00
5a091ccad9 feat(open): the open names the STEP that cost the time, not just the phase
A phase that costs a minute and names only itself tells an operator where to
look but not what to look at. MEASURED on a real 14,056-noun / 72,679-verb
store, the warm reopen's generation-store phase cost 55,538 ms with nothing
inside it named — the fold was skipped (the close was clean), so the cost was
somewhere else entirely and the breakdown could not say where.

Six steps inside the open now report their own wall with their own cause when
they exceed the phase threshold: the generation store's open (manifest,
committed ranges, fact log, packed tier, crash replay), the entity-tree stamp
verification, the brain-format read, the pre-upgrade backup, the derived-index
gate, and the VFS init. Silent under the threshold, so a fast open says nothing
extra. Same always-visible channel as the phase lines.
2026-08-28 11:02:57 -07:00
4a67aa0fb9 perf(vfs): the old-root sweep runs once per store, not once per open
MEASURED on a 14,056-noun / 72,679-verb production-shaped store, measured
solo under an exclusive lock: the vfs-bootstrap phase cost 43,021 ms of a cold open and 52,696 ms of
a WARM REOPEN. What dominates it is a migration sweep — a filtered find() over
the whole store hunting for root directories created before the fixed root id
existed. A store either carries such duplicates or never will, and the sweep
ran on every open, forever, in the foreground.

It is now caused by the store's state instead of by the open count: a durable
marker under _system/ records that the sweep has run, and a store carrying it
never sweeps again. A store without one sweeps in the BACKGROUND — the sweep
only removes duplicate roots, nothing serves from them, and it was already
declared non-critical — narrated at both ends, with whenRootSweepSettled() for
anyone who needs to observe rather than race it. An adapter with no raw-object
door keeps the old behaviour: correctness over cost, never a silent skip.

Pins: tests/integration/vfs-root-sweep-once.test.ts — the sweep runs on the
first open and never on the second or third; a sweep slowed to 4s does not
delay the open.
2026-08-28 11:01:43 -07:00
c1f0972395 chore: keep the generated neural stamps at main's values
The build regenerates these from the git commit time; a local rebuild moved
only the stamp. Restored so the branch carries no incidental churn.
2026-08-28 10:58:00 -07:00
48802ba385 feat(contract): declare contract 1, serve three operators, refuse four by name
Open Brainy's side of the API contract the accelerated engine published.

DECLARED: package.json carries "brainyContract": 1 and the engine states its
own via contractVersion() / BRAINY_CONTRACT_VERSION — two engines compare an
integer instead of probing prototypes, and a tool reads the package field
without importing the engine. Pinned so the two can never drift apart.

SERVED: hasAll, noneOf and excludes now work on the index path. The defect
underneath was worse than the reported divergence — the metadata index's
operator switch had NO DEFAULT CASE, so any operator without a case left the
field's match set at its initial [] and find() returned an empty page.
Documented, validator-accepted, matcher-implemented operators answering
silently wrong. hasAll intersects each element's posting set (an empty operand
is vacuously true of every row that has the field), noneOf complements their
union, excludes complements contains.

REFUSED BY NAME: startsWith, endsWith, matches and length raise
INVALID_QUERY naming the operator, the field and the reason. An equality/range
posting index cannot evaluate a substring, a pattern or an array length without
reading every row — which is the cost this path exists to avoid — so it refuses
rather than answering an empty page. Both engines now agree on all 25 tokens
and contract 1 has no remaining operator divergence. This is a visible change
for a consumer calling those four through find({ where }): an empty page
becomes a typed refusal.

EMITTED: scripts/emit-contract-manifest.mjs generates docs/api-contract.json
from the BUILT surface — prototype doors, exported error classes, the operator
sets read out of their single definitions, the field-addressing vocabulary, the
health verdicts. Nothing hand-maintained, so a diff between two manifests is a
diff between two engines. `--check` fails on a stale manifest, which makes the
announce-every-addition duty mechanical rather than remembered.

RATIFIED in docs/contract-1-ratification.md: the 41-of-57 required split with
the promise spelled out (a refusal is part of a door; deprecation is not
removal), the serving-withholding list confirmed exhaustive and identical, the
minor/major rule adopted with the announcement duty, the 30 storage seam
methods committed as supported surface until Stage 2, and a finding filed
against the spec — is / isNot / greaterEqual / lessEqual are listed there as
served aliases and have never existed in this engine, which throws
INVALID_QUERY on all four.
2026-08-28 10:57:43 -07:00
50676c02f4 fix(open): a provider rebuilding itself is a third state, not a CRITICAL
Follower to the self-rebuild deference. The open gate's consistency check —
"metadata index has 0 entries but storage has N entities" → CRITICAL + a forced
second rebuild — knew two states, migrating and not. A provider whose rebuild()
returns once the rebuild is OWNED AND RUNNING ONLINE (its doors refusing by
name while other families serve) legitimately reports 0 entries there, so every
first contact printed a false CRITICAL and kicked a redundant second rebuild.

The exemption rides the rebuild-progress hook, NOT isMigrating() — widening
that would hold every write and 503 the whole brain through the migration
snapshot, which is worse than the false alarm. The check's real class is
untouched: a provider reporting 0 entries with no rebuild in progress still
trips it.

The crash-recovery rebuild kick gets the same deference: a provider already
rebuilding itself from canonical is doing exactly that work, and the fold ran
in the generation store's open before any provider existed, so what it is
reading is the repaired canonical.

Pin: a provider stub reporting a rebuild and 0 entries opens with no CRITICAL
line and no second rebuild; the vacuous-stub case fails loudly.
2026-08-28 10:50:26 -07:00
131daa08cd feat(open): open never waits for a provider that is rebuilding itself
MEASURED on a production store: a metadata provider that had to rebuild made
init() pay the ENTIRE rebuild on the foreground — 641 seconds — with every
other family idle behind it. The cause is a missing distinction: a provider
reporting serving:false because it is BUSY BUILDING ITSELF and one reporting
serving:false because it is BROKEN looked identical through healthReport(),
and both were answered the same way — call rebuild(), and wait for it.

The contract that tells them apart is one optional, synchronous, O(1) hook:
`rebuildInProgress(): ProviderRebuildProgress | null`, reporting a phase name
and whatever the provider actually measures (done/total/startedAt) — never an
estimate dressed as a fact. A provider without the hook behaves exactly as
before.

With it, a provider owns its own rebuild:
- the open gate neither starts a second rebuild nor waits for the provider's,
  and narrates that it is not waiting and what will refuse meanwhile;
- init() returns and every other family serves;
- that family's doors refuse BY NAME, carrying the provider's own progress,
  and say plainly that the door opens by itself and no action is needed —
  distinct from a broken index, which names repairIndex();
- the epoch stamp does not advance while any family is still being built.

Nothing is ever served empty: a not-serving family refuses, as it already did.

Pins: tests/integration/open-does-not-wait-for-a-rebuilding-provider.test.ts —
init() returns in milliseconds against a provider claiming a 6s rebuild, brainy
starts no rebuild of its own, a filtered read refuses naming the phase and the
4,096/14,056 progress, and the door answers once the provider reports serving.
The pin fails loudly rather than vacuously if its stub never installs.
2026-08-28 10:48:52 -07:00
f5a6cb3f61 perf(flush): an idle brain does no work — no periodic flush without a write
REPORTED from the field: a process holding many stores, with no writes for ten
minutes, printed "All indexes flushed to disk in 216-601ms" per store every
~35 seconds and burned over a core at idle. Every one of those flushes re-persisted
state identical to what was already on disk — the provider flushes, the
watermark stamps, the generation counter, the entity-tree stamp — because
flush() never asked whether anything had changed.

- flush() over a clean brain is now O(1) and silent: a dirty witness is set by
  every committed write (both commit paths end at noteWriteForPersistence, and
  the deferred-embed worker lands through the single-op path) and cleared by a
  flush that runs. A write landing DURING a flush sets it again, so no write's
  work is ever skipped — it is done by the next flush. Set before the policy
  check, so a `'manual'` consumer's explicit flush is never a no-op it didn't
  ask for.
- An explicit flush now tells the cadence it happened. It didn't, so the very
  next write saw "30s since the last flush" and kicked a background flush with
  nothing to do, and the idle timer fired two seconds later over writes the
  explicit flush had already persisted.
- The graph adjacency index's auto-flush asks before it acts: two O(1) reads
  of the LSM MemTables, and a tick over a quiet index returns without calling
  into the trees at all.

assessProviderHealth is NOT timer-driven — it is a synchronous O(1) read of a
provider's own healthReport(), called on the read gate, so it costs nothing on
an idle brain. No change needed there.

Pins: tests/integration/idle-costs-nothing.test.ts — 90 idle seconds produce
zero flushes, zero provider calls and zero log lines; three explicit flushes
over a clean brain call no provider; one write earns exactly one flush.
2026-08-28 10:44:38 -07:00
3fffd9c6e6 feat(repair): repairIndex narrates every phase and its receipt carries the walls
Some checks are pending
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
On a production store (14,647 nouns / 73,070 verbs) a repairIndex() ran for
more than thirty minutes at roughly a full core with ZERO log lines between
its start and its end, while the read doors kept serving. The operator could
tell it was alive only from `top`, and could not tell which of its
single-threaded walks it was inside.

Same law as the open, applied to the repair:
- every phase announces itself BEFORE it works, naming what it is about to
  walk (each canonical walk, the VFS containment reconciliation, each
  provider's invariant pass);
- an unref'd heartbeat names the phase still running every 5s, for as long as
  it runs;
- every phase reports its own wall, and that wall is carried in the TYPED
  receipt as RepairFamilyReport.durationMs — a receipt that cannot say where
  the time went is not a receipt;
- the whole repair's narration moves to the always-visible channel, so a
  production log level cannot silence it.

The phases move into runRepairIndexPhases() so the heartbeat can live in a
finally around them; the public door and its report shape are unchanged apart
from the added durationMs.

Pins: tests/integration/repair-narration.test.ts — every checked family has a
start line, a finish line with its wall, and a numeric durationMs in the
receipt; a phase slowed to 6.5s produces a heartbeat naming it, with the
logger clamped to ERROR.
2026-08-28 10:31:42 -07:00
f4e2d34b4e fix(storage): a suspect count ledger heals itself, and counts.json is written atomically
Some checks are pending
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
MEASURED on a real store: the ALL-visibility ledger read 14,231 nouns against
14,056 identity records and 72,729 verbs against 72,679 — exactly that store's
25 noun and 50 verb SCAR directories. Two copies of the same archive derived
different numbers (14,231 and 14,081), because each had been persisted at a
different moment under the old rule that counted one entity per id DIRECTORY.
A downstream index heal subtracted against that denominator and reported
remaining work that did not exist.

The scan already applies the right predicate — one entity per IDENTITY RECORD
(the metadata content leg), shared with pruneOrphanedEntities so the two agree
by construction. What was missing is that a ledger persisted under the old rule
was only FLAGGED suspect and then went on serving its wrong numbers for the
life of the store, waiting for an operator to run repairIndex.

- The ledger now derives itself honestly in the BACKGROUND after the open,
  narrating start and finish with the correction it made. Background because
  these scalars are denominators — no read is served from them — and because
  walks exactly like these are how a 24,898-id store spent minutes of a
  restart in silence. Observable via whenCountLedgerSettled(); nothing in the
  read path waits on it.
- A derivation that raced a write refuses to stamp its number "exact": one
  retry on a quiet store, then the ledger stays SUSPECT and says so, naming
  repairIndex as the door that recounts under a barrier.
- The one derivation that CANNOT leave the foreground says why it cannot:
  getNounCount()/getVerbCount() are served from it, and a background walk
  would make a populated store answer "0 entities" — a wrong answer, not a
  slow one. It narrates its start and its wall instead.
- counts.json is written temp+rename. A truncating write left a window —
  measured at roughly 750ms after a flush or close — in which a concurrent
  reader saw the file EMPTY; an unparseable ledger sends the next open down
  the full-rescan path, so the cheapest file in the store was buying the most
  expensive recovery.
- The writer lock's clean-close record is now consulted before the
  same-process branch too: a restart reported "Re-acquiring writer lock ...
  this is a bug" immediately after a clean close, sending an operator after a
  leak that did not exist.

Pins: tests/integration/count-ledger-identity-record.test.ts (background
correction with scar and ghost fixtures, two copies of one archive agreeing,
counts.json never observed unparseable across 40 persists);
tests/integration/ledger-derivation-identity.test.ts updated to the new law —
the OPEN still never walks (proved by slowing the walk 1.2s and timing the
open), and the ledger heals behind it.
2026-08-28 10:28:25 -07:00
afe08a1ff9 feat(open): the open narrates itself, on a channel production cannot clamp
Some checks failed
CI / Node 22 (push) Has been cancelled
CI / Node 24 (push) Has been cancelled
CI / Integration + conformance (Node 22) (push) Has been cancelled
CI / Bun (latest) (push) Has been cancelled
An operator watched a production service open a 16 GB store and print nothing
for three minutes before its first line of work. Two defects, both fixed here.

The narration was written to `prodLog.warn`, and every environment that looks
like production clamps the logger to ERROR — so the phase breakdown that would
have named the slow phase was composed and thrown away. `prodLog.narrate` is
always visible, like `error`: it carries the two things an operator is
entitled to hear from a database regardless of a cost setting — why it is slow
and what it is doing about it. `silent: true` still silences it; that is a
request, not a default.

And nothing spoke DURING a phase, only after the whole open. init() now runs
an unref'd heartbeat that every 5s names the phase currently running, its
elapsed wall and what it is paying for, plus one line per phase as it ends for
any phase over 2s. The generation-log fold's own progress and completion lines
move to the same channel and now carry their wall — they were invisible in
production, which is how an operator came to restart a converging fold three
times.

Pins: tests/integration/open-narration.test.ts — narrate() survives the clamp
that silences warn(); a 6.5s storage-init produces a heartbeat naming the
phase and a completion line naming its wall, with the logger clamped to ERROR.
2026-08-28 10:19:55 -07:00
e652162c1f fix(storage): a clean close is recorded, and the writer lock is always given up
Some checks are pending
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
A production restart made this necessary: a service stopped with exit code 0,
having awaited close() on every pooled brain, and its next boot announced
"Overwriting stale writer lock ... appears dead" for every store it owned.
Nothing had crashed. "The recorded pid is gone" is equally true of an orderly
restart and of a crash, so the verdict could not tell an operator which one
they had — and when the OS recycles a pid it fails the other way, refusing to
open a store whose writer died days ago.

Three changes, all at the law:

- close() is two parts, and the second is unconditional. The durable steps
  (flush, markers, component close, plugin deactivate, buffer drain) move to
  closeDurableSteps(); the terminal releases — the flush-request watcher, the
  WRITER LOCK, the VFS timers, the terminal `closed` flag — always run. The
  original failure is narrated with what it costs the next open, then rethrown.
- releaseWriterLock() writes a CLEAN-CLOSE RECORD (`locks/_writer.close`)
  naming the lock generation it released; the next claim consumes it, so a
  record can never vouch for a later crash. An open reads the record instead
  of guessing: recorded → nothing to recover; absent → say so, and name the
  crash recovery this open will now run.
- The signal path stops failing in a batch. It was one try around a loop over
  every open brain, so the first instance whose flush rejected stranded every
  remaining brain's lock and markers — at exit code 0. Now: per-instance
  isolation, the generation store's close (the clean-shutdown marker, without
  which the next open folds the whole log) is part of shutdown, the lock is
  given up in a finally, and the handler no longer calls process.exit() when
  the host application has its own signal handler — that race truncated the
  host's own close() mid-flight.

Pins: tests/integration/writer-lock-clean-close.test.ts — completed close
leaves no lock and a consumed-once record with a silent reopen; a failing
durable step still releases and still rethrows; SIGKILL leaves the lock with
no record and the reopen names the crash; a host SIGTERM handler runs to
completion.

Branch plan (10 lines):
 1. writer lock: clean-close record + always-release close  [this commit]
 2. open narration: an always-on channel; production clamps prodLog to ERROR,
    which is why a three-minute open printed nothing
 3. open narration: per-phase lines as each phase ENDS, with progress cadence
 4. measure both real-store fixtures on the box, before/after
 5. move the generation-log fold out of the foreground where the serving law
    allows; durable resumable progress marker
 6. same for the VFS bootstrap
 7. counts: a legacy container-rule ledger must not keep serving wrong
    denominators; counts.json written atomically
 8. counts pin with scar directories; two copies of one archive agree
 9. docs/canonical-layout-ratification.md — 12 facts confirmed/corrected
10. report: MEASURED before/after, findings, and whether this is 10.4.4
2026-08-28 10:17:20 -07:00
38c3397b60 docs: repository links point at soulcraftlabs/open-brainy — the soulcraft/brainy path becomes the native engine's repo tonight
All checks were successful
CI / Node 22 (push) Successful in 12m20s
CI / Node 24 (push) Successful in 12m15s
CI / Bun (latest) (push) Successful in 12m24s
CI / Integration + conformance (Node 22) (push) Successful in 19m54s
2026-08-27 17:26:44 -07:00
384f4b6b9c chore(release): 10.4.3
Some checks failed
CI / Node 24 (push) Successful in 12m32s
CI / Node 22 (push) Successful in 12m34s
Publish (The Source) / Publish to The Source registry (push) Successful in 12m49s
CI / Bun (latest) (push) Has been cancelled
CI / Integration + conformance (Node 22) (push) Has been cancelled
2026-08-27 17:10:30 -07:00
a58372f03f Merge branch 'next/open-brainy-rename'
Some checks failed
CI / Node 22 (push) Has been cancelled
CI / Integration + conformance (Node 22) (push) Has been cancelled
CI / Node 24 (push) Has been cancelled
CI / Bun (latest) (push) Has been cancelled
2026-08-27 17:08:30 -07:00
a99b1e83c4 chore: rename to @soulcraftlabs/brainy for Open Brainy on The Source
Prepares the repo for its new home at soulcraftlabs/open-brainy ahead
of the Forgejo transfer: package name, publish registry, release
script, and every install/import reference across docs, src, tests,
examples, and integrations now point at @soulcraftlabs/brainy on
The Source. The npmjs storefront leg and byte-identity pair
verification are stripped from the release script — The Source is
now the only publish target. README gains an Open Brainy explainer
and a registry note for consumers.

@soulcraft/brainy 10.4.2 was the last release under the old name.
2026-08-27 17:07:09 -07:00
9f248b2495 docs(releases): 10.4.3 — Open Brainy's first release under the new name, same engine as 10.4.2; The Source is the one registry
Some checks failed
CI / Integration + conformance (Node 22) (push) Has been cancelled
CI / Node 24 (push) Has been cancelled
CI / Node 22 (push) Has been cancelled
CI / Bun (latest) (push) Has been cancelled
2026-08-27 16:56:45 -07:00
1f34fc6ce4 chore(release): 10.4.2
All checks were successful
Publish (The Source) / Publish to The Source registry (push) Successful in 12m36s
CI / Node 22 (push) Successful in 12m15s
CI / Node 24 (push) Successful in 12m30s
CI / Integration + conformance (Node 22) (push) Successful in 19m42s
CI / Bun (latest) (push) Successful in 12m20s
2026-08-27 15:14:03 -07:00
a082e0efdd docs(releases): 10.4.1 and 10.4.2 consumer notes; 10.4.2 is the last MIT release under this name, Open Brainy continues at @soulcraftlabs/brainy
Some checks failed
CI / Node 22 (push) Successful in 12m27s
CI / Node 24 (push) Successful in 12m20s
CI / Integration + conformance (Node 22) (push) Failing after 14m54s
CI / Bun (latest) (push) Successful in 12m25s
2026-08-27 15:07:25 -07:00
6f93108648 chore(release): 10.4.2-rc.1
All checks were successful
Publish (The Source) / Publish to The Source registry (push) Successful in 12m27s
CI / Node 22 (push) Successful in 12m20s
CI / Node 24 (push) Successful in 12m16s
CI / Bun (latest) (push) Successful in 12m23s
CI / Integration + conformance (Node 22) (push) Successful in 19m42s
2026-08-27 14:47:48 -07:00
9b84ef5b02 Merge branch 'next/zero-norm-unvector-door'
All checks were successful
CI / Node 22 (push) Successful in 12m24s
CI / Node 24 (push) Successful in 12m6s
CI / Bun (latest) (push) Successful in 12m32s
CI / Integration + conformance (Node 22) (push) Successful in 19m41s
# Conflicts:
#	src/hnsw/hnswIndex.ts
2026-08-27 13:54:06 -07:00
0de7665930 fix(vectors): a zero-norm vector is not a vector, canonical side included, plus the sanctioned unvector door
Some checks failed
CI / Node 22 (push) Successful in 12m22s
CI / Node 24 (push) Successful in 12m15s
CI / Integration + conformance (Node 22) (push) Failing after 14m49s
CI / Bun (latest) (push) Successful in 12m28s
The engine-pair seam law: a zero-norm vector never crosses an engine
boundary. The index belt already refused to insert one, but the canonical
write and the vectored-noun ledger still counted it, so a near-empty store
whose only vectored row was zero-norm read "1 canonical vectored vs 0
indexed" and threw a not-ready error at open, and a store's own legacy
zero-norm VFS root could trip the same gate before its VFS-init-time cure
ever ran.

- add()/update() (single and transact()) now normalize an explicit
  real all-zero vector to the unvectored [] shape before the dimension
  pin, the ledger flag, and the index ops ever see it (loud, one warn per
  write, canonical write still succeeds).
- The legacy counts.json derivation walk (scanVectoredNounCount) excludes
  a persisted zero-norm row, matching the live ledger's definition.
- A legacy zero-norm VFS root now migrates at open, before the vector-leg
  gate evaluates, via one O(1) fixed-path read (torn-tolerant — skips
  rather than aborting init on a torn root, letting the recovery walk
  heal it) — independent of whether a VirtualFileSystem is ever
  constructed this session.
- update({ id, vector: [] }) (and the same op inside transact()) is now
  the sanctioned, idempotent unvector door: index removal, exactly-once
  ledger decrement, no re-embed, and it clears a pending deferred-embed
  marker rather than leaving it to re-vectorize the row later. The
  combination with deferEmbedding is a typed refusal.
- JsHnswVectorIndex.rebuild() now skips a zero-norm/empty persisted
  vector when repopulating from canonical (the same belt the live
  add/replace paths already had), and health()'s index-parity check now
  compares HNSW size against the vectored-noun ledger rather than the
  raw metadata-entry count, since a store's VFS root is permanently
  unvectored by design.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-27 13:53:09 -07:00
8fc553b126 fix(hnsw): skip unvectored rows on rebuild; refuse empty vectors in the index
Some checks failed
CI / Node 22 (push) Successful in 12m14s
CI / Node 24 (push) Successful in 12m15s
CI / Integration + conformance (Node 22) (push) Failing after 14m55s
CI / Bun (latest) (push) Successful in 12m24s
A canonical row persisted with vector: [] (a system row, a deferred embed
not yet landed, or any other legitimately-unvectored record) is a normal,
enumerable row -- but rebuild()'s storage walk had no guard against it.
storage.getVectorIndexData() derives its answer from the row's own record,
so it returns non-null for any existing noun whether or not that noun was
ever actually indexed -- rebuild() admitted such rows into the live graph
with a length-0 vector. A vector-less node could become the entry point (or
occupy any graph position); the next real insert then ran a distance
calculation against it and blew up with a dimension mismatch.

Fix at two layers in src/hnsw/hnswIndex.ts:
- rebuild() now skips any row whose vector.length === 0 before it ever
  becomes a graph node (one summary count line, never per-row spam), and
  restores the pinned dimension from the first real vector it loads --
  previously the pin stayed null across a restart, since addItem/updateItem
  are the only sites that set it and rebuild() never goes through either.
- addItem/updateItem now refuse a length-0 vector with a typed
  EmptyVectorIndexError instead of ever pinning dimension to 0 or storing a
  vector-less node, so no future fill/rebuild/load path can poison the index
  silently. getVectorSafe's lazy-load "not found" check also missed that an
  empty array is truthy -- tightened to catch it.

IndexOperations.ts's ReplaceInVectorIndexOperation rollback paths now skip
re-adding an oldVector of length 0 (never a legal index member) instead of
attempting an illegal empty re-insert on rollback.

biography.test.ts's final ledger-exactness assertion assumed every noun the
lane creates is vectored, including the VFS root counted in
vfsBaselineNouns -- but the root is deliberately persisted unvectored.
Corrected the expected formula to exclude it.

Adds tests/integration/index-skips-unvectored.test.ts pinning: rebuild()
indexes only vectored rows with the dimension pinned correctly; clear()
then real adds never trip a dimension mismatch; addItem/updateItem refuse a
length-0 vector; and a crash/repair cycle stays dimension-consistent.
2026-08-27 13:29:11 -07:00
fd6b4ce4ff fix(storage): derive the canonical count ledger from identity records, stamp the derivation rule, and mark legacy-derived ledgers suspect at load
Some checks failed
CI / Node 22 (push) Successful in 12m18s
CI / Node 24 (push) Successful in 12m21s
CI / Integration + conformance (Node 22) (push) Failing after 14m51s
CI / Bun (latest) (push) Successful in 12m28s
2026-08-27 13:00:44 -07:00
204d74c161 Merge branch 'next/enumeration-identity-rekey' 2026-08-27 12:39:56 -07:00
f8d8ce16b9 fix(storage): enumeration re-keys on the identity record, not the vector leg
Some checks failed
CI / Node 22 (push) Successful in 12m19s
CI / Node 24 (push) Successful in 12m16s
CI / Integration + conformance (Node 22) (push) Failing after 14m47s
CI / Bun (latest) (push) Successful in 12m20s
The noun/verb pagination walks (getNounsWithPagination,
getNounIdsWithPagination, getVerbsWithPagination) listed shard contents by
filtering for vectors.json, while the canonical count ledger has always
counted a row by its metadata.json presence alone. A row with metadata and
no vector file was therefore counted by the ledger but never yielded by the
walk — a permanent "counted but invisible" phantom for any downstream
consumer that iterates the walk to account for the ledger's total.

Nouns now enumerate by metadata.json and hydrate the vector leg optionally,
yielding the sanctioned unvectored shape (vector: []) when it's absent.
Verbs enumerate the same way, but a metadata-only verb row can only be fully
reconstructed when sourceId/targetId happen to be recoverable from metadata
(never true for a current production write — those fields live only in the
vector leg); otherwise the row is counted but loudly skipped rather than
fabricated, since a phantom edge with fake endpoints would be worse than the
original defect.

Separately, GenerationStore's recovery-fold replay (replayFact) now applies
preserve-if-absent: a metadata-only after-image replayed over an already-
vectored row carries the existing vector forward instead of deleting it via
writeNounRaw/writeVerbRaw's exact-restore null-means-delete contract (which
must stay exact for transaction-abort rollback). A genuine tombstone still
removes both legs.
2026-08-27 12:38:52 -07:00
2496e09aeb fix(init): rethrow plugin activation failures with the original error as cause so the originating frame survives to the caller 2026-08-27 12:07:09 -07:00
4c7b0fab7a Merge branch 'next/vfs-root-zero-norm' 2026-08-27 11:49:50 -07:00
c6cc0de955 fix(vfs): the VFS root never persists a zero-norm vector
Some checks failed
CI / Node 22 (push) Successful in 12m23s
CI / Node 24 (push) Successful in 12m12s
CI / Integration + conformance (Node 22) (push) Failing after 14m47s
CI / Bun (latest) (push) Successful in 12m38s
A zero-norm vector is lawful inside brainy (cosine distance scores it at
maximum, never a false top hit) but a false attractor for a downstream
engine serving squared-euclidean distance, which cannot tell a real
all-zero vector apart from a legitimate origin point.

- The VFS root now persists with vector [] (the existing "unvectored"
  shape) instead of a real all-zero 384-dim placeholder, and is never
  routed into the deferred-embed pipeline.
- A one-time migration in the root-init path detects a pre-fix store's
  all-zero placeholder root (by norm, not length) and rewrites it to []
  through a new sanctioned Brainy method that keeps the canonical
  vectored-noun ledger honest and removes the row from the vector index.
- The vector-index write seam (AddToVectorIndexOperation,
  ReplaceInVectorIndexOperation, and the generation materializer's direct
  insert) now refuses any real all-zero vector before it reaches a
  provider, loudly naming the entity, while the canonical write still
  lands.
- add()'s dimension-pinning and HNSW-insert gates, and the add-params
  validator, now treat any empty vector as carrying no dimension
  information, closing a latent trap where an explicit `vector: []`
  would have pinned dimensions to 0.
2026-08-27 09:28:44 -07:00
8a5c1245a7 build: derive generated-file stamps from git commit time, not wall clock
All checks were successful
CI / Node 22 (push) Successful in 12m16s
CI / Node 24 (push) Successful in 12m12s
CI / Bun (latest) (push) Successful in 12m19s
CI / Integration + conformance (Node 22) (push) Successful in 19m52s
Two builds of the same source tree could publish different artifacts
because buildEmbeddedPatterns.ts and buildTypeEmbeddings.ts stamped
their generated output with new Date().toISOString(). Route both
generators' "Generated:" header and the generatedAt runtime field
through a shared resolver: newest git commit time among the
generator's inputs (script + source data), falling back to the stamp
already present in the previous output when git is unavailable (e.g.
a published tarball build), and finally to a fixed epoch value. Every
fallback logs to stderr so degradation is never silent.

Regenerated both committed output files once so the tree carries
deterministic stamps; no other content changed.
2026-08-27 09:18:45 -07:00
aad9e2eeb1 Merge remote-tracking branch 'origin/release/10.4.1'
All checks were successful
CI / Node 22 (push) Successful in 12m17s
CI / Node 24 (push) Successful in 12m13s
CI / Bun (latest) (push) Successful in 12m25s
CI / Integration + conformance (Node 22) (push) Successful in 19m47s
# Conflicts:
#	CHANGELOG.md
#	package-lock.json
#	package.json
2026-08-26 15:56:41 -07:00
0a19bbd8a7 chore(release): 10.4.1
All checks were successful
Publish (The Source) / Publish to The Source registry (push) Successful in 12m34s
CI / Node 22 (push) Successful in 12m21s
CI / Node 24 (push) Successful in 12m16s
CI / Bun (latest) (push) Successful in 12m24s
CI / Integration + conformance (Node 22) (push) Successful in 19m37s
2026-08-26 15:19:54 -07:00
2914e0eb42 docs(concepts): the serving law — a failure is graded by whether an answer could be wrong, never by the cost of the fix; reads refuse per family
All checks were successful
CI / Node 22 (push) Successful in 12m14s
CI / Node 24 (push) Successful in 12m13s
CI / Integration + conformance (Node 22) (push) Successful in 19m40s
CI / Bun (latest) (push) Successful in 12m19s
2026-08-26 14:48:26 -07:00
7870dc4092 chore(release): 10.4.1-rc.1
All checks were successful
Publish (The Source) / Publish to The Source registry (push) Successful in 12m31s
CI / Node 22 (push) Successful in 12m13s
CI / Node 24 (push) Successful in 12m9s
CI / Bun (latest) (push) Successful in 12m24s
CI / Integration + conformance (Node 22) (push) Successful in 19m42s
2026-08-26 14:29:20 -07:00
c039411e08 fix(reads): the read gate is per-family; a write carrying unchanged data never re-embeds
All checks were successful
CI / Node 22 (push) Successful in 12m15s
CI / Node 24 (push) Successful in 12m12s
CI / Bun (latest) (push) Successful in 12m25s
CI / Integration + conformance (Node 22) (push) Successful in 19m47s
Two cures from the pair's first production adoption, both measured live.

THE READ GATE IS PER-FAMILY. The report-driven gate refused on ANY
provider's not-ready verdict at every read choke point — so a pure
metadata find({ where }) was refused because the VECTOR leg was not
serving, and a deployment's badge reads returned errors for a verdict that
had nothing to do with them. A read may only be refused by the family it
actually consults: metadata reads by the metadata leg (plus graph for a
`connected` filter), vector search by the vector leg, traversal by the
graph leg. Callers name what they need; the existing narration-once-per-
generation and typed-refusal laws are unchanged within a family.

NO RE-EMBED ON UNCHANGED DATA. update() — and its transact() planner —
treated any write that carried `data` as a data change: with
deferEmbedding it queued a landing, and the worker re-embedded and re-landed
a vector for content that had not changed. A host heartbeat re-writing an
unchanged row every few seconds therefore fed a live index-row loop on a
production store. A write carrying the row's current data (structural
compare, key order normalized) is now not a data change: no re-embed, no
deferred landing, no vector rewrite; the metadata write itself still
commits. A real change re-embeds exactly as before.

Pinned in tests/integration/read-gate-scope-and-no-reembed.test.ts — both
pins red-proved against the unfixed code with the production shapes
verbatim. Two health-gate pins that encoded the old brain-global scope are
re-pointed to the family their reads consult.
2026-08-26 13:55:11 -07:00
21e506e802 docs(guide): the docs pipeline publishes through the ingest API — the separate deploy step is retired
All checks were successful
CI / Node 22 (push) Successful in 12m20s
CI / Node 24 (push) Successful in 12m13s
CI / Bun (latest) (push) Successful in 12m24s
CI / Integration + conformance (Node 22) (push) Successful in 19m42s
2026-08-26 10:19:20 -07:00
efa042b52c chore(release): 10.4.0
Some checks failed
Publish (The Source) / Publish to The Source registry (push) Successful in 12m48s
CI / Node 24 (push) Has been cancelled
CI / Bun (latest) (push) Has been cancelled
CI / Integration + conformance (Node 22) (push) Has been cancelled
CI / Node 22 (push) Successful in 12m28s
2026-08-26 09:44:47 -07:00
834149ed90 docs(releases): the 10.4.0 entry catches up to the late trains — repair routing, the vector ledger and open-gate leg, the loud config guard, the JSON-safe crossing
All checks were successful
CI / Node 24 (push) Successful in 12m21s
CI / Node 22 (push) Successful in 12m32s
CI / Integration + conformance (Node 22) (push) Successful in 19m46s
CI / Bun (latest) (push) Successful in 12m16s
2026-08-26 08:03:48 -07:00
d3aacfaf24 chore(release): 10.4.0-rc.4
All checks were successful
Publish (The Source) / Publish to The Source registry (push) Successful in 12m47s
CI / Node 24 (push) Successful in 12m11s
CI / Node 22 (push) Successful in 12m25s
CI / Bun (latest) (push) Successful in 12m23s
CI / Integration + conformance (Node 22) (push) Successful in 19m38s
2026-08-25 16:00:16 -07:00
9730835bdf feat(vector): the vectored-noun scalar joins the count ledger; the open gate closes the vector leg
All checks were successful
CI / Node 24 (push) Successful in 12m22s
CI / Node 22 (push) Successful in 12m32s
CI / Integration + conformance (Node 22) (push) Successful in 19m42s
CI / Bun (latest) (push) Successful in 12m15s
The coverage denominator the health-by-accounting ratification named for
the vector family — never built until now, and its absence was measured as
the exact outage class it existed to prevent: a migrated store with
canonical vectors and no derived index opened with the vector leg EMPTY,
served [] from vector search with no error, and the report-driven read gate
had nothing to refuse on (the provider's coverage invariant was honestly
unledgered — the denominator was ours to supply).

- getCanonicalCounts() gains vectors: { all } — the count of canonical
  nouns holding a REAL vector. Incremented where a vector lands (the
  isNew-gated metadata seam for explicit vectors — the same discipline that
  keeps HNSW neighbor-link re-saves from inflating counts; a narrow
  noteVectorLanded hook for the deferred-embed landing, gated on the
  worker's own pre-embed read). Decremented on a proven delete of a
  vectored noun; a vector-uncertain delete marks the ledger suspect rather
  than guessing (no new reads on the delete path). Recounted by the
  sanctioned recount; legacy counts.json derives it once (a deferred noun's
  vector file exists with an empty vector, so presence requires one
  content read at derivation — never on the hot path).
- The open gate's vector leg: when a health-reporting provider claims
  serving while the index holds zero nodes and the ledger proves vectored
  canonical rows exist, open BUILDS (narrated) — routed through the
  provider's idempotent fillFromCanonical() when exposed (the joint door;
  a partial shortfall stays repair()'s operator business), the JS rebuild
  otherwise — or fails typed pre-serve. Scoped exactly: bare isReady()
  providers, migrating providers, and white-box size stubs open as before.

Pinned end-to-end from the partner gate's probe shape (store with vectored
canonical rows, no derived index, reopen → search serves N, never []),
red-proved against the pre-fix path; the inverse (zero vectored rows) opens
without building and serves [] honestly.
2026-08-25 15:31:19 -07:00
bce2593e24 chore(release): 10.4.0-rc.3
All checks were successful
Publish (The Source) / Publish to The Source registry (push) Successful in 12m47s
CI / Node 24 (push) Successful in 12m28s
CI / Node 22 (push) Successful in 12m29s
CI / Bun (latest) (push) Successful in 12m24s
CI / Integration + conformance (Node 22) (push) Successful in 19m26s
2026-08-25 12:51:19 -07:00
f4780c8e88 fix(update-seam): the metadata crossing never carries BigInt endpoint ints
All checks were successful
CI / Node 22 (push) Successful in 12m19s
CI / Node 24 (push) Successful in 12m25s
CI / Bun (latest) (push) Successful in 12m24s
CI / Integration + conformance (Node 22) (push) Successful in 19m14s
resolveVerbEndpointInts mirrors the resolved u64 endpoint ints onto the
verb object itself as BigInt (verb.sourceInt/targetInt) for the graph legs'
own params. The live verb path's delete legs then reused that same object
as the metadata-index crossing — and the seam's metadata is JSON-safe by
contract (a native provider serializes it; u64 as Number corrupts above
2^53), so JSON.stringify threw and the whole transaction aborted. Found by
the first joint pair gate; four downstream suites red from one crossing.

The crossing now routes through a JSON-safe view that drops BigInt-valued
top-level keys — endpoint ints ride their own op params on the graph legs,
exactly as designed, and never the metadata crossing. Applied at the
retraction helper (cascade + unrelate + transact mirrors) and
updateRelation's remove leg.

Pinned by driving the exact shape (relate resolves ints, remove cascades
the same object) through a provider shim enforcing the JSON contract —
red-proved against the unfixed path (the joint gate's verbatim error),
green with the fix.
2026-08-25 12:07:28 -07:00
f14da34b27 Merge branch 'worktree-agent-ad3aff0dffd17a6eb'
Some checks failed
CI / Node 22 (push) Successful in 12m19s
CI / Node 24 (push) Has been cancelled
CI / Integration + conformance (Node 22) (push) Has been cancelled
CI / Bun (latest) (push) Has been cancelled
2026-08-25 11:47:25 -07:00
0e1286e321 chore(release): 10.4.0-rc.2
All checks were successful
Publish (The Source) / Publish to The Source registry (push) Successful in 12m30s
CI / Node 22 (push) Successful in 12m19s
CI / Node 24 (push) Successful in 12m8s
CI / Bun (latest) (push) Successful in 12m25s
CI / Integration + conformance (Node 22) (push) Successful in 19m30s
2026-08-25 11:38:06 -07:00
39b916a3c0 test(readiness): the report helper's clock freezes — two independently-built reports compared across a millisecond tick made the plant lane red
All checks were successful
CI / Node 22 (push) Successful in 12m14s
CI / Node 24 (push) Successful in 12m12s
CI / Bun (latest) (push) Successful in 12m23s
CI / Integration + conformance (Node 22) (push) Successful in 19m22s
2026-08-25 11:04:30 -07:00
553e0d97ae feat(repair): a heal:'repair' verdict routes to the provider's own incremental repair()
Some checks failed
CI / Node 24 (push) Failing after 7m47s
CI / Node 22 (push) Successful in 12m19s
CI / Bun (latest) (push) Successful in 12m20s
CI / Integration + conformance (Node 22) (push) Successful in 19m22s
repairIndex() acted only on heal:'rebuild' — an invariant asking for the
INCREMENTAL heal (re-post exactly what the ledger names, O(missing), never
a store-sized rebuild) did nothing on brainy's side. A failing 'repair'
verdict now routes to the provider's feature-detected repair(); the
post-heal RE-READ of the report decides success (the acceptance meta-pin's
law — run the named heal once, re-read, nothing may still fail the same
way), and a repair that does not converge is recorded with the escalation
named: repairIndex({ rebuild: [family] }).
2026-08-25 10:47:51 -07:00
ddd5e71928 fix(storage): an unknown nested storage config can never silently land on the shared default root
The factory loudly rejects every REMOVED pre-8.0 path key, but an unknown
nested `config` object (e.g. `storage: { config: { baseDir } }` — a shape
that was never supported) fell through SILENTLY to the zero-config default
directory. Every instance constructed with such a shape wrote to ONE shared
on-disk root while its caller believed each had its own — found live when
two integration tests' brains shared a store across an entire
single-process CI run and a health probe refused on the foreign edges it
sampled. A nested `config` carrying any path-shaped key now throws the same
loud migration error, naming the canonical `path` rename. The two tests are
repaired to the supported shape (and now actually test isolated stores, for
the first time since 8.0).
2026-08-25 10:47:51 -07:00
258e9042af fix(add): empty string is real data, not a missing field
validateAddParams() treated '' as falsy and rejected it with "Missing
required field 'data'" — so a legitimate empty file's first write always
failed. Only null/undefined data (with no vector either) is genuinely
absent; '' is real content. Fixed the check, plus the identical bug in
validateUpdateParams() (truncating a file to empty via overwrite hit the
same falsy check) and in update()/transact()'s update planner, where a
plain `Boolean(params.data)`/truthy check on the resolved vector would have
silently skipped both the deferred-embed marker and the eager re-embed for
an emptied value — a stale vector with no path to ever correct itself.

Verified end-to-end: vfs.writeFile('/empty.txt', '') now succeeds,
readFile() returns '', the file lists, and stat() reports size 0; the
existing "should reject empty string as data" tests (unit + integration)
asserted the old buggy behavior and are updated to assert the fixed
contract instead.
2026-08-25 10:10:19 -07:00
fc516da6eb feat(vfs): implement readdir's recursive option — typed since 7.30, never read
vfs.readdir()'s ReaddirOptions.recursive was typed but silently ignored: a
recursive request behaved identically to a non-recursive one. Implemented
properly: recursive listing walks every descendant (files and directories,
any depth) via the same graph-traversal + one-batch-fetch path
getTreeStructure()/getDescendants() already use, and reports each entry as
a path relative to the queried directory (Node's fs.readdir(dir,
{ recursive: true }) convention) — 'sub/file.txt', not just 'file.txt'.

With withFileTypes: true, each VFSDirent.name carries that same relative
path when recursive (matching the string-array form byte for byte);
VFSDirent.path stays the absolute VFS path either way, so no information is
lost. Filter/sort/pagination compose unchanged, now over the full recursive
set. Non-recursive behavior (direct children, named by basename) is
unchanged.
2026-08-25 10:10:01 -07:00
96624f408c feat(open-path): init never gates on the embedding model; open goes concurrent; slow opens narrate
A production restart storm measured 90,017ms for a single brain init vs
1,117ms quiet (~80x contention multiplier), traced to performInit() eagerly
awaiting the process-global WASM embedding engine before the VFS root even
existed. Every writer's open() queued on the one throttled model compile
(90-140s on throttled CPUs).

- VirtualFileSystem.doInitializeRoot() no longer embeds '/'. The root is
  system-tier plumbing nothing ever searches; when the default WASM engine
  is active it now gets an explicit all-zero placeholder vector
  (cosineDistance returns max distance for a zero vector, so it never ranks
  ahead of real content). deferEmbedding was considered and rejected: its
  landing path kicks the embed worker synchronously right after commit,
  which would still force the cold compile within milliseconds — just off
  the awaited path, not avoided. A registered native 'embeddings' provider
  (no cold-start cost, possibly a different dimension) still embeds the
  root for real, via the new Brainy.usesDefaultWasmEmbedder() seam.

- performInit()'s eager-embedding step now only STARTS the WASM engine warm
  in the background instead of awaiting it inline. embed()/embeddingManager
  already serialize concurrent callers on one shared init promise, so the
  first real embed() converges correctly either way; a failed warm narrates
  loudly instead of surfacing as a silent latency spike or an unhandled
  rejection. eagerEmbeddings: false still means no warm at all.

- FileSystemStorage.init() batches its ~8 independent bootstrap mkdirs
  (each creates its own full subtree via recursive:true, so none depend on
  the others existing) into one Promise.all. The restore-completion step
  and initializeCounts() stay strictly sequential — they have real order
  dependencies on rootDir and systemDir respectively.

- performInit() now times five phases (storage init / generation-store
  open+fold / index init+gate / VFS bootstrap / embedding-warm-started) and
  logs one warning with the per-phase breakdown when total open exceeds
  2000ms; silent otherwise.
2026-08-25 10:09:45 -07:00
8cced871a0 docs(release): the 10.4.0 entry, the index-health concept doc, and the API surfaces — written from the tree, not the plan
Some checks failed
CI / Node 22 (push) Successful in 12m19s
CI / Node 24 (push) Successful in 12m16s
CI / Integration + conformance (Node 22) (push) Failing after 15m12s
CI / Bun (latest) (push) Successful in 12m24s
2026-08-25 10:02:25 -07:00
b9ba50fbec fix(plugins): the silent-degrade doors close — a broken accelerator install can never read as absent
The auto-detection "not installed" heuristic accepted any resolution failure
whose message merely CONTAINED the package name, unterminated — so a missing
platform-binary sibling package (what a deploy replacing node_modules
mid-restart leaves behind) read as "the accelerator is not installed", and
brainy silently served the default WASM engines with zero journal lines. A
production restart storm paid 90 seconds of throttled WASM compile behind
exactly that hole. The name must now terminate where it ends (quote,
whitespace, punctuation, end) — a sibling package, an inner file path, or a
dependency failure is a broken install and init() throws, as the guard's own
law always stated.

Second door: activate() returning false (the documented graceful decline)
warned on console.warn, which `silent: true` patches away — an invisible
degrade. The decline now narrates via the always-on channel.

Also exports CanonicalCounts from the public surface (the coverage-ledger
denominator type consumers read through getCanonicalCounts()).

Pinned in tests/unit/plugin-activation-loudness.test.ts (five error shapes;
the decline warn under silent: true).
2026-08-25 10:01:56 -07:00
18f172e098 feat(recovery): the catchup verdict is consumed; verb rows go live; the metadata rebuild goes online
Three cures on the JS metadata index, one seam:

- THE CATCHUP WIRING. The index computed its three-way watermark verdict at
  open and nothing consumed it — after a crash + adopt reopen, find() served
  the pre-crash index while canonical reads and counts recovered (caught by
  the lifecycle lane's first run). The open path now consumes the verdict:
  'adopt' is a no-op, 'catchup' folds the fact window (stamped, committed]
  through the index legs — nouns and verbs, remove-then-add, one mechanism
  for add and update — and 'rescan' runs the explicit rebuild, each narrated.
  The lane's Ch4–6 release-blocking marker comes off: the contract holds.
  Bonus root-cause: close() never stamped the projection watermarks (only
  flush() did), so any close without a prior flush verdicted a needless
  'rescan' on reopen — both doors now stamp.

- THE LIVE VERB PATH. Verb rows entered the metadata index only via rebuild
  walks, so every rebuilt store minted phantom/stale verb postings from its
  first live relate(). relate()/unrelate()/updateRelation() and remove()'s
  cascade now post/retract the verb's row in the same commit as the graph
  leg — transact() planners mirror identically — using the exact record
  shape the rebuild walk uses, so live and rebuilt populations agree.

- THE ONLINE REBUILD. rebuild() was clear-then-walk — every metadata read
  empty for the duration. rebuildMetadataIndexOnline builds a fresh manager
  beside the serving one (shared identity, in-memory build, dual-write via
  a shadow seam with zero call-site changes), atomically swaps the
  reference, and persists exactly once post-swap. A find() polled ~200x
  during a 2k-noun rebuild never dropped below its baseline.
  repairIndex({ rebuild: ['metadata'] }) uses it automatically.
2026-08-25 10:01:56 -07:00
f8f64780b1 feat(health): the gate reads the named report — reads refuse loudly, never rebuild; open serves before it returns; the ceremony door
All checks were successful
CI / Node 22 (push) Successful in 12m19s
CI / Node 24 (push) Successful in 12m16s
CI / Integration + conformance (Node 22) (push) Successful in 18m41s
CI / Bun (latest) (push) Successful in 12m20s
The read gate stops consulting the unnamed isReady() boolean: every provider
may expose healthReport() (sync, O(1), composed from exact ledgers —
HealthReport with a monotonic generation, per-invariant source
ledger|deep|unledgered, missing {count, sample}), and one readiness
authority (assessProviderHealth) derives the verdict. Unledgered families
are UNKNOWN — never healthy, never broken; a report that throws is a loud
not-ready, never a shrug. Reads at the four index choke points refuse with
the typed NotReady errors, narrated once per (provider, generation) — a
read NEVER starts a store walk:

- the first-read lazy build retires (open builds instead, regardless of
  size — the ≥10k deferral and the "lazy loading on first query" branch go;
  disableAutoRebuild is re-meant honestly in its docs);
- the verify*Live read-path rebuild triggers retire (refuse-or-serve);
- the read-time consistency probe that could launch a dark rebuild from an
  ordinary find() retires;
- repairIndex({ rebuild: ['metadata'|'graph'|'vector'] | 'all' }) is the
  one explicit door: rebuilds the named leg unconditionally and reports
  rebuilt per family; bare repairIndex() stays report-driven.

test(lifecycle): the biography lane — a store's whole life, refereed

tests/lifecycle/: an independent shadow model referees every read after
every chapter (founding, a working day, clean restart, crash, repair,
second life). Chapters 1-3 green. Chapters 4-6 assert the true contract and
are marked .fails as a release-blocking finding (the kill-matrix
convention): after a crash + adopt reopen the metadata index computes its
'catchup' watermark verdict and nothing consumes it — find() serves the
pre-crash index while canonical and counts recover. The catchup wiring is
the cure; a passing .fails will force the marker's removal. The lane runs
in the integration gate (config + coverage guard).
2026-08-24 12:45:51 -07:00
a8b5ca0c8f chore(release): 10.4.0-rc.1
All checks were successful
Publish (The Source) / Publish to The Source registry (push) Successful in 12m33s
CI / Node 22 (push) Successful in 12m19s
CI / Node 24 (push) Successful in 12m8s
CI / Bun (latest) (push) Successful in 12m24s
CI / Integration + conformance (Node 22) (push) Successful in 18m28s
2026-08-24 10:46:55 -07:00
a1376e4a2c ci(publish): the home dist-tag follows the version — a prerelease publishes under 'rc' and never moves 'latest'
Some checks failed
CI / Node 22 (push) Successful in 12m14s
CI / Node 24 (push) Successful in 12m9s
CI / Bun (latest) (push) Has been cancelled
CI / Integration + conformance (Node 22) (push) Has been cancelled
2026-08-24 10:46:07 -07:00
dcbad1765a chore(release): --source-only — a home-only prerelease mode (The Source, never the storefront)
All checks were successful
CI / Node 22 (push) Successful in 12m18s
CI / Node 24 (push) Successful in 12m15s
CI / Bun (latest) (push) Successful in 12m23s
CI / Integration + conformance (Node 22) (push) Successful in 18m24s
A prerelease the other engine devDeps from our own registry while the two
are being proven together must never reach npmjs: --source-only tags, lets
CI publish to The Source, creates the release page, and SKIPS the storefront
publish, the cross-registry verification and the docs push — loudly, at
each step. Refused for a non-prerelease version: a public floor always ships
byte-identical on both registries.
2026-08-24 10:06:23 -07:00
4176439ba3 test(fold-checkpoint): the ARM-AT-FLIP pin arms its crash instead of racing the pending-flush timer
Some checks are pending
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
The pin wrote a post-flip row, abandoned the brain as crashed, and then
deleted the row's canonical bytes to prove the first post-flip boot folds
BOUNDED above the flip's stamp. Between the write's ack and the abandon sat
the store's 50ms pending-flush timer: on a loaded box (the plant lane) the
flush won, barrier-synced the row and advanced the checkpoint over it — and
the fold, correctly bounded, did not restore bytes the test had destroyed
after they were stamped durable. Green locally, red on the plant: the
engine was right, the pin was timing-dependent.

The crash is now armed at exactly singleop-after-fact-append: the fact is
appended and at-ack synced, no flush is ever scheduled, the stamp provably
still reads the flip's value when the pre-flip bytes are dropped, and the
post-flip row's bytes — which lived only in the pending tier's RAM — are
lost for real, not synthetically. The reopen must re-materialize it from
its fact and must not restore the pre-flip row.
2026-08-24 09:58:45 -07:00
116550eb16 fix(health): one contract for a throwing probe — heal is none, serving is not withheld; repair report gains missing/rebuilt/reason
Some checks are pending
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
A validateInvariants() that threw was re-synthesized by the host's catch as
heal:'rebuild' with serving:false — a rebuild lever one transient exception
away, while the native provider's own composer reports the same event as
heal:'none'. Two components disagreeing on what a thrown check means is how
a flaky probe becomes an outage. Both now agree: the report is named
('validate-invariants-threw'), loud (healthy:false, the error in detail),
unverified — and it never buys a rebuild and never withholds serving; the
provider's serving verdict is the provider's to compose, not inferred from a
probe that failed to run. The synthesized report also carries the provider's
name instead of 'unknown'.

RepairFamilyReport gains `missing: {count, sample}` (an exact count plus a
capped id sample — a verdict, not a dump), `rebuilt` (a full generational
rebuild ran, as opposed to an incremental heal) and `reason`, aligning the
receipt's shape with the provider-side health report.

Pinned in tests/unit/validate-invariants-delegation.test.ts.
2026-08-24 09:54:25 -07:00
161 changed files with 14711 additions and 1529 deletions

View file

@ -2,7 +2,7 @@
## What Is Brainy
@soulcraft/brainy (v7.17.0) is a Universal Knowledge Protocol -- a Triple Intelligence database combining vector search, graph traversal, and metadata filtering in a single library. Published to npm as a public MIT-licensed package.
@soulcraftlabs/brainy (v7.17.0) is a Universal Knowledge Protocol -- a Triple Intelligence database combining vector search, graph traversal, and metadata filtering in a single library. Published to npm as a public MIT-licensed package.
## Core Architecture

View file

@ -5,6 +5,10 @@ name: CI
# sequential, so tag-triggered matrix jobs (~22 min) would queue AHEAD of the
# tag's publish-source run and starve every release (observed on 8.10.3 and
# 9.0.0: the publish sat behind the tag's own redundant CI).
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
on:
push:
branches: ['**']

View file

@ -12,6 +12,11 @@ on:
push:
tags:
- 'v*'
workflow_dispatch:
inputs:
ref_reason:
description: 'why this manual run (e.g. tag event dropped)'
required: false
jobs:
publish:
@ -32,22 +37,31 @@ jobs:
run: |
set -eo pipefail
SOURCE_NPM_REG="https://source.soulcraft.com/api/packages/soulcraft/npm/"
SOURCE_NPM_REG="https://source.soulcraft.com/api/packages/soulcraftlabs/npm/"
VERSION="$(node -p "require('./package.json').version")"
echo "Publishing @soulcraft/brainy@${VERSION} to The Source registry..."
# The dist-tag follows the version: a prerelease (any hyphen —
# 10.4.0-rc.1) publishes under 'rc' and must NEVER move 'latest' —
# every consumer resolving 'latest' from this registry would otherwise
# be handed a release candidate. Same rule scripts/release.sh applies
# to the storefront leg.
NPM_TAG="latest"
case "$VERSION" in
*-*) NPM_TAG="rc" ;;
esac
echo "Publishing @soulcraftlabs/brainy@${VERSION} to The Source registry (dist-tag: ${NPM_TAG})..."
TMPRC="$(mktemp)"
chmod 600 "$TMPRC"
{
echo "@soulcraft:registry=${SOURCE_NPM_REG}"
echo "//source.soulcraft.com/api/packages/soulcraft/npm/:_authToken=${FORGE_NPM_TOKEN}"
echo "@soulcraftlabs:registry=${SOURCE_NPM_REG}"
echo "//source.soulcraft.com/api/packages/soulcraftlabs/npm/:_authToken=${FORGE_NPM_TOKEN}"
} > "$TMPRC"
# The release script bumps package.json's version before it tags, so
# this tag's checkout already carries the version being published —
# nothing here re-derives it from the tag name.
PUBLISH_OK=true
if ! npm publish --tag latest --userconfig "$TMPRC"; then
if ! npm publish --tag "$NPM_TAG" --userconfig "$TMPRC"; then
PUBLISH_OK=false
fi
@ -55,7 +69,7 @@ jobs:
# exit code: a benign duplicate publish (a prior run, or a mirror, already
# landed this exact version) reports failure even though the registry
# already holds the right content.
LANDED_VERSION="$(npm view "@soulcraft/brainy@${VERSION}" version --userconfig "$TMPRC" 2>/dev/null || echo "")"
LANDED_VERSION="$(npm view "@soulcraftlabs/brainy@${VERSION}" version --userconfig "$TMPRC" 2>/dev/null || echo "")"
rm -f "$TMPRC"
if [ "$LANDED_VERSION" != "$VERSION" ]; then
@ -64,7 +78,7 @@ jobs:
fi
if [ "$PUBLISH_OK" = true ]; then
echo "Published and verified @soulcraft/brainy@${VERSION} on The Source registry."
echo "Published and verified @soulcraftlabs/brainy@${VERSION} on The Source registry."
else
echo "::warning::npm publish reported failure, but readback confirms @soulcraft/brainy@${VERSION} is already live on The Source (a prior run or mirror landed it) — treating this run as successful, since the registry content is correct. Any OTHER failure mode would have failed the readback check above instead."
echo "::warning::npm publish reported failure, but readback confirms @soulcraftlabs/brainy@${VERSION} is already live on The Source (a prior run or mirror landed it) — treating this run as successful, since the registry content is correct. Any OTHER failure mode would have failed the readback check above instead."
fi

View file

@ -2,13 +2,120 @@
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
### [10.3.1](https://source.soulcraft.com/soulcraft/brainy/compare/v10.3.0...v10.3.1) (2026-08-18)
### [10.4.4](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.3...v10.4.4) (2026-08-28)
- fix(vfs): the old-root sweep narrates only when it has something to say (d49148e1)
- fix(tests): the health-gate pin follows the verdict, and the VFS suite uses its own store (42e2da25)
- Merge branch 'next/open-lazy-open-and-counts' (5ebd3b40)
- docs: the contract manifest stands alone; public docs describe this engine only (a8c724a2)
- docs(releases): 10.4.4 consumer notes — correctness and observability, with the performance line stated exactly (61a46927)
- docs: measurements in public history carry numbers, not provenance (02c61636)
- feat(open): name the two steps that hold the vfs-bootstrap phase (2cf38010)
- fix(storage): a dead flush watch falls back to the 500ms poll, not the 30s sweep (5c22f950)
- fix(storage): the flush watcher cannot arm twice in its async window (16d2e1a9)
- perf(idle): the flush-request watch is event-driven; the heartbeat is observability (fb1da1c5)
- perf(open): answer "are there any entities?" with one directory read (417ddb51)
- perf(generations): discover generations by directory name, not by walking the log (9dd39921)
- fix(flush): clear() and repairIndex() set the dirty witness themselves (e4c27fbc)
- feat(open): the open names the STEP that cost the time, not just the phase (5a091cca)
- perf(vfs): the old-root sweep runs once per store, not once per open (4a67aa0f)
- chore: keep the generated neural stamps at main's values (c1f09723)
- feat(contract): declare contract 1, serve three operators, refuse four by name (48802ba3)
- fix(open): a provider rebuilding itself is a third state, not a CRITICAL (50676c02)
- feat(open): open never waits for a provider that is rebuilding itself (131daa08)
- perf(flush): an idle brain does no work — no periodic flush without a write (f5a6cb3f)
- feat(repair): repairIndex narrates every phase and its receipt carries the walls (3fffd9c6)
- fix(storage): a suspect count ledger heals itself, and counts.json is written atomically (f4e2d34b)
- feat(open): the open narrates itself, on a channel production cannot clamp (afe08a1f)
- fix(storage): a clean close is recorded, and the writer lock is always given up (e652162c)
- docs: repository links point at soulcraftlabs/open-brainy — the soulcraft/brainy path becomes the native engine's repo tonight (38c3397b)
### [10.4.3](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.2...v10.4.3) (2026-08-27)
- Merge branch 'next/open-brainy-rename' (a58372f0)
- chore: rename to @soulcraftlabs/brainy for Open Brainy on The Source (a99b1e83)
- docs(releases): 10.4.3 — Open Brainy's first release under the new name, same engine as 10.4.2; The Source is the one registry (9f248b24)
### [10.4.2](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.2-rc.1...v10.4.2) (2026-08-27)
- docs(releases): 10.4.1 and 10.4.2 consumer notes; 10.4.2 is the last MIT release under this name, Open Brainy continues at @soulcraftlabs/brainy (a082e0ef)
### [10.4.2-rc.1](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.1...v10.4.2-rc.1) (2026-08-27)
- Merge branch 'next/zero-norm-unvector-door' (9b84ef5b)
- fix(vectors): a zero-norm vector is not a vector, canonical side included, plus the sanctioned unvector door (0de76659)
- fix(hnsw): skip unvectored rows on rebuild; refuse empty vectors in the index (8fc553b1)
- fix(storage): derive the canonical count ledger from identity records, stamp the derivation rule, and mark legacy-derived ledgers suspect at load (fd6b4ce4)
- Merge branch 'next/enumeration-identity-rekey' (204d74c1)
- fix(storage): enumeration re-keys on the identity record, not the vector leg (f8d8ce16)
- fix(init): rethrow plugin activation failures with the original error as cause so the originating frame survives to the caller (2496e09a)
- Merge branch 'next/vfs-root-zero-norm' (4c7b0fab)
- fix(vfs): the VFS root never persists a zero-norm vector (c6cc0de9)
- build: derive generated-file stamps from git commit time, not wall clock (8a5c1245)
- Merge remote-tracking branch 'origin/release/10.4.1' (aad9e2ee)
- docs(concepts): the serving law — a failure is graded by whether an answer could be wrong, never by the cost of the fix; reads refuse per family (2914e0eb)
- chore(release): 10.4.1-rc.1 (7870dc40)
### [10.4.1](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.0...v10.4.1) (2026-08-26)
- fix(reads): the read gate is per-family; a write carrying unchanged data never re-embeds (c039411e)
- docs(guide): the docs pipeline publishes through the ingest API — the separate deploy step is retired (21e506e8)
### [10.4.0](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.0-rc.4...v10.4.0) (2026-08-26)
- docs(releases): the 10.4.0 entry catches up to the late trains — repair routing, the vector ledger and open-gate leg, the loud config guard, the JSON-safe crossing (834149ed)
### [10.4.0-rc.4](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.0-rc.3...v10.4.0-rc.4) (2026-08-25)
- feat(vector): the vectored-noun scalar joins the count ledger; the open gate closes the vector leg (9730835b)
### [10.4.0-rc.3](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.0-rc.2...v10.4.0-rc.3) (2026-08-25)
- fix(update-seam): the metadata crossing never carries BigInt endpoint ints (f4780c8e)
- Merge branch 'worktree-agent-ad3aff0dffd17a6eb' (f14da34b)
- fix(add): empty string is real data, not a missing field (258e9042)
- feat(vfs): implement readdir's recursive option — typed since 7.30, never read (fc516da6)
- feat(open-path): init never gates on the embedding model; open goes concurrent; slow opens narrate (96624f40)
### [10.4.0-rc.2](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.0-rc.1...v10.4.0-rc.2) (2026-08-25)
- test(readiness): the report helper's clock freezes — two independently-built reports compared across a millisecond tick made the plant lane red (39b916a3)
- feat(repair): a heal:'repair' verdict routes to the provider's own incremental repair() (553e0d97)
- fix(storage): an unknown nested storage config can never silently land on the shared default root (ddd5e719)
- docs(release): the 10.4.0 entry, the index-health concept doc, and the API surfaces — written from the tree, not the plan (8cced871)
- fix(plugins): the silent-degrade doors close — a broken accelerator install can never read as absent (b9ba50fb)
- feat(recovery): the catchup verdict is consumed; verb rows go live; the metadata rebuild goes online (18f172e0)
- feat(health): the gate reads the named report — reads refuse loudly, never rebuild; open serves before it returns; the ceremony door (f8f64780)
### [10.4.0-rc.1](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.3.1...v10.4.0-rc.1) (2026-08-24)
- ci(publish): the home dist-tag follows the version — a prerelease publishes under 'rc' and never moves 'latest' (a1376e4a)
- chore(release): --source-only — a home-only prerelease mode (The Source, never the storefront) (dcbad176)
- test(fold-checkpoint): the ARM-AT-FLIP pin arms its crash instead of racing the pending-flush timer (4176439b)
- fix(health): one contract for a throwing probe — heal is none, serving is not withheld; repair report gains missing/rebuilt/reason (116550eb)
- feat(storage): the canonical count ledger — ALL-visibility scalars, unclamped totals, suspect-on-unprovable-delete (7c8c8be3)
- fix(delete): the null-metadata skip closes — index legs run id-keyed or narrate, never silently strand postings (607e9f54)
- feat(repair): repairIndex returns the per-family receipt and narrates its summary (8d45f964)
- fix(reads): the readiness gate guards every index read surface — serving empty from a not-ready provider is unrepresentable (40e7119b)
- ci(gate): the machine-health preflight and the truncation verdict guard (1e046aa1)
### [10.3.1](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.3.0...v10.3.1) (2026-08-18)
- docs(releases): the 10.3.1 consumer entry — the fold that behaves (900cc895)
- fix(recovery): the fold streams and narrates; the checkpoint chain arms at the flip (ed7d1db9)
### [10.3.0](https://source.soulcraft.com/soulcraft/brainy/compare/v10.2.0...v10.3.0) (2026-08-18)
### [10.3.0](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.2.0...v10.3.0) (2026-08-18)
- docs(releases): the 10.3.0 consumer entry — the trust-and-provenance release (97d75649)
- fix(locks): the fence keys ownership on pid+hostname — a same-process re-open never fences its predecessor (0991cf28)
@ -17,14 +124,14 @@ All notable changes to this project will be documented in this file. See [standa
- feat(log): system commits carry their origin; the attested per-id reconcile door (9ac9e706)
### [10.2.0](https://source.soulcraft.com/soulcraft/brainy/compare/v10.1.0...v10.2.0) (2026-08-17)
### [10.2.0](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.1.0...v10.2.0) (2026-08-17)
- docs(releases): the 10.2.0 consumer entry — adoption completes in one call (97538e1f)
- ci: the correctness plant runs integration + conformance on every push — a release never waits on a second machine (b17fdc8e)
- fix(adoption): the baseline backfill runs to completion — one call adopts a pre-log baseline of any size (a5a18838)
### [10.1.0](https://source.soulcraft.com/soulcraft/brainy/compare/v10.0.0...v10.1.0) (2026-08-13)
### [10.1.0](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.0.0...v10.1.0) (2026-08-13)
- docs(releases): the 10.1.0 consumer entry — bounded recovery, restore founding, the two write-path cures (7d3c8696)
- fix(restore): a restore is an unclean event — the swap runs quiesced and the snapshot's durability stamps never survive it (9ca80667)
@ -33,7 +140,7 @@ All notable changes to this project will be documented in this file. See [standa
- feat(query): the sparse-store cut — where on a never-carried field serves operator truth, never a refusal (7b67db4d)
### [10.0.0](https://source.soulcraft.com/soulcraft/brainy/compare/v9.0.0...v10.0.0) (2026-08-12)
### [10.0.0](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v9.0.0...v10.0.0) (2026-08-12)
- fix(adoption): the baseline backfill cures hydration-law drift — existing brains reach the crash-safe default with zero operator steps (25f0dd96)
- fix(adoption): the reserved-root mint exemption — int 0 is legitimate for exactly one id (2abe8b38)
@ -65,7 +172,7 @@ All notable changes to this project will be documented in this file. See [standa
- test: version-coupling pins go major-agnostic — the 8.x literals broke at the 9.0.0 bump while the coupling law itself behaved correctly (8a6807e8)
### [9.0.0](https://source.soulcraft.com/soulcraft/brainy/compare/v8.11.0...v9.0.0) (2026-08-04)
### [9.0.0](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v8.11.0...v9.0.0) (2026-08-04)
- docs: 9.0 namespace-migration guide — the simple story + the mechanical sweep checklist, published for humans and tooling alike (61ab9db2)
- fix(release): storefront leg republishes CI's exact forge artifact — byte-identity by construction, verified by cross-registry shasum before the ceremony reports success (d89df2ed)
@ -100,7 +207,7 @@ All notable changes to this project will be documented in this file. See [standa
- feat: scanFacts liveness contract — first batch or loud failure within a documented bound (f8e6da2b)
### [8.11.0](https://source.soulcraft.com/soulcraft/brainy/compare/v8.10.1...v8.11.0) (2026-07-27)
### [8.11.0](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v8.10.1...v8.11.0) (2026-07-27)
- docs: the last two archived-host links point home (91ef1c8b)
- feat: includeHidden — export carries every visibility tier for migration-grade canon completeness (63c1eeb9)
@ -109,19 +216,19 @@ All notable changes to this project will be documented in this file. See [standa
- ci: run the pipeline on the forge (999d0ebb)
### [8.10.3](https://source.soulcraft.com/soulcraft/brainy/compare/v8.10.2...v8.10.3) (2026-08-03)
### [8.10.3](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v8.10.2...v8.10.3) (2026-08-03)
- docs: dedupe the 8.10.2 release-notes entry the cherry doubled onto the branch (8c956608)
- fix: user metadata named 'level' is a real field everywhere — the engine-internal node layer no longer shadows it in sort/filter/aggregation, and the indexing views stop stamping a phantom 0 into its column; index epoch 2 rebuilds existing brains at first open (958a0859)
### [8.10.2](https://source.soulcraft.com/soulcraft/brainy/compare/v8.10.1...v8.10.2) (2026-07-29)
### [8.10.2](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v8.10.1...v8.10.2) (2026-07-29)
- docs: 8.10.2 consumer release notes — update() write granularity, PathResolver idle-log fix, graph-lsm key recognition (a0123b5b)
- fix: metadata-only update() never rewrites the noun record — the unconditional whole-vector save turned per-entity stat touches into full rewrites+fsync, amplifying read-heavy sweeps into disk saturation on a production deployment (5b65eb82)
### [8.10.1](https://source.soulcraft.com/soulcraft/brainy/compare/v8.10.0...v8.10.1) (2026-07-24)
### [8.10.1](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v8.10.0...v8.10.1) (2026-07-24)
- refactor: remove the orphaned transaction-result type left behind by the dead-path removal (edf123a5)
- fix: warm() metadata surface routes through the active provider (warm hook added to the metadata contract); add maintenanceDebt() observability surface (5b2cbf74)

View file

@ -12,13 +12,13 @@ Handoff file: `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md`
**Brainy's current open actions:** None. MIT open-source — no platform-specific actions.
**Current version:** run `npm view @soulcraft/brainy version` (never trust a hardcoded number here — this line went stale for months); consumer-facing changes tracked in `RELEASES.md`
**Current version:** run `npm view @soulcraftlabs/brainy version --registry https://source.soulcraft.com/api/packages/soulcraftlabs/npm/` (never trust a hardcoded number here — this line went stale for months); consumer-facing changes tracked in `RELEASES.md`
---
## Project Overview
Brainy is a Universal Knowledge Protocol -- a Triple Intelligence database that combines vector similarity search, graph traversal, and metadata filtering into a single TypeScript library. Published as `@soulcraft/brainy` on npm under the MIT license.
Brainy is a Universal Knowledge Protocol -- a Triple Intelligence database that combines vector similarity search, graph traversal, and metadata filtering into a single TypeScript library. Published as `@soulcraftlabs/brainy` on The Source (source.soulcraft.com registry) under the MIT license.
## Getting Started
@ -91,7 +91,7 @@ test: add/update tests (patch version bump)
## Docs Pipeline — soulcraft.com/docs
Docs in `docs/**/*.md` are published with the npm package (included in `files`) and synced to soulcraft.com/docs on every portal deploy. Frontmatter controls what appears publicly.
Docs in `docs/**/*.md` are published with the npm package (included in `files`) and go live on soulcraft.com/docs via the docs ingest API: the release script's `scripts/push-docs.js` step POSTs every public doc to `https://soulcraft.com/api/docs/ingest` (auth: `DOCS_INGEST_SECRET` in the environment). No separate deploy step is involved (the old deploy-to-publish flow was retired in a platform change, 2026-08). Frontmatter controls what appears publicly.
### Docs check triggers
@ -161,9 +161,9 @@ npm run release:major # Breaking changes (rare, manual decision)
The script: verifies clean git state, builds, tests, bumps version, updates CHANGELOG.md, commits, tags, pushes, publishes to npm, and creates a GitHub release.
After a successful release, remind the user:
> "Published. Deploy portal to pick up the new docs → go to the portal project and deploy."
> "Published. Docs are live on soulcraft.com/docs (pushed via the ingest API during the release) — spot-check a changed page with curl."
Do NOT deploy portal from here. Portal is always deployed separately from within the portal project.
There is no separate deploy step anymore. If the docs push failed (the script warns loudly), re-run `node scripts/push-docs.js` with `DOCS_INGEST_SECRET` set.
## Closed-Source Product Names — HARD RULE

View file

@ -6,7 +6,7 @@ may find elsewhere in the repo's history.
## Where the project lives
The source of truth is a self-hosted forge: **source.soulcraft.com/soulcraft/brainy**.
The source of truth is a self-hosted forge: **source.soulcraft.com/soulcraftlabs/open-brainy**.
It's anonymously readable and cloneable — no account needed to browse, clone,
or build.
@ -31,7 +31,7 @@ fine) to talk through the approach saves everyone rework.
## Development setup
```bash
git clone https://source.soulcraft.com/soulcraft/brainy.git
git clone https://source.soulcraft.com/soulcraftlabs/open-brainy.git
cd brainy
npm install
npm run build
@ -57,6 +57,17 @@ see `package.json` for `test:integration`, `test:coverage`, and friends.
description states a number, cite the benchmark that produced it (see
[docs/performance-envelopes.md](docs/performance-envelopes.md) for the
pattern). Don't state an estimate as if it were measured.
- **Measurements carry numbers, not provenance.** Public commit messages and
docs give the SHAPE a number was taken at and never where it was taken: no
hostnames, no store or deployment identities, no operational anecdotes about
someone's running system. "A 14,056-noun / 72,679-verb production-shaped
store, measured solo under an exclusive lock" tells a reader everything the
number depends on; the machine it ran on and whose data it was tell them
nothing except where somebody's infrastructure lives.
- **Documents that answer or reference a confidential specification never enter
this repository, even summarized.** The public docs describe THIS engine and
the published contract, and nothing else — a summary of a private document is
still that document's contents.
## License

View file

@ -1,5 +1,5 @@
<p align="center">
<img src="https://source.soulcraft.com/soulcraft/brainy/raw/branch/main/brainy.png" alt="Brainy" width="180">
<img src="https://source.soulcraft.com/soulcraftlabs/open-brainy/raw/branch/main/brainy.png" alt="Brainy" width="180">
</p>
<h1 align="center">Brainy</h1>
@ -11,9 +11,9 @@
</p>
<p align="center">
<a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/v/@soulcraft/brainy.svg" alt="npm version"></a>
<a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/dm/@soulcraft/brainy.svg" alt="npm downloads"></a>
<a href="https://source.soulcraft.com/soulcraft/brainy/actions"><img src="https://source.soulcraft.com/soulcraft/brainy/actions/workflows/ci.yml/badge.svg?branch=main" alt="CI"></a>
<a href="https://source.soulcraft.com/soulcraftlabs/-/packages/npm/brainy"><img src="https://img.shields.io/badge/package-The%20Source-2c3e50.svg" alt="Package on The Source"></a>
<a href="https://source.soulcraft.com/soulcraftlabs/open-brainy"><img src="https://img.shields.io/badge/repo-open--brainy-2c3e50.svg" alt="Repository"></a>
<a href="https://source.soulcraft.com/soulcraftlabs/open-brainy/actions"><img src="https://source.soulcraft.com/soulcraftlabs/open-brainy/actions/workflows/ci.yml/badge.svg?branch=main" alt="CI"></a>
<a href="https://soulcraft.com/docs"><img src="https://img.shields.io/badge/docs-soulcraft.com-blue.svg" alt="Documentation"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License"></a>
<a href="https://www.typescriptlang.org/"><img src="https://img.shields.io/badge/%3C%2F%3E-TypeScript-%230074c1.svg" alt="TypeScript"></a>
@ -30,6 +30,8 @@
---
**Open Brainy** is the MIT engine — the open API, client library, types, and protocol; an openly specified canonical on-disk format; and this TypeScript reference engine, scoped as a single-node engine for stores up to roughly one million rows. `@soulcraft/brainy` 10.4.2 was the last release under the old package name — the name passes to the native engine, **Brainy**, at 11.0.0: the same API over the same open format at production scale, and it requires a license.
Built because we were tired of stitching a vector store to a graph database to a document store — and spending weeks on plumbing before writing a line of business logic. Brainy indexes every fact **three ways at once** and lets one call query them together:
| You write | Brainy indexes it as | You query it with |
@ -45,12 +47,14 @@ It runs **inside your process** — no server, no Docker, nothing to operate —
## Quick start
```bash
bun add @soulcraft/brainy # Bun ≥ 1.1 — recommended
npm install @soulcraft/brainy # Node.js ≥ 22
bun add @soulcraftlabs/brainy # Bun ≥ 1.1 — recommended
npm install @soulcraftlabs/brainy # Node.js ≥ 22
```
> **Registry**: add `@soulcraftlabs:registry=https://source.soulcraft.com/api/packages/soulcraftlabs/npm/` to your `.npmrc` (anonymous read).
```javascript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
import { Brainy, NounType, VerbType } from '@soulcraftlabs/brainy'
const brain = new Brainy() // in-memory; one line swaps to disk
await brain.init()

View file

@ -1,7 +1,7 @@
# @soulcraft/brainy — Release Notes for Consumers
This file is the **quick reference for downstream sessions** tracking Brainy changes.
Full auto-generated changelog: `CHANGELOG.md` · Releases: https://source.soulcraft.com/soulcraft/brainy/releases
Full auto-generated changelog: `CHANGELOG.md` · Releases: https://source.soulcraft.com/soulcraftlabs/open-brainy/releases
**How to use:** Brainy is the underlying data engine for downstream applications. Read this when:
- Upgrading `@soulcraft/brainy` in your application
@ -31,6 +31,330 @@ is sometimes cited as a 7.x removal — those methods never existed on 7.x; the
---
## v10.4.4 — 2026-08-28
**A correctness and observability release.** The headline is not speed: it is that a
restart now tells you the truth about itself, a store stops lying about how much it
holds, and the engine stops doing work nobody asked for. There is a performance
improvement and it is modest; it is stated exactly below rather than rounded up.
### The dark restart — fixed at the root
A service could stop cleanly, exit 0, having awaited `close()` on every store it held,
and its next boot would announce `Overwriting stale writer lock … appears dead` for
every one of them. Nothing had crashed. Two deployments hit this; the same defect also
made those boots pay a crash-recovery fold they did not owe.
The cause was not the lock. `close()` released it correctly — when it got there. A
failure part-way through close skipped both the release AND the clean-shutdown marker,
and "the recorded pid is gone" reads identically for an orderly restart and a crash.
- `close()` is now two parts and the second is unconditional: the flush-request watcher,
the **writer lock**, the VFS timers and the terminal `closed` flag are released whether
the durable steps succeeded or not. The original failure is narrated with what it costs
the next open, then rethrown.
- Releasing the lock writes a **clean-close record** naming the lock generation it gave
up. The next open reads that record instead of guessing: recorded → nothing to recover;
absent → it says so, and names the recovery it is about to run. This also ends two
long-standing false alarms — a recycled pid locking a store out of its own reopen, and
`Re-acquiring writer lock … this is a bug` after a perfectly clean close.
- The signal path stopped failing in a batch. One store's failing flush used to strand
every remaining store's lock and markers — at exit code 0. Now: per-store isolation, the
generation store's close (the marker) is part of shutdown, the lock goes in a `finally`,
and the handler no longer calls `process.exit()` when the host application has its own
signal handler, a race that truncated the host's own shutdown mid-flight.
### The count ledger stops lying, and `counts.json` is written atomically
The all-tier scalars are the denominator a coverage check subtracts against. A ledger
derived under the old rule — one entity per id DIRECTORY — counted ghost and scar
containers as rows, and was only FLAGGED suspect: it went on serving wrong numbers for
the life of the store. Two copies of one archive could disagree, and a downstream index
heal reported remaining work that did not exist.
- Such a ledger now derives itself honestly **in the background** after the open, counting
identity records, and persists the correction stamped. Nothing waits for it, because no
read is served from a denominator.
- A derivation that raced a write refuses to stamp its number: one retry on a quiet store,
then the ledger stays SUSPECT and names `repairIndex()` as the door that recounts under
a barrier.
- `counts.json` is written temp+rename. A truncating write left a window in which a
concurrent reader saw the file EMPTY — and an unparseable ledger sends the next open
down the full-rescan path, so the cheapest file in the store was buying the most
expensive recovery.
### An open and a repair narrate themselves — on a channel a log level cannot silence
A store could open for three minutes and print nothing at all. The phase timings existed;
they were written to a channel that every production-looking environment clamps away.
- Narration moved to an always-visible channel. An open now heartbeats the phase it is in,
names each phase as it ends with what it was paying for, and names the expensive STEP
inside a phase. `repairIndex()` does the same and its receipt carries a per-family
`durationMs` — a repair that ran for half an hour with no output could only be watched
through `top`.
- A brain nobody has written to now does nothing: a flush over a clean store is a no-op
and says nothing, the graph index's auto-flush asks before it acts, and the
cross-process flush-request watch is **event-driven** (`fs.watch`) instead of polling a
directory every 500 ms per store forever, with a slow safety sweep behind it and a
narrated fall back to polling where a filesystem cannot be watched.
- A provider that is REBUILDING ITSELF is no longer confused with a broken one. `init()`
does not wait for it, every other family serves, and that family's doors refuse **by
name, carrying the provider's own progress**, saying plainly that they open by
themselves and no action is needed. Health narration dedupes by content, so an unchanged
verdict is silent however a provider's generation counter moves.
### For operators — one behaviour change
**Four `where` operators that previously returned an empty page now raise
`INVALID_QUERY`:** `startsWith`, `endsWith`, `matches` and `length`. An equality/range
posting index cannot evaluate a substring, a pattern or an array length without reading
every row, and it now refuses by name instead of answering with an empty result that
looks like an answer.
**Three that previously returned an empty page are now SERVED:** `hasAll`, `noneOf` and
`excludes`. All 25 accepted operator tokens now agree between this engine and its
accelerated counterpart.
### Performance — stated exactly
Measured on a 14,056-noun / 72,679-verb production-shaped store, both builds solo under
an exclusive lock:
- **Warm reopen after a clean close: 85.7 s → 77.0 s (10.2%).** The whole of that gain is
one fix — generation discovery reads directory NAMES instead of recursively walking the
entire generation log (9.2 s, and it scales with history rather than row count). The
VFS phase is **unchanged**.
- **Cold open: 31.4 s** (518.1 s → 486.7 s), of which the count-ledger derivation moving
off the critical path accounts for storage-init dropping 5,941 ms → 25 ms.
- **A dominant ~38 s remains, diagnosed and NOT fixed.** It is not the VFS — the VFS's own
init is under 2 s of that phase. It is the log-authority adoption and/or the
pending-embed log recovery, both now instrumented so the next measurement names the
culprit outright.
Continuing work, named so nobody has to rediscover it: that ~38 s term; making the
generation store's committed-range set lazy; the hydration path that substitutes
`Date.now()` for an unreadable stored timestamp (inventing data); and a VFS path-prefix
filter built with a `$startsWith` spelling no operator set accepts, so
`searchFiles({ path })` throws today.
---
## v10.4.3 — 2026-08-27 (Open Brainy's first release)
**`@soulcraftlabs/brainy` 10.4.3 is the same engine as `@soulcraft/brainy` 10.4.2, byte for
byte — only the name, the registry, and the pointers changed.** Install:
```bash
npm install @soulcraftlabs/brainy
```
with the registry line in your `.npmrc` (anonymous read):
```
@soulcraftlabs:registry=https://source.soulcraft.com/api/packages/soulcraftlabs/npm/
```
- **The Source is the one registry.** Open Brainy publishes to source.soulcraft.com only; the
npmjs republish step is retired from the release rail. Existing npmjs versions of
`@soulcraft/brainy` stay as they are and receive no new versions.
- **The repository moved** to `soulcraftlabs/open-brainy` on The Source; the old path redirects.
- **No engine change.** Everything in the 10.4.2 notes applies unchanged; adoption is one
install-line change (`@soulcraft/brainy``@soulcraftlabs/brainy`), which downstream
applications make together with their native-engine bump.
## v10.4.2 — 2026-08-27 (a zero-norm vector is not a vector)
**This is the last release of the MIT engine under the `@soulcraft/brainy` name.**
The MIT package continues as **Open Brainy**`@soulcraftlabs/brainy`: the open API,
client library, types and protocol, an openly specified canonical format, and the TypeScript
reference engine, scoped honestly as a single-node engine for stores up to roughly one
million rows. The `@soulcraft/brainy` name passes to the native engine, **Brainy**, at a
major version bump; that engine implements the same API over the same open format at
production scale, requires a license, and refuses loudly without one. Nothing changes
for existing installs until that major ships; the move is announced with it.
Six fixes, one law: a vector with no magnitude carries no information, so it must
never reach a vector index — in any engine — and the canonical store must say so.
- **The permanently-unvectored row.** `add({ ..., vector: [] })` (and the same item
shape in `addMany` / `transact`) is now the sanctioned "no vector" row: persisted
with an empty vector leg, never embedded, never indexed, counted as unvectored in
the canonical ledger. Metadata-only rows — telemetry tallies, counters, plumbing —
no longer need a placeholder vector and never enter the vector leg. `vector: []`
together with `deferEmbedding: true` is refused with a typed error (a supplied
vector has nothing to defer). Previously `vector: []` threw a dimension error.
- **The unvector door.** `update({ id, vector: [] })` (and its `transact()` twin) is
the sanctioned way to strip a vector from an existing row: canonical vector → `[]`,
removal from the vector index, the vectored ledger decremented exactly once — and
idempotent, so a resumed cleanup pass may simply re-issue. It never re-embeds, and
it clears a pending deferred-embed marker durably so the background worker cannot
re-vector the row later. Note that a rebuild never sheds vectors (it re-derives the
index from canonical rows); shedding historical vectors needs this door.
- **Zero-norm vectors are normalized at the write.** An explicit all-zero vector on
any write path is persisted as unvectored (`[]`) with one warning naming the row;
the vector-index operations keep their own refusal as a second line. The engine's
own VFS root, which used to persist a deliberate all-zero placeholder (harmless
under cosine distance, a false attractor under a downstream engine's
squared-euclidean serving — a production incident this week), is now created
unvectored, and an existing store's legacy root is migrated on open by a single
fixed-path read before the health gate runs — never a walk.
- **Enumeration keys on the identity record.** `getNouns()` / `getVerbs()` and the
cursor walks behind them enumerate by the metadata record, the same key the
canonical ledger counts by — previously the walk keyed on the vector file, so a
row holding metadata but no vector was counted yet never yielded (a permanent
"missing" phantom in coverage math), while an orphaned vector-only directory
could be yielded as a phantom id. The recovery fold also never deletes an existing
vector when it replays a metadata-only after-image (preserve-if-absent). One
documented gap remains: a verb's endpoints live only in its vector leg, so a
metadata-only verb is counted and loudly skipped, never fabricated — the fix is a
canonical-format change and lands with the open format.
- **The ledger's one-time derivation counts identity records.** Stores upgraded from
pre-ledger versions derived their ALL-visibility scalars once by counting id
directories, which included ghost and scar containers left by an old partial-delete
defect — an inflated denominator whose coverage row could never reach exact. The
derivation now counts only directories holding a metadata record, `counts.json`
carries a derivation-rule stamp, and a ledger derived under the old rule is marked
`suspect` at open (one O(1) field read, one warning) so the online `repairIndex()`
path clears it with a real recount.
- **The vector index refuses what it cannot hold.** `rebuild()` skips unvectored and
zero-norm rows (one summary line), re-pins the vector dimension from the first real
vector after a restart (previously a restart left the pin unset, so a wrong-length
insert became the new pin instead of being rejected), and `addItem` / `updateItem`
throw a typed `EmptyVectorIndexError` on a length-0 vector instead of ever storing
a vector-less node.
- **Smaller:** a failing plugin activation now rethrows with the original error as
`cause` (the originating file and line survive to the caller's log); build
generators stamp from the repository history of their inputs instead of wall clock,
so two builds of the same tree are byte-identical.
Adoption: one restart, paired with its native-engine release. The first open of an
existing store runs the legacy-root migration (one narrated line) and, on stores that
upgraded from pre-ledger versions, marks the ledger suspect until the next sanctioned
recount — no rebuild in either case.
## v10.4.1 — 2026-08-26 (reads refuse per family; an unchanged write never re-embeds)
Two production defects from the same week, fixed together as a patch to 10.4.0.
- **The read gate is per family.** A read now refuses only when the index family it
actually consults is unhealthy: a metadata filter is served while the vector leg is
rebuilding; a semantic query is refused only by the vector family; a graph
traversal only by the graph family. Previously any unhealthy family refused every
read on the brain — under a long vector rebuild, a production deployment's
metadata-only reads were refused for the duration, and the retries became a write
pump of their own.
- **Unchanged data never re-embeds.** `update()` compares the incoming `data`
structurally with the stored record; an update carrying identical data (a common
shape for periodic upserts) no longer embeds again and no longer churns the vector
leg. Previously every such update re-embedded and re-inserted, which under load
saturated the vector index with near-identical vectors.
Adoption: one restart, paired with its native-engine release.
## v10.4.0 — 2026-08-25 (the health report has a name)
Three related cures, one root cause: an index deciding whether it could be trusted
by sampling itself instead of by exact accounting. This release replaces every
sampled self-probe with ledger-derived truth, and a read against an unhealthy index
now refuses loudly instead of guessing.
- **The canonical count ledger.** Storage now tracks two scalars per family
(nouns/verbs) on the write path: the user-facing `counted` total — unchanged,
still what `getNounCount()` / `getVerbCount()` return — and a new ALL-visibility
`all` total covering every tier, the real denominator a derived index's own
coverage math needs. The unfiltered storage-level `totalCount` returned by
`getNouns()` / `getVerbs()` is now this unclamped ALL scalar; previously it could
only ever move up (`Math.max(scalar, scanned)`), so an inflated counter could
never self-correct. A delete that cannot prove the record it removed actually
existed (no canonical read, no prior image available) no longer decrements on
faith — it marks the ledger `suspect` (narrated once per session) instead of
silently drifting, and the next `repairIndex()` clears the flag with a real
recount.
- **One contract for a throwing health probe.** A provider's `validateInvariants()`
is documented to never throw — but if one does anyway (a bug, a transient fault),
it is now read the same way everywhere: `heal: 'none'`, the error named in the
report, never synthesized into a rebuild trigger and never swallowed into "looks
fine." A flaky check can no longer buy itself a rebuild. `repairIndex()`'s
per-family receipt also gains `missing` (an exact count plus a capped id sample),
`rebuilt` (a full rebuild ran, vs. an incremental heal), and `reason`.
- **The named health report; reads refuse instead of rebuilding.** Any index
provider may now expose a synchronous, O(1) `healthReport()` — composed from the
provider's own exact ledgers, never a sample — and this is the one signal
Brainy's read gate trusts. The first-query lazy-build path is gone: `brain.init()`
now runs every needed rebuild to completion before it returns, always, regardless
of dataset size. A read that lands on a provider whose health report says it
isn't serving throws a typed error instead of triggering a rebuild mid-query —
`GraphIndexNotReadyError`, `MetadataIndexNotReadyError`, or
`VectorIndexNotReadyError` (all exported from `@soulcraft/brainy`), naming the
reasons. `repairIndex({ rebuild: ['metadata' | 'graph' | 'vector'] | 'all' })` is
the new explicit operator door: it rebuilds the named family unconditionally, no
health check consulted — reach for it when you have independent reason to
distrust a family regardless of what it self-reports. Bare `repairIndex()` is
unchanged in spirit: report-driven, heals only what its own checks say needs it.
- New concept doc: [Index Health](docs/concepts/index-health.md) walks the whole
story from a consumer's side — degraded-but-serving vs. not-ready, what
`repairIndex()` checks and heals per family, what `suspect` counts mean.
**Nothing to change to adopt this.** No API removed, no signature narrowed —
`repairIndex()` gains an optional options bag and its return value gains fields,
both additive. The honest notes: if your code ever relied on a `find()` against a
cold/not-yet-built index quietly triggering a rebuild and returning results a beat
later, that behavior is gone — it now throws one of the three typed
`*NotReadyError` classes instead (catch them if you need to distinguish "not ready
yet" from "no results"). And `disableAutoRebuild: true` no longer defers index
construction to the first query — a needed rebuild always runs at `open()` now;
the flag has no effect on timing. Full manual control still lives in
`repairIndex({ rebuild: [...] })`.
- **Crash-reopen catchup.** After an unclean shutdown, the metadata index now
folds the exact fact window it missed — `find()` serves every acked write on
reopen, closing the gap where canonical reads and counts recovered a
crash-window write but the index kept serving its pre-crash state until the
next full rebuild. Related root-cause fixed alongside: `close()` never
stamped the index watermarks (only `flush()` did), so a close without a
prior flush caused a needless full rescan verdict on the next open.
- **Relation rows are live in the metadata index.** Previously verb rows
entered the metadata index only during a rebuild — so a rebuilt store's
relation postings went stale from the first `relate()` after it. Relations
are now posted and retracted on the live write path (relate / unrelate /
updateRelation / remove's cascade, and their `transact()` forms), in the
same commit as the graph leg.
- **The metadata rebuild is online.** `rebuild()` for the metadata family no
longer clears and rebuilds in place (reads went empty for the duration): it
builds a complete replacement beside the serving index, mirrors concurrent
writes to both, swaps atomically, and persists once after the swap. Reads
never observe a partial index. `repairIndex({ rebuild: ['metadata'] })` uses
it automatically.
- **Incremental heal is routed.** A provider invariant that asks for the
incremental heal (`heal: 'repair'`) now routes to the provider's own
`repair()` when it exposes one — re-posting exactly what its ledger names,
never a store-sized rebuild — and the post-heal re-read of the report decides
success; a repair that doesn't converge is recorded with the escalation named.
- **The vector family joins the count ledger.** `getCanonicalCounts()` gains
`vectors: { all }` — the count of canonical entities holding a real vector
(deferred-embed entities count when their vector lands). And the open gate
closes the vector leg: a store whose canonical rows hold vectors but whose
derived vector index is empty now builds at `open()` (or refuses with the
typed error) instead of silently serving empty vector-search results.
- **An unknown storage config shape fails loudly.** A nested `config` object
carrying a path-shaped key (a shape that was never supported) used to fall
through silently to the default shared directory — every instance writing one
store while callers believed each had its own. It now throws, naming the
canonical `path` key.
- **Relation index rows are JSON-safe.** Internal endpoint identifiers can no
longer ride the metadata-index crossing (a native provider serializes it);
they stay on the graph operations where they belong.
- **A broken accelerator install can never read as "not installed."** The
auto-detection free pass now requires the resolution error to name the
accelerator package itself, exactly — a missing platform-binary sibling
package, an inner file path, or a dependency failure is a broken install and
`init()` throws loudly. And a plugin that declines activation is narrated on
the always-on log channel, so `silent: true` can no longer hide a fallback
to the default engines.
---
## v10.3.1 — 2026-08-18 (the fold that behaves)
Three recovery cures from one production first-boot incident (a brain's first

View file

@ -30,7 +30,7 @@ commit to backporting fixes to unsupported lines.
## Scope
This policy covers the `@soulcraft/brainy` package itself — the code in
This policy covers the `@soulcraftlabs/brainy` package itself — the code in
this repository. If you're evaluating a deployment that also uses
`@soulcraft/cor`, report issues in that package the same way, to the same
address; we'll route internally.

View file

@ -3,7 +3,7 @@
/**
* Modern TypeScript CLI Runner
*
* This is the entry point after npm install @soulcraft/brainy
* This is the entry point after npm install @soulcraftlabs/brainy
* It runs the compiled TypeScript CLI code
*/

View file

@ -3,7 +3,7 @@
"configVersion": 0,
"workspaces": {
"": {
"name": "@soulcraft/brainy",
"name": "@soulcraftlabs/brainy",
"dependencies": {
"@aws-sdk/client-s3": "^3.540.0",
"@azure/identity": "^4.0.0",

View file

@ -25,13 +25,13 @@
### Prerequisites
```bash
npm install @soulcraft/brainy
npm install @soulcraftlabs/brainy
```
### Your First Neural Database
```typescript
import { Brainy, NounType } from '@soulcraft/brainy'
import { Brainy, NounType } from '@soulcraftlabs/brainy'
// Step 1: Create and initialize Brainy
const brain = new Brainy({
@ -143,7 +143,7 @@ Once you're comfortable with basic operations, move to **Level 2** to learn abou
### Building a Knowledge Graph
```typescript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
import { Brainy, NounType, VerbType } from '@soulcraftlabs/brainy'
const brain = new Brainy({ storage: { type: 'memory' } })
await brain.init()
@ -314,7 +314,7 @@ Ready for AI-powered search and clustering? Move to **Level 3**.
### Triple Intelligence in Action
```typescript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
import { Brainy, NounType, VerbType } from '@soulcraftlabs/brainy'
const brain = new Brainy({ storage: { type: 'memory' } })
await brain.init()
@ -529,7 +529,7 @@ Want to treat files as intelligent entities? Learn the **Virtual Filesystem** in
### Files as Intelligent Entities
```typescript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
import { Brainy, NounType, VerbType } from '@soulcraftlabs/brainy'
const brain = new Brainy({ storage: { type: 'memory' } })
await brain.init()
@ -832,7 +832,7 @@ Ready for production deployment? Level 5 covers **planet-scale architecture**.
### Production-Ready Deployment
```typescript
import { Brainy, NounType } from '@soulcraft/brainy'
import { Brainy, NounType } from '@soulcraftlabs/brainy'
// 1. PRODUCTION STORAGE - Filesystem with off-site snapshots
console.log('Initializing production storage...\n')

View file

@ -1217,7 +1217,7 @@ where: {
await brain.find({ type: 'Document' })
// ✅ Correct: Use NounType enum
import { NounType } from '@soulcraft/brainy'
import { NounType } from '@soulcraftlabs/brainy'
await brain.find({ type: NounType.Document })
// ❌ Error: Operator not recognized

View file

@ -153,13 +153,13 @@ brainy-data/
### Step 1: Update Brainy Package
```bash
npm install @soulcraft/brainy@latest
npm install @soulcraftlabs/brainy@latest
```
**Check your version:**
```bash
npm list @soulcraft/brainy
# Should show: @soulcraft/brainy@4.0.0
npm list @soulcraftlabs/brainy
# Should show: @soulcraftlabs/brainy@4.0.0
```
### Step 2: No Code Changes Required! ✅
@ -374,7 +374,7 @@ If you encounter issues, you can rollback:
```bash
# Reinstall v3
npm install @soulcraft/brainy@^3.50.0
npm install @soulcraftlabs/brainy@^3.50.0
# Restart application
```
@ -389,7 +389,7 @@ rm -rf ./data
cp -r ./data-backup ./data
# Reinstall v3
npm install @soulcraft/brainy@^3.50.0
npm install @soulcraftlabs/brainy@^3.50.0
```
## Common Migration Scenarios
@ -539,7 +539,7 @@ console.log('Storage type:', status.type)
**Migration Checklist:**
- ✅ Backup data
- ✅ Update npm package (`npm install @soulcraft/brainy@latest`)
- ✅ Update npm package (`npm install @soulcraftlabs/brainy@latest`)
- ✅ Restart application (automatic migration)
- ✅ Verify data integrity
- ✅ Enable lifecycle policies

View file

@ -323,58 +323,24 @@ Only the graph adjacency index carries a committed scale assertion:
- ✅ **Single-Node by Design**: One process owns one `path`; scale out at the service layer
- ✅ **Zero Stubs**: Every line of code is production-ready
## Lazy Loading Performance
## Index Build at Open (10.4+)
Brainy supports two initialization modes for optimal performance across different use cases:
As of 10.4, `brain.init()` runs every needed index rebuild to completion before
it returns — always, regardless of dataset size. There is no lazy,
first-query rebuild path: a brain either finishes opening healthy, or `init()`
fails loudly. `disableAutoRebuild` no longer defers index construction to a
first query; it has no effect on *when* a rebuild runs. Manual control over
rebuilds is `repairIndex({ rebuild: [...] })`. See
[Index Health](concepts/index-health.md) for the full read-gate contract
(providers self-report readiness via `healthReport()`; a read against a
not-serving provider throws a typed `*NotReadyError` rather than rebuilding
mid-query).
### Mode 1: Auto-Rebuild (Default)
```javascript
const brain = new Brainy()
await brain.init() // Rebuilds indexes during init (~500ms-3s for 10K entities)
```
**Performance:**
- Init time: 500ms-3s (depends on dataset size)
- First query: Instant (indexes already loaded)
- Use case: Traditional applications, long-running servers
### Mode 2: Lazy Loading
```javascript
const brain = new Brainy({ disableAutoRebuild: true })
await brain.init() // Returns instantly (0-10ms)
const results = await brain.find({ limit: 10 }) // First query triggers rebuild (~50-200ms)
const more = await brain.find({ limit: 100 }) // Subsequent queries instant (0ms check)
```
**Performance:**
- Init time: 0-10ms (instant)
- First query: 50-200ms (includes index rebuild for 1K-10K entities)
- Subsequent queries: 0ms check (instant)
- Concurrent queries: Wait for same rebuild (mutex prevents duplicates)
**Concurrency Safety:**
```javascript
// 100 concurrent queries immediately after init
await brain.init()
const promises = Array.from({ length: 100 }, () =>
brain.find({ limit: 10 })
)
const results = await Promise.all(promises)
// ✅ Only 1 rebuild triggered (mutex)
// ✅ All 100 queries return correct results
// ✅ Total time: ~60ms (not 6000ms!)
```
**Use Cases for Lazy Loading:**
- **Serverless/Edge**: Minimize cold start time (0-10ms init)
- **Development**: Faster restarts during development
- **Large datasets**: Defer index loading until needed
- **Read-heavy workloads**: Writes don't wait for index rebuild
<!-- The pre-10.4 "Mode 2: Lazy Loading on First Query" section previously
documented here (disableAutoRebuild deferring index construction to the
first find() call) described a real, now-retired code path. Removed
rather than left to mislead; the concept doc above is the current
contract. -->
## Zero Configuration Required
@ -384,10 +350,6 @@ Brainy is designed to be **smart enough to tune itself dynamically**. No configu
// That's it. Brainy handles everything.
const brain = new Brainy()
await brain.init()
// Or with lazy loading for serverless
const brain = new Brainy({ disableAutoRebuild: true })
await brain.init() // Instant (0-10ms)
```
### Automatic Self-Tuning
@ -395,7 +357,6 @@ await brain.init() // Instant (0-10ms)
- **Metadata Index**: Auto-builds sorted indices for range queries on first use
- **Graph Index**: Auto-flushes every 30 seconds
- **Default Tuning**: Research-based vector index defaults
- **Lazy Loading**: Indices built only when needed
- **Cache Management**: LRU caches with TTL
### Intelligent Defaults

View file

@ -10,7 +10,7 @@ next:
- guides/storage-adapters
---
# Plugin Development Guide
# Plugin System
Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cor` provides optional native acceleration, and it's the same system available to any developer.
@ -46,7 +46,7 @@ If no plugin provides a given key, brainy uses its built-in JavaScript implement
### 1. Implement the `BrainyPlugin` interface
```typescript
import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin'
import type { BrainyPlugin, BrainyPluginContext } from '@soulcraftlabs/brainy/plugin'
const myPlugin: BrainyPlugin = {
name: 'my-brainy-plugin', // Must be unique (typically your npm package name)
@ -90,7 +90,7 @@ await brain.init()
**Programmatic registration:** For plugins not installed as npm packages, use `brain.use()`:
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
import myPlugin from './my-plugin.js'
const brain = new Brainy()
@ -200,15 +200,30 @@ members so a warm reopen never pays a redundant rebuild-from-canonical:
- **`init?(): Promise<void>`** — eager cold-load. Brainy awaits it once during
`brain.init()`, after the metadata provider's `init()` (the id-mapper hydrates first)
and **before the rebuild gate**.
- **`isReady?(): boolean`** — honest durability signal. `true` ⇔ the persisted index is
loaded (or cheaply demand-loadable) and consistent with what was last persisted. When
exposed, the rebuild gate defers to this signal **instead of** the `size() === 0` /
`totalEntries === 0` heuristics — a disk-native index may report 0 resident entries
while fully durable. Never return `true` if the durable state failed to load: the
signal is honest in both directions, and a not-ready provider gets its rebuild even
when `size() > 0`.
- **`healthReport?(): HealthReport`** — the PREFERRED signal (10.4+). A named,
synchronous, O(1) verdict derived from the provider's own exact ledgers — never a
sample, never I/O, must never throw for a well-formed provider. Brainy's read gate
(`assessProviderHealth()`) reads this INSTEAD of `isReady()` / size heuristics when
present: `serving: false` refuses the read with a typed `*NotReadyError` rather than
triggering a rebuild — a read never starts a store walk. `healthy` marks every
*verified* invariant holding; a family named in `unledgered` counts as neither
healthy nor broken. See `HealthReport` / `LedgerInvariantResult` /
`InvariantSource` in `src/plugin.ts`, and
[Index Health](concepts/index-health.md) for the consumer-facing story.
- **`isReady?(): boolean`** — honest durability signal, the fallback when
`healthReport()` is absent. `true` ⇔ the persisted index is loaded (or cheaply
demand-loadable) and consistent with what was last persisted. When exposed, the
gate defers to this signal **instead of** the `size() === 0` / `totalEntries === 0`
heuristics — a disk-native index may report 0 resident entries while fully durable.
Never return `true` if the durable state failed to load: the signal is honest in
both directions, and a not-ready provider gets its rebuild even when `size() > 0`.
- **`isMigrating?(): boolean`** — while `true`, the provider owns its index (background
migration); brainy skips its rebuild entirely.
- **`validateInvariants?(): Promise<ProviderInvariantReport>`** — the async DEEP
diagnostic (full scans allowed), distinct from the bounded, sync `healthReport()`.
Must never throw — a failure is `healthy: false` data, not an exception; a provider
that throws anyway is read as a loud, unverified failure (never as "healthy") by
every caller, never silently retried into a rebuild.
Providers that implement none of these keep the size/count heuristics — correct for
engines whose `rebuild()` *is* their load path (like brainy's built-in JS vector index).
@ -257,10 +272,10 @@ When provided by an optional native acceleration plugin (such as `@soulcraft/cor
#### `cache`
**Type:** `UnifiedCache`
Replaces the global `UnifiedCache` singleton used for VFS path resolution, semantic caching, and vector index caching. Must implement the `UnifiedCache` interface (available from `@soulcraft/brainy/internals`).
Replaces the global `UnifiedCache` singleton used for VFS path resolution, semantic caching, and vector index caching. Must implement the `UnifiedCache` interface (available from `@soulcraftlabs/brainy/internals`).
```typescript
import type { UnifiedCache } from '@soulcraft/brainy/internals'
import type { UnifiedCache } from '@soulcraftlabs/brainy/internals'
context.registerProvider('cache', myNativeCache)
```
@ -310,8 +325,8 @@ Plugins can register custom storage backends that users reference by name.
### Implementing a Storage Adapter
```typescript
import type { StorageAdapterFactory } from '@soulcraft/brainy/plugin'
import type { StorageAdapter } from '@soulcraft/brainy'
import type { StorageAdapterFactory } from '@soulcraftlabs/brainy/plugin'
import type { StorageAdapter } from '@soulcraftlabs/brainy'
class MyStorageAdapter implements StorageAdapter {
async init(): Promise<void> { /* ... */ }
@ -345,9 +360,9 @@ Brainy provides three entry points for plugin developers:
| Import Path | Contents | Stability |
|-------------|----------|-----------|
| `@soulcraft/brainy` | Public API, types, StorageAdapter | Stable (semver) |
| `@soulcraft/brainy/plugin` | BrainyPlugin, BrainyPluginContext, StorageAdapterFactory | Stable (semver) |
| `@soulcraft/brainy/internals` | UnifiedCache, EntityIdMapper, logger utilities | Internal (may change between minor versions) |
| `@soulcraftlabs/brainy` | Public API, types, StorageAdapter | Stable (semver) |
| `@soulcraftlabs/brainy/plugin` | BrainyPlugin, BrainyPluginContext, StorageAdapterFactory | Stable (semver) |
| `@soulcraftlabs/brainy/internals` | UnifiedCache, EntityIdMapper, logger utilities | Internal (may change between minor versions) |
## Diagnostics
@ -425,7 +440,7 @@ A minimal but useful plugin that provides SIMD-accelerated distance calculations
```typescript
// simd-distance-plugin/src/plugin.ts
import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin'
import type { BrainyPlugin, BrainyPluginContext } from '@soulcraftlabs/brainy/plugin'
// Hypothetical native module
import { simdCosineDistance } from './native.js'
@ -455,7 +470,7 @@ export default simdDistancePlugin
"main": "./dist/plugin.js",
"types": "./dist/plugin.d.ts",
"peerDependencies": {
"@soulcraft/brainy": ">=7.0.0"
"@soulcraftlabs/brainy": ">=7.0.0"
}
}
```
@ -463,7 +478,7 @@ export default simdDistancePlugin
Usage:
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const brain = new Brainy({ plugins: ['brainy-simd-distance'] })
await brain.init()

View file

@ -54,7 +54,7 @@ After 40 API calls:
```typescript
// server.ts
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
// SINGLETON INSTANCE
let brainInstance: Brainy | null = null
@ -174,7 +174,7 @@ process.on('SIGTERM', async () => {
```typescript
// server.ts - Clean Bun implementation
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
let brain: Brainy | null = null

View file

@ -5,7 +5,7 @@
## Quick Start
```typescript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
import { Brainy, NounType, VerbType } from '@soulcraftlabs/brainy'
const brain = new Brainy()
await brain.init()

View file

@ -99,7 +99,7 @@ Examples:
```bash
# 1. Deprecate wrong version on npm
npm deprecate @soulcraft/brainy@X.X.X "Incorrect version - use Y.Y.Y"
npm deprecate @soulcraftlabs/brainy@X.X.X "Incorrect version - use Y.Y.Y"
# 2. Fix version in package.json
# 3. Republish correct version

View file

@ -13,7 +13,7 @@
### In-Memory
```typescript
import Brainy from '@soulcraft/brainy'
import Brainy from '@soulcraftlabs/brainy'
const brain = new Brainy({ storage: { type: 'memory' } })
```
@ -43,7 +43,7 @@ The native vector provider (via the optional `@soulcraft/cor` package) extends t
Numbers below are **measured** by `tests/benchmarks/find-composition-scale.js` (a single
Node 22 process, in-memory storage, 384-dim vectors, `balanced` recall). They are the
open-core (pure-TypeScript) path — what you get from `@soulcraft/brainy` with no native
open-core (pure-TypeScript) path — what you get from `@soulcraftlabs/brainy` with no native
provider installed. Run it yourself: `node --max-old-space-size=8192 tests/benchmarks/find-composition-scale.js 100000`.
`find()` query latency, p50 / p95 (200 queries each):

1544
docs/api-contract.json Normal file

File diff suppressed because it is too large Load diff

View file

@ -24,7 +24,7 @@ next:
## Quick Start
```typescript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
import { Brainy, NounType, VerbType } from '@soulcraftlabs/brainy'
const brain = new Brainy() // Zero config!
await brain.init() // VFS auto-initialized!
@ -1010,7 +1010,7 @@ await db.release() // unpin + free cached materialization
### Db API errors
All exported from `@soulcraft/brainy`:
All exported from `@soulcraftlabs/brainy`:
| Error | Thrown by | Meaning |
|---|---|---|
@ -1451,6 +1451,34 @@ const count = await brain.getVerbCount()
---
### The canonical count ledger (`StorageAdapter.getCanonicalCounts()`)
An OPTIONAL method on the `StorageAdapter` interface (implemented by both
built-in adapters), not a method on `Brainy` itself — relevant if you're
writing a custom storage adapter or composing a provider's own
`healthReport()`. O(1), no I/O. Per family (`nouns`/`verbs`):
```typescript
interface CanonicalCounts {
nouns: { counted: number; all: number }
verbs: { counted: number; all: number }
suspect: boolean
}
```
- `counted` mirrors `getNounCount()` / `getVerbCount()` (public + internal tiers).
- `all` is the ALL-visibility scalar — every tier, including system/internal
records — the denominator a derived index's own coverage math is measured
against.
- `suspect` is `true` when an unprovable delete has left `all` unverified since
the last recount; `brain.repairIndex()` clears it with a real canonical walk.
Adapters without the ledger omit the method; treat absence as "no
denominator," never as zero. See
**[Index Health](../concepts/index-health.md)** for the full story.
---
### Subtype & facet APIs
Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**.
@ -1831,6 +1859,104 @@ const semanticOnly = await brain.getStats({ excludeVFS: true })
---
### `repairIndex(options?)``Promise<RepairReport>`
The ceremony door for index repair. Bare `repairIndex()` is report-driven: it
prunes orphaned containers, recomputes count rollups, reconciles VFS
containment, and rebuilds only a derived-index family whose own health check
asks for it. Pass `options.rebuild` to force one or more families to rebuild
UNCONDITIONALLY — no health check is consulted — when an operator has
independent reason to reconcile a family regardless of what it self-reports.
```typescript
// Report-driven: only heals what actually needs it
const report = await brain.repairIndex()
console.log(report.healedTotal, report.families)
// Explicit: force the graph adjacency to rebuild from canonical, unconditionally
await brain.repairIndex({ rebuild: ['graph'] })
// Explicit: force all three derived indexes to rebuild
await brain.repairIndex({ rebuild: 'all' })
```
**`RepairReport`:**
- `families: RepairFamilyReport[]` — one row per family checked
- `healedTotal: number` — items healed across every family
- `durationMs: number`
**`RepairFamilyReport`** (one row):
- `family: string` — e.g. `'orphaned-containers'`, `'count-rollups'`,
`'vfs-containment'`, `'metadata-corruption'`, `'provider:metadata'`,
`'provider:graph'`, `'provider:vector'`
- `checked: boolean` — was this family actually examined (`false` ⇒ see `skipped`)
- `healed: number` — items re-posted/corrected in place (the incremental heal count)
- `missing?: { count: number; sample: string[] }` — exact count plus a capped id
sample when the check can name what diverged (never the full list)
- `rebuilt?: boolean` — a full generational rebuild ran (vs. an incremental heal)
- `detail?: string` / `reason?: string` — narration
- `skipped?: string` — why the family wasn't checked
Full walkthrough — what each family checks, degraded-but-serving vs. not-ready,
and what `suspect` counts mean — in
**[Index Health](../concepts/index-health.md)**.
---
### Index readiness: typed errors, `healthReport()`, `disableAutoRebuild`
Every derived-index provider (vector, graph, metadata) may expose a named,
synchronous, O(1) `healthReport()` composed from its own exact ledgers — the
signal Brainy's read gate trusts over sampling or size heuristics. `init()`
brings every provider to serving before it returns; there is no first-query
lazy-rebuild path. A read that reaches a provider whose health report says it
isn't serving throws instead of rebuilding mid-query:
| Error | Thrown by | Meaning |
|---|---|---|
| `GraphIndexNotReadyError` | `find({ connected })`, `neighbors()`, `related()` | Graph adjacency isn't serving |
| `MetadataIndexNotReadyError` | `find({ where })` | Metadata/field index isn't serving |
| `VectorIndexNotReadyError` | `find({ query })`, `similar()` | Vector index isn't serving |
All three are exported from `@soulcraftlabs/brainy`. Catch them to distinguish
"index not ready" from a genuine empty result:
```typescript
import { MetadataIndexNotReadyError } from '@soulcraftlabs/brainy'
try {
const rows = await brain.find({ where: { status: 'active' } })
} catch (err) {
if (err instanceof MetadataIndexNotReadyError) {
// reconcile: await brain.repairIndex(), then retry
} else {
throw err
}
}
```
**`disableAutoRebuild`** no longer defers index construction to the first
query. A needed rebuild always runs at `open()`, regardless of this flag or
dataset size; the flag has no effect on *when* a rebuild runs. Full manual
control lives in `repairIndex({ rebuild: [...] })`, above.
### `validateIndexConsistency()``Promise<...>`
The deep, async diagnostic counterpart to `healthReport()` — safe to run on a
live brain, but does more work (a provider's `validateInvariants()` may run a
full scan, not just read a ledger). Aggregates the JS metadata index's own
consistency check with every derived-index provider's invariant report.
```typescript
const validation = await brain.validateIndexConsistency()
if (!validation.healthy) {
console.log(validation.recommendation) // what to run, e.g. repairIndex()
console.log(validation.providers) // each provider's own invariant report, when exposed
}
```
---
## Lifecycle
### Initialization
@ -2082,7 +2208,7 @@ For the full taxonomy with all 169 types and their descriptions, see:
- **📖 Documentation:** [Full Documentation](../)
- **🐛 Issues:** [GitHub Issues](https://github.com/soulcraftlabs/brainy/issues)
- **💬 Discussions:** [GitHub Discussions](https://github.com/soulcraftlabs/brainy/discussions)
- **📦 NPM:** [@soulcraft/brainy](https://www.npmjs.com/package/@soulcraft/brainy)
- **📦 NPM:** [@soulcraftlabs/brainy](https://www.npmjs.com/package/@soulcraftlabs/brainy)
- **⭐ GitHub:** [Star us](https://github.com/soulcraftlabs/brainy)
---

View file

@ -268,7 +268,7 @@ locks/_flush_responses/ # writer answers with <uuid>.ack
| **Counts/statistics** | Per-type and per-subtype maps | `_system/{type,subtype,verb-subtype}-statistics.json.gz`, `counts.json` | Recomputable by scanning entities (`brainy inspect repair`) |
A pluggable index provider (the 8.0 plugin contract in
`@soulcraft/brainy/plugin`) may replace any of the JS implementations; the
`@soulcraftlabs/brainy/plugin`) may replace any of the JS implementations; the
persisted formats above are contract-bound so JS and native implementations
can interleave on the same directory.

View file

@ -126,7 +126,7 @@ class TypeAwareMetadataIndex {
**The Design**: Specify types clearly in your API calls:
```typescript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
import { Brainy, NounType, VerbType } from '@soulcraftlabs/brainy'
// Add entity with explicit type
await brain.add({
@ -231,7 +231,7 @@ class OrgEnrichmentAugmentation {
**Brainy's Approach**: Extract **typed** concepts:
```typescript
import { NaturalLanguageProcessor } from '@soulcraft/brainy'
import { NaturalLanguageProcessor } from '@soulcraftlabs/brainy'
const nlp = new NaturalLanguageProcessor()
const concepts = await nlp.extractConcepts("Alice works at Google in San Francisco")
@ -382,7 +382,7 @@ import {
getVerbTypes,
BrainyTypes,
suggestType
} from '@soulcraft/brainy'
} from '@soulcraftlabs/brainy'
// Get all available noun types
const nounTypes = getNounTypes()

View file

@ -723,6 +723,14 @@ async stats(): Promise<Statistics> {
### 5. Index Rebuilding (Lazy Loading Support)
> **Stale as of 10.4 — "Mode 2: Lazy Loading on First Query" below is
> RETIRED.** `disableAutoRebuild` no longer defers index construction to a
> first query; `brain.init()` now runs every needed rebuild to completion
> before it returns, unconditionally, and a read against a not-serving
> provider throws a typed `*NotReadyError` instead of rebuilding mid-query.
> See `docs/concepts/index-health.md` for the current contract. Left below
> as historical background on the rebuild mechanics.
**Two modes of index loading:**
#### Mode 1: Auto-Rebuild on init() (default)

View file

@ -1,5 +1,15 @@
# Initialization and Rebuild Processes
> **Stale as of 10.4 — "Mode 2: Lazy Loading on First Query" below is RETIRED.**
> `disableAutoRebuild` no longer defers index construction to a first query;
> `brain.init()` now runs every needed rebuild to completion before it
> returns, unconditionally. A read against a not-serving provider throws a
> typed `*NotReadyError` instead of rebuilding mid-query. See
> `docs/concepts/index-health.md` for the current contract; this document's
> line-number references to `src/brainy.ts` also predate the file's current
> size and are unreliable. Left as historical background on the rebuild
> mechanics, not as a current API description.
This document explains how Brainy's four indexes (MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
## Core Principle: All Indexes Are Disk-Based

View file

@ -127,7 +127,7 @@ For reference, a clean migration path:
`isMultiProcessSafe` type-guard. Keep `hasStorageMethod` for
build/install artifact protection.
5. Document the new contract in `concepts/storage-adapters.md`.
6. Major-version-bump the `@soulcraft/brainy` peerDep range expected by
6. Major-version-bump the `@soulcraftlabs/brainy` peerDep range expected by
plugins.
Estimated work: ~half a day of code, ~2 hours of doc/example updates,

View file

@ -20,7 +20,7 @@ next:
Every example on this page is written against the real Brainy 8.0 API. The setup is always the same:
```typescript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
import { Brainy, NounType, VerbType } from '@soulcraftlabs/brainy'
const brain = new Brainy()
await brain.init()
@ -40,7 +40,7 @@ Brainy's **Noun-Verb Taxonomy** achieves broad coverage of human knowledge throu
- **Multi-hop Graph Traversals = Relationship Complexity**
- **Result: Model data across virtually any industry**
Every piece of information can be represented as entities (nouns) connected by relationships (verbs) carrying properties (metadata). The standardized type system from `@soulcraft/brainy` (`NounType`, `VerbType`) gives those nouns and verbs a stable, shared name.
Every piece of information can be represented as entities (nouns) connected by relationships (verbs) carrying properties (metadata). The standardized type system from `@soulcraftlabs/brainy` (`NounType`, `VerbType`) gives those nouns and verbs a stable, shared name.
## The Power of Standardization: Universal Interoperability

View file

@ -35,7 +35,7 @@ constructor and `init()`.
## Instant Start
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
// That's it. No config needed.
const brain = new Brainy()

View file

@ -167,7 +167,7 @@ await brain.find({ orderBy: 'createdAt' })
`UnresolvableFieldError` is exported from the package root:
```typescript
import { UnresolvableFieldError } from '@soulcraft/brainy'
import { UnresolvableFieldError } from '@soulcraftlabs/brainy'
try {
await brain.find({ orderBy: 'createdAt' })

View file

@ -0,0 +1,217 @@
---
title: Index Health
slug: concepts/index-health
public: true
category: concepts
template: concept
order: 8
description: How Brainy knows whether a derived index can be trusted — exact accounting instead of sampling, the named health report, degraded-but-serving vs. not-ready, and what repairIndex() checks, heals, and rebuilds.
next:
- concepts/generation-fact-log
- guides/inspection
---
# Index Health
Brainy keeps one **canonical** copy of every entity and relationship, and three
**derived** indexes built from it — vector, metadata, and graph — so `find()` can
answer semantically, by filter, and by traversal without re-deriving the answer from
scratch on every query. A derived index is a cache with a serving structure: it can
be present but stale, present but only partially loaded, or fully out of sync with
canonical after a crash. This page is about how Brainy decides whether to trust one,
what it does when it can't, and how you reconcile the two.
## Exact accounting instead of sampling
Older health checks worked by inference: does `size()` return something greater
than zero, does a spot-check on one known item come back correct. Both are proxies.
A cold index can report a nonzero count while its actual serving structure never
loaded, and a spot-check only proves the one item it happened to ask about.
Every derived-index provider may now expose a named, synchronous, O(1)
`healthReport()` — composed from the provider's own **exact ledgers** (real counters
it already maintains on the write path), never a sample or a walk. This is the one
signal Brainy's read gate consults. A provider that doesn't yet expose one falls
back to an honest `isReady()` boolean, and finally to a size heuristic for engines
with neither — but wherever a `healthReport()` exists, it wins.
Underneath, storage itself keeps an analogous **canonical count ledger**: a
`counted` scalar (the user-facing total — what `getNounCount()` / `getVerbCount()`
return) and an `all` scalar (every tier, including internal records a derived
index's own coverage math needs to compare against). This is the real denominator
a provider's `healthReport()` measures itself by, rather than a total that can only
ever ratchet upward. See [What `suspect` counts mean](#what-suspect-counts-mean)
below for the one case that ledger can't stay exact through on its own.
## The named report
A `HealthReport` carries, per provider (`'vector'` / `'graph'` / `'metadata'`):
- **`healthy`** — `true` iff every *verified* invariant holds. An invariant whose
family has no ledger yet is `unledgered`, never counted either way — unknown,
not passing.
- **`serving`** — can this provider answer a query right now. A failing invariant
graded `heal: 'repair'` or `heal: 'none'` still leaves `serving: true` — this is
**degraded-but-serving**: something is off (say, a stale rollup on an
`employee` record's relationship count) but reads keep working. Only a failure
graded `heal: 'rebuild'` flips `serving` to `false`**not-ready** — because the
provider itself is telling you its serving structure cannot answer correctly.
- **`invariants`** — each checked condition, with its provenance
(`source: 'ledger'` — an exact count; `'deep'` — a full scan, diagnostic-only;
`'unledgered'` — not yet tracked) and, for a failing one, an exact `missing`
count plus a capped sample of the affected ids — a verdict, never a dump.
- **`generation`** — bumps on every ledger mutation and rebuild, so a caller can
cache a verdict per generation instead of re-deriving it.
The distinction that matters day to day: `healthy: false` can be entirely benign —
a maintenance window, a divergence `repairIndex()` will clean up on its own
schedule. `serving: false` is not benign. It means this provider is refusing to
answer, on its own word, right now.
**How a failure gets its grade — the serving law.** A provider grades `heal` by
one question only: *could an answer be wrong?* — never *how expensive is the
fix?* A missing-postings shortfall, however large, is `heal: 'repair'` (re-post
exactly what the ledger names, reads serving throughout); it can never withhold
serving just because healing it takes work. `serving` is withheld only by a
small, named set of rebuild-graded conditions — the index not initialized, its
durable state absent, a manifest naming files that are not resident, a replay
that did not complete cleanly — the states in which an answer could genuinely be
wrong. And a read is only ever refused by the family it actually consults: a
metadata filter is answered by the metadata index alone, vector search by the
vector index, traversal by the graph index — one family's refusal never blocks
another family's reads.
## Reads refuse — they never rebuild
A query that reaches a not-serving provider does not trigger a rebuild from inside
the read. Brainy retired that path deliberately: a rebuild kicked off by an ordinary
`find({ where: { status: 'active' } })` call is a dark, unpredictable cost hiding
behind a request that looks like a cheap read. Instead, the read throws a typed,
catchable error naming the reason:
| Error | Thrown when | Meaning |
|---|---|---|
| `GraphIndexNotReadyError` | `find({ connected })`, `neighbors()`, `related()` | The graph adjacency index isn't serving — traversal would otherwise return `[]` indistinguishable from "no relationships" |
| `MetadataIndexNotReadyError` | `find({ where })` | The metadata/field index isn't serving — a filtered read would otherwise return `[]` indistinguishable from "no matches" |
| `VectorIndexNotReadyError` | `find({ query })`, `similar()` | The vector index isn't serving — a semantic search would otherwise return `[]` indistinguishable from "nothing similar" |
All three are exported from `@soulcraftlabs/brainy`. Catch them where your application
needs to distinguish "this index isn't ready yet" from "there's genuinely nothing
here" — a health dashboard, a retry policy, an operator alert. The fix is always
the same: reconcile the index, either by reopening the brain (which brings every
provider to serving before `init()` returns — see the next section) or by calling
`repairIndex()` explicitly.
```typescript
try {
const active = await brain.find({ where: { status: 'active' } })
} catch (err) {
if (err instanceof MetadataIndexNotReadyError) {
// not a "no results" — the index itself refused; alert or retry after repair
} else {
throw err
}
}
```
### Rebuilds happen at open, not on first query
`brain.init()` runs every needed rebuild to completion **before it returns**,
unconditionally, regardless of dataset size. There is no lazy, first-query
rebuild path anymore — a brain either finishes opening healthy, or it fails
open loudly. `disableAutoRebuild: true` no longer defers index construction to
the first query: it has no effect on *when* a needed rebuild runs. Full manual
control over rebuilds is `repairIndex({ rebuild: [...] })` (below), not this flag.
## `repairIndex()` — checking and healing
Bare `repairIndex()` is **report-driven**: it only heals what its own checks say
actually needs it, and it always returns a full per-family receipt.
```typescript
const report = await brain.repairIndex()
report.healedTotal // total items healed across every family
report.durationMs
report.families // one row per family checked
```
Each `RepairFamilyReport` row names what happened:
- **`checked`** — was this family actually examined (`false` means skipped —
see `skipped` for why).
- **`healed`** — items re-posted or corrected in place.
- **`missing`** — when the check can name what diverged: an exact `count` plus a
capped `sample` of ids.
- **`rebuilt`** — a full generational rebuild ran (as opposed to an incremental
heal).
- **`detail`** / **`reason`** / **`skipped`** — the receipt's narration; a row is
always either checked or explains why it wasn't. Nothing is silent.
On every call, bare `repairIndex()`:
1. Prunes orphaned canonical containers left by a partial delete.
2. Recomputes the count rollups from one canonical walk (unconditional — this is
also what clears a `suspect` ledger; see below).
3. Reconciles VFS containment edges, if the VFS is initialized.
4. Runs the metadata index's own corruption detection pass.
5. Consults each of the three derived-index providers' own health check and
rebuilds only a family whose failing invariant actually asks for it
(`heal: 'rebuild'`) — never a provider that reports `healthy` or a lesser
grade.
### The explicit rebuild door
`options.rebuild` skips the health check and rebuilds one or more families
**unconditionally** — the operator override for when you have independent reason
to distrust a family regardless of what it self-reports (a suspicious deploy, a
storage-layer incident, a support ticket that doesn't match what the health report
says):
```typescript
// Force the graph adjacency to rebuild from canonical, no invariant consulted
await brain.repairIndex({ rebuild: ['graph'] })
// Force all three derived indexes
await brain.repairIndex({ rebuild: 'all' })
```
A family named this way is recorded with `rebuilt: true` and
`reason: 'explicit rebuild requested'`, and is skipped by the normal
health-driven pass in the same call — it was already rebuilt unconditionally.
Reach for the explicit door when you need certainty regardless of self-report;
reach for bare `repairIndex()` for routine maintenance and after any incident
where you're not sure which family (if any) needs it.
## What `suspect` counts mean
Storage's canonical count ledger increments the ALL-visibility total on every new
record and decrements it on every *proven* delete — one where the record was read,
or the caller supplied its prior image. A delete that cannot prove what it removed
existed doesn't guess: it flags the ledger `suspect` (an operator-visible
`console.warn`, narrated once per session, not once per delete) rather than risk
decrementing a total that was never incremented for that record in the first
place. This is intentionally rare — it's a defensive fallback for callers on an
unusual removal path, not a per-delete cost.
`suspect` is not directly exposed on any `Brainy` method today — it lives on the
`StorageAdapter`'s optional `getCanonicalCounts()`, primarily consulted by
`repairIndex()`'s recount step and by custom storage adapters composing their own
`healthReport()`. What matters for an application: a `suspect` ledger is not
incorrect, just *unverified since the last recount* — and `repairIndex()`'s
unconditional count-rollup step (step 2, above) recomputes the ALL scalars from a
real canonical walk on every call, clearing the flag with proof either way.
## Practical guidance
- **On a normal restart**, do nothing — `init()` brings every provider to
serving before it returns, or fails loudly.
- **On a `*NotReadyError`** from a live read, reconcile with `repairIndex()`
(report-driven is almost always sufficient) and retry.
- **After an incident** where you distrust a specific family regardless of what
it reports healthy — a storage-layer fault, a suspicious restore — use the
explicit door: `repairIndex({ rebuild: ['metadata' | 'graph' | 'vector'] })`.
- **To audit before trusting a report**, `brain.auditGraph()` walks every stored
relationship and proves (or disproves) that reads return canonical truth,
independent of what any provider self-reports — see
[Inspecting a Live Brainy](../guides/inspection.md).

View file

@ -61,7 +61,7 @@ The only required override is the capability flag. Returning `true` from
to call `acquireWriterLock()` at init.
```typescript
import { FileSystemStorage } from '@soulcraft/brainy'
import { FileSystemStorage } from '@soulcraftlabs/brainy'
export class MmapFileSystemStorage extends FileSystemStorage {
public supportsMultiProcessLocking(): boolean {
@ -79,7 +79,7 @@ If your storage is **not filesystem-backed** (a custom
network backend), extend `BaseStorage` directly:
```typescript
import { BaseStorage } from '@soulcraft/brainy'
import { BaseStorage } from '@soulcraftlabs/brainy'
export class MyCloudStorage extends BaseStorage {
// BaseStorage's default no-op implementations of the multi-process
@ -101,7 +101,7 @@ The defensive check at every new-storage-method call site (`brainy.ts`,
`hasStorageMethod(name)`) does **not** exist to handle "plugin bundles a
stale BaseStorage." Plugins ship a dist that preserves the dynamic ESM
import (verify in your plugin's `dist/`: `import { FileSystemStorage } from
'@soulcraft/brainy'` is not rewritten to a vendored copy). The prototype
'@soulcraftlabs/brainy'` is not rewritten to a vendored copy). The prototype
chain at runtime resolves to whatever Brainy version your consumer has
installed.
@ -109,8 +109,8 @@ installed.
the prototype chain at the consumer-app level:
- **Stale `node_modules`** — a lingering install from before the consumer
upgraded Brainy. The package.json says `@soulcraft/brainy@7.22.0` but
`node_modules/@soulcraft/brainy` is still 7.20.x.
upgraded Brainy. The package.json says `@soulcraftlabs/brainy@7.22.0` but
`node_modules/@soulcraftlabs/brainy` is still 7.20.x.
- **Lockfile drift**`bun.lockb` / `package-lock.json` pins a brainy
version older than the package.json range, and `bun install` honors the
lockfile.
@ -131,7 +131,7 @@ and the warning names the adapter class plus a remediation hint:
methods on its prototype chain. Writer locking and the flush-request RPC are
disabled for this directory. Likely fix: clean install (`rm -rf node_modules
bun.lockb && bun install`) or rebuild your container image to refresh
`@soulcraft/brainy` to ≥7.21. See docs/concepts/storage-adapters.md.
`@soulcraftlabs/brainy` to ≥7.21. See docs/concepts/storage-adapters.md.
```
## Authoring a new storage adapter — minimum checklist
@ -168,7 +168,7 @@ bun.lockb && bun install`) or rebuild your container image to refresh
install time — fix install, not your plugin.
6. **Pin your peer dep generously.** `"peerDependencies": {
"@soulcraft/brainy": "^7.21.0" }` accepts any compatible 7.x. Don't pin
"@soulcraftlabs/brainy": "^7.21.0" }` accepts any compatible 7.x. Don't pin
to an exact patch unless you're tracking a known regression.
## Future direction
@ -185,5 +185,5 @@ follow-up; consumers don't need to anticipate the change.
heartbeat semantics, what the lock protects.
- [`guides/inspection`](../guides/inspection.md) — `brainy inspect` and the
read-only mode.
- `node_modules/@soulcraft/brainy/dist/storage/baseStorage.d.ts` — the
- `node_modules/@soulcraftlabs/brainy/dist/storage/baseStorage.d.ts` — the
authoritative type signatures for every method this page references.

View file

@ -22,7 +22,7 @@ they share a single scan.
## Quick Start
```typescript
import { Brainy, NounType } from '@soulcraft/brainy'
import { Brainy, NounType } from '@soulcraftlabs/brainy'
const brain = new Brainy()
await brain.init()

View file

@ -8,7 +8,7 @@ Brainy is **framework-friendly** - designed to drop into the server side of any
Brainy embeds an HNSW vector index, a graph engine, and a filesystem-backed persistence layer. These belong on the server:
- **Zero configuration**: Just `import { Brainy } from '@soulcraft/brainy'`
- **Zero configuration**: Just `import { Brainy } from '@soulcraftlabs/brainy'`
- **Auto storage detection**: `new Brainy()` auto-selects filesystem persistence on Node
- **Cleaner code**: No browser polyfills, no conditional client/server imports
- **Better DX**: One instance shared across your server routes
@ -18,13 +18,13 @@ Brainy embeds an HNSW vector index, a graph engine, and a filesystem-backed pers
### Install Brainy
```bash
npm install @soulcraft/brainy
npm install @soulcraftlabs/brainy
```
### Basic Integration
```javascript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
// Run on the server (API route, server component, backend service)
// new Brainy() auto-detects filesystem persistence on Node
@ -105,7 +105,7 @@ On the server, create one Brainy instance and reuse it across requests. This mod
```javascript
// lib/brain.server.js
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
let brainPromise
@ -163,7 +163,7 @@ On the server, create one Brainy instance and reuse it across requests:
```javascript
// server/brain.js (server-only module)
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
let brainPromise
@ -248,7 +248,7 @@ The matching backend endpoint uses Brainy directly (Node/Bun):
```typescript
// server: api/search
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const brain = new Brainy() // auto-detects filesystem persistence on Node
await brain.init()
@ -266,7 +266,7 @@ In Next.js, Brainy lives in server code only: API routes, server components, or
```javascript
// lib/brain.server.js (imported only by server code)
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
let brainPromise
@ -318,7 +318,7 @@ Brainy runs in a server-only module (`*.server.js`); the component fetches resul
```javascript
// src/lib/server/brain.js (server-only — note the .server suffix)
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
let brainPromise
@ -432,7 +432,7 @@ import { defineConfig } from 'vite'
export default defineConfig({
ssr: {
external: ['@soulcraft/brainy']
external: ['@soulcraftlabs/brainy']
}
})
```
@ -440,7 +440,7 @@ export default defineConfig({
```javascript
// rollup.config.js (server bundle)
export default {
external: ['@soulcraft/brainy', 'node:fs', 'node:path', 'node:crypto']
external: ['@soulcraftlabs/brainy', 'node:fs', 'node:path', 'node:crypto']
}
```
@ -466,7 +466,7 @@ export async function load({ url }) {
```javascript
// For build-time usage (runs in Node during the build)
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
export async function generateStaticProps() {
const brain = new Brainy({
@ -513,7 +513,7 @@ export async function generateStaticProps() {
### Issue: Large client bundle size
**Cause**: A client module is pulling in Brainy.
**Solution**: Move the `import { Brainy } from '@soulcraft/brainy'` into a server-only module so it never reaches the browser bundle.
**Solution**: Move the `import { Brainy } from '@soulcraftlabs/brainy'` into a server-only module so it never reaches the browser bundle.
### Issue: SSR hydration mismatch
**Solution**: Run the search on the server (loader / server action / API route) and pass the results down as props, so server and client render the same markup.

View file

@ -9,7 +9,7 @@ Brainy's import is **ONE magical method** that understands EVERYTHING:
## The Ultimate Simplicity
```javascript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const brain = new Brainy()
await brain.init()

View file

@ -13,7 +13,7 @@ Brainy provides real-time progress tracking for **all 7 supported file formats**
### Basic Progress Tracking
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
import * as fs from 'fs'
const brain = await Brainy.create()

View file

@ -7,7 +7,7 @@
## Basic Import
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const brain = new Brainy()
await brain.init()
@ -187,7 +187,7 @@ await brain.import(file, {
## Complete Example
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
import * as fs from 'fs'
async function importCatalog() {

View file

@ -108,7 +108,7 @@ check fails — useful for piping into monitoring or CI.
## Programmatic inspection
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const reader = await Brainy.openReadOnly({
storage: { type: 'filesystem', path: '/data/brain' }

View file

@ -21,21 +21,21 @@ next:
## Install
```bash
npm install @soulcraft/brainy
npm install @soulcraftlabs/brainy
```
Or with your preferred package manager:
```bash
bun add @soulcraft/brainy
yarn add @soulcraft/brainy
pnpm add @soulcraft/brainy
bun add @soulcraftlabs/brainy
yarn add @soulcraftlabs/brainy
pnpm add @soulcraftlabs/brainy
```
## Verify
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const brain = new Brainy()
await brain.init()
@ -52,7 +52,7 @@ npm install @soulcraft/cor
```
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const brain = new Brainy({ plugins: ['@soulcraft/cor'] })
await brain.init() // native providers registered during init
@ -71,7 +71,7 @@ remains available on npm if you need it.
Brainy ships with full TypeScript types. No `@types/` package needed:
```typescript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
import { Brainy, NounType, VerbType } from '@soulcraftlabs/brainy'
const brain = new Brainy()
await brain.init()

View file

@ -66,7 +66,7 @@ const results = await brain.search("query")
**New diagnostics for capacity planning and performance tuning.**
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const brain = new Brainy()
await brain.init()
@ -112,7 +112,7 @@ Recommendations: ${stats.recommendations.join(', ')}
### Step 1: Update Package
```bash
npm install @soulcraft/brainy@latest
npm install @soulcraftlabs/brainy@latest
```
### Step 2: Restart Your Application
@ -134,7 +134,7 @@ npm run start
### Check Adaptive Sizing is Working
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const brain = new Brainy()
await brain.init()
@ -218,7 +218,7 @@ For debugging or compatibility testing:
If you need to rollback to v3.35.0:
```bash
npm install @soulcraft/brainy@3.35.0
npm install @soulcraftlabs/brainy@3.35.0
```
**Note:** We don't anticipate any issues, but rollback is straightforward if needed.
@ -367,7 +367,7 @@ if (stats.fairness.fairnessViolation) {
## Next Steps
1. ✅ **Upgrade:** `npm install @soulcraft/brainy@latest`
1. ✅ **Upgrade:** `npm install @soulcraftlabs/brainy@latest`
2. 📊 **Monitor:** Use `getCacheStats()` to verify performance improvements
3. 🎯 **Tune:** Adjust based on recommendations (if needed)
4. 📖 **Read:** [Operations Guide](../operations/capacity-planning.md) for capacity planning

View file

@ -37,7 +37,7 @@ This single WASM file contains everything needed for sentence embeddings.
```bash
# Bun as a runtime — supported and recommended
bun add @soulcraft/brainy
bun add @soulcraftlabs/brainy
bun run server.ts
```

View file

@ -80,7 +80,7 @@ If you read raw stored records (fact-log scanners, export tooling), use
the exported shape-aware splitters — they handle both record eras:
```typescript
import { splitNounMetadataRecord } from '@soulcraft/brainy'
import { splitNounMetadataRecord } from '@soulcraftlabs/brainy'
const { reserved, custom } = splitNounMetadataRecord(rawRecord)
// reserved = engine fields · custom = the user's bag, ANY names
```
@ -88,7 +88,7 @@ const { reserved, custom } = splitNounMetadataRecord(rawRecord)
Feature detection (never version-sniff):
```typescript
import * as brainy from '@soulcraft/brainy'
import * as brainy from '@soulcraftlabs/brainy'
const lawActive = 'FIELD_ADDRESSING_CAPABILITY' in brainy // 'field-addressing/v1'
```

View file

@ -9,7 +9,7 @@ Complete guide to integrating Brainy with Next.js applications, covering App Rou
```bash
npx create-next-app@latest my-brainy-app
cd my-brainy-app
npm install @soulcraft/brainy
npm install @soulcraftlabs/brainy
```
### Basic Setup
@ -18,7 +18,7 @@ npm install @soulcraft/brainy
// app/components/BrainyProvider.jsx
'use client'
import { createContext, useContext, useEffect, useState } from 'react'
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const BrainyContext = createContext()
@ -271,7 +271,7 @@ export default function SearchPage() {
```javascript
// app/api/search/route.js (App Router)
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
let brain = null
@ -332,7 +332,7 @@ export async function GET() {
```javascript
// pages/api/search.js (Pages Router)
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
let brain = null
@ -374,7 +374,7 @@ export default async function handler(req, res) {
```javascript
// app/api/data/route.js
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
let brain = null
@ -418,7 +418,7 @@ export async function POST(request) {
```jsx
// app/actions/brainy.js
'use server'
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
let brain = null
@ -630,7 +630,7 @@ CMD ["npm", "start"]
/** @type {import('next').NextConfig} */
const nextConfig = {
experimental: {
serverComponentsExternalPackages: ['@soulcraft/brainy']
serverComponentsExternalPackages: ['@soulcraftlabs/brainy']
},
webpack: (config, { isServer }) => {
if (!isServer) {
@ -797,7 +797,7 @@ export function rateLimit(req, limit = 100, window = 60000) {
// app/contexts/BrainyContext.jsx
'use client'
import { createContext, useContext, useReducer, useEffect } from 'react'
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const BrainyContext = createContext()
@ -873,7 +873,7 @@ import { BrainyProvider } from '../app/components/BrainyProvider'
import { Search } from '../app/components/Search'
// Mock Brainy
jest.mock('@soulcraft/brainy', () => ({
jest.mock('@soulcraftlabs/brainy', () => ({
Brainy: jest.fn().mockImplementation(() => ({
init: jest.fn().mockResolvedValue(undefined),
find: jest.fn().mockResolvedValue([

View file

@ -32,7 +32,7 @@ Brainy 7.31.0 adds a per-entity revision counter so multiple writers can coordin
Every distributed-job scheduler eventually wants this exact loop:
```ts
import { Brainy, RevisionConflictError } from '@soulcraft/brainy'
import { Brainy, RevisionConflictError } from '@soulcraftlabs/brainy'
const LOCK_ID = '...uuid for this job slot...'
@ -137,7 +137,7 @@ await brain.addIfMissing({ // ← not a real API
It's race-prone as a plain read-then-write: two concurrent imports both see "not found," both insert, you get duplicates. Without a unique-index primitive (which Brainy doesn't have today), close the race with whole-store CAS — read at a pinned generation, then commit only if nothing moved:
```ts
import { GenerationConflictError } from '@soulcraft/brainy'
import { GenerationConflictError } from '@soulcraftlabs/brainy'
async function addIfMissingByEmail(email: string, data: string) {
for (let attempt = 0; attempt < 5; attempt++) {

View file

@ -18,13 +18,13 @@ Get Brainy running in under a minute.
## 1. Install
```bash
npm install @soulcraft/brainy
npm install @soulcraftlabs/brainy
```
## 2. Initialize
```typescript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
import { Brainy, NounType, VerbType } from '@soulcraftlabs/brainy'
const brain = new Brainy()
await brain.init()
@ -67,7 +67,7 @@ await brain.relate({
## 5. Query with Triple Intelligence
```typescript
import type { Result } from '@soulcraft/brainy'
import type { Result } from '@soulcraftlabs/brainy'
// All three search paradigms in one call
const results: Result[] = await brain.find({

View file

@ -11,7 +11,7 @@
### One Interface for Everything
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const brain = await Brainy.create()
@ -78,7 +78,7 @@ interface ImportProgress {
```typescript
import { useState } from 'react'
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
function UniversalImportProgress({ file }: { file: File }) {
const [progress, setProgress] = useState({
@ -177,7 +177,7 @@ function UniversalImportProgress({ file }: { file: File }) {
```typescript
import ora from 'ora'
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
async function importWithProgress(filePath: string) {
const spinner = ora('Starting import...').start()

View file

@ -28,7 +28,7 @@ on-disk layout (memory's "disk" is a JS Map).
## Quick start
```ts
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
// Filesystem (recommended for any persistent workload):
const brain = new Brainy({
@ -134,7 +134,7 @@ config; the `type` is optional.
If you want to skip the factory:
```ts
import { FileSystemStorage, MemoryStorage } from '@soulcraft/brainy'
import { FileSystemStorage, MemoryStorage } from '@soulcraftlabs/brainy'
const fsStorage = new FileSystemStorage('./brainy-data')
const memStorage = new MemoryStorage()

View file

@ -34,7 +34,7 @@ Three layers solve this:
### Write
```typescript
import { Brainy, NounType } from '@soulcraft/brainy'
import { Brainy, NounType } from '@soulcraftlabs/brainy'
const brain = new Brainy()
await brain.init()
@ -240,7 +240,7 @@ await brain.migrateField({
A realistic adoption sequence for a brain that started without these primitives:
```typescript
import { Brainy, NounType } from '@soulcraft/brainy'
import { Brainy, NounType } from '@soulcraftlabs/brainy'
const brain = new Brainy({ storage: { type: 'filesystem', path: './brain-data' } })
await brain.init()

View file

@ -25,7 +25,7 @@ content — and how 8.0 recovers it for you.
## TL;DR
- **Just upgrade to `@soulcraft/brainy@8.0.12` (or later) and open the store.**
- **Just upgrade to `@soulcraftlabs/brainy@8.0.12` (or later) and open the store.**
If a previous upgrade left VFS content stranded, 8.0.12 **heals it on open**,
with no operator action.
- Want to force or script it? Call **`await brain.vfs.adoptOrphanedBlobs()`**.
@ -90,7 +90,7 @@ So the operator action for a stranded store is simply: **upgrade to 8.0.12 and
open it.**
```ts
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
// Opening the store is all that is required — recovery runs during init().
const brain = new Brainy({ storage: { type: 'filesystem', path: '/data/my-store' } })
@ -182,5 +182,5 @@ and opening each store is sufficient.
The recovery is copy-only, so no rollback of the recovery itself is ever needed.
If you need to roll back the **whole** 7→8 upgrade, restore the directory from
your pre-upgrade backup (retained automatically while recovery is incomplete, or
your own snapshot) and pin `@soulcraft/brainy@7.x`. 8.0 does not keep the old
your own snapshot) and pin `@soulcraftlabs/brainy@7.x`. 8.0 does not keep the old
branch layout in place, so a directory-level restore is the rollback path.

View file

@ -12,7 +12,7 @@ Complete guide to integrating Brainy with Vue.js applications, covering Vue 3, N
npm create vue@latest my-brainy-app
cd my-brainy-app
npm install
npm install @soulcraft/brainy
npm install @soulcraftlabs/brainy
```
### Basic Setup
@ -574,7 +574,7 @@ Nuxt's server engine (Nitro) is the natural home for Brainy: it runs on Node/Bun
```javascript
// server/utils/brain.js (server-only — Nitro never bundles this into the client)
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
let brainPromise
@ -1201,7 +1201,7 @@ import vue from '@vitejs/plugin-vue'
export default defineConfig({
plugins: [vue()],
ssr: {
external: ['@soulcraft/brainy']
external: ['@soulcraftlabs/brainy']
}
})
```

View file

@ -24,7 +24,7 @@ Brainy's neural extraction system uses a **4-signal ensemble architecture** to c
### Method 1: Brain Instance (Recommended)
```typescript
import { Brainy, NounType } from '@soulcraft/brainy'
import { Brainy, NounType } from '@soulcraftlabs/brainy'
const brain = new Brainy()
await brain.init()
@ -62,9 +62,9 @@ const people = await brain.extractEntities('...', {
import {
SmartExtractor,
SmartRelationshipExtractor
} from '@soulcraft/brainy'
} from '@soulcraftlabs/brainy'
// Or use subpath imports:
import { SmartExtractor } from '@soulcraft/brainy/neural/SmartExtractor'
import { SmartExtractor } from '@soulcraftlabs/brainy/neural/SmartExtractor'
const brain = new Brainy()
await brain.init()
@ -176,7 +176,7 @@ const withVectors = await brain.extractEntities(text, {
**Direct entity type classifier.** Use when you have pre-detected candidates or need custom configuration.
```typescript
import { SmartExtractor, FormatContext } from '@soulcraft/brainy'
import { SmartExtractor, FormatContext } from '@soulcraftlabs/brainy'
const extractor = new SmartExtractor(brain, {
minConfidence: 0.7, // Threshold
@ -229,7 +229,7 @@ interface ExtractionResult {
**Relationship type classifier.** Determines verb/relationship types between entities.
```typescript
import { SmartRelationshipExtractor } from '@soulcraft/brainy'
import { SmartRelationshipExtractor } from '@soulcraftlabs/brainy'
const relExtractor = new SmartRelationshipExtractor(brain, {
minConfidence: 0.6,
@ -286,7 +286,7 @@ const rel = await relExtractor.infer(
**Full extraction orchestrator.** Handles candidate detection, classification, and deduplication.
```typescript
import { NeuralEntityExtractor } from '@soulcraft/brainy'
import { NeuralEntityExtractor } from '@soulcraftlabs/brainy'
const extractor = new NeuralEntityExtractor(brain)
@ -607,7 +607,7 @@ const locations = entities.filter(e => e.type === NounType.Location)
### Example 2: Excel Data Classification
```typescript
import { SmartExtractor } from '@soulcraft/brainy'
import { SmartExtractor } from '@soulcraftlabs/brainy'
const extractor = new SmartExtractor(brain)
@ -629,7 +629,7 @@ for (let i = 0; i < cells.length; i++) {
### Example 3: Relationship Extraction
```typescript
import { SmartRelationshipExtractor } from '@soulcraft/brainy'
import { SmartRelationshipExtractor } from '@soulcraftlabs/brainy'
const relExtractor = new SmartRelationshipExtractor(brain)

View file

@ -204,8 +204,8 @@ await brain.add({ data: { name: 'Entity' }, type: NounType.Thing })
### Basic Add Operation
```typescript
import { Brainy } from '@soulcraft/brainy'
import { NounType } from '@soulcraft/brainy/types'
import { Brainy } from '@soulcraftlabs/brainy'
import { NounType } from '@soulcraftlabs/brainy/types'
const brain = new Brainy()
await brain.init()
@ -428,7 +428,7 @@ await brain.relate({ ... }) // a crash here leaves the entity unlinked
```typescript
import { describe, it, expect } from 'vitest'
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
describe('Transaction Tests', () => {
it('should rollback on failure', async () => {

View file

@ -23,7 +23,7 @@ The Universal Display Augmentation is a powerful AI-powered system that automati
### Basic Usage
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const brainy = new Brainy()
await brainy.init()

View file

@ -71,9 +71,9 @@ Let's build a projection that organizes files by priority (high, medium, low):
### Step 1: Create the Strategy Class
```typescript
import { BaseProjectionStrategy } from '@soulcraft/brainy/vfs/semantic'
import { Brainy } from '@soulcraft/brainy'
import { VirtualFileSystem, VFSEntity } from '@soulcraft/brainy/vfs'
import { BaseProjectionStrategy } from '@soulcraftlabs/brainy/vfs/semantic'
import { Brainy } from '@soulcraftlabs/brainy'
import { VirtualFileSystem, VFSEntity } from '@soulcraftlabs/brainy/vfs'
export class PriorityProjection extends BaseProjectionStrategy {
readonly name = 'priority'
@ -141,7 +141,7 @@ export class PriorityProjection extends BaseProjectionStrategy {
### Step 2: Register the Strategy
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
import { PriorityProjection } from './PriorityProjection'
const brain = new Brainy()
@ -537,7 +537,7 @@ Use the projection's resolve cache:
```typescript
import { describe, it, expect, beforeAll } from 'vitest'
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
import { PriorityProjection } from './PriorityProjection'
describe('PriorityProjection', () => {
@ -714,7 +714,7 @@ async resolve(brain, vfs, value: string) {
3. Use appropriate limits: Don't fetch more than needed
### Type errors
1. Import correct types: `import { Brainy, VirtualFileSystem } from '@soulcraft/brainy'`
1. Import correct types: `import { Brainy, VirtualFileSystem } from '@soulcraftlabs/brainy'`
2. Use `as VFSEntity` when mapping results
3. Check BaseProjectionStrategy import

View file

@ -14,11 +14,11 @@ A file explorer that:
## ⚡ Step 1: Basic Setup (1 minute)
```bash
npm install @soulcraft/brainy
npm install @soulcraftlabs/brainy
```
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
// ✅ CORRECT: Use filesystem storage for production
const brain = new Brainy({
@ -115,7 +115,7 @@ Here's a complete React component using the correct patterns:
```tsx
import React, { useState, useEffect } from 'react'
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
export function FileExplorer() {
const [brain, setBrain] = useState(null)
@ -288,8 +288,8 @@ Your file explorer is now working! Here's what to explore next:
### "Module not found" errors
```bash
# Make sure you're using the right import
npm ls @soulcraft/brainy # Check version
npm install @soulcraft/brainy@latest # Update if needed
npm ls @soulcraftlabs/brainy # Check version
npm install @soulcraftlabs/brainy@latest # Update if needed
```
### "VFS not initialized" errors

View file

@ -24,7 +24,7 @@ Brainy VFS is a revolutionary virtual filesystem that runs on top of Brainy's ne
## Quick Start
```javascript
import { VirtualFileSystem } from '@soulcraft/brainy/vfs'
import { VirtualFileSystem } from '@soulcraftlabs/brainy/vfs'
// Initialize the VFS
const vfs = new VirtualFileSystem({
@ -381,7 +381,7 @@ Brainy VFS fully leverages Brainy's revolutionary Triple Intelligence system:
## Installation
```bash
npm install @soulcraft/brainy
npm install @soulcraftlabs/brainy
```
## Requirements

View file

@ -135,7 +135,7 @@ Mount VFS as a native filesystem on Linux/Mac/Windows.
```typescript
// Planned (research phase)
import { mountVFS } from '@soulcraft/brainy/vfs/fuse'
import { mountVFS } from '@soulcraftlabs/brainy/vfs/fuse'
await mountVFS(vfs, {
mountPoint: '/mnt/brainy',
@ -160,7 +160,7 @@ These features would benefit from community contributions. If you're interested
### Express.js Static Middleware
```typescript
// Wanted: Community contribution
import { createStaticMiddleware } from '@soulcraft/brainy/vfs/express'
import { createStaticMiddleware } from '@soulcraftlabs/brainy/vfs/express'
app.use('/files', createStaticMiddleware(vfs, {
index: ['index.html', 'index.md'],
@ -172,7 +172,7 @@ app.use('/files', createStaticMiddleware(vfs, {
### VSCode Extension
```typescript
// Wanted: Community contribution
import { VFSProvider } from '@soulcraft/brainy/vfs/vscode'
import { VFSProvider } from '@soulcraftlabs/brainy/vfs/vscode'
const provider = new VFSProvider(vfs)
vscode.workspace.registerFileSystemProvider('brainy', provider)

View file

@ -327,7 +327,7 @@ console.log(id1 === id2 && id2 === id3) // true
Create your own semantic dimensions:
```typescript
import { BaseProjectionStrategy } from '@soulcraft/brainy/vfs/semantic'
import { BaseProjectionStrategy } from '@soulcraftlabs/brainy/vfs/semantic'
class PriorityProjection extends BaseProjectionStrategy {
readonly name = 'priority'

View file

@ -7,7 +7,7 @@ Brainy's Virtual Filesystem (VFS) provides a POSIX-like filesystem interface tha
## Quick Start
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
// Initialize Brainy
const brain = new Brainy({
@ -598,7 +598,7 @@ const user = await store.findById('users', 'user123')
VFS uses standard POSIX-style errors:
```typescript
import { VFSError, VFSErrorCode } from '@soulcraft/brainy'
import { VFSError, VFSErrorCode } from '@soulcraftlabs/brainy'
try {
await vfs.readFile('/nonexistent.txt')

View file

@ -280,7 +280,7 @@ GitBridge provides Git import/export capabilities:
#### GitBridge Usage
```javascript
// Import and instantiate GitBridge
import { GitBridge } from '@soulcraft/brainy'
import { GitBridge } from '@soulcraftlabs/brainy'
const gitBridge = new GitBridge(vfs, brain)
// Export VFS to Git repository structure
@ -452,7 +452,7 @@ This ordering prevents race conditions where file writes might fail because pare
## Complete Example
```javascript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
async function vfsExample() {
// Initialize

View file

@ -196,5 +196,5 @@ await brain.relate({
Always import and use the type enums:
```javascript
import { NounType, VerbType } from '@soulcraft/brainy'
import { NounType, VerbType } from '@soulcraftlabs/brainy'
```

View file

@ -5,7 +5,7 @@
The Brainy VFS is automatically initialized during `brain.init()`. No separate initialization needed!
```javascript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
// Create and initialize Brainy
const brain = new Brainy({
@ -71,7 +71,7 @@ VFS stores files as entities and relationships in the same graph as everything e
## Complete Example
```javascript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
async function useVFS() {
// Initialize Brainy
@ -100,7 +100,7 @@ useVFS().catch(console.error)
## TypeScript Usage
```typescript
import { Brainy, VirtualFileSystem } from '@soulcraft/brainy'
import { Brainy, VirtualFileSystem } from '@soulcraftlabs/brainy'
class FileManager {
private brain: Brainy

View file

@ -37,7 +37,7 @@ Brainy VFS provides safe, tree-aware methods that prevent these issues:
### Method 1: Use `getDirectChildren()` (Recommended)
```typescript
import { Brainy, VirtualFileSystem } from '@soulcraft/brainy'
import { Brainy, VirtualFileSystem } from '@soulcraftlabs/brainy'
const brain = new Brainy()
await brain.init()
@ -97,7 +97,7 @@ Here's a complete example using React:
```tsx
import React, { useState, useEffect } from 'react'
import { VirtualFileSystem } from '@soulcraft/brainy'
import { VirtualFileSystem } from '@soulcraftlabs/brainy'
interface FileNode {
name: string
@ -177,7 +177,7 @@ function TreeView({ node, onToggle, expanded }) {
If you must build trees manually from flat lists, use the `VFSTreeUtils`:
```typescript
import { VFSTreeUtils } from '@soulcraft/brainy/vfs'
import { VFSTreeUtils } from '@soulcraftlabs/brainy/vfs'
// Get all entities somehow
const allEntities = await vfs.getDescendants('/root')

View file

@ -7,7 +7,7 @@
* the Bluesky firehose with Brainy's distributed architecture
*/
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
import { WebSocket } from 'ws'
// =====================================================

View file

@ -14,7 +14,7 @@
* ts-node examples/monitor-cache-performance.ts
*/
import { Brainy, NounType } from '@soulcraft/brainy'
import { Brainy, NounType } from '@soulcraftlabs/brainy'
// ANSI color codes for pretty output
const colors = {

View file

@ -5,7 +5,7 @@ Connect Brainy to spreadsheets, BI tools, and external systems with zero configu
## Quick Start
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const brain = new Brainy({ integrations: true })
await brain.init()
@ -178,7 +178,7 @@ Webhooks include `X-Brainy-Signature` header with HMAC-SHA256 signature.
### Minimal (in-memory):
```typescript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const brain = new Brainy({ integrations: true })
await brain.init()
@ -194,7 +194,7 @@ console.log(brain.hub.getInstructions())
```typescript
import express from 'express'
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const app = express()
const brain = new Brainy({
@ -232,7 +232,7 @@ app.listen(3000, () => {
```typescript
import { Hono } from 'hono'
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const app = new Hono()

View file

@ -99,7 +99,7 @@ Add the `BRAINY_URL` script property in Apps Script settings.
The simplest way to enable all integrations:
```javascript
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const brain = new Brainy({ integrations: true })
await brain.init()
@ -112,7 +112,7 @@ With Express:
```javascript
import express from 'express'
import { Brainy } from '@soulcraft/brainy'
import { Brainy } from '@soulcraftlabs/brainy'
const app = express()
const brain = new Brainy({ integrations: true })

8
package-lock.json generated
View file

@ -1,12 +1,12 @@
{
"name": "@soulcraft/brainy",
"version": "10.3.1",
"name": "@soulcraftlabs/brainy",
"version": "10.4.4",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@soulcraft/brainy",
"version": "10.3.1",
"name": "@soulcraftlabs/brainy",
"version": "10.4.4",
"license": "MIT",
"dependencies": {
"@msgpack/msgpack": "^3.1.2",

View file

@ -1,6 +1,7 @@
{
"name": "@soulcraft/brainy",
"version": "10.3.1",
"name": "@soulcraftlabs/brainy",
"version": "10.4.4",
"brainyContract": 1,
"description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.",
"main": "dist/index.js",
"module": "dist/index.js",
@ -126,15 +127,16 @@
"license": "MIT",
"private": false,
"publishConfig": {
"access": "public"
"access": "public",
"registry": "https://source.soulcraft.com/api/packages/soulcraftlabs/npm/"
},
"homepage": "https://source.soulcraft.com/soulcraft/brainy",
"homepage": "https://source.soulcraft.com/soulcraftlabs/open-brainy",
"bugs": {
"url": "https://source.soulcraft.com/soulcraft/brainy/issues"
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/issues"
},
"repository": {
"type": "git",
"url": "git+https://source.soulcraft.com/soulcraft/brainy.git"
"url": "git+https://source.soulcraft.com/soulcraftlabs/open-brainy.git"
},
"files": [
"dist/**/*.js",

76
releases/brainy.json Normal file
View file

@ -0,0 +1,76 @@
{
"product": "brainy",
"entries": [
{
"version": "11.0.5",
"date": "2026-09-02",
"headline": "Graph-first finds in production, and opens that stop rescanning history",
"items": [
"find({ connected, where }) now walks the neighbours first and filters only those rows through a native door — correct at every page and O(neighbours), never the whole store.",
"related() with a list of verb types returns every requested kind (a fast path had silently kept only the first).",
"Deferred-embedding recovery resumes from a low-water mark instead of rescanning the whole generation log at every open — measured at two minutes on a large brain, now milliseconds."
],
"url": null,
"thumb": null
},
{
"version": "11.0.4",
"date": "2026-09-01",
"headline": "Closes in milliseconds, index rebuilds without the disk-sync storm",
"items": [
"close() no longer pays deferred compaction or waits out an in-flight rebuild — measured 8 ms against the 4-minute closes it replaces; deferred work resumes at the next open, in the background.",
"The metadata index's rebuild syncs to disk per shard instead of per row, and the durability point moved to the publish step — the same guarantee, a fraction of the disk traffic.",
"A new native filter door evaluates queries over exactly the candidate rows a graph walk found, never the whole store."
],
"url": null,
"thumb": null
},
{
"version": "11.0.3",
"date": "2026-09-01",
"headline": "The embedding upgrade ceremony runs on every brain",
"items": [
"A brain opened through the standard plugin now carries its embedding-model identity, so the full-precision upgrade ceremony can run on it.",
"A one-fix release; nothing else changed."
],
"url": null,
"thumb": null
},
{
"version": "11.0.2",
"date": "2026-08-31",
"headline": "One embedding quality everywhere, 34× faster imports",
"items": [
"Every runtime embeds with the same full-precision model — search quality no longer depends on where you run.",
"Bulk embedding measured 3.14.2× faster, and an online re-embed ceremony upgrades existing stores without downtime.",
"The engine's change feed is documented, with the SSE/WebSocket fan-out pattern for realtime surfaces."
],
"url": null,
"thumb": null
},
{
"version": "11.0.1",
"date": "2026-08-31",
"headline": "Deletes inside transactions are safe",
"items": [
"Deleting relations inside a transact() no longer corrupts index bookkeeping.",
"A store that deletes its last relation keeps serving instead of refusing."
],
"url": null,
"thumb": null
},
{
"version": "11.0.0",
"date": "2026-08-28",
"headline": "One install, one engine — Brainy",
"items": [
"The former two-package pair is one package: the native engine under the familiar API. One import is the whole install.",
"A missing native build refuses loudly with its cures named; nothing falls back silently.",
"Stores open in place — no migration."
],
"url": null,
"thumb": null
}
],
"history": "The version line continues from the 4.3.x native-engine releases; their record lives in the product repository's CHANGELOG.md."
}

136
releases/open-brainy.json Normal file
View file

@ -0,0 +1,136 @@
{
"product": "open-brainy",
"entries": [
{
"version": "10.4.11",
"date": "2026-09-02",
"headline": "Hybrid finds filter before they hydrate, one owner per shutdown, and a faster open",
"items": [
"Hybrid finds (query/vector combined with a filter, including connected and fusion finds) now filter first and hydrate only the page — one batchGet of exactly the requested rows, instead of hydrating everything the search side found. Fixes a bug where any page after the first came back empty.",
"A brain now has exactly one shutdown owner — a host and its engine no longer race to close the same store, and a follow-up flush requested during a running flush is handed off cleanly instead of ever risking a stall.",
"find({ path }) and other path-scoped VFS searches now serve a real range over the indexed path (O(log n)) instead of refusing the query outright — both scoped and recursive:false searches were silently broken before this.",
"Open no longer rescans a brain's whole fact log on every open — sealed segments the manifest already accounts for are skipped, collapsing a multi-second open term to near-zero on large brains.",
"commitTransaction() now refuses by name if single-ops are still pending, and a read-only open no longer writes clean-shutdown evidence it didn't earn — two correctness invariants that were previously assumed, not enforced."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.11",
"thumb": null
},
{
"version": "10.4.10",
"date": "2026-09-02",
"headline": "A planner door for indexes, batched containment repair, and a fixed near()",
"items": [
"An optional planFindPage door lets an index plan a find() and answer it in one call, instead of the engine assembling the plan itself.",
"repairContainment's reconcile pass now walks paged edges once instead of issuing one graph call per file.",
"find({ near }) now searches around the anchor's own vector and refuses by name when none is available, instead of silently querying with no vector at all."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.10",
"thumb": null
},
{
"version": "10.4.9",
"date": "2026-09-02",
"headline": "Graph-first finds, honest verb arrays, and opens that stop rescanning history",
"items": [
"find({ connected, where }) now walks the neighbours first and filters only those rows — correct at every page, and O(neighbours) instead of O(store).",
"related() with a list of verb types (or sources, or targets) returns every requested kind — four fast paths silently kept only the first.",
"Deferred-embedding recovery resumes from a low-water mark instead of rescanning the whole generation log at every open — measured at two minutes on a large brain, now milliseconds."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.9",
"thumb": null
},
{
"version": "10.4.7",
"date": "2026-09-01",
"headline": "Count ledgers can no longer race themselves",
"items": [
"Concurrent count flushes coalesce into one writer with a trailing pass — parallel flushes can no longer corrupt a store's count ledger.",
"Atomic writes carry a per-process sequence, so two processes' temp files can never collide."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.7",
"thumb": null
},
{
"version": "10.4.6",
"date": "2026-08-31",
"headline": "Transactions cross the index seam safely",
"items": [
"Deleting relations inside a transact() no longer fails against the metadata index — operations take a JSON-safe view at the moment they execute.",
"Fixes a class of transaction failures on stores with integer-mapped relation endpoints."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.6",
"thumb": null
},
{
"version": "10.4.5",
"date": "2026-08-31",
"headline": "Recovery tells the truth, docs live at home",
"items": [
"A torn generation-log tail is a terminal verdict with a named cure — never an endless wait at open.",
"A sealed segment declares only the generations it actually holds.",
"The engine's documentation now publishes from its own repository."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.5",
"thumb": null
},
{
"version": "10.4.4",
"date": "2026-08-28",
"headline": "Faster opens, quieter idle",
"items": [
"Opening a store discovers generations from directory names instead of walking the log, and answers \"any entities?\" with one directory read.",
"The flush-request watch is event-driven; idle stores stop paying a polling heartbeat.",
"A slow open now names the exact step it is in, so operators see what is being paid and why."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.4",
"thumb": null
},
{
"version": "10.4.3",
"date": "2026-08-27",
"headline": "Open Brainy, under its own name",
"items": [
"The same engine as 10.4.2, now published as @soulcraftlabs/brainy — the MIT reference engine, on The Source.",
"No code changes; your imports change once and everything else stays put."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.3",
"thumb": null
},
{
"version": "10.4.2",
"date": "2026-08-27",
"headline": "Vectors that lie are refused, counts that drift are caught",
"items": [
"A zero-norm vector is not a vector: the index refuses them, rebuilds skip them, and a sanctioned unvector door removes them cleanly.",
"The canonical count ledger derives from identity records and marks legacy-derived ledgers suspect at load.",
"Plugin activation failures keep their original error as cause, so the real frame reaches your logs."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.2",
"thumb": null
},
{
"version": "10.4.1",
"date": "2026-08-26",
"headline": "Writes that change nothing cost nothing",
"items": [
"The read gate is per index family, and a write carrying unchanged data never re-embeds.",
"The vectored-row count joins the ledger, so vector coverage is a number you can read, not a guess."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.1",
"thumb": null
},
{
"version": "10.4.0",
"date": "2026-08-26",
"headline": "Repair routing, the vector ledger, and honest empties",
"items": [
"Repairs route to the index that owns the damage, and the open gate closes the vector leg until coverage is proven.",
"An empty string is real data, not a missing field.",
"The metadata crossing never carries raw integer relation endpoints — a whole class of serialization faults closed."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.0",
"thumb": null
}
],
"history": "Earlier releases are recorded in CHANGELOG.md in this repository."
}

View file

@ -10,6 +10,7 @@ import { TransformerEmbedding } from '../src/utils/embedding.js'
import * as fs from 'fs/promises'
import * as path from 'path'
import { fileURLToPath } from 'url'
import { resolveDeterministicStamp } from './lib/deterministicStamp.js'
const __dirname = path.dirname(fileURLToPath(import.meta.url))
@ -97,13 +98,22 @@ async function buildEmbeddedPatterns() {
// Convert to base64 for embedding in TypeScript
const uint8 = new Uint8Array(buffer)
const base64 = Buffer.from(uint8).toString('base64')
// Deterministic stamp: derived from the git commit time of this
// generator's inputs, never from wall-clock time — two builds of the
// same source tree must produce byte-identical output.
const outputPath = path.join(__dirname, '..', 'src', 'neural', 'embeddedPatterns.ts')
const generatedStamp = resolveDeterministicStamp(
[path.join(__dirname, 'buildEmbeddedPatterns.ts'), libraryPath],
outputPath
)
// Generate TypeScript file with everything embedded
const tsContent = `/**
* 🧠 BRAINY EMBEDDED PATTERNS
*
* AUTO-GENERATED - DO NOT EDIT
* Generated: ${new Date().toISOString()}
* Generated: ${generatedStamp}
* Patterns: ${libraryData.patterns.length}
* Coverage: 94-98% of all queries
*
@ -197,7 +207,6 @@ prodLog.info(\`🧠 Brainy Pattern Library loaded: \${EMBEDDED_PATTERNS.length}
`
// Write the TypeScript file
const outputPath = path.join(__dirname, '..', 'src', 'neural', 'embeddedPatterns.ts')
await fs.writeFile(outputPath, tsContent)
// Report statistics

View file

@ -11,6 +11,7 @@ import * as fs from 'fs/promises'
import * as path from 'path'
import { fileURLToPath } from 'url'
import { NounType, VerbType } from '../src/types/graphTypes.js'
import { resolveDeterministicStamp } from './lib/deterministicStamp.js'
const __dirname = path.dirname(fileURLToPath(import.meta.url))
@ -373,12 +374,24 @@ async function buildTypeEmbeddings() {
const uint8 = new Uint8Array(buffer)
const base64 = Buffer.from(uint8).toString('base64')
// Deterministic stamp: derived from the git commit time of this
// generator's inputs, never from wall-clock time — two builds of the
// same source tree must produce byte-identical output.
const outputPath = path.join(__dirname, '..', 'src', 'neural', 'embeddedTypeEmbeddings.ts')
const generatedStamp = resolveDeterministicStamp(
[
path.join(__dirname, 'buildTypeEmbeddings.ts'),
path.join(__dirname, '..', 'src', 'types', 'graphTypes.ts')
],
outputPath
)
// Generate TypeScript file
const tsContent = `/**
* 🧠 BRAINY EMBEDDED TYPE EMBEDDINGS
*
* AUTO-GENERATED - DO NOT EDIT
* Generated: ${new Date().toISOString()}
* Generated: ${generatedStamp}
* Noun Types: ${nounTypes.length}
* Verb Types: ${verbTypes.length}
*
@ -395,7 +408,7 @@ export const TYPE_METADATA = {
verbTypes: ${verbTypes.length},
totalTypes: ${totalTypes},
embeddingDimensions: ${embeddingDim},
generatedAt: "${new Date().toISOString()}",
generatedAt: "${generatedStamp}",
sizeBytes: {
embeddings: ${buffer.byteLength},
base64: ${base64.length}
@ -494,7 +507,6 @@ prodLog.info(\`🧠 Brainy Type Embeddings loaded: \${TYPE_METADATA.nounTypes} n
`
// Write the TypeScript file
const outputPath = path.join(__dirname, '..', 'src', 'neural', 'embeddedTypeEmbeddings.ts')
await fs.writeFile(outputPath, tsContent)
// Report statistics

View file

@ -0,0 +1,128 @@
#!/usr/bin/env node
/**
* Emit this build's API-contract manifest to docs/api-contract.json.
*
* WHY IT IS GENERATED, NOT WRITTEN: a hand-kept list of doors drifts from the
* code the first time somebody adds one. This reads the surface the build
* actually exposes the prototype's own methods and accessors, the exported
* error classes, the `where` operator sets, the field-addressing vocabulary,
* the health verdicts so a diff between two engines' manifests is a diff
* between two engines, never between two authors.
*
* Requirement marking (required / optional per door) is NOT derivable from the
* surface it is a commitment, recorded with the contract's owner rather than
* here. This manifest carries the surface; the promise lives with the contract.
*
* Usage: node scripts/emit-contract-manifest.mjs [--check]
* --check exits non-zero when the committed manifest is stale.
*/
import { writeFileSync, readFileSync, existsSync } from 'node:fs'
import { join, dirname } from 'node:path'
import { fileURLToPath } from 'node:url'
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..')
const OUT = join(ROOT, 'docs', 'api-contract.json')
const { Brainy } = await import(join(ROOT, 'dist', 'brainy.js'))
const errorsModule = await import(join(ROOT, 'dist', 'errors', 'brainyError.js'))
const versionModule = await import(join(ROOT, 'dist', 'utils', 'version.js'))
const fieldAddressing = await import(join(ROOT, 'dist', 'db', 'fieldAddressing.js'))
/** Every own method and accessor on the class's prototype, minus the private ones. */
function surfaceOf(ctor) {
const doors = []
for (const name of Object.getOwnPropertyNames(ctor.prototype)) {
if (name === 'constructor' || name.startsWith('_')) continue
const descriptor = Object.getOwnPropertyDescriptor(ctor.prototype, name)
if (!descriptor) continue
if (typeof descriptor.value === 'function') {
doors.push({ name, kind: 'method', arity: descriptor.value.length })
} else if (descriptor.get) {
doors.push({ name, kind: 'accessor' })
}
}
return doors.sort((a, b) => a.name.localeCompare(b.name))
}
const errors = Object.entries(errorsModule)
.filter(([name, value]) => typeof value === 'function' && /Error$/.test(name))
.map(([name]) => name)
.sort()
// The operator sets, read from the engine's own refusal message so the
// manifest can never disagree with the validator.
const filterSource = readFileSync(join(ROOT, 'src', 'utils', 'metadataFilter.ts'), 'utf-8')
const acceptedMatch = filterSource.match(/const VALUE_OPERATORS = new Set<string>\(\[([\s\S]*?)\]\)/)
if (!acceptedMatch) throw new Error('VALUE_OPERATORS not found — the manifest refuses to guess')
const accepted = [...acceptedMatch[1].matchAll(/'([^']+)'/g)].map((m) => m[1]).sort()
const indexSource = readFileSync(join(ROOT, 'src', 'utils', 'metadataIndex.ts'), 'utf-8')
const refusedByIndex = ['endsWith', 'length', 'matches', 'startsWith'].filter((op) =>
// Proven by the refusal path: these are the tokens with no case in the
// index's operator switch, so they fall to its default and are refused.
!new RegExp(`case '${op}':`).test(indexSource)
)
const servedOnIndex = accepted.filter((op) => !refusedByIndex.includes(op))
const manifest = {
contractVersion: versionModule.contractVersion(),
engine: '@soulcraftlabs/brainy',
compatibility: {
minor:
'additive — a new optional door, a new served operator, a new error class; every existing implementation still conforms',
major:
'breaking — a door removed, an answer narrowed, an ordering law changed, an optional door promoted to required, or an operator moved from served to refused'
},
doors: surfaceOf(Brainy),
errors,
operators: {
accepted,
servedOnIndexPath: servedOnIndex,
refusedByIndexPath: refusedByIndex,
combinators: ['allOf', 'anyOf', 'not']
},
fieldAddressing: {
systemKeyPrefix: 'system.',
systemEntityScalars: [...(fieldAddressing.SYSTEM_ENTITY_SCALARS ?? [])].sort(),
systemRelationScalars: [...(fieldAddressing.SYSTEM_RELATION_SCALARS ?? [])].sort(),
plumbingFields: [...(fieldAddressing.PLUMBING_FIELDS ?? [])].sort()
},
health: {
verdicts: ['pass', 'warn', 'fail'],
healKinds: ['none', 'repair', 'rebuild'],
servingWithholdingInvariants: [
'index-initialized',
'durable-state-present',
'manifest-residency',
'replay-clean',
'strand-latch'
]
}
}
const rendered = `${JSON.stringify(manifest, null, 2)}\n`
if (process.argv.includes('--check')) {
if (!existsSync(OUT)) {
console.error(`docs/api-contract.json is missing — run: node scripts/emit-contract-manifest.mjs`)
process.exit(1)
}
if (readFileSync(OUT, 'utf-8') !== rendered) {
console.error(
`docs/api-contract.json is STALE — the public surface changed. Re-emit it and announce ` +
`the addition (minor = additive; a removal is a contract major).`
)
process.exit(1)
}
console.log(`docs/api-contract.json is current (${manifest.doors.length} doors, contract ${manifest.contractVersion}).`)
process.exit(0)
}
writeFileSync(OUT, rendered)
console.log(
`Wrote docs/api-contract.json — contract ${manifest.contractVersion}, ` +
`${manifest.doors.length} doors, ${manifest.errors.length} error classes, ` +
`${manifest.operators.accepted.length} operators ` +
`(${manifest.operators.refusedByIndexPath.length} refused by the index path).`
)

View file

@ -0,0 +1,118 @@
/**
* Deterministic generation-stamp resolution for Brainy's build-time code
* generators.
*
* Two builds of the same source tree must produce byte-identical output.
* A wall-clock stamp (`new Date()`) breaks that guarantee, so every
* generator that writes a "Generated:" header or a `generatedAt` field
* into its output must resolve the stamp through this module instead.
*
* Resolution order:
* 1. The newest git commit timestamp among the generator's input files
* (the generator script itself always counts as an input).
* 2. If git metadata is unavailable (for example, building from a
* published npm tarball with no `.git` directory), the stamp already
* recorded in the previously generated output file.
* 3. If neither is available, the fixed epoch string
* `1970-01-01T00:00:00.000Z`.
*
* Every fallback logs a line to stderr deterministic degradation is
* loud, never a silent divergence.
*/
import { execFileSync } from 'child_process'
import * as fs from 'fs'
const EPOCH_STAMP = '1970-01-01T00:00:00.000Z'
const STAMP_PATTERN = /\*\s*Generated:\s*(\S+)/
/**
* Resolve the deterministic stamp for a generator run.
*
* @param inputPaths Absolute paths to every file whose content determines
* the generator's output, including the generator script itself.
* @param previousOutputPath Absolute path to the previously generated
* file, used for the existing-stamp fallback when git is unavailable.
* @returns An ISO-8601 timestamp string that is deterministic for a given
* source tree.
*/
export function resolveDeterministicStamp(
inputPaths: string[],
previousOutputPath: string
): string {
const gitStamp = newestGitCommitTimestamp(inputPaths)
if (gitStamp) {
return gitStamp
}
const existingStamp = readExistingStamp(previousOutputPath)
if (existingStamp) {
process.stderr.write(
`[deterministic-stamp] no git commit history found for generator inputs; ` +
`reusing existing stamp from ${previousOutputPath}: ${existingStamp}\n`
)
return existingStamp
}
process.stderr.write(
`[deterministic-stamp] no git commit history and no previous output at ` +
`${previousOutputPath}; falling back to fixed epoch stamp ${EPOCH_STAMP}\n`
)
return EPOCH_STAMP
}
/**
* Find the newest git commit timestamp among the given input paths.
* Returns null if git is unavailable, the tree is not a git repository,
* or none of the inputs have any commit history yet.
*/
function newestGitCommitTimestamp(inputPaths: string[]): string | null {
let newest: string | null = null
for (const inputPath of inputPaths) {
if (!fs.existsSync(inputPath)) {
continue
}
let out: string
try {
out = execFileSync(
'git',
['log', '-1', '--format=%cI', '--', inputPath],
{ stdio: ['ignore', 'pipe', 'ignore'] }
)
.toString()
.trim()
} catch {
// git missing, not a repository, or no permissions — handled by the
// caller's fallback chain.
continue
}
if (!out) {
// Path exists but has no commit history yet (e.g. newly created,
// uncommitted file).
continue
}
if (!newest || new Date(out).getTime() > new Date(newest).getTime()) {
newest = out
}
}
return newest
}
/**
* Parse the `* Generated: <ISO timestamp>` header out of a previously
* generated file, if one exists.
*/
function readExistingStamp(outputPath: string): string | null {
if (!fs.existsSync(outputPath)) {
return null
}
const content = fs.readFileSync(outputPath, 'utf-8')
const match = content.match(STAMP_PATTERN)
return match ? match[1] : null
}

View file

@ -15,6 +15,12 @@ NC='\033[0m' # No Color
RELEASE_TYPE="${1:-patch}" # patch, minor, or major
SKIP_TESTS=false
DRY_RUN=false
# --source-only is now a no-op: The Source is the one registry, so every
# release already ships Source-only — tag, CI's publish to The Source, the
# release page, and the docs push, with no separate storefront leg to skip.
# The flag is still accepted (for backward-compatible invocations) and just
# prints a notice; it no longer changes behavior.
SOURCE_ONLY=false
for arg in "$@"; do
case $arg in
@ -24,6 +30,9 @@ for arg in "$@"; do
--dry-run)
DRY_RUN=true
;;
--source-only)
SOURCE_ONLY=true
;;
esac
done
@ -100,7 +109,7 @@ else
;;
*)
echo -e "${RED}❌ Invalid release type: ${RELEASE_TYPE}${NC}"
echo "Usage: ./scripts/release.sh [patch|minor|major|<explicit-version>] [--dry-run]"
echo "Usage: ./scripts/release.sh [patch|minor|major|<explicit-version>] [--dry-run] [--source-only (no-op; The Source is the one registry)]"
exit 1
;;
esac
@ -119,6 +128,9 @@ echo -e "${BLUE}New version: ${NEW_VERSION}${NC}"
if [ "$PRERELEASE" = true ]; then
echo -e "${YELLOW}⚠️ Prerelease → npm dist-tag '${NPM_TAG}', GitHub prerelease${NC}"
fi
if [ "$SOURCE_ONLY" = true ]; then
echo -e "${YELLOW}⚠️ The Source is the one registry; --source-only is implied${NC}"
fi
echo ""
if [ "$DRY_RUN" = true ]; then
@ -142,7 +154,7 @@ else
fi
# Create new changelog entry
CHANGELOG_ENTRY="### [${NEW_VERSION}](https://source.soulcraft.com/soulcraft/brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d))
CHANGELOG_ENTRY="### [${NEW_VERSION}](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d))
${COMMITS}
"
@ -193,9 +205,9 @@ echo -e "${GREEN}✅ Pushed to origin${NC}\n"
# .forgejo/workflows/publish-source.yml, which builds and publishes on The
# Source's own runner (datacenter-side: seconds, not the laptop's WAN timing
# out on an 87MB tarball PUT). The laptop holds no home-registry publish
# credential anymore; it only waits for CI's result before trusting the
# home/npmjs pair enough to publish the storefront leg.
SOURCE_NPM_REG="https://source.soulcraft.com/api/packages/soulcraft/npm/"
# credential anymore; it only waits for CI's result before continuing on to
# the release page and the docs push.
SOURCE_NPM_REG="https://source.soulcraft.com/api/packages/soulcraftlabs/npm/"
SOURCE_POLL_INTERVAL_S=15
SOURCE_POLL_MAX_ATTEMPTS=200 # 200 × 15s = 50 minutes — the runner is sequential and a busy day's ci.yml
# backlog has twice exceeded the old 20-minute window (8.10.3, 9.0.0);
@ -203,7 +215,7 @@ SOURCE_POLL_MAX_ATTEMPTS=200 # 200 × 15s = 50 minutes — the runner is sequen
echo -e "${BLUE}9⃣ Waiting for CI to publish v${NEW_VERSION} to The Source registry (home)...${NC}"
SOURCE_LANDED=false
for ((attempt = 1; attempt <= SOURCE_POLL_MAX_ATTEMPTS; attempt++)); do
LANDED_VERSION=$(npm view "@soulcraft/brainy@${NEW_VERSION}" version "--@soulcraft:registry=${SOURCE_NPM_REG}" 2>/dev/null || echo "")
LANDED_VERSION=$(npm view "@soulcraftlabs/brainy@${NEW_VERSION}" version "--@soulcraftlabs:registry=${SOURCE_NPM_REG}" 2>/dev/null || echo "")
if [ "$LANDED_VERSION" = "$NEW_VERSION" ]; then
SOURCE_LANDED=true
break
@ -216,50 +228,8 @@ if [ "$SOURCE_LANDED" = true ]; then
echo -e "${GREEN}✅ CI published v${NEW_VERSION} to The Source${NC}\n"
else
echo -e "${RED}❌ CI's home publish did not land — check the workflow run on The Source; the pair must not diverge.${NC}"
echo -e "${RED} v${NEW_VERSION} was tagged and pushed, but @soulcraft/brainy@${NEW_VERSION} never became visible on the${NC}"
echo -e "${RED} Source registry after ${SOURCE_POLL_MAX_ATTEMPTS} attempts, ${SOURCE_POLL_INTERVAL_S}s apart. Aborting before npmjs.${NC}"
exit 1
fi
echo -e "${BLUE}9⃣½ Publishing to npmjs (storefront, dist-tag: ${NPM_TAG})...${NC}"
# BYTE-IDENTITY LAW: the storefront republishes CI's EXACT artifact — download
# the tarball The Source serves and publish that file, never a fresh local pack
# (a local rebuild can differ byte-wise, and the fleet verifies the pair by
# shasum across registries).
STOREFRONT_TMP="$(mktemp -d)"
(cd "$STOREFRONT_TMP" && npm pack "@soulcraft/brainy@${NEW_VERSION}" "--@soulcraft:registry=${SOURCE_NPM_REG}" >/dev/null)
SOURCE_TARBALL="$(ls "$STOREFRONT_TMP"/soulcraft-brainy-*.tgz)"
echo -e "${BLUE} home artifact: $(sha256sum "$SOURCE_TARBALL" | cut -d' ' -f1)${NC}"
npm publish "$SOURCE_TARBALL" --tag "$NPM_TAG" "--@soulcraft:registry=https://registry.npmjs.org/"
rm -rf "$STOREFRONT_TMP"
# Brainy is the only PUBLIC @soulcraft package — verify visibility after every publish.
npm access get status @soulcraft/brainy "--@soulcraft:registry=https://registry.npmjs.org/" || true
# Verify the pair is byte-identical by registry-reported shasum — divergence
# here means the storefront leg must be treated as failed, loudly. RETRIED
# with raw curl: npmjs metadata propagates with a lag measured in minutes,
# and a one-shot npm-view probe fired a false DIVERGENCE on 10.0.0 while a
# raw curl of the registry document already confirmed byte-identity. The
# probe now reads the registry JSON directly (no npm cache in the path) and
# gives propagation up to 5 minutes before calling the pair divergent.
NPMJS_VERIFY_ATTEMPTS=20
NPMJS_VERIFY_INTERVAL_S=15 # 20 × 15s = 5 minutes of propagation grace
SOURCE_SHA=$(npm view "@soulcraft/brainy@${NEW_VERSION}" dist.shasum "--@soulcraft:registry=${SOURCE_NPM_REG}" 2>/dev/null || echo "source-unavailable")
PAIR_IDENTICAL=false
for ((attempt = 1; attempt <= NPMJS_VERIFY_ATTEMPTS; attempt++)); do
NPMJS_SHA=$(curl -fsSL "https://registry.npmjs.org/@soulcraft%2Fbrainy" 2>/dev/null \
| node -e "let d='';process.stdin.on('data',c=>d+=c).on('end',()=>{try{const v=JSON.parse(d).versions[process.argv[1]];console.log(v?v.dist.shasum:'')}catch{console.log('')}})" "${NEW_VERSION}" \
|| echo "")
if [ -n "$NPMJS_SHA" ] && [ "$SOURCE_SHA" = "$NPMJS_SHA" ]; then
PAIR_IDENTICAL=true
break
fi
echo -e "${YELLOW} … npmjs metadata not settled (attempt ${attempt}/${NPMJS_VERIFY_ATTEMPTS}: '${NPMJS_SHA:-absent}' vs '${SOURCE_SHA}'); retrying in ${NPMJS_VERIFY_INTERVAL_S}s${NC}"
sleep "$NPMJS_VERIFY_INTERVAL_S"
done
if [ "$PAIR_IDENTICAL" = true ]; then
echo -e "${GREEN}✅ Published to npmjs — byte-identical pair (shasum ${NPMJS_SHA})${NC}\n"
else
echo -e "${RED}❌ REGISTRY DIVERGENCE: The Source shasum ${SOURCE_SHA} != npmjs shasum ${NPMJS_SHA} after ${NPMJS_VERIFY_ATTEMPTS} attempts — investigate before announcing${NC}\n"
echo -e "${RED} v${NEW_VERSION} was tagged and pushed, but @soulcraftlabs/brainy@${NEW_VERSION} never became visible on the${NC}"
echo -e "${RED} Source registry after ${SOURCE_POLL_MAX_ATTEMPTS} attempts, ${SOURCE_POLL_INTERVAL_S}s apart. Aborting.${NC}"
exit 1
fi
@ -267,7 +237,7 @@ fi
# and RELEASES.md are the record; this just gives The Source's UI a release page).
echo -e "${BLUE}🔟 Creating release page on The Source...${NC}"
if [ -n "${FORGEJO_RELEASE_TOKEN:-}" ]; then
if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraft/brainy/releases" \
if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraftlabs/open-brainy/releases" \
-H "Authorization: token ${FORGEJO_RELEASE_TOKEN}" -H "Content-Type: application/json" \
-d "{\"tag_name\":\"v${NEW_VERSION}\",\"name\":\"v${NEW_VERSION}\",\"prerelease\":${PRERELEASE}}" >/dev/null; then
echo -e "${GREEN}✅ Release page created on The Source${NC}\n"
@ -278,21 +248,15 @@ else
echo -e "${RED}⚠️ FORGEJO_RELEASE_TOKEN unset — no release page created; tag + CHANGELOG remain the record${NC}\n"
fi
# Step 12: Push public docs to the soulcraft.com docs ingest door
# (VENUE-DOCS-RELEASE-PUSH). Skips with a loud warning when
# DOCS_INGEST_SECRET is unset; fails loudly (without undoing the publish —
# that already happened) when a push errors, so the docs site never
# silently trails npm.
echo -e "${BLUE}1⃣2⃣ Pushing public docs to soulcraft.com/docs...${NC}"
if node scripts/push-docs.js; then
echo -e "${GREEN}✅ Docs push step done${NC}\n"
else
echo -e "${RED}❌ Docs push FAILED — soulcraft.com/docs trails npm until re-run or interim sync${NC}\n"
fi
# Step 12 RETIRED (2026-08-31, CORTEX-SITE-BRAINY-RENAME round 12, David-ruled):
# soulcraft.com/docs carries the paid product's documentation only. This
# engine's documentation home is THIS repository — README and docs/ — and the
# site serves 301s for the slugs this rail used to push. The push script stays
# in the tree for history; the rail no longer calls it.
echo -e "${BLUE}Docs step: this engine documents itself in its own repo (site push retired 2026-08-31)${NC}"
echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo -e "${GREEN}🎉 Release ${NEW_VERSION} complete!${NC}"
echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo ""
echo -e "📦 npm: ${BLUE}https://www.npmjs.com/package/@soulcraft/brainy/v/${NEW_VERSION}${NC}"
echo -e "🏠 The Source: ${BLUE}https://source.soulcraft.com/soulcraft/brainy/releases/tag/v${NEW_VERSION}${NC}"
echo -e "🏠 The Source: ${BLUE}https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v${NEW_VERSION}${NC}"

File diff suppressed because it is too large Load diff

View file

@ -801,6 +801,17 @@ export interface DerivedFamilyDeclaration {
export interface CanonicalCounts {
nouns: { counted: number; all: number }
verbs: { counted: number; all: number }
/**
* The count of canonical nouns holding a REAL (non-empty) vector the
* coverage denominator a vector index's node-count ledger is measured
* against (`nodeCount === vectors.all` is the whole-store coverage
* verdict for the vector leg, the vector-side mirror of `nouns.all` for
* metadata/graph). A deferred-embed noun (`add({ deferEmbedding: true })`)
* counts only once its vector actually LANDS its canonical record exists
* (counted in `nouns.all`) with an empty vector until then, so it is
* deliberately NOT counted here in the interim.
*/
vectors: { all: number }
/** An unprovable delete has left the `all` scalars unverified since the last recount. */
suspect: boolean
}
@ -819,14 +830,68 @@ export interface StorageAdapter {
* Save noun metadata separately
* @param id Noun ID
* @param metadata Noun metadata
* @param hasVector - OPTIONAL vectored-noun ledger hint: `true` when this
* write is a FRESH insert (`isNew`) whose vector is a real, non-empty
* array the caller already knows this for free (the insert's own
* `vector` local), so the increment rides the SAME isNew gate that
* already protects `totalNounCountAll` from double-counting on HNSW
* neighbor-link re-saves (`saveNoun_internal` re-runs on every link
* change; this metadata seam does not). Absent/`false` no ledger
* action. A deferred-embed insert passes `false` (its vector lands
* later see {@link StorageAdapter.noteVectorLanded}).
*/
saveNounMetadata(id: string, metadata: NounMetadata): Promise<void>
saveNounMetadata(id: string, metadata: NounMetadata, hasVector?: boolean): Promise<void>
/**
* Delete noun metadata
* @param id Noun ID
* @param priorRecord - OPTIONAL already-known metadata (the caller's
* pre-delete read) see {@link StorageAdapter.deleteNoun}.
* @param hadVector - OPTIONAL vectored-noun ledger hint: `true`/`false`
* when the caller already knows (read as a side effect of ITS OWN delete
* flow e.g. `remove()`'s pre-read for the vector-index removal never
* a read added FOR this ledger), `undefined` when genuinely unknown. A
* known `true` decrements the vectored-noun ledger; a known `false` is a
* no-op (it was never counted); `undefined` marks the ledger SUSPECT
* rather than guessing the delete path must never add a canonical read
* to answer this question.
*/
deleteNounMetadata(id: string): Promise<void>
deleteNounMetadata(id: string, priorRecord?: NounMetadata | null, hadVector?: boolean): Promise<void>
/**
* OPTIONAL narrow ledger hook: record that a canonical noun's vector just
* LANDED for the first time. Exists ONLY for the deferred-embedding
* lifecycle the landing commit (`system:embed-landing`) carries a vector
* write with no accompanying metadata operation, so the normal
* `saveNounMetadata(..., hasVector)` seam never fires for it. Callers MUST
* call this only when the noun held NO real vector before this write (the
* deferred-embed worker already holds that fact for free, from its own
* pre-embed read never an added read). A backend without vectored-noun
* tracking is a no-op via this method's absence (feature-detected).
* @param id - The noun whose vector just landed.
*/
noteVectorLanded?(id: string): Promise<void>
/**
* OPTIONAL narrow ledger hook, the mirror of {@link noteVectorLanded}:
* record that a canonical noun's vector was just REMOVED rewritten from
* a real (non-empty) vector to the "unvectored" empty-array shape. Exists
* for the ONE sanctioned reverse migration this engine supports: the VFS
* root's zero-norm fix (see `VirtualFileSystem.doInitializeRoot()` and
* `Brainy.unvectorNounForRootMigration()`), which rewrites a pre-fix
* store's all-zero placeholder root vector to `[]` and must decrement
* `vectors.all` through this hook so the coverage ledger never drifts.
* NOT a general-purpose "I removed a vector" callback ordinary
* application data has no sanctioned path from vectored back to
* unvectored (`update()` refuses an empty vector as a dimension
* mismatch by design). Callers MUST call this only when the noun held a
* REAL vector immediately before this write (the caller already holds
* that fact for free, from its own pre-write read never an added read).
* A backend without vectored-noun tracking is a no-op via this method's
* absence (feature-detected).
* @param id - The noun whose vector was just removed.
*/
noteVectorUnlanded?(id: string): Promise<void>
/**
* Get noun with metadata combined
@ -875,8 +940,11 @@ export interface StorageAdapter {
* REQUIRE re-reading the record being removed: when the internal read
* returns `null` (replace race, or a ghost left by an earlier version) the
* decrement falls back to this record instead of being silently skipped.
* @param hadVector OPTIONAL vectored-noun ledger hint see
* {@link StorageAdapter.deleteNounMetadata}'s `hadVector` param, which
* this forwards to unchanged.
*/
deleteNoun(id: string, priorMetadata?: NounMetadata | null): Promise<void>
deleteNoun(id: string, priorMetadata?: NounMetadata | null, hadVector?: boolean): Promise<void>
/**
* Save verb - Pure HNSW verb with core fields only

View file

@ -28,7 +28,7 @@
* speculative `with()` overlay; the canonical storage walk only ever answers
* "what is live right now."
*
* All are exported from the package root (`@soulcraft/brainy`).
* All are exported from the package root (`@soulcraftlabs/brainy`).
*/
/**

View file

@ -12,9 +12,11 @@
* the verified surface is a small set of rollup invariants (entity/
* relationship counts) plus `sourceGeneration`.
*
* `sourceGeneration` is the generation of the source-of-truth log this
* projection reflects open-time coherence becomes a COMPARISON (stamp vs
* log head), not a walk:
* `sourceGeneration` is the COMMITTED generation of the source-of-truth log
* this projection reflects never the allocated counter, which names a
* generation that may never commit (see {@link StampVerdict.torn}) so
* open-time coherence becomes a COMPARISON (stamp vs committed head), not a
* walk:
*
* - equal + invariants hold coherent, serve.
* - behind the projection missed the tail (crash between commit and stamp);
@ -24,6 +26,9 @@
* - invariants FAIL at equal generation genuine incoherence: loud, and the
* repair ritual (`repairIndex()`, whose recount rebuilds the rollups from a
* canonical walk) heals it.
* - AHEAD a torn generation-log tail: the stamp's fsync outlived the log
* tail's. TERMINAL, never a wait the generation the stamp names does not
* exist to arrive.
*
* Stamps are JSON on purpose every incident gets debugged by reading a
* stamp in a terminal.
@ -70,6 +75,12 @@ export type StampVerdict =
| { state: 'coherent' }
| { state: 'absent' } // legacy store — first stamp writes at the next flush
| { state: 'behind'; stampSource: number; head: number }
/**
* TORN GENERATION-LOG TAIL: the stamp witnesses a source generation the
* store's committed watermark can no longer show. TERMINAL there is no
* generation to wait for, so the open demotes (or refuses) and never spins.
*/
| { state: 'torn'; stampSource: number; head: number }
| { state: 'incoherent'; failures: string[] }
| { state: 'unverifiable'; reason: string } // a FAULT reading the stamp — never conflated with absence
@ -118,12 +129,15 @@ export function verifyFamilyStamp(
): StampVerdict {
if (stamp === null) return { state: 'absent' }
if (stamp.sourceGeneration > head) {
// A stamp AHEAD of the log claims state that never committed — the
// projection was stamped against truth that a crash rolled back.
return {
state: 'incoherent',
failures: [`sourceGeneration ${stamp.sourceGeneration} is ahead of the log head ${head}`]
}
// A stamp AHEAD of committed truth witnesses a generation the store can no
// longer show: the stamp's fsync survived a crash that the log tail did
// not. This is the TORN GENERATION-LOG TAIL — its own class, never folded
// in with `incoherent` (a count that drifted at a generation both sides
// agree on), because the two have opposite cures: incoherence is recounted,
// a tear is DEMOTED. It is also terminal by construction — there is no
// generation the open can wait for, because the one the stamp names is
// gone.
return { state: 'torn', stampSource: stamp.sourceGeneration, head }
}
if (stamp.sourceGeneration < head) {
return { state: 'behind', stampSource: stamp.sourceGeneration, head }

View file

@ -147,6 +147,60 @@ export class GenerationSegmentStore {
return this.coveringSegment(gen) !== null
}
/**
* @description True when `meta` declares more generations than it holds
* frames a segment sealed by a writer that folded across a hole. The
* manifest records `frames` at fold time, so this is an O(1) comparison
* against the declared span and needs no I/O.
*/
private isSparse(meta: SegmentMeta): boolean {
return meta.lastGeneration - meta.firstGeneration + 1 !== meta.frames
}
/**
* @description The generations this tier ACTUALLY holds, as coalesced
* ascending intervals not what the segments declare.
*
* Dense segments (every one a current writer produces) contribute their
* declared range with no I/O. A SPARSE segment one sealed before the
* density law was enforced, whose declared range spans generations it has
* no frame for has its real generation list read from its sidecar and
* contributed instead, with the discrepancy narrated once.
*
* This is what keeps a store that already carries the damage from wedging.
* `open()` seeds `committedRanges` from these intervals, so a hole is never
* re-admitted as a committed generation, and the auto-compaction pass that
* used to fail on every run with "packed history is damaged" simply never
* asks for the missing frame.
*
* @returns Ascending, non-overlapping `[first, last]` intervals.
*/
async actualRanges(): Promise<Array<[number, number]>> {
const out: Array<[number, number]> = []
for (const meta of this.manifest.segments) {
if (!this.isSparse(meta)) {
out.push([meta.firstGeneration, meta.lastGeneration])
continue
}
const missing = meta.lastGeneration - meta.firstGeneration + 1 - meta.frames
prodLog.warn(
`[GenerationSegments] sealed segment ${meta.file} declares generations ` +
`${meta.firstGeneration}..${meta.lastGeneration} but holds only ${meta.frames} ` +
`frame(s) — ${missing} generation(s) in that span were never folded into it. ` +
`Serving the frames it actually holds; the declared span is not treated as ` +
`committed history. (Written by a pre-density-law writer that folded across a ` +
`gap; the segment itself is intact and no record is lost.)`
)
const idx = await this.sidecarFor(meta)
for (const [gen] of idx.generations) {
const last = out[out.length - 1]
if (last !== undefined && gen === last[1] + 1) last[1] = gen
else out.push([gen, gen])
}
}
return out
}
/**
* Fold consecutive generations into ONE new sealed segment + sidecar and
* append it to the manifest atomically. Caller guarantees: `gens` is
@ -164,6 +218,38 @@ export class GenerationSegmentStore {
throw new Error('[GenerationSegments] fold() input must be strictly ascending')
}
}
// THE DENSITY LAW, MADE MECHANICAL.
//
// A sealed segment declares a CONTIGUOUS range [firstGeneration,
// lastGeneration] and every reader treats that range as containment:
// `coveringSegment` is an interval test, `hasGeneration` returns true for
// anything inside it, and `open()` seeds committedRanges from it. So a
// segment folded from a SPARSE input silently claims generations it does
// not hold, and the first read of one of those holes throws
// "inside sealed segment ... but has no frame — packed history is damaged".
//
// That is exactly how the damage was produced. `repackHistory` skipped
// generations mid-batch — ones absent from committedRanges, ones still in
// the pending buffer, ones whose tx.json would not read — and handed the
// survivors here, where the range was computed from the first and last of
// them. Worse, the mis-declared range was then merged back into
// committedRanges at the next open, which is what turned a quiet hole into
// a repeating auto-compaction failure on every subsequent run.
//
// Callers now split at discontinuities; this refusal is what keeps any
// future caller from reintroducing the class. A refusal here loses
// nothing — the generations stay in the live tier, readable, and the next
// pass folds them correctly.
for (let i = 1; i < gens.length; i++) {
if (gens[i].generation !== gens[i - 1].generation + 1) {
throw new Error(
`[GenerationSegments] fold() input is not contiguous: ${gens[i - 1].generation}` +
`${gens[i].generation} skips ${gens[i].generation - gens[i - 1].generation - 1} ` +
`generation(s). A sealed segment declares a dense range, so folding a sparse ` +
`batch would claim generations it does not hold. Split the batch at the gap.`
)
}
}
const last = this.manifest.segments[this.manifest.segments.length - 1]
if (last && gens[0].generation <= last.lastGeneration) {
throw new Error(
@ -364,12 +450,37 @@ export class GenerationSegmentStore {
return this.decodeFrame(payload)
}
}
// In the covering range but not present: the packed tier is dense by
// construction (fold packs every generation it is handed, including
// record-less ones) — absence inside a sealed range is damage.
// Inside the covering range but with no frame. Two very different causes,
// and conflating them is what made this class wedge every maintenance pass
// on the affected stores.
//
// (1) A SPARSE SEGMENT — the manifest's own `frames` count is smaller than
// the span it declares. That segment was sealed by a writer that
// folded across a hole (the class this file's density law now bars).
// The segment is INTACT and nothing is lost; it simply never held this
// generation. Answering "not packed" is the honest answer, and it lets
// the caller's two-tier read decide what a genuinely absent generation
// means, instead of every compaction pass dying on a repeating throw.
// `actualRanges()` keeps such holes out of committedRanges at open, so
// in a healed store nobody asks this question in the first place.
//
// (2) A DENSE SEGMENT missing a frame it says it has — the manifest and
// the sidecar disagree about a segment that claims to be complete.
// That IS damage, and it stays loud.
if (this.isSparse(meta)) {
prodLog.warn(
`[GenerationSegments] generation ${gen} falls inside sealed segment ${meta.file}'s ` +
`declared range ${meta.firstGeneration}..${meta.lastGeneration}, but that segment ` +
`holds ${meta.frames} frame(s) for a ${meta.lastGeneration - meta.firstGeneration + 1}` +
`-generation span — it was sealed across a gap and never held this generation. ` +
`Reporting it as unpacked rather than as damage; no record is lost.`
)
return null
}
throw new Error(
`[GenerationSegments] generation ${gen} is inside sealed segment ${meta.file}'s declared ` +
`range but has no frame — packed history is damaged`
`range but has no frame, and that segment declares a complete ${meta.frames}-frame ` +
`span — the manifest and the sidecar disagree; packed history is damaged`
)
}

View file

@ -96,6 +96,35 @@ export const FOLD_CHECKPOINT_PATH = '_system/fold-checkpoint.json'
/** Storage-root-relative prefix of the per-generation record directories. */
export const GENERATIONS_PREFIX = '_generations'
/**
* @description Split an ascending list of fold candidates into maximal
* CONTIGUOUS runs `[7,8,9,12,13]` becomes `[[7,8,9],[12,13]]`.
*
* A sealed segment declares one dense range `[firstGeneration,
* lastGeneration]`, and every reader treats that range as containment. So a
* batch with a hole in it must never become one segment: it would claim a
* generation it does not hold, and the first read of that hole reports the
* packed history as damaged. One run, one segment the ranges then describe
* exactly what the segments contain.
*
* @param gens - Fold candidates, strictly ascending by generation.
* @returns One array per contiguous run, in ascending order. Empty in, empty out.
*/
export function contiguousRuns(gens: FoldGeneration[]): FoldGeneration[][] {
const runs: FoldGeneration[][] = []
let run: FoldGeneration[] = []
for (const g of gens) {
const prev = run[run.length - 1]
if (prev !== undefined && g.generation !== prev.generation + 1) {
runs.push(run)
run = []
}
run.push(g)
}
if (run.length > 0) runs.push(run)
return runs
}
/**
* @description Phases of the {@link GenerationStore.commitTransaction} commit
* protocol at which a test-only fault injector can simulate a process crash.
@ -537,12 +566,29 @@ export class GenerationStore {
this.horizonGen = finiteGen(manifest?.horizon, 'manifest horizon')
this.counter = Math.max(finiteGen(counterFile?.generation, 'generation counter'), this.committed)
// Discover existing generation record directories.
const recordPaths = await this.storage.listRawObjects(GENERATIONS_PREFIX)
// Discover existing generation record directories — BY DIRECTORY NAME.
// This used to call listRawObjects(), which recurses the whole
// `_generations/` tree and returns every file in every generation, to
// extract a set of integers the top-level directory names already spell.
// MEASURED on a real store with an 11 GB generation history: the phase
// this sits in cost 55,538 ms of a WARM REOPEN after a clean close, with
// no fold to blame — this walk is what it was doing. An adapter without
// the one-level door falls back to the recursive listing, unchanged.
const seenGens = new Set<number>()
for (const p of recordPaths) {
const gen = parseGenerationFromPath(p)
if (gen !== null) seenGens.add(gen)
const oneLevel = (
this.storage as { listRawPrefixes?: (prefix: string) => Promise<string[]> }
).listRawPrefixes
if (typeof oneLevel === 'function') {
for (const name of await oneLevel.call(this.storage, GENERATIONS_PREFIX)) {
const gen = Number(name)
if (Number.isSafeInteger(gen) && gen >= 0) seenGens.add(gen)
}
} else {
const recordPaths = await this.storage.listRawObjects(GENERATIONS_PREFIX)
for (const p of recordPaths) {
const gen = parseGenerationFromPath(p)
if (gen !== null) seenGens.add(gen)
}
}
let rolledBack = 0
@ -652,21 +698,56 @@ export class GenerationStore {
: 'WHOLE-LOG fold'
: 'above-manifest replay'
let replayed = 0
const foldStartedAt = Date.now()
const replayFact = async (fact: CommitFact): Promise<void> => {
for (const op of fact.ops) {
const image =
op.record === null
? { metadata: null, vector: null }
: { metadata: op.record.metadata, vector: op.record.vector }
let image: { metadata: unknown | null; vector: unknown | null }
if (op.record === null) {
// A genuine tombstone (both legs absent) — the fold removes
// both legs, exactly like `writeNounRaw`/`writeVerbRaw`'s raw
// exact-restore contract.
image = { metadata: null, vector: null }
} else if (
op.record.metadata !== null &&
(op.record.vector === null || op.record.vector === undefined)
) {
// PRESERVE-IF-ABSENT (population law, ADR-008 G1 — the fold's
// half): a metadata-only after-image must never DELETE an
// existing vector leg through the fold. `writeNounRaw`/
// `writeVerbRaw` are exact-restore primitives — a `vector:
// null` there means "delete", which is exactly right for
// `rollBackUncommittedGeneration`'s before-image restore (a
// transaction abort legitimately un-writes a vector the failed
// transaction added). It is NOT right here: this fold replays
// AFTER-IMAGES, and re-applying an already-intact record must
// be byte-safe (this module's own invariant, see the log-authority
// comment above) — silently erasing a landed vector because one
// replayed fact's vector leg came back null is the exact defect
// that left metadata-counted, never-enumerated rows in a
// production store (confirmed root cause: the enumeration walk
// used to key on the vector leg, so a preserved-but-then-deleted
// vector made the row invisible while the ledger still counted
// it by metadata). A genuine "unvector" has its own sanctioned,
// ledger-correct path (`Brainy.unvectorNounForRootMigration`) —
// never this raw primitive, and never the fold.
const current =
op.kind === 'verb'
? await this.storage.readVerbRaw(op.id)
: await this.storage.readNounRaw(op.id)
image = { metadata: op.record.metadata, vector: current.vector ?? null }
} else {
image = { metadata: op.record.metadata, vector: op.record.vector }
}
if (op.kind === 'verb') await this.storage.writeVerbRaw(op.id, image)
else await this.storage.writeNounRaw(op.id, image)
this.noteCheckpointDirty(op.kind, op.id)
}
replayed++
if (replayed % 1000 === 0) {
prodLog.warn(
prodLog.narrate(
`[GenerationStore] recovery fold in progress — ${replayed} fact(s) folded ` +
`(at generation ${fact.generation}); do not restart, the fold is finite`
`in ${Date.now() - foldStartedAt}ms (at generation ${fact.generation}); ` +
`do not restart, the fold is finite`
)
}
if (fact.generation > this.committed) {
@ -681,7 +762,7 @@ export class GenerationStore {
}
}
if (uncleanOpen) {
prodLog.warn(
prodLog.narrate(
`[GenerationStore] log-authority recovery: ${foldKind} beginning ` +
`(unclean shutdown detected) — streaming replay, bounded memory, ` +
`progress every 1000 facts. Do not restart the process; a restart ` +
@ -704,9 +785,10 @@ export class GenerationStore {
}
await this.storage.writeRawObject(MANIFEST_PATH, manifest)
await this.storage.syncRawObjects([MANIFEST_PATH])
prodLog.warn(
prodLog.narrate(
`[GenerationStore] log-authority recovery replayed ${replayed} fact(s) into ` +
`canonical (${foldKind}; committed at ${this.committed}) — an acked write is never lost`
`canonical in ${Date.now() - foldStartedAt}ms (${foldKind}; committed at ` +
`${this.committed}) — an acked write is never lost`
)
}
// A recovery fold re-applied (and the barrier below re-syncs) every
@ -731,9 +813,15 @@ export class GenerationStore {
if (storageSupportsFactLog(this.storage)) {
this.segments = new GenerationSegmentStore(this.storage)
await this.segments.open()
const packedRanges = this.segments
.segments()
.map((s): [number, number] => [s.firstGeneration, Math.min(s.lastGeneration, this.committed)])
// ACTUAL ranges, not declared ones. A segment sealed by a pre-density-law
// writer can declare a span wider than the frames it holds; seeding
// committedRanges from the declared span re-admits those holes as
// committed generations, and every later maintenance pass then asks for a
// frame that was never written. `actualRanges()` reads the real
// generation list from the sidecar for exactly those segments (and does
// no I/O for the dense ones, which is all of them on a healthy store).
const packedRanges = (await this.segments.actualRanges())
.map((r): [number, number] => [r[0], Math.min(r[1], this.committed)])
.filter(([lo, hi]) => lo <= hi)
if (packedRanges.length > 0) {
// Merge packed (older) + live (newer) interval sets — both ascending;
@ -3068,13 +3156,26 @@ export class GenerationStore {
foldInput.push({ generation: gen, timestamp: delta.timestamp, delta, records })
}
if (foldInput.length === 0) continue
await segments.fold(foldInput)
segmentsCreated++
// Segment + manifest durable → the live copies retire.
for (const g of foldInput) {
await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${g.generation}`)
// SPLIT AT DISCONTINUITIES. `eligible` is NOT contiguous — three
// filters above punch holes in it: a generation missing from
// committedRanges never appears, one still in the pending buffer is
// skipped, and one whose tx.json will not read is skipped. A sealed
// segment declares a DENSE range, so folding across such a hole makes
// the segment claim a generation it does not hold; the next open
// merges that mis-declared range into committedRanges, and every
// subsequent auto-compaction pass then asks for the missing frame and
// fails with "packed history is damaged". Fold each contiguous RUN as
// its own segment instead — same bytes, honest ranges.
for (const run of contiguousRuns(foldInput)) {
if (deadline !== undefined && Date.now() >= deadline) break
await segments.fold(run)
segmentsCreated++
// Segment + manifest durable → the live copies retire.
for (const g of run) {
await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${g.generation}`)
}
folded += run.length
}
folded += foldInput.length
}
if (folded > 0) {
prodLog.info(

View file

@ -450,6 +450,21 @@ export interface GenerationStorage {
deleteRawObject(path: string): Promise<void>
/** List raw object paths under a prefix (normalized, `.gz`-stripped). */
listRawObjects(prefix: string): Promise<string[]>
/**
* OPTIONAL: the IMMEDIATE child directory names under a prefix one level,
* no recursion, no file paths.
*
* Why it exists: discovering which generations are on disk needs only the
* top-level directory NAMES under `_generations/`, but the only door for it
* was `listRawObjects`, which recurses the whole tree and returns every file
* in every generation. On a store with a long history that is a full walk of
* the entire generation log, paid on EVERY open, to learn a set of integers
* the directory names already spell out.
*
* An adapter without this door keeps working the caller falls back to the
* recursive listing.
*/
listRawPrefixes?(prefix: string): Promise<string[]>
/** Remove every object under a prefix (and the directory itself on disk). */
removeRawPrefix(prefix: string): Promise<void>
/** Durability barrier: fsync the given object paths (no-op in memory). */

View file

@ -128,7 +128,7 @@ async function loadBunAssets(): Promise<ModelAssets> {
}
// Strategy 2: node_modules path relative to CWD (for installed packages)
const nmPath = './node_modules/@soulcraft/brainy/assets/models/all-MiniLM-L6-v2'
const nmPath = './node_modules/@soulcraftlabs/brainy/assets/models/all-MiniLM-L6-v2'
pathsToTry.push([
`${nmPath}/model.safetensors`,
`${nmPath}/tokenizer.json`,
@ -168,9 +168,9 @@ async function loadBunAssets(): Promise<ModelAssets> {
// If all strategies fail, provide helpful error message
throw new Error(
'Could not load model assets. For bun --compile, ensure model files are accessible:\n' +
' Option 1: Keep node_modules/@soulcraft/brainy/assets/ alongside your binary\n' +
' Option 1: Keep node_modules/@soulcraftlabs/brainy/assets/ alongside your binary\n' +
' Option 2: Copy assets/ folder to your working directory\n' +
' Option 3: Use --asset flag: bun build --compile --asset="./node_modules/@soulcraft/brainy/assets/**/*"'
' Option 3: Use --asset flag: bun build --compile --asset="./node_modules/@soulcraftlabs/brainy/assets/**/*"'
)
}
@ -190,7 +190,7 @@ async function loadNodeAssets(): Promise<ModelAssets> {
if (!fs.existsSync(assetsDir)) {
throw new Error(
`Model assets not found: ${assetsDir}\n` +
`Ensure @soulcraft/brainy is installed correctly.`
`Ensure @soulcraftlabs/brainy is installed correctly.`
)
}

View file

@ -14,7 +14,7 @@
* - {@link RelationNotFoundError} a referenced relationship (verb) does
* not exist.
*
* Both are exported from the package root (`@soulcraft/brainy`).
* Both are exported from the package root (`@soulcraftlabs/brainy`).
*/
/**

View file

@ -1052,6 +1052,17 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
*/
private startAutoFlush(): void {
this.flushTimer = setInterval(async () => {
// NO PERIODIC WORK WITHOUT A CAUSE. Ask first, in two O(1) reads: an
// index nobody has written to since the last flush has nothing to
// write, and calling into the trees (and their logging) on a cadence
// over a quiet store is exactly the idle cost this law exists to
// remove.
if (
!this.lsmTreeVerbsBySource.hasPendingWrites() &&
!this.lsmTreeVerbsByTarget.hasPendingWrites()
) {
return
}
await this.flush()
}, this.config.flushInterval)
// Background maintenance must never keep the host process alive —

View file

@ -687,6 +687,17 @@ export class LSMTree {
}
}
/**
* @description Whether this tree holds anything a flush would write
* the MemTable is non-empty. Synchronous and O(1), so a background cadence
* can ask before it does anything at all: the engine does no periodic work
* without a cause.
* @returns true when a flush would write; false when it would be a no-op.
*/
hasPendingWrites(): boolean {
return !this.memTable.isEmpty()
}
async close(): Promise<void> {
this.stopCompactionTimer()

View file

@ -10,7 +10,7 @@ import {
Vector,
VectorDocument
} from '../coreTypes.js'
import { euclideanDistance, calculateDistancesBatch } from '../utils/index.js'
import { euclideanDistance, calculateDistancesBatch, isZeroNormVector } from '../utils/index.js'
import type { BaseStorage } from '../storage/baseStorage.js'
import { getGlobalCache, UnifiedCache } from '../utils/unifiedCache.js'
import { prodLog } from '../utils/logger.js'
@ -64,6 +64,34 @@ export class HnswFlushError extends Error {
}
}
/**
* @description Thrown by {@link JsHnswVectorIndex.addItem} / {@link
* JsHnswVectorIndex.updateItem} when handed a length-0 vector. A length-0
* vector is the sanctioned "unvectored" shape for a canonical noun record
* (class-J: a VFS-system row, a deferred embed not yet landed, or any other
* legitimately-vector-less row) but it is NEVER a legal INDEX insert. The
* index itself has no concept of "unvectored"; deciding that a row is
* unvectored and therefore skippable is the FILL/REBUILD/LOAD consumer's job
* (see {@link JsHnswVectorIndex.rebuild}), done BEFORE ever calling addItem.
* A length-0 vector reaching this point is a caller bug: silently accepting
* it would pin `this.dimension = 0` on an empty index (poisoning every real
* insert thereafter with a dimension mismatch) or store a vector-less node
* that a distance calculation can never safely compare against. Loud errors,
* never quiet losses this throws instead of either.
*/
export class EmptyVectorIndexError extends Error {
constructor(public readonly id: string, operation: 'addItem' | 'updateItem') {
super(
`${operation}(${id}): refusing to index a length-0 vector — a length-0 vector is the ` +
`sanctioned "unvectored" shape for a canonical row, but it is never a legal index ` +
`insert. Callers that fill/rebuild/load the index must skip vector.length === 0 rows ` +
`themselves (unvectored = nothing to index, not an error at that layer); reaching ` +
`here with one is a caller bug.`
)
this.name = 'EmptyVectorIndexError'
}
}
/**
* Implements {@link VectorIndexProvider}: the vector-index surface Brainy calls
* on whatever the `'vector'` factory returns (its own `JsHnswVectorIndex`, or a native
@ -580,6 +608,15 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
throw new Error('Vector is undefined or null')
}
// THE INDEX REFUSES A LENGTH-0 VECTOR (see EmptyVectorIndexError's JSDoc):
// an empty vector is the sanctioned "unvectored" shape at the canonical
// layer, never a legal index member. Refusing here — loudly, before the
// dimension pin below — means no future fill/rebuild/load path can ever
// poison `this.dimension` to 0 or park a vector-less node in the graph.
if (vector.length === 0) {
throw new EmptyVectorIndexError(id, 'addItem')
}
// Set dimension on first insert
if (this.dimension === null) {
this.dimension = vector.length
@ -954,6 +991,13 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
return
}
// Same refusal as addItem (see EmptyVectorIndexError's JSDoc) — an
// in-place relink must never rewrite an already-indexed node down to the
// unvectored shape or poison the pinned dimension.
if (vector.length === 0) {
throw new EmptyVectorIndexError(id, 'updateItem')
}
if (this.dimension === null) {
this.dimension = vector.length
} else if (vector.length !== this.dimension) {
@ -1555,7 +1599,15 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
}
const loaded = await this.storage.getNounVector(noun.id)
if (!loaded) {
// `loaded` is a length-0 array (not null/undefined) for a canonical row
// that is legitimately unvectored — `![]` is FALSE (an empty array is
// truthy), so the bare `!loaded` check below would silently accept it
// as "found" and hand a dimension-0 vector to a distance calculation.
// A node only reaches this lazy-load path because it is a MEMBER of
// the live index (rebuild() now refuses to admit unvectored rows — see
// its JSDoc), so an empty vector here is never legitimate: treat it
// exactly like "not found", loudly.
if (!loaded || loaded.length === 0) {
throw new Error(`Vector not found for noun ${noun.id}`)
}
@ -1765,9 +1817,56 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
totalCount = result.totalCount || result.items.length
// UNVECTORED ROWS ARE NOT AN INDEX MEMBER (the class-J law): a canonical
// noun whose vector leg is `[]` (a VFS-root-style system row, a
// deferred embed not yet landed, or a best-effort fallback for an
// unreadable vector leg) is a normal, enumerable, countable row — it
// is simply not indexed. `storage.getVectorIndexData()` derives its
// {level, connections} answer straight from the noun's OWN record, so
// it returns non-null for every existing noun regardless of whether
// that noun ever actually reached `addItem()` — it cannot be used to
// decide indexability. `nounData.vector.length === 0` is the one
// truthful signal (mirrors the `noun.vector.length > 0` guards in
// {@link getVectorSafe} / {@link getVectorSync}): skip here, counted
// once in a summary line, never per-row spam.
let skippedUnvectored = 0
// Process all nouns at once
for (const nounData of result.items) {
try {
if (!Array.isArray(nounData.vector) || nounData.vector.length === 0) {
skippedUnvectored++
continue
}
// THE ZERO-NORM LAW — bulk-rebuild leg: a persisted zero-norm
// vector (a pre-10.4.2 row the canonical write has not yet
// normalized) must never enter the index either, mirroring the
// belt AddToVectorIndexOperation enforces on the live write path.
// Only the canonical vector is authoritative here — persisted
// HNSW graph metadata (level/connections) can outlive an unvector.
if (isZeroNormVector(nounData.vector)) {
prodLog.warn(
`[HNSW] rebuild(): skipping entity ${nounData.id} — persisted vector is ` +
`zero-norm (a zero-norm vector is not a vector and never crosses an ` +
`engine boundary)`
)
continue
}
// Restore the pinned dimension from the first real vector this
// rebuild loads. `addItem`/`updateItem` only pin `this.dimension`
// on a LIVE insert — a fresh rebuild from storage never goes
// through either, so without this the pin stays `null` across a
// restart. A `null` pin means the very next insert (correct OR
// wrong length) silently BECOMES the new pin instead of being
// checked against the store's real dimension — the wrong-length
// case then fails much later and less clearly, inside a distance
// calculation against an already-loaded node, instead of here,
// immediately, with a named expected-vs-got mismatch.
if (this.dimension === null) {
this.dimension = nounData.vector.length
}
// Load HNSW graph data for this entity
const hnswData = await this.storage.getVectorIndexData(nounData.id)
@ -1815,7 +1914,10 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
options.onProgress(loadedCount, totalCount)
}
prodLog.info(`HNSW: Loaded ${loadedCount.toLocaleString()} nodes (${storageType})`)
prodLog.info(
`HNSW: Loaded ${loadedCount.toLocaleString()} nodes (${storageType})` +
(skippedUnvectored > 0 ? `${skippedUnvectored.toLocaleString()} unvectored row(s) skipped` : '')
)
}
// Step 5: CRITICAL - Recover entry point if missing)

View file

@ -184,6 +184,7 @@ export {
// Export version utilities
export { getBrainyVersion } from './utils/version.js'
export { contractVersion, BRAINY_CONTRACT_VERSION } from './utils/version.js'
// Export plugin system
export type { BrainyPlugin, BrainyPluginContext, StorageAdapterFactory } from './plugin.js'
@ -269,6 +270,10 @@ export type { FamilyStamp, StampMembers, StampVerdict } from './db/familyStamp.j
export { isVersionedIndexProvider } from './plugin.js'
export type { VersionedIndexProvider } from './plugin.js'
export type { ProviderInvariantReport, InvariantResult, InvariantHeal } from './plugin.js'
// The named, synchronous, O(1) health-report contract (the read gate's ONLY
// source of truth for "can I serve right now") — see HealthReport's
// derivation laws in plugin.ts.
export type { HealthReport, LedgerInvariantResult, InvariantSource } from './plugin.js'
// Optional provider self-report of outstanding background maintenance work
// (compaction, deferred writes, etc.) — the payload type for
// brain.maintenanceDebt(). See the measure-only-what-you-track contract on
@ -385,7 +390,10 @@ import type {
HNSWVerb,
HNSWConfig,
StorageAdapter,
DerivedFamilyDeclaration
DerivedFamilyDeclaration,
// The canonical count ledger a storage adapter maintains (counted + ALL-visibility
// scalars per family, the coverage-ledger denominators) — see StorageAdapter.getCanonicalCounts.
CanonicalCounts
} from './coreTypes.js'
// Export vector index implementation (the JS HNSW path)

View file

@ -9,7 +9,7 @@
*
* @example Enable integrations (recommended)
* ```typescript
* import { Brainy } from '@soulcraft/brainy'
* import { Brainy } from '@soulcraftlabs/brainy'
*
* const brain = new Brainy({ integrations: true })
* await brain.init()

View file

@ -41,7 +41,7 @@ The `BrainyMCPService` has been refactored to separate the core functionality fr
### In Any Environment (Browser, Node.js, Server)
```typescript
import { Brainy, BrainyMCPAdapter, MCPAugmentationToolset } from '@soulcraft/brainy'
import { Brainy, BrainyMCPAdapter, MCPAugmentationToolset } from '@soulcraftlabs/brainy'
// Create a Brainy instance
const brainyData = new Brainy()
@ -81,7 +81,7 @@ const toolResponse = await toolset.handleRequest({
### In Browser Environment (Core Functionality Only)
```typescript
import { Brainy, BrainyMCPService } from '@soulcraft/brainy'
import { Brainy, BrainyMCPService } from '@soulcraftlabs/brainy'
// Create a Brainy instance
const brainyData = new Brainy()

View file

@ -2,7 +2,7 @@
* 🧠 BRAINY EMBEDDED PATTERNS
*
* AUTO-GENERATED - DO NOT EDIT
* Generated: 2026-07-02T21:43:26.976Z
* Generated: 2025-09-29T10:10:00-07:00
* Patterns: 220
* Coverage: 94-98% of all queries
*

View file

@ -2,7 +2,7 @@
* 🧠 BRAINY EMBEDDED TYPE EMBEDDINGS
*
* AUTO-GENERATED - DO NOT EDIT
* Generated: 2026-02-09T16:59:48.867Z
* Generated: 2026-06-29T10:04:19-07:00
* Noun Types: 42
* Verb Types: 127
*
@ -19,7 +19,7 @@ export const TYPE_METADATA = {
verbTypes: 127,
totalTypes: 169,
embeddingDimensions: 384,
generatedAt: "2026-02-09T16:59:48.867Z",
generatedAt: "2026-06-29T10:04:19-07:00",
sizeBytes: {
embeddings: 259584,
base64: 346112

View file

@ -9,6 +9,7 @@
* registered manually via `brain.use()` there is no implicit detection.
*/
import { prodLog } from './utils/logger.js'
import type {
StorageAdapter,
Vector,
@ -21,7 +22,7 @@ import type { GraphIndexStats } from './graph/graphAdjacencyIndex.js'
// Re-export the provider contracts that already live closer to their
// implementations so a plugin author (Cor) can import the *entire*
// provider surface from one stable entrypoint: `@soulcraft/brainy/plugin`.
// provider surface from one stable entrypoint: `@soulcraftlabs/brainy/plugin`.
export type { ColumnStoreProvider } from './indexes/columnStore/types.js'
export type {
AggregationProvider,
@ -40,7 +41,7 @@ export interface BrainyPlugin {
name: string
/**
* Optional semver range of `@soulcraft/brainy` this plugin supports
* Optional semver range of `@soulcraftlabs/brainy` this plugin supports
* (e.g. `'>=8.0.0 <9.0.0'` or `'^8.0.0'`). When set and the running brainy is
* OUTSIDE the range, `init()` THROWS rather than silently falling back to the
* default JS engine. This is the version-coupling guard for the native
@ -171,6 +172,66 @@ export interface ProviderInvariantReport {
durationMs: number
}
/**
* @description Where a {@link LedgerInvariantResult} verdict came from:
* - `'ledger'` decided from an exact, durable ledger (a real count, not a sample).
* - `'deep'` decided by a full/expensive scan (the `validateInvariants()` diagnostic path only).
* - `'unledgered'` this family has no ledger yet; the verdict is UNKNOWN, never healthy and never broken.
*/
export type InvariantSource = 'ledger' | 'deep' | 'unledgered'
/**
* @description One invariant verdict inside a {@link HealthReport}. Extends
* {@link InvariantResult} with the provenance of the verdict ({@link InvariantSource})
* and, for a failing set-membership invariant, an exact count plus a capped sample
* of the diverging ids a VERDICT, never a dump. `sample` MUST be capped at 16 ids;
* `count` is the exact number even when `sample` is truncated.
*/
export interface LedgerInvariantResult extends InvariantResult {
/** Provenance of this verdict — see {@link InvariantSource}. */
source: InvariantSource
/** Exact count of diverging/missing items plus a capped (≤16 ids) sample. Present only on a failing set-membership invariant. */
missing?: { count: number; sample: string[] }
}
/**
* @description The NAMED, SYNCHRONOUS, O(1) health report a provider exposes via
* {@link MetadataIndexProvider.healthReport} / {@link GraphIndexProvider.healthReport} /
* {@link VectorIndexProvider.healthReport}. This is the read gate's ONLY source of
* truth for "can I serve right now" it replaces sampled self-probes and the
* unnamed `isReady()` latch with an exact, ledger-derived verdict.
*
* Derivation laws (a provider MUST honor these; brainy's read gate assumes them):
* - `healthy` = every VERIFIED invariant in {@link invariants} holds. An invariant
* whose family is named in {@link unledgered} is NEVER counted toward `healthy`
* either way it is unknown, not passing.
* - `serving` = no verified invariant in {@link invariants} FAILS with `heal: 'rebuild'`.
* A failure with `heal: 'repair'` or `heal: 'none'` is degraded-but-serving
* `serving` stays `true`. Only a `'rebuild'`-grade failure makes `serving` `false`.
* - `validateInvariants()` remains the async DEEP diagnostic (full scans allowed,
* `source: 'deep'` results); `healthReport()` MUST be synchronous, O(1) from
* exact ledgers/counters, and MUST NOT throw for a well-formed provider a
* provider that cannot produce a safe verdict reports it as a failing invariant,
* it does not throw (a throw is read by the gate as a CONTRACT VIOLATION, not as
* "unknown").
*/
export interface HealthReport extends ProviderInvariantReport {
/**
* Monotonic per provider: bumps on every ledger mutation and every rebuild
* boundary. Consumers (the read gate's narration dedup, external callers) may
* cache a verdict per generation.
*/
generation: number
/** Each checked invariant, with provenance — see {@link LedgerInvariantResult}. */
invariants: LedgerInvariantResult[]
/**
* Families with no ledger yet. NAMED here so an operator can see what is not
* yet tracked NEVER counted as healthy (they are not verified) and NEVER
* counted as broken (there is nothing to fail).
*/
unledgered: string[]
}
/**
* @description A provider's self-report of its own outstanding background
* maintenance work (compaction, deferred writes, a build-newverifyswap in
@ -266,6 +327,20 @@ export interface MetadataIndexProvider {
*/
validateInvariants?(): Promise<ProviderInvariantReport>
/**
* @description OPTIONAL. The named, SYNCHRONOUS, O(1) health verdict this
* provider derives from its own exact ledgers see {@link HealthReport} for
* the full derivation laws. MUST NOT perform I/O and MUST NOT throw for a
* well-formed provider (brainy treats a throw as a CONTRACT VIOLATION, never
* as "unknown"). When present, brainy's read gate (`assessProviderHealth()`)
* reads THIS instead of `isReady()` / size heuristics: `serving` decides
* whether reads may proceed; a `false` refuses the read loudly rather than
* triggering a rebuild. Absent the gate falls back to `isReady?()` / the
* size heuristic (this train's JS built-in providers stay on that interim
* path).
*/
healthReport?(): HealthReport
/**
* @description OPTIONAL. A native provider returns true from the moment its
* `init()` detects a large epoch-drift until its background
@ -462,6 +537,20 @@ export interface GraphIndexProvider {
*/
validateInvariants?(): Promise<ProviderInvariantReport>
/**
* @description OPTIONAL. The named, SYNCHRONOUS, O(1) health verdict this
* provider derives from its own exact ledgers see {@link HealthReport} for
* the full derivation laws. MUST NOT perform I/O and MUST NOT throw for a
* well-formed provider (brainy treats a throw as a CONTRACT VIOLATION, never
* as "unknown"). When present, brainy's read gate (`assessProviderHealth()`)
* reads THIS instead of `isReady()` / size heuristics: `serving` decides
* whether reads may proceed; a `false` refuses the read loudly rather than
* triggering a rebuild. Absent the gate falls back to `isReady?()` / the
* size heuristic (this train's JS built-in providers stay on that interim
* path).
*/
healthReport?(): HealthReport
/**
* @description OPTIONAL eager cold-load. Called once during brain init AFTER
* the metadata provider's `init()` (so the id-mapper is hydrated; a native int
@ -1225,6 +1314,20 @@ export interface VectorIndexProvider {
*/
validateInvariants?(): Promise<ProviderInvariantReport>
/**
* @description OPTIONAL. The named, SYNCHRONOUS, O(1) health verdict this
* provider derives from its own exact ledgers see {@link HealthReport} for
* the full derivation laws. MUST NOT perform I/O and MUST NOT throw for a
* well-formed provider (brainy treats a throw as a CONTRACT VIOLATION, never
* as "unknown"). When present, brainy's read gate (`assessProviderHealth()`)
* reads THIS instead of `isReady()` / size heuristics: `serving` decides
* whether reads may proceed; a `false` refuses the read loudly rather than
* triggering a rebuild. Absent the gate falls back to `isReady?()` / the
* size heuristic (this train's JS built-in providers stay on that interim
* path).
*/
healthReport?(): HealthReport
/**
* @description OPTIONAL. A native provider returns true from the moment its
* `init()` detects a large epoch-drift until its background
@ -1472,9 +1575,13 @@ export class PluginRegistry {
this.activated.add(name)
activated.push(name)
} else {
// Documented graceful decline (activate() → false). Surface it loudly so
// a silent degrade to the default engine never goes unnoticed.
console.warn(
// Documented graceful decline (activate() → false). Surface it on the
// ALWAYS-ON channel: `silent: true` patches console, and a declined
// accelerator warned into a patched console is a silent degrade to the
// default engines — the exact invisible-fallback class this registry
// exists to prevent (a production storm ran the WASM engine for 90s
// behind one suppressed warn).
prodLog.warn(
`[brainy] Plugin "${name}" declined activation (activate() returned false); ` +
`the default engine is in use for its providers.`
)

View file

@ -1041,16 +1041,43 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
*/
protected totalNounCountAll = 0
protected totalVerbCountAll = 0
/**
* The count of canonical nouns holding a REAL (non-empty) vector the
* vector-side mirror of `totalNounCountAll` and the coverage denominator a
* vector index's node-count ledger is measured against. A deferred-embed
* noun (`add({ deferEmbedding: true })`) counts only once its vector
* LANDS (the `system:embed-landing` commit) its canonical record exists
* (already counted in `totalNounCountAll`) with an empty vector until
* then. Maintained on the write path (a fresh insert whose vector is
* non-empty +1, a deferred embed's landing +1, a PROVEN delete of a
* vectored noun 1), persisted beside the other ALL scalars, recomputed by
* the sanctioned recount. Shares `allCountsSuspect` no separate flag.
*/
protected totalVectoredNounCount = 0
/**
* `true` when a delete could not prove whether the record existed (no
* canonical read, no caller-provided prior) the ALL scalar may be off by
* the unprovable deletes since. Loud, persisted, and cleared only by the
* sanctioned recount; a consumer reading the scalar as a ledger denominator
* must treat a suspect scalar as unverified, never as exact.
* must treat a suspect scalar as unverified, never as exact. Also covers
* `totalVectoredNounCount` a delete whose vector-presence fact was
* unknowable marks this SAME flag rather than minting a second one.
*/
protected allCountsSuspect = false
/** One narration per session for the suspect transition (never per delete). */
private allCountsSuspectNarrated = false
/**
* Which rule produced the ALL scalars currently in memory. `'identity-record'`
* means one counted entity per metadata content leg the honest rule: a
* bare id-directory (a ghost or scar left by a partial-delete defect, no
* content leg) counts zero. Set by the one-time derivation and by the
* sanctioned recount, alongside `allCountsSuspect = false`; left `undefined`
* when a loaded counts.json carries the ALL scalars but no stamp the
* legacy container-rule derivation, which forces `allCountsSuspect = true`
* at load instead. A filesystem concern: `MemoryStorage` has no counts.json
* and never sets this.
*/
protected allCountsDerivedBy?: 'identity-record'
protected entityCounts: Map<string, number> = new Map() // type -> count
protected verbCounts: Map<string, number> = new Map() // verb type -> count
protected countCache: Map<string, { count: number; timestamp: number }> = new Map()
@ -1083,15 +1110,19 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* The canonical count ledger O(1), no I/O. `counted` is the user-facing
* scalar (public/internal tiers, what `getNounCount()` returns); `all` is
* the ALL-visibility scalar every unfiltered storage walk is measured
* against (the coverage-ledger denominator for derived-index providers).
* `suspect` is `true` when an unprovable delete has made `all` unverified
* since the last sanctioned recount (`rebuildTypeCounts`).
* @returns Both scalars per family plus the suspect flag.
* against (the coverage-ledger denominator for derived-index providers);
* `vectors.all` is the vectored-noun scalar the coverage denominator for
* a vector index's node-count ledger specifically.
* `suspect` is `true` when an unprovable delete has made `all` (any
* family, including `vectors`) unverified since the last sanctioned
* recount (`rebuildTypeCounts`).
* @returns All scalars per family plus the suspect flag.
*/
async getCanonicalCounts(): Promise<CanonicalCounts> {
return {
nouns: { counted: this.totalNounCount, all: this.totalNounCountAll },
verbs: { counted: this.totalVerbCount, all: this.totalVerbCountAll },
vectors: { all: this.totalVectoredNounCount },
suspect: this.allCountsSuspect
}
}
@ -1103,7 +1134,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* @param family - Which family's delete was unprovable.
* @param id - The id whose existence could not be established.
*/
protected markAllCountsSuspect(family: 'noun' | 'verb', id: string): void {
protected markAllCountsSuspect(family: 'noun' | 'verb' | 'noun-vector', id: string): void {
this.allCountsSuspect = true
if (!this.allCountsSuspectNarrated) {
this.allCountsSuspectNarrated = true
@ -1116,6 +1147,41 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
}
}
/**
* OPTIONAL narrow ledger hook (see {@link StorageAdapter.noteVectorLanded}):
* record a deferred-embed noun's FIRST real vector landing. The caller
* (the deferred-embed worker) proves this is a genuine landing not a
* re-embed of an already-vectored row by observing its own pre-embed
* read's vector was empty, at no added storage cost.
* @param id - The noun whose vector just landed (retained for a future
* narration seam; the count itself needs no id-keyed state).
*/
async noteVectorLanded(id: string): Promise<void> {
void id
this.totalVectoredNounCount++
this.scheduleCountPersist().catch(() => {
// Ignore persist errors — the in-memory count is authoritative; a later op retries.
})
}
/**
* OPTIONAL narrow ledger hook (see {@link StorageAdapter.noteVectorUnlanded}):
* the mirror of {@link noteVectorLanded} record a noun's vector was just
* REMOVED (rewritten to the unvectored `[]` shape). Never below zero: a
* caller that (incorrectly) fires this for a noun already unvectored would
* otherwise drive the ledger negative clamped defensively, matching the
* delete path's `if (this.totalVectoredNounCount > 0)` guard.
* @param id - The noun whose vector was just removed (retained for a
* future narration seam; the count itself needs no id-keyed state).
*/
async noteVectorUnlanded(id: string): Promise<void> {
void id
if (this.totalVectoredNounCount > 0) this.totalVectoredNounCount--
this.scheduleCountPersist().catch(() => {
// Ignore persist errors — the in-memory count is authoritative; a later op retries.
})
}
/**
* Increment count for entity type - O(1) operation.
* Concurrency is handled by the process-global mutex

View file

@ -14,10 +14,13 @@ import {
StorageBatchConfig,
SYSTEM_DIR,
STATISTICS_KEY,
WriterLockInfo
WriterLockInfo,
WriterCloseRecord
} from '../baseStorage.js'
import { getBrainyVersion } from '../../utils/index.js'
import { isAbsentError } from '../../utils/errorClassification.js'
import { prodLog } from '../../utils/logger.js'
import { isZeroNormVector } from '../../utils/distance.js'
import {
TornRecordError,
isUnparseablePayloadError,
@ -97,7 +100,30 @@ export class FileSystemStorage extends BaseStorage {
// timer rewrites the lock every 10s so stale-lock detection can tell a dead
// writer from a slow one. The constant name matches the file path used.
private static readonly WRITER_LOCK_FILE = '_writer.lock'
private static readonly WRITER_HEARTBEAT_MS = 10_000
/**
* The clean-close record at `locks/_writer.close` (see
* {@link WriterCloseRecord}). Written when the lock is released, consumed by
* the next claim, so an open can distinguish "the previous writer left" from
* "the previous writer died" without inferring either from a pid.
*/
private static readonly WRITER_CLOSE_FILE = '_writer.close'
/**
* How often the lock file's `lastHeartbeat` is rewritten.
*
* THIS IS OBSERVABILITY ONLY, and the cadence follows from that. Staleness
* is decided by PID LIVENESS alone (see isWriterLockStale) and the fence
* compares pid + hostname no decision anywhere reads this timestamp. It
* exists so an operator inspecting a lock file, or reading the
* BRAINY_WRITER_LOCKED error, can judge liveness themselves.
*
* At 10s it was a lock-file WRITE every ten seconds per brain, forever: 2.1
* writes/s across a production process holding 21 idle brains, for a
* human-readable timestamp nothing computes with. At 60s an operator still
* sees a heartbeat inside the minute, at a sixth of the cost. With the
* clean-close record now recording orderly releases explicitly, the
* heartbeat carries even less weight than it did.
*/
private static readonly WRITER_HEARTBEAT_MS = 60_000
private static readonly WRITER_STALE_THRESHOLD_MS = 60_000
private writerLockHeartbeat?: NodeJS.Timeout
private writerLockInfo?: WriterLockInfo
@ -110,6 +136,13 @@ export class FileSystemStorage extends BaseStorage {
*/
private writerHeartbeatInFlight?: Promise<void>
/**
* The in-flight background count-ledger derivation, if one was needed at
* open. See {@link scheduleCountLedgerDerivation} awaited only by
* {@link whenCountLedgerSettled}, never by a read.
*/
private countLedgerDerivation?: Promise<void>
// Flush-request RPC state. The writer polls `locks/_flush_requests/` for
// new `.req` files and emits `.ack` files in `locks/_flush_responses/` after
// flushing. Inspectors call `requestFlushOverFilesystem` to drop a request
@ -118,9 +151,16 @@ export class FileSystemStorage extends BaseStorage {
private static readonly FLUSH_REQUEST_DIR = '_flush_requests'
private static readonly FLUSH_RESPONSE_DIR = '_flush_responses'
private static readonly FLUSH_WATCH_INTERVAL_MS = 500
/**
* The safety sweep behind the fs.watch: catches events an exotic filesystem
* dropped, and runs the stale-request GC. See startFlushRequestWatcher.
*/
private static readonly FLUSH_SAFETY_SWEEP_MS = 30_000
private static readonly FLUSH_POLL_INTERVAL_MS = 100
private static readonly FLUSH_REQUEST_TTL_MS = 60_000
private flushWatcherInterval?: NodeJS.Timeout
/** The inotify-backed watch on the request directory, when the FS supports one. */
private flushWatcher?: import('node:fs').FSWatcher
private flushWatcherInFlight = false
private flushWatcherOnRequest?: () => Promise<void>
@ -239,38 +279,54 @@ export class FileSystemStorage extends BaseStorage {
// Finish any restore interrupted by a crash (resume the staged swap, or
// discard an uncommitted staging area) BEFORE counts/derived state load,
// so the rest of startup sees the completed store.
// so the rest of startup sees the completed store. ORDER-DEPENDENT:
// `swapStagedRestoreIn()` reads `fs.readdir(rootDir)` and then
// removes/renames rootDir's own TOP-LEVEL entries to place the staged
// copy — racing that against the directory-creation batch below (which
// also touches rootDir's children) could see a half-created directory
// mid-swap or a mkdir racing a concurrent rm/rename on the same path.
// Stays strictly sequential, never folded into the OPEN-PATH batch.
await this.completeInterruptedRestore()
// Create the nouns directory if it doesn't exist
await this.ensureDirectoryExists(this.nounsDir)
// OPEN-PATH FIX: the remaining bootstrap directories are mutually
// independent — each is its own subtree under rootDir, and
// `fs.mkdir(dir, { recursive: true })` creates every intermediate
// segment of ITS OWN path in one call, so it never depends on any
// sibling here existing first. Nothing between here and
// `initializeCounts()` reads any of them, so batching collapses what
// was up to 8 sequential mkdir round-trips (each a real syscall+await)
// into one wave — this is what serialized an N-writer restart storm on
// filesystem I/O it never structurally needed. `initializeCounts()`
// right after DOES depend on `systemDir` (which the batch creates), so
// it stays outside, awaited only once every directory has landed.
await Promise.all([
// Create the nouns directory if it doesn't exist
this.ensureDirectoryExists(this.nounsDir),
// Create the verbs directory if it doesn't exist
this.ensureDirectoryExists(this.verbsDir),
// Create the metadata directory if it doesn't exist
this.ensureDirectoryExists(this.metadataDir),
// Create the noun metadata directory if it doesn't exist
this.ensureDirectoryExists(this.nounMetadataDir),
// Create the verb metadata directory if it doesn't exist
this.ensureDirectoryExists(this.verbMetadataDir),
// Create both directories for backward compatibility
this.ensureDirectoryExists(this.systemDir),
// Only create legacy directory if it exists (don't create new legacy
// dirs) — a read-then-maybe-write, but on its own subtree, so it's
// still independent of every other entry in this batch.
(async () => {
if (await this.directoryExists(this.indexDir)) {
await this.ensureDirectoryExists(this.indexDir)
}
})(),
// Create the locks directory if it doesn't exist
this.ensureDirectoryExists(this.lockDir),
// Create the binary blobs directory if it doesn't exist
this.ensureDirectoryExists(this.blobsDir)
])
// Create the verbs directory if it doesn't exist
await this.ensureDirectoryExists(this.verbsDir)
// Create the metadata directory if it doesn't exist
await this.ensureDirectoryExists(this.metadataDir)
// Create the noun metadata directory if it doesn't exist
await this.ensureDirectoryExists(this.nounMetadataDir)
// Create the verb metadata directory if it doesn't exist
await this.ensureDirectoryExists(this.verbMetadataDir)
// Create both directories for backward compatibility
await this.ensureDirectoryExists(this.systemDir)
// Only create legacy directory if it exists (don't create new legacy dirs)
if (await this.directoryExists(this.indexDir)) {
await this.ensureDirectoryExists(this.indexDir)
}
// Create the locks directory if it doesn't exist
await this.ensureDirectoryExists(this.lockDir)
// Create the binary blobs directory if it doesn't exist
await this.ensureDirectoryExists(this.blobsDir)
// Initialize count management
// Initialize count management — depends on systemDir, created above.
this.countsFilePath = path.join(this.systemDir, 'counts.json')
await this.initializeCounts()
@ -586,6 +642,20 @@ export class FileSystemStorage extends BaseStorage {
* automatically. Returns the pruned container ids so the caller can recompute
* counts.
*/
/**
* @description Whether an id directory's file legs include the metadata
* CONTENT leg (`metadata.json` or its `.json.gz` variant) the single
* test that decides whether an `entities/<kind>/<shard>/<id>/` container is
* a live entity or a ghost/scar orphan left by the pre-8.3.1 partial-delete
* defect (see {@link pruneOrphanedEntities}). Shared by the orphan prune
* and {@link scanCanonicalEntities} so the two agree by construction one
* counted entity per identity record, never per bare container.
* @param legs - File names in one `entities/<kind>/<shard>/<id>/` directory.
*/
private hasMetadataContentLeg(legs: string[]): boolean {
return legs.some((f) => f.startsWith('metadata.json'))
}
public async pruneOrphanedEntities(): Promise<{ nouns: string[]; verbs: string[] }> {
await this.ensureInitialized()
const pruned: { nouns: string[]; verbs: string[] } = { nouns: [], verbs: [] }
@ -625,7 +695,7 @@ export class FileSystemStorage extends BaseStorage {
}
// A live entity has its metadata content leg. No content leg → a
// vector-only ghost or an empty scar → prune the whole container.
if (legs.some((f) => f.startsWith('metadata.json'))) continue
if (this.hasMetadataContentLeg(legs)) continue
await fs.promises.rm(idAbs, { recursive: true, force: true })
pruned[kind].push(entry.name)
console.warn(
@ -639,6 +709,30 @@ export class FileSystemStorage extends BaseStorage {
return pruned
}
/**
* @description The IMMEDIATE child directory names under a prefix ONE
* `readdir`, no recursion, no file paths. See the seam's JSDoc
* (`src/db/types.ts`) for what this replaced: discovering the generations on
* disk walked the entire generation log on every open, reading out every
* file in every generation, to learn the set of integers the top-level
* directory names already spell.
* @param prefix - Storage-root-relative directory prefix.
* @returns The child directory names (not paths); empty when the prefix does
* not exist.
*/
public override async listRawPrefixes(prefix: string): Promise<string[]> {
await this.ensureInitialized()
const fullPath = path.join(this.rootDir, prefix)
try {
const entries = await fs.promises.readdir(fullPath, { withFileTypes: true })
return entries.filter((e: { isDirectory: () => boolean }) => e.isDirectory())
.map((e: { name: string }) => e.name)
} catch (error: any) {
if (error?.code === 'ENOENT') return []
throw error
}
}
/**
* Primitive operation: List objects under path prefix
* All metadata operations use this internally via base class routing
@ -1849,18 +1943,41 @@ export class FileSystemStorage extends BaseStorage {
}
}
// THE CLEAN-CLOSE RECORD IS READ BEFORE ANY VERDICT (see
// WriterCloseRecord). A lock file whose release was RECORDED is
// bookkeeping left by an orderly shutdown, not evidence of anything —
// and that is true whether the previous holder was another process or
// an earlier instance in THIS one. A production restart reported
// "Re-acquiring writer lock ... this is a bug" immediately after a clean
// close, sending an operator hunting for a leak that did not exist.
const closeRecord = existing ? await this.readWriterCloseRecord() : null
const releasedCleanly =
existing !== null &&
closeRecord !== null &&
this.closeRecordVouchesFor(closeRecord, existing)
if (existing) {
// Same-process re-open: a second Brainy instance in this Node process
// (e.g. test "simulate server restart" patterns, or a consumer that
// explicitly re-instantiates without closing first). This isn't the
// dangerous cross-process case the lock exists to prevent — the two
// instances share a memory space and can't silently diverge from each
// other beyond what their callers already see. Warn and take over.
// other beyond what their callers already see. Warn and take over —
// unless the record proves the previous instance already let go, in
// which case there is nothing to warn about.
if (existing.pid === myPid && existing.hostname === hostname && !options?.force) {
console.warn(
`[brainy] Re-acquiring writer lock for ${this.rootDir} held by the same process (PID ${existing.pid}). ` +
`If you intended to keep the previous Brainy instance alive, this is a bug — close it first.`
)
if (releasedCleanly) {
console.warn(
`[brainy] Clearing the leftover writer lock for ${this.rootDir} — an earlier ` +
`instance in this process (PID ${existing.pid}) RELEASED it cleanly at ` +
`${closeRecord!.closedAt} but could not remove the file. Nothing to recover.`
)
} else {
console.warn(
`[brainy] Re-acquiring writer lock for ${this.rootDir} held by the same process (PID ${existing.pid}). ` +
`If you intended to keep the previous Brainy instance alive, this is a bug — close it first.`
)
}
const info: WriterLockInfo = {
pid: myPid,
hostname,
@ -1870,11 +1987,18 @@ export class FileSystemStorage extends BaseStorage {
rootDir: this.rootDir
}
await this.writeFileAtomic(lockFile, JSON.stringify(info, null, 2))
await this.clearWriterCloseRecord()
this.installWriterLock(info)
return info
}
const stale = !options?.force && (await this.isWriterLockStale(existing))
// A cleanly-released lock is stale by RECORD, not by inference. Only
// when no record vouches for this lock do we fall back to pid
// liveness, and then we say THAT honestly too: an unrecorded lock
// means the writer did not complete its close, so the store was not
// closed cleanly and this open pays recovery.
const stale =
releasedCleanly || (!options?.force && (await this.isWriterLockStale(existing)))
if (!options?.force && !stale) {
// Consumer-facing error contract: callers detect this case via
// err.code and read the holder's details from err.lockInfo.
@ -1885,8 +2009,16 @@ export class FileSystemStorage extends BaseStorage {
options?.force
? `[brainy] Force-overwriting writer lock for ${this.rootDir} ` +
`(was held by PID ${existing.pid} on ${existing.hostname}).`
: `[brainy] Overwriting stale writer lock for ${this.rootDir} ` +
`(PID ${existing.pid} on ${existing.hostname} appears dead).`
: releasedCleanly
? `[brainy] Clearing the leftover writer lock for ${this.rootDir}` +
`PID ${existing.pid} on ${existing.hostname} RELEASED it cleanly at ` +
`${closeRecord!.closedAt} but could not remove the file. ` +
`Nothing to recover.`
: `[brainy] Overwriting stale writer lock for ${this.rootDir} ` +
`(PID ${existing.pid} on ${existing.hostname} is gone and left NO ` +
`clean-close record — that writer did not finish closing, so this ` +
`store was not closed cleanly; open will run crash recovery and ` +
`report its wall).`
)
// Takeover: verify the file still holds the lock we judged (a live
// successor may have claimed meanwhile), then remove it and fall
@ -1940,6 +2072,12 @@ export class FileSystemStorage extends BaseStorage {
await fs.promises.unlink(claimTmp).catch(() => {})
}
// CONSUME the previous writer's clean-close record. It described the
// lock generation that just ended; leaving it in place would let it
// vouch for OUR lock if this process later dies without closing —
// turning a real crash into a "closed cleanly" verdict. One unlink.
await this.clearWriterCloseRecord()
this.installWriterLock(info)
return info
}
@ -2063,13 +2201,27 @@ export class FileSystemStorage extends BaseStorage {
return
}
const lockFile = path.join(this.lockDir, FileSystemStorage.WRITER_LOCK_FILE)
const released = this.writerLockInfo
try {
// Only delete if we still own it — avoid clobbering a successor that
// claimed the lock via force-override.
const current = await this.readWriterLock()
if (current && current.pid === this.writerLockInfo.pid && current.hostname === this.writerLockInfo.hostname) {
const ours =
current === null ||
(current.pid === released.pid && current.hostname === released.hostname)
if (current && ours) {
await fs.promises.unlink(lockFile)
}
// THE CLEAN-CLOSE RECORD (see WriterCloseRecord). Written whenever this
// instance gives up a lock nobody else has taken — the unlink above
// having succeeded OR the file already being gone. The next open reads
// it instead of guessing from pid liveness: a recorded release is an
// orderly shutdown, an absent record is a writer that never finished
// closing. Not written when a successor holds the lock: our release is
// then a no-op and a record would slander their live lock.
if (ours) {
await this.writeWriterCloseRecord(released)
}
} catch (err: any) {
if (err.code !== 'ENOENT') {
console.warn('[brainy] Failed to release writer lock file:', err)
@ -2079,6 +2231,97 @@ export class FileSystemStorage extends BaseStorage {
}
}
/**
* @description Read the clean-close record at `locks/_writer.close`, or
* `null` when it is absent or unparseable. A torn record is treated as
* absent the conservative direction, since an unreadable record can
* vouch for nothing.
* @returns The record, or null.
*/
public async readWriterCloseRecord(): Promise<WriterCloseRecord | null> {
await this.ensureInitialized()
const recordFile = path.join(this.lockDir, FileSystemStorage.WRITER_CLOSE_FILE)
try {
const raw = await fs.promises.readFile(recordFile, 'utf-8')
const parsed = JSON.parse(raw) as WriterCloseRecord
if (
typeof parsed?.pid !== 'number' ||
typeof parsed?.hostname !== 'string' ||
typeof parsed?.startedAt !== 'string' ||
typeof parsed?.closedAt !== 'string'
) {
return null
}
return parsed
} catch (err: any) {
if (err.code === 'ENOENT') return null
return null
}
}
/**
* @description Whether a clean-close record describes the very lock
* generation `lock` represents. The match is pid + hostname + `startedAt`:
* `startedAt` is the lock generation's identity, so a record can never
* vouch for a LATER lock taken by the same pid on the same host (the
* same-process re-open path mints a fresh `startedAt`).
* @param record - The clean-close record read from disk.
* @param lock - The lock file's contents.
*/
private closeRecordVouchesFor(record: WriterCloseRecord, lock: WriterLockInfo): boolean {
return (
record.pid === lock.pid &&
record.hostname === lock.hostname &&
record.startedAt === lock.startedAt
)
}
/**
* @description Write the clean-close record for a lock this instance just
* released. Atomic (temp + rename) so a concurrent opener never reads half
* a record. A failure here costs the next open nothing but the honest
* fallback (pid liveness), so it warns rather than failing the close.
* @param released - The lock info this instance held.
*/
private async writeWriterCloseRecord(released: WriterLockInfo): Promise<void> {
const record: WriterCloseRecord = {
pid: released.pid,
hostname: released.hostname,
startedAt: released.startedAt,
closedAt: new Date().toISOString(),
version: released.version
}
const recordFile = path.join(this.lockDir, FileSystemStorage.WRITER_CLOSE_FILE)
try {
await this.writeFileAtomic(recordFile, JSON.stringify(record, null, 2))
} catch (err) {
// ENOENT = the lock directory is gone, i.e. the whole store was removed
// under us. There is no next open to inform.
if ((err as NodeJS.ErrnoException)?.code === 'ENOENT') return
console.warn(
`[brainy] Failed to write the writer clean-close record for ${this.rootDir}` +
`the next open will fall back to pid liveness and may report this orderly ` +
`shutdown as a crash:`,
err
)
}
}
/**
* @description Remove the clean-close record. Called by every successful
* lock claim so a record never outlives the lock generation it describes.
*/
private async clearWriterCloseRecord(): Promise<void> {
const recordFile = path.join(this.lockDir, FileSystemStorage.WRITER_CLOSE_FILE)
try {
await fs.promises.unlink(recordFile)
} catch (err: any) {
if (err.code !== 'ENOENT') {
console.warn('[brainy] Failed to clear the writer clean-close record:', err)
}
}
}
public override async readWriterLock(): Promise<WriterLockInfo | null> {
await this.ensureInitialized()
const lockFile = path.join(this.lockDir, FileSystemStorage.WRITER_LOCK_FILE)
@ -2165,36 +2408,115 @@ export class FileSystemStorage extends BaseStorage {
/**
* Start watching for cross-process flush requests. Called by Brainy.init()
* in writer mode. Polls `locks/_flush_requests/` every
* FLUSH_WATCH_INTERVAL_MS each new `.req` file triggers the supplied
* callback (`brain.flush()`), after which an `.ack` is written to
* `locks/_flush_responses/` with the same request ID. Stale `.req` files
* (>FLUSH_REQUEST_TTL_MS) are garbage-collected on every tick.
* in writer mode. Each new `.req` file in `locks/_flush_requests/` triggers
* the supplied callback (`brain.flush()`), after which an `.ack` is written
* to `locks/_flush_responses/` with the same request ID. Stale `.req` files
* (>FLUSH_REQUEST_TTL_MS) are garbage-collected on each sweep.
*
* THE WATCH IS EVENT-DRIVEN, NOT A POLL. It used to `readdir` the request
* directory every 500 ms, per brain, for the entire life of every writer
* armed on every non-reader brain whether or not any inspector process
* existed. MEASURED on a production process holding 21 brains: 42 directory
* reads per second on a completely idle service, plus a stale-request GC
* pass on every one of them. The engine does no periodic work without a
* cause, and a request that has not been made is not a cause.
*
* `fs.watch` (inotify on Linux) delivers the arrival itself, so a request is
* seen SOONER than the old poll saw it. Two honest concessions ride with it:
* - a slow SAFETY SWEEP (FLUSH_SAFETY_SWEEP_MS) still runs, because
* `fs.watch` can miss events on network and fuse filesystems and because
* the stale-request GC needs some tick of its own. At 30s that is 0.7
* reads/s across 21 brains where the poll cost 42.
* - a filesystem that cannot watch at all falls back to the ORIGINAL
* 500 ms poll, narrated once, because correctness outranks idle cost:
* an inspector whose request is never seen waits forever.
*/
public override startFlushRequestWatcher(onRequest: () => Promise<void>): void {
if (this.flushWatcherInterval) return // already watching
// Already watching — or already ARMING. The arm is asynchronous (the
// request directory is created before it can be watched), so neither the
// watcher nor the interval exists yet during that window; the callback is
// the flag that covers it. Without this a second call in the window would
// leave two watchers and two sweeps running for the life of the store.
if (this.flushWatcherInterval || this.flushWatcher || this.flushWatcherOnRequest) return
this.flushWatcherOnRequest = onRequest
const reqDir = path.join(this.lockDir, FileSystemStorage.FLUSH_REQUEST_DIR)
const ackDir = path.join(this.lockDir, FileSystemStorage.FLUSH_RESPONSE_DIR)
// Ensure both dirs exist up front so the first .req drop doesn't race with mkdir.
this.ensureDirectoryExists(reqDir).catch(() => {})
this.ensureDirectoryExists(ackDir).catch(() => {})
this.flushWatcherInterval = setInterval(() => {
if (this.flushWatcherInFlight) return // skip overlapping tick
const sweep = (): void => {
if (this.flushWatcherInFlight) return // skip overlapping sweep
this.flushWatcherInFlight = true
this.processFlushRequests(reqDir, ackDir).finally(() => {
this.flushWatcherInFlight = false
})
}, FileSystemStorage.FLUSH_WATCH_INTERVAL_MS)
}
// Ensure both dirs exist up front so the first .req drop doesn't race with
// mkdir — and so there is a directory to watch.
void this.ensureDirectoryExists(reqDir)
.then(() => this.ensureDirectoryExists(ackDir))
.then(() => {
if (this.flushWatcherOnRequest !== onRequest) return // stopped meanwhile
try {
const watcher = fs.watch(reqDir, () => sweep())
this.flushWatcher = watcher
watcher.on('error', (err: Error) => {
// A watch that dies mid-life must not leave the door deaf.
console.warn(
`[brainy] Flush-request watch failed (${err.message}) — falling back to polling.`
)
this.flushWatcher?.close()
this.flushWatcher = undefined
// The SAFETY sweep must go first. It is already armed at 30s, and
// startFlushRequestPolling() declines to arm over an existing
// interval — so leaving it would quietly leave this store answering
// flush requests on a 30s cadence instead of the 500ms one the door
// promises. A degrade nobody asked for is still a degrade.
if (this.flushWatcherInterval) {
clearInterval(this.flushWatcherInterval)
this.flushWatcherInterval = undefined
}
this.startFlushRequestPolling(sweep)
})
if (typeof watcher.unref === 'function') watcher.unref()
// The safety sweep: missed events on exotic filesystems, and the
// stale-request GC.
this.flushWatcherInterval = setInterval(sweep, FileSystemStorage.FLUSH_SAFETY_SWEEP_MS)
if (typeof this.flushWatcherInterval.unref === 'function') {
this.flushWatcherInterval.unref()
}
// One sweep now: a request may have been dropped before the watch armed.
sweep()
} catch (err) {
console.warn(
`[brainy] Flush-request directory cannot be watched on this filesystem ` +
`(${(err as Error).message}) — polling every ` +
`${FileSystemStorage.FLUSH_WATCH_INTERVAL_MS}ms instead.`
)
this.startFlushRequestPolling(sweep)
}
})
.catch(() => {
// The request directory could not be created; nothing to watch. A
// cross-process flush request cannot be made either, so there is
// nothing to miss.
})
}
/** The original 500 ms poll — the fallback when a directory cannot be watched. */
private startFlushRequestPolling(sweep: () => void): void {
if (this.flushWatcherInterval) return
this.flushWatcherInterval = setInterval(sweep, FileSystemStorage.FLUSH_WATCH_INTERVAL_MS)
if (typeof this.flushWatcherInterval.unref === 'function') {
this.flushWatcherInterval.unref()
}
}
public override stopFlushRequestWatcher(): void {
if (this.flushWatcher) {
this.flushWatcher.close()
this.flushWatcher = undefined
}
if (this.flushWatcherInterval) {
clearInterval(this.flushWatcherInterval)
this.flushWatcherInterval = undefined
@ -2567,24 +2889,73 @@ export class FileSystemStorage extends BaseStorage {
// record reads), persist, and never scan again. Absent keys are a
// legacy file, not a zero — a zero here would make every provider's
// coverage ledger read "over-posted" on a populated store.
let needsPersist = false
if (
typeof counts.totalNounCountAll === 'number' &&
typeof counts.totalVerbCountAll === 'number'
) {
this.totalNounCountAll = counts.totalNounCountAll
this.totalVerbCountAll = counts.totalVerbCountAll
this.allCountsSuspect = counts.allCountsSuspect === true
if (counts.allCountsDerivedBy === 'identity-record') {
// Derived (or recounted) under the honest rule — one counted
// entity per metadata content leg. Trust the persisted suspect
// flag as-is; an unprovable delete since may still have set it.
this.allCountsDerivedBy = 'identity-record'
this.allCountsSuspect = counts.allCountsSuspect === true
} else {
// The ALL scalars exist but predate the identity-record stamp —
// they were derived under the legacy rule that counted one
// entity per id DIRECTORY, so orphaned ghost/scar containers (a
// pre-8.3.1 partial-delete defect — see pruneOrphanedEntities())
// were counted as entities too. O(1) field read, NEVER a walk
// here: force suspect and name it loudly. A sanctioned recount
// (repairIndex) restores exact denominators and clears this.
this.allCountsDerivedBy = undefined
this.allCountsSuspect = true
needsPersist = true
prodLog.narrate(
'[FileSystemStorage] canonical count ledger was derived under the legacy ' +
'container rule — it counts one entity per id DIRECTORY, so every ghost/scar ' +
'container inflates it. Marked suspect, and an honest recount is scheduled to ' +
'run in the background after this open; until it lands, do not subtract ' +
'against these ALL scalars.'
)
// A suspect ledger used to stay wrong for the life of the store,
// waiting for an operator to run repairIndex. A downstream index
// heal took its "remaining" figure from these inflated
// denominators and reported work that did not exist. The ledger
// now HEALS ITSELF — in the background, because a denominator is
// a derived scalar and no read is ever served from it.
this.scheduleCountLedgerDerivation('legacy container-rule ledger')
}
} else {
const nouns = await this.scanCanonicalEntities('nouns')
const verbs = await this.scanCanonicalEntities('verbs')
this.totalNounCountAll = nouns.count
this.totalVerbCountAll = verbs.count
this.allCountsSuspect = false
console.warn(
`[FileSystemStorage] counts.json predates the ALL-visibility count ledger — ` +
`derived once from the canonical id tree (${nouns.count} nouns, ${verbs.count} verbs, ` +
`every tier) and persisted; no further scan.`
)
// No ALL scalars at all. There is nothing to serve in the meantime —
// a zero would read as an empty store — so the scalars stay unknown
// and SUSPECT until the background derivation lands. The open does
// not wait for it: an id-tree walk is O(ids) and this file has been
// the whole reason a 24k-id store opened in silence.
this.allCountsSuspect = true
this.scheduleCountLedgerDerivation('counts.json predates the ALL-visibility ledger')
}
// The vectored-noun scalar (shipped after the ALL scalars above — a
// counts.json can carry `totalNounCountAll`/`totalVerbCountAll` but
// still predate THIS key). Unlike the ALL scalars, presence cannot be
// decided from the id-directory listing alone: a deferred-embed
// noun's `vectors.json` EXISTS with an empty `vector: []` until its
// embed lands, so this derivation reads every noun's `vectors.json`
// ONCE (O(nouns) reads, not O(ids) listing) — honest, one-time cost.
if (typeof counts.totalVectoredNounCount === 'number') {
this.totalVectoredNounCount = counts.totalVectoredNounCount
} else {
// O(nouns) CONTENT reads — the most expensive derivation of the
// three, and the one most likely to have been the silent minutes at
// the front of a large store's open. Background, suspect until it
// lands, same as the ALL scalars.
this.allCountsSuspect = true
this.scheduleCountLedgerDerivation('counts.json predates the vectored-noun ledger')
}
if (needsPersist) {
await this.persistCounts()
}
@ -2611,6 +2982,22 @@ export class FileSystemStorage extends BaseStorage {
* Initialize counts by scanning disk (only done once)
*/
private async initializeCountsFromDisk(): Promise<void> {
const startedAt = Date.now()
// THIS ONE CANNOT LEAVE THE FOREGROUND, and the reason is worth stating:
// it derives `totalNounCount` / `totalVerbCount`, the scalars
// `getNounCount()` and `getVerbCount()` RETURN. Backgrounding it would
// make a populated store answer "0 entities" until the walk landed — a
// wrong answer, not a slow one, and the serving law grades a failure by
// whether an answer could be wrong. The ALL-visibility denominators, which
// no read is served from, DO run in the background (see
// scheduleCountLedgerDerivation). What this walk owes the operator instead
// is narration: it announces itself, and reports its wall.
prodLog.narrate(
`[FileSystemStorage] no usable counts.json — deriving the entity counters from ` +
`the canonical id tree now. This is O(ids) listings plus one vectors.json read ` +
`per noun, and it BLOCKS the open because getNounCount()/getVerbCount() are ` +
`served from it. It runs once; the result is persisted.`
)
try {
// Count the CANONICAL 8.0 layout (`entities/<kind>/<shard>/<id>/…`) —
// the tree saveNoun/getNouns actually read and write. The previous scan
@ -2627,6 +3014,13 @@ export class FileSystemStorage extends BaseStorage {
this.totalNounCountAll = nouns.count
this.totalVerbCountAll = verbs.count
this.allCountsSuspect = false
this.allCountsDerivedBy = 'identity-record'
// Vectored-noun scalar: presence needs each noun's vectors.json CONTENT
// (a deferred-embed noun's file exists but holds an empty vector until
// its embed lands), so this is a full O(nouns) content scan — see
// scanVectoredNounCount()'s JSDoc for the cost note. Paid once, here,
// alongside the rest of this from-disk recovery.
this.totalVectoredNounCount = await this.scanVectoredNounCount()
// Sample some entities for the type distribution (don't read all).
// Read the metadata files DIRECTLY with fs — this runs inside init(),
@ -2651,6 +3045,11 @@ export class FileSystemStorage extends BaseStorage {
}
await this.persistCounts()
prodLog.narrate(
`[FileSystemStorage] counter derivation from the canonical id tree finished in ` +
`${Date.now() - startedAt}ms: ${this.totalNounCount} nouns, ${this.totalVerbCount} verbs, ` +
`${this.totalVectoredNounCount} vectored nouns — persisted, stamped identity-record.`
)
} catch (error) {
console.error('Error initializing counts from disk:', error)
}
@ -2658,11 +3057,132 @@ export class FileSystemStorage extends BaseStorage {
/**
* Walk the canonical `entities/<kind>/<2-hex-shard>/<id>/` tree, counting
* one entity per id directory (the layout `getNounVectorPath`/`getNouns`
* use). Returns up to 100 sampled entity directories (absolute paths)
* nouns feed the type-distribution estimate above. An absent tree (fresh
* store) counts zero.
* one entity per id directory that holds the metadata CONTENT leg
* (`metadata.json` or its `.json.gz` variant see
* {@link hasMetadataContentLeg}). A bare container a ghost (a stale
* `vectors.json` left with no metadata leg) or a scar (an empty directory),
* both artifacts of the pre-8.3.1 partial-delete defect counts ZERO: the
* identity record IS the population (ADR-008 G1), never the directory.
* This is the ONE-TIME legacy derivation walk (see callers); a prior
* version of this scan counted every id directory regardless of content,
* over-counting any store carrying orphaned containers see
* `allCountsDerivedBy` for how a counts.json derived under that old rule is
* marked suspect on load. Returns up to 100 sampled *counted* entity
* directories (absolute paths) nouns feed the type-distribution estimate
* above. An absent tree (fresh store) counts zero.
*/
/**
* @description Derive the ALL-visibility count ledger honestly one entity
* per IDENTITY RECORD, never per id directory IN THE BACKGROUND, once,
* and persist the result stamped `identity-record`.
*
* Why background: these scalars are DENOMINATORS. No read is served from
* them, so deriving them cannot be allowed to hold an open hostage a
* store with 24,898 ids spent minutes of a production restart inside walks
* exactly like these, in silence, before serving anything. Why at all: a
* ledger derived under the old container rule stayed wrong for the life of
* the store, and a downstream index heal subtracted against it and reported
* remaining work that did not exist (measured on a real store: 14,231
* derived against 14,056 identity records precisely the store's 25 noun
* scar directories; verbs 72,729 against 72,679, its 50 verb scars).
*
* Idempotent: a second call while one is in flight joins the first.
* @param reason - What made the ledger untrustworthy, quoted in narration.
* @returns Nothing; observe completion with {@link whenCountLedgerSettled}.
*/
private scheduleCountLedgerDerivation(reason: string): void {
if (this.countLedgerDerivation) return
this.countLedgerDerivation = (async () => {
const startedAt = Date.now()
prodLog.narrate(
`[FileSystemStorage] count-ledger derivation started in the background ` +
`(${reason}) — counting identity records, not id directories; the open does ` +
`not wait for it and no read is served from these scalars.`
)
try {
const beforeNouns = this.totalNounCountAll
const beforeVerbs = this.totalVerbCountAll
const beforeVectored = this.totalVectoredNounCount
// A walk that RACED A WRITE cannot prove its number: a row that landed
// mid-walk may or may not have been in the shard the walk had already
// passed. Rather than persist a figure that might be off by one and
// stamp it "exact", the walk is repeated once on a quiet store, and if
// the store is never quiet the ledger stays SUSPECT and says so. One
// retry, never a spin.
let attempt = 0
let derived: { nouns: number; verbs: number; vectored: number } | null = null
while (attempt < 2 && derived === null) {
attempt++
const activityBefore = this.ledgerActivityStamp()
const nouns = await this.scanCanonicalEntities('nouns')
const verbs = await this.scanCanonicalEntities('verbs')
const vectored = await this.scanVectoredNounCount()
if (this.ledgerActivityStamp() === activityBefore) {
derived = { nouns: nouns.count, verbs: verbs.count, vectored }
}
}
if (derived === null) {
this.allCountsSuspect = true
prodLog.narrate(
`[FileSystemStorage] count-ledger derivation could not finish on a quiet store ` +
`after ${attempt} attempts (${Date.now() - startedAt}ms) — writes landed during ` +
`every walk. The ALL-visibility scalars stay SUSPECT and must not be subtracted ` +
`against; brain.repairIndex() derives them under a recount barrier.`
)
return
}
this.totalNounCountAll = derived.nouns
this.totalVerbCountAll = derived.verbs
this.totalVectoredNounCount = derived.vectored
this.allCountsDerivedBy = 'identity-record'
this.allCountsSuspect = false
await this.persistCounts()
prodLog.narrate(
`[FileSystemStorage] count-ledger derivation finished in ${Date.now() - startedAt}ms: ` +
`${derived.nouns} nouns / ${derived.verbs} verbs / ${derived.vectored} vectored nouns` +
(beforeNouns !== derived.nouns ||
beforeVerbs !== derived.verbs ||
beforeVectored !== derived.vectored
? ` (corrected from ${beforeNouns} / ${beforeVerbs} / ${beforeVectored} — the ` +
`difference is ghost and scar containers the old rule counted as entities)`
: ' (unchanged)') +
` — persisted, stamped identity-record, no longer suspect.`
)
} catch (error) {
// The ledger stays suspect and the next open retries. Loud: a
// denominator nobody can derive is a fact an operator must have.
this.allCountsSuspect = true
prodLog.error(
`[FileSystemStorage] count-ledger derivation FAILED after ` +
`${Date.now() - startedAt}ms — the ALL-visibility scalars remain SUSPECT ` +
`and must not be subtracted against; the next open retries:`,
error
)
}
})()
}
/**
* @description A cheap witness that the ledger changed while a walk was
* running. Every landed write moves one of these live counters, so an
* unchanged stamp across a walk means no write landed during it.
* @returns A value that differs whenever the live ALL scalars have moved.
*/
private ledgerActivityStamp(): string {
return `${this.totalNounCountAll}:${this.totalVerbCountAll}:${this.totalVectoredNounCount}`
}
/**
* @description Resolve once any background count-ledger derivation has
* settled (succeeded or failed). Resolves immediately when none was needed.
* Exists so tests and operators can observe the ledger's honest value rather
* than race it; nothing in the read path waits on this.
* @returns A promise that settles with the derivation.
*/
public async whenCountLedgerSettled(): Promise<void> {
await this.countLedgerDerivation
}
private async scanCanonicalEntities(
kind: 'nouns' | 'verbs'
): Promise<{ count: number; sampleDirs: string[] }> {
@ -2678,9 +3198,21 @@ export class FileSystemStorage extends BaseStorage {
const ids = await fs.promises.readdir(shardPath, { withFileTypes: true })
for (const entry of ids) {
if (!entry.isDirectory()) continue
const idAbs = path.join(shardPath, entry.name)
let legs: string[]
try {
legs = await fs.promises.readdir(idAbs)
} catch (error: any) {
if (error?.code === 'ENOENT') continue
throw error
}
// No metadata content leg → a ghost or scar container → not an
// entity. Same test pruneOrphanedEntities() uses, so the two agree
// by construction.
if (!this.hasMetadataContentLeg(legs)) continue
count++
if (sampleDirs.length < SAMPLE_MAX) {
sampleDirs.push(path.join(shardPath, entry.name))
sampleDirs.push(idAbs)
}
}
}
@ -2712,6 +3244,72 @@ export class FileSystemStorage extends BaseStorage {
}
}
/**
* Read one canonical noun's `vectors.json` (or `.json.gz`) directly with fs
* the vector-side mirror of {@link readEntityMetadataRaw}, same
* reentrancy reason (bypasses `getNoun()`'s `ensureInitialized()`).
* @param entityDir - Absolute `entities/nouns/<shard>/<id>` directory.
* @returns The parsed vector record, or null when absent/unreadable.
*/
private async readEntityVectorRaw(entityDir: string): Promise<any | null> {
const base = path.join(entityDir, 'vectors.json')
try {
return JSON.parse(await fs.promises.readFile(base, 'utf-8'))
} catch {
// fall through to the compressed variant
}
try {
const gz = await fs.promises.readFile(`${base}.gz`)
return JSON.parse(zlib.gunzipSync(gz).toString('utf-8'))
} catch {
return null
}
}
/**
* Count canonical nouns holding a REAL (non-empty, non-zero-norm) vector
* the vectored-noun ledger scalar. UNLIKE {@link scanCanonicalEntities},
* presence cannot be decided from the id-directory listing alone: a
* deferred-embed noun's `vectors.json` EXISTS (written at `add()` time
* with `vector: []`) until its embed LANDS, so this walk reads every
* noun's `vectors.json` CONTENT O(nouns) reads, not O(ids) listing.
* ZERO-NORM LAW: a real all-zero vector is not a vector it never counts
* here either (Brainy's write paths normalize an explicit zero-norm
* vector to `[]` at write time, but a store created before that fix may
* still carry legacy all-zero rows on disk; this derivation must agree
* with the live ledger's definition of "vectored" regardless of when the
* row was written). Used ONLY for a one-time legacy-counts.json derivation
* or a lost/corrupted counts.json recovery; the result is persisted so
* this scan never repeats.
*/
private async scanVectoredNounCount(): Promise<number> {
const base = path.join(this.rootDir, 'entities', 'nouns')
let vectored = 0
try {
const shards = await fs.promises.readdir(base, { withFileTypes: true })
for (const shard of shards) {
if (!shard.isDirectory() || !/^[0-9a-f]{2}$/i.test(shard.name)) continue
const shardPath = path.join(base, shard.name)
const ids = await fs.promises.readdir(shardPath, { withFileTypes: true })
for (const entry of ids) {
if (!entry.isDirectory()) continue
const record = await this.readEntityVectorRaw(path.join(shardPath, entry.name))
if (
record &&
Array.isArray(record.vector) &&
record.vector.length > 0 &&
!isZeroNormVector(record.vector)
) {
vectored++
}
}
}
} catch (error: any) {
if (error?.code !== 'ENOENT') throw error
}
return vectored
}
/**
* Persist counts to filesystem storage
*/
@ -2728,14 +3326,30 @@ export class FileSystemStorage extends BaseStorage {
// written before the ledger existed; initializeCounts() derives them once.
totalNounCountAll: this.totalNounCountAll,
totalVerbCountAll: this.totalVerbCountAll,
// Vectored-noun ledger scalar — absent in files written before it
// existed; initializeCounts() derives it once (a content scan, see
// scanVectoredNounCount()'s JSDoc).
totalVectoredNounCount: this.totalVectoredNounCount,
allCountsSuspect: this.allCountsSuspect,
// Derivation-rule stamp for the ALL scalars above — 'identity-record'
// when they were counted one-per-metadata-content-leg (the honest
// rule); omitted (JSON.stringify drops `undefined`) when the current
// in-memory scalars came from a legacy container-rule counts.json
// that hasn't been through a sanctioned recount yet, so a future load
// keeps naming them suspect rather than trusting an unproven value.
allCountsDerivedBy: this.allCountsDerivedBy,
lastUpdated: new Date().toISOString()
}
await fs.promises.writeFile(
this.countsFilePath,
JSON.stringify(counts, null, 2)
)
// ATOMIC (temp + rename), never a plain writeFile. A direct write
// truncates the file first, so every persist opened a window — measured
// at roughly 750ms after a flush or close on a real store — in which a
// concurrent reader saw counts.json EMPTY. An empty file is unparseable,
// and an unparseable ledger sends the next open down the full-rescan
// path: the cheapest file in the store was costing the most expensive
// recovery. The rename is atomic, so a reader sees the old ledger or the
// new one, never neither.
await this.writeFileAtomic(this.countsFilePath, JSON.stringify(counts, null, 2))
} catch (error) {
console.error('Error persisting counts:', error)
}

View file

@ -520,6 +520,12 @@ export class MemoryStorage extends BaseStorage {
let totalNouns = 0
let totalVerbs = 0
// Vectored-noun scalar: unlike the bare presence check above, this needs
// the vectors.json RECORD'S content — a deferred-embed noun's record
// exists with an empty `vector: []` until its embed lands. In-memory this
// is a free field access (no I/O), unlike the filesystem adapter's
// per-noun disk read.
let totalVectoredNouns = 0
// Scan all paths in objectStore
for (const path of this.objectStore.keys()) {
@ -528,6 +534,10 @@ export class MemoryStorage extends BaseStorage {
if (nounMatch) {
// Type is in metadata, not path - just count total
totalNouns++
const record = this.objectStore.get(path) as { vector?: unknown } | undefined
if (Array.isArray(record?.vector) && record.vector.length > 0) {
totalVectoredNouns++
}
}
// Count verbs (entities/verbs/{shard}/{id}/vectors.json)
@ -543,6 +553,7 @@ export class MemoryStorage extends BaseStorage {
// A scan of every canonical record IS the ALL-visibility count.
this.totalNounCountAll = totalNouns
this.totalVerbCountAll = totalVerbs
this.totalVectoredNounCount = totalVectoredNouns
this.allCountsSuspect = false
}

View file

@ -125,6 +125,36 @@ export interface WriterLockInfo {
rootDir?: string // Convenience for log lines / error messages
}
/**
* THE CLEAN-CLOSE RECORD. Written by `releaseWriterLock()` at the instant it
* gives up the writer lock, naming the lock identity it released. The next
* `acquireWriterLock()` reads it and can then say from a RECORD, not from a
* guess whether the previous writer left on purpose.
*
* Why a record and not PID liveness: "the recorded PID is no longer alive" is
* true of every orderly restart AND of every crash, so the two were reported
* identically ("appears dead") and neither could be trusted. Worse, the same
* inference fails the other way when the operating system RECYCLES the pid
* a live unrelated process makes a long-dead writer's lock look held, and the
* store refuses to open naming a pid that was never Brainy. A record settles
* both: matched the previous writer closed cleanly, nothing to recover;
* absent say so, and name what recovery the open will now run.
*
* Lifecycle: written at release, consumed (deleted) by the next successful
* lock claim a record must never outlive the lock generation it describes,
* or it would vouch for a later crash.
*/
export interface WriterCloseRecord {
pid: number
hostname: string
/** `startedAt` of the lock this close released — the identity match key. */
startedAt: string
/** ISO timestamp at which the lock was released. */
closedAt: string
/** Brainy version that performed the close. */
version: string
}
/**
* FNV-1a hash returning a 2-char hex bucket (00-ff).
* Distributes system keys across 256 sub-prefixes to avoid
@ -203,6 +233,40 @@ function idFromVectorPath(path: string): string {
return lastSlash >= 0 ? withoutSuffix.slice(lastSlash + 1) : withoutSuffix
}
/**
* @description Extract the entity id embedded in a metadata path
* (`entities/{nouns|verbs}/{shard}/{id}/metadata.json`) the IDENTITY-RECORD
* mirror of {@link idFromVectorPath}. The cursored noun/verb walks key their
* population on this file (ADR-008 G1: the metadata record IS the population;
* the vector leg is optional), so walk ordering and cursor resume derive the
* id from THIS path, never the vector path a row with metadata and no
* vector file must still be listed, ordered, and resumable.
* @param path - A metadata path (full or prefix-relative; must end with `/metadata.json`).
* @returns The entity id (the path segment immediately before `/metadata.json`).
*/
function idFromMetadataPath(path: string): string {
const withoutSuffix = path.replace(/\/metadata\.json$/, '')
const lastSlash = withoutSuffix.lastIndexOf('/')
return lastSlash >= 0 ? withoutSuffix.slice(lastSlash + 1) : withoutSuffix
}
/**
* @description The sanctioned UNVECTORED shape for a noun hydrated during
* enumeration when its identity record (metadata.json) exists but its vector
* leg (vectors.json) does not a fold-born metadata-only after-image, or any
* row genuinely without a vector yet. Mirrors the shape
* `unvectorNounForRootMigration` (src/brainy.ts) writes for the sanctioned
* unvector path (`{ vector: [], connections: new Map(), level: 0 }`), so a
* walk-yielded unvectored row is byte-shape-identical to one produced by that
* migration. Callers already handle `vector: []` as first-class
* (validateAddParams exempts it; index gates key on `length > 0`).
* @param id - The noun id.
* @returns A structurally-valid, vector-empty `HNSWNoun`.
*/
function unvectoredNoun(id: string): HNSWNoun {
return { id, vector: [], connections: new Map<number, Set<string>>(), level: 0 }
}
/**
* Get ID-first path for verb metadata
* No type parameter needed - direct O(1) lookup by ID
@ -1373,6 +1437,29 @@ export abstract class BaseStorage extends BaseStorageAdapter {
return this.listObjectsUnderPath(prefix)
}
/**
* @description The IMMEDIATE child directory names under a prefix one
* level, no recursion. See the seam's JSDoc (`db/types.ts`) for why a
* separate door exists. This default derives them from the recursive
* listing, so it is never WRONG, only never faster; the filesystem adapter
* overrides it with a single directory read.
* @param prefix - Storage-root-relative directory prefix.
* @returns The child directory names (not paths), in listing order.
*/
public async listRawPrefixes(prefix: string): Promise<string[]> {
await this.ensureInitialized()
const paths = await this.listObjectsUnderPath(prefix)
const normalizedPrefix = prefix.endsWith('/') ? prefix : `${prefix}/`
const names = new Set<string>()
for (const p of paths) {
const rest = p.startsWith(normalizedPrefix) ? p.slice(normalizedPrefix.length) : null
if (rest === null) continue
const slash = rest.search(/[/\\]/)
if (slash > 0) names.add(rest.slice(0, slash))
}
return [...names]
}
/**
* Remove every object under a storage-root-relative prefix. The filesystem
* adapter overrides this with a recursive directory removal; this default
@ -1453,6 +1540,18 @@ export abstract class BaseStorage extends BaseStorageAdapter {
* rollups are derived state with their own rebuild paths
* (`rebuildTypeCounts()` / `rebuildSubtypeCounts()`).
*
* EXACT-RESTORE PRIMITIVE `vector: null` DELETES the vector leg, on
* purpose: `GenerationStore.rollBackUncommittedGeneration()` depends on
* this to legitimately un-write a vector a failed transaction added. This
* is deliberately NOT "preserve if absent" a caller replaying an
* AFTER-IMAGE (the recovery fold, `GenerationStore`'s `replayFact`) must
* apply preserve-if-absent itself BEFORE calling this, by reading the
* current vector and carrying it forward when the after-image's own
* vector leg is null/undefined but its metadata is not (see `replayFact`
* for the implementation and full rationale). A caller that genuinely
* wants to unvector a row uses the sanctioned, ledger-correct path
* (`Brainy.unvectorNounForRootMigration`) never this primitive.
*
* @param id - The entity id.
* @param record - Raw stored objects as returned by {@link BaseStorage.readNounRaw}.
*/
@ -1489,7 +1588,9 @@ export abstract class BaseStorage extends BaseStorageAdapter {
/**
* Restore a relationship's raw stored objects byte-for-byte (verb-side
* mirror of {@link BaseStorage.writeNounRaw}; same bookkeeping caveats).
* mirror of {@link BaseStorage.writeNounRaw}; same bookkeeping caveats,
* same EXACT-RESTORE contract `vector: null` deletes, on purpose; the
* fold's preserve-if-absent logic lives at its call site, not here).
*
* @param id - The relationship id.
* @param record - Raw stored objects as returned by {@link BaseStorage.readVerbRaw}.
@ -1762,8 +1863,10 @@ export abstract class BaseStorage extends BaseStorageAdapter {
/**
* Delete a noun from storage
* @param hadVector - OPTIONAL vectored-noun ledger hint, forwarded to
* {@link deleteNounMetadata} unchanged see its JSDoc.
*/
public async deleteNoun(id: string, priorMetadata?: NounMetadata | null): Promise<void> {
public async deleteNoun(id: string, priorMetadata?: NounMetadata | null, hadVector?: boolean): Promise<void> {
await this.ensureInitialized()
// FULL removal (live-HEAD hygiene): remove BOTH canonical legs AND the
@ -1780,7 +1883,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// LONGER wrapped in a blind catch that masked faults as "file didn't exist".
// `priorMetadata` (the caller's pre-delete read) keeps the decrement honest
// even when the canonical read inside returns null (replace race / ghost).
await this.deleteNounMetadata(id, priorMetadata)
await this.deleteNounMetadata(id, priorMetadata, hadVector)
// Remove the now-empty entity container (a no-op for key/prefix stores).
await this.removeCanonicalContainer(getNounVectorPath(id))
@ -2181,9 +2284,18 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// Stable within-shard order (by noun id) so offset windows and cursor resume
// are deterministic; ids come from the path so skipped nouns are never read.
//
// IDENTITY-KEYED WALK (population law, ADR-008 G1): the metadata record
// (not the vector) IS the population — a noun with metadata and no vector
// file (a fold-born after-image, see writeNounRaw's preserve-if-absent
// contract) must still enumerate. Keying on metadata.json here means the
// ledger recount (rebuildTypeCounts' `allNouns`, also metadata.json-keyed)
// and this walk agree on population by construction. Ordering is
// unaffected for a healthy store: every vectored noun has both legs, so
// the id set and sort order are identical to the old vectors.json keying.
const entries = nounFiles
.filter((p) => p.includes('/vectors.json'))
.map((p) => ({ path: p, id: idFromVectorPath(p) }))
.filter((p) => p.includes('/metadata.json'))
.map((p) => ({ path: p, id: idFromMetadataPath(p) }))
.sort((a, b) => (a.id < b.id ? -1 : a.id > b.id ? 1 : 0))
// Resume: in the cursor's own shard, skip up to AND INCLUDING the cursor
@ -2208,13 +2320,24 @@ export abstract class BaseStorage extends BaseStorageAdapter {
) {
const batch = toHydrate.slice(i, i + BaseStorage.HYDRATE_CONCURRENCY)
const hydrated = await Promise.all(
batch.map(async ({ path: nounPath }) => {
batch.map(async ({ path: metadataPath, id }) => {
try {
const noun = await this.readCanonicalObject(nounPath)
if (!noun) return null
const deserialized = this.deserializeNoun(noun)
const metadata = await this.getNounMetadata(deserialized.id)
const metadata = await this.readCanonicalObject(metadataPath)
if (!metadata) return null
// The vector leg is OPTIONAL (population law): a metadata-only
// row hydrates with the sanctioned unvectored shape rather than
// being dropped from the walk. A fault reading the vector leg
// is treated the same as absence — best-effort, matching the
// canonical recount's tolerance for an unreadable vectors.json
// (rebuildTypeCounts) — a vector-leg problem never hides an
// otherwise-good identity record.
let deserialized: HNSWNoun
try {
const vectorRecord = await this.readCanonicalObject(getNounVectorPath(id))
deserialized = vectorRecord ? this.deserializeNoun(vectorRecord) : unvectoredNoun(id)
} catch {
deserialized = unvectoredNoun(id)
}
return { deserialized, metadata }
} catch (error) {
// A TORN record must surface typed — a paginated read that
@ -2224,7 +2347,9 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
// Identity point-reads (get-by-id) still throw typed upstream.
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
// Skip nouns that fail to load
// Skip nouns whose IDENTITY record fails to load (the metadata
// read above) — that is the one leg this walk cannot proceed
// without.
return null
}
})
@ -2345,9 +2470,14 @@ export abstract class BaseStorage extends BaseStorageAdapter {
const shardDir = `entities/nouns/${shardHex}`
try {
const nounFiles = await this.listCanonicalObjects(shardDir)
// IDENTITY-KEYED WALK (population law, ADR-008 G1) — see the matching
// comment in getNounsWithPagination: metadata.json is the population;
// the vector leg is optional, so a metadata-only row must still be
// listed (and here, for the unfiltered case, needs ZERO reads either
// way — the id comes straight from the path).
const entries = nounFiles
.filter((p) => p.includes('/vectors.json'))
.map((p) => idFromVectorPath(p))
.filter((p) => p.includes('/metadata.json'))
.map((p) => idFromMetadataPath(p))
.sort((a, b) => (a < b ? -1 : a > b ? 1 : 0))
const toWalk =
cursor && shard === cursor.shard ? entries.filter((id) => id > cursor.id) : entries
@ -2558,23 +2688,79 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// Stable within-shard order (by verb id) so offset windows and cursor resume
// are deterministic and consistent across calls. Ids come from the path, so
// verbs skipped by the cursor are never read.
//
// IDENTITY-KEYED WALK (population law, ADR-008 G1) — the noun mirror of
// this comment in getNounsWithPagination applies here too: metadata.json
// is the population; keying on it here means this walk and the ledger
// recount (rebuildTypeCounts' `allVerbs`, already metadata.json-keyed)
// agree on population by construction. Unchanged for a healthy store —
// `relate()` always writes both legs of a verb in the same commit, so
// the id set and order match the old vectors.json keying exactly; this
// only additionally surfaces a fold-born metadata-only row (see
// writeVerbRaw's preserve-if-absent contract).
const entries = verbFiles
.filter((p) => p.includes('/vectors.json'))
.map((p) => ({ path: p, id: idFromVectorPath(p) }))
.filter((p) => p.includes('/metadata.json'))
.map((p) => ({ path: p, id: idFromMetadataPath(p) }))
.sort((a, b) => (a.id < b.id ? -1 : a.id > b.id ? 1 : 0))
for (const { path: verbPath, id: verbId } of entries) {
for (const { path: metadataPath, id: verbId } of entries) {
if (collected.length >= peekCount) break
// Resume: in the cursor's own shard, skip up to AND INCLUDING the cursor id
// (later shards are processed in full). No read for skipped verbs.
if (cursor && shard === cursor.shard && verbId <= cursor.id) continue
try {
const rawVerb = await this.readCanonicalObject(verbPath)
if (!rawVerb) continue
// Identity leg first — required. A verb this walk cannot read
// metadata for cannot be hydrated at all (same as before).
const metadata = await this.readCanonicalObject(metadataPath)
if (!metadata) continue
// Deserialize connections Map from JSON storage format
const verb = this.deserializeVerb(rawVerb)
// The vector leg is the verb's STRUCTURAL core (verb/sourceId/
// targetId live there — see coreTypes.ts HNSWVerb), unlike a
// noun's vector, which is pure embedding data. `relate()` always
// writes both legs atomically and verbs have no deferred-embed
// path, so a healthy store's verbs always have both. A vector-leg
// absence here can only be a fold-born after-image (see
// writeVerbRaw's preserve-if-absent contract) — and unlike a
// noun, this walk cannot safely FABRICATE sourceId/targetId to
// synthesize a structurally-valid verb (an empty-string endpoint
// would silently create a phantom edge — worse than omission).
// If the metadata record happens to carry its own sourceId/
// targetId (never true for current production writes, but not
// disallowed — e.g. a future schema or a repair tool could
// populate them), reconstruct from those; otherwise this row is
// loudly skipped — counted by the ledger, but not returned as an
// item, until a repair can supply the missing endpoints.
const rawVerb = await this.readCanonicalObject(getVerbVectorPath(verbId))
let verb: HNSWVerb
if (rawVerb) {
verb = this.deserializeVerb(rawVerb)
} else {
const metaSourceId = (metadata as Record<string, unknown>).sourceId
const metaTargetId = (metadata as Record<string, unknown>).targetId
const metaVerbType = (metadata as Record<string, unknown>).verb
if (
typeof metaSourceId === 'string' && metaSourceId.length > 0 &&
typeof metaTargetId === 'string' && metaTargetId.length > 0 &&
typeof metaVerbType === 'string' && metaVerbType.length > 0
) {
verb = {
id: verbId,
vector: [],
connections: new Map<number, Set<string>>(),
verb: metaVerbType as VerbType,
sourceId: metaSourceId,
targetId: metaTargetId
}
} else {
prodLog.error(
`[BaseStorage] getVerbsWithPagination: verb ${verbId} has a metadata ` +
`record but no vector leg and no recoverable sourceId/targetId — ` +
`skipping (counted by the ledger, not yielded; needs repair).`
)
continue
}
}
// Apply type filter
if (filterVerbTypes && !filterVerbTypes.has(verb.verb)) {
@ -2591,9 +2777,6 @@ export abstract class BaseStorage extends BaseStorageAdapter {
continue
}
// Load metadata
const metadata = await this.getVerbMetadata(verb.id)
// Apply subtype filter (requires metadata — checked AFTER load)
if (filterSubtypes) {
const subtype = metadata?.subtype as string | undefined
@ -3392,11 +3575,12 @@ export abstract class BaseStorage extends BaseStorageAdapter {
/**
* Save noun metadata to storage (now typed)
* Routes to correct sharded location based on UUID
* @param hasVector - See {@link StorageAdapter.saveNounMetadata}'s JSDoc.
*/
public async saveNounMetadata(id: string, metadata: NounMetadata): Promise<void> {
public async saveNounMetadata(id: string, metadata: NounMetadata, hasVector?: boolean): Promise<void> {
// Validate noun type in metadata - storage boundary protection
validateNounType(metadata.noun)
return this.saveNounMetadata_internal(id, metadata)
return this.saveNounMetadata_internal(id, metadata, hasVector)
}
/**
@ -3407,9 +3591,10 @@ export abstract class BaseStorage extends BaseStorageAdapter {
* This ensures counts are updated AFTER metadata exists, fixing the race condition
* where storage adapters tried to read metadata before it was saved.
*
* @param hasVector - See {@link StorageAdapter.saveNounMetadata}'s JSDoc.
* @protected
*/
protected async saveNounMetadata_internal(id: string, metadata: NounMetadata): Promise<void> {
protected async saveNounMetadata_internal(id: string, metadata: NounMetadata, hasVector?: boolean): Promise<void> {
await this.ensureInitialized()
// ID-first path - no type needed!
@ -3465,6 +3650,14 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// record persists here so the ALL scalar never lags the tree.
if (isNew) {
this.totalNounCountAll++
// Vectored-noun ledger: rides the SAME isNew gate (once per id, at
// creation) — this seam is metadata-write-driven and never re-runs on
// the HNSW neighbor-link re-saves that hit saveNoun_internal, so it
// cannot double-count. A deferred-embed insert passes hasVector=false
// (or omits it); its vector lands later via noteVectorLanded().
if (hasVector) {
this.totalVectoredNounCount++
}
if (!(metadata.noun && isCounted)) {
this.scheduleCountPersist().catch(() => {
// Ignore persist errors — the in-memory count is authoritative; a later op retries.
@ -3860,8 +4053,18 @@ export abstract class BaseStorage extends BaseStorageAdapter {
* the skip permanently inflated the persisted totals (adds counted, paired
* removals not decremented), and `Math.max(totalNounCount, scanned)` made
* the inflation unfixable by any disk cleanup.
* @param hadVector - OPTIONAL vectored-noun ledger hint see
* {@link StorageAdapter.deleteNounMetadata}'s JSDoc. This method never
* reads `vectors.json` to answer the question itself (a canonical read
* the delete path must never add); a caller that cannot supply the fact
* for free leaves it `undefined`, and the ledger goes SUSPECT rather
* than guessing.
*/
public async deleteNounMetadata(id: string, priorRecord?: NounMetadata | null): Promise<void> {
public async deleteNounMetadata(
id: string,
priorRecord?: NounMetadata | null,
hadVector?: boolean
): Promise<void> {
await this.ensureInitialized()
// Direct O(1) delete with ID-first path. Read the canonical record BEFORE
@ -3885,6 +4088,15 @@ export abstract class BaseStorage extends BaseStorageAdapter {
} else {
this.markAllCountsSuspect('noun', id)
}
// Vectored-noun ledger: a KNOWN vector fact decrements (or no-ops);
// an UNKNOWN one goes suspect rather than guessing — see @param hadVector.
if (hadVector === true) {
if (this.totalVectoredNounCount > 0) this.totalVectoredNounCount--
else this.markAllCountsSuspect('noun-vector', id)
} else if (hadVector === undefined) {
this.markAllCountsSuspect('noun-vector', id)
}
this.scheduleCountPersist().catch(() => {
// Ignore persist errors — the in-memory count is authoritative; a later op retries.
})
@ -4549,6 +4761,16 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// construction.
let allNouns = 0
let allVerbs = 0
// Vectored-noun scalar: unlike `allNouns` (decided from the metadata.json
// LISTING alone), presence cannot be decided from the vectors.json
// listing alone — a deferred-embed noun's vectors.json EXISTS with an
// empty `vector: []` until its embed lands, so the file's CONTENT must be
// read. This walk already lists every path per shard (including
// vectors.json entries — `listCanonicalObjects` yields both legs), so
// reading them here costs one EXTRA read per noun beyond the metadata.json
// read above (doubling this walk's per-noun I/O) — honest cost, paid only
// by this diagnostic/repair recount, never on the hot path.
let allVectoredNouns = 0
// Scan noun shards
for (let shard = 0; shard < 256; shard++) {
@ -4559,6 +4781,18 @@ export abstract class BaseStorage extends BaseStorageAdapter {
const paths = await this.listCanonicalObjects(shardDir)
for (const path of paths) {
if (path.includes('/vectors.json')) {
try {
const vectorRecord = await this.readCanonicalObject(path)
if (vectorRecord && Array.isArray(vectorRecord.vector) && vectorRecord.vector.length > 0) {
allVectoredNouns++
}
} catch (error) {
// Skip vector records that fail to load — best-effort ground truth,
// same as the metadata read below.
}
continue
}
if (!path.includes('/metadata.json')) continue
allNouns++
@ -4634,17 +4868,23 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// IS the proof an unprovable delete could not give.
const nounsAllBefore = this.totalNounCountAll
const verbsAllBefore = this.totalVerbCountAll
const vectoredBefore = this.totalVectoredNounCount
this.totalNounCountAll = allNouns
this.totalVerbCountAll = allVerbs
this.totalVectoredNounCount = allVectoredNouns
this.allCountsSuspect = false
// This walk counts one entity per metadata.json record (never per bare
// container) — the identity-record rule. Stamp it so a future load
// trusts these scalars instead of naming them suspect at open.
this.allCountsDerivedBy = 'identity-record'
this.countCache.clear()
await this.persistCounts()
prodLog.info(
`[BaseStorage] Rebuilt counts: ${totalNouns} nouns, ${totalVerbs} verbs (user-facing); ` +
`ALL-visibility ledger ${allNouns} nouns / ${allVerbs} verbs` +
(nounsAllBefore !== allNouns || verbsAllBefore !== allVerbs
? ` (corrected from ${nounsAllBefore} / ${verbsAllBefore})`
`ALL-visibility ledger ${allNouns} nouns / ${allVerbs} verbs / ${allVectoredNouns} vectored nouns` +
(nounsAllBefore !== allNouns || verbsAllBefore !== allVerbs || vectoredBefore !== allVectoredNouns
? ` (corrected from ${nounsAllBefore} / ${verbsAllBefore} / ${vectoredBefore})`
: ' (unchanged)') +
` — scalar + per-type persisted`
)

View file

@ -154,6 +154,21 @@ export function resolveFilesystemRoot(
) {
throwRemovedStorageKey('fileSystemStorage.path')
}
// A nested `config` object carrying a path-shaped key is the same hazard in
// a shape nobody ever supported: it used to fall through SILENTLY to the
// shared default root — every instance writing one directory while its
// caller believed each had its own. (Found live: an integration test's
// brains shared one store across a whole single-process run and a health
// probe refused on the foreign edges it sampled.) Loud, with the rename.
const nested = (config as Record<string, unknown>).config
if (nested && typeof nested === 'object') {
const pathish = ['path', 'baseDir', 'rootDirectory', 'rootDir', 'dir', 'directory']
const hit = pathish.find(
(k) => typeof (nested as Record<string, unknown>)[k] === 'string' &&
((nested as Record<string, unknown>)[k] as string).length > 0
)
if (hit) throwRemovedStorageKey(`config.${hit}`)
}
// 3. Zero-config default. A `type: 'filesystem'` with no path lands here
// intentionally ("persist, default location").

View file

@ -13,6 +13,8 @@ import type { VectorIndexProvider, GraphIndexProvider } from '../../plugin.js'
import type { MetadataIndexManager } from '../../utils/metadataIndex.js'
import type { GraphVerb } from '../../coreTypes.js'
import type { Operation, RollbackAction } from '../types.js'
import { isZeroNormVector } from '../../utils/distance.js'
import { prodLog } from '../../utils/logger.js'
/**
* Backend identity stamped into an operation's emitted `name` string (e.g.
@ -88,6 +90,30 @@ export class AddToVectorIndexOperation implements Operation {
}
async execute(): Promise<RollbackAction> {
// THE ZERO-NORM LAW (the live provider-write seam's belt): a zero-norm
// vector is not a vector — it never crosses an engine boundary. This
// engine's own cosine distance treats an all-zero vector safely (a
// zero-norm operand always scores MAXIMUM distance, see
// {@link isZeroNormVector}'s JSDoc), but a downstream engine serving
// squared-euclidean distance cannot tell it apart from a legitimate
// origin point — a false attractor that silently darkens real results.
// The canonical write already landed (SaveNoun/SaveNounMetadata
// operations are staged ahead of this one in every caller) — only the
// INDEX INSERT is refused here, loudly, never a throw. A length-0
// vector is the unrelated "unvectored" shape and is skipped silently
// (the same contract callers already rely on for deferred embeds).
if (this.vector.length === 0) {
return async () => {}
}
if (isZeroNormVector(this.vector)) {
prodLog.warn(
`[vector-index] refusing to index a zero-norm vector for entity ${this.id}` +
`a zero-norm vector is not a vector and never crosses an engine boundary ` +
`(the canonical write is unaffected; only the vector-index insert is skipped)`
)
return async () => {}
}
// Check if item already exists (for rollback decision)
const existed = await this.itemExists(this.id)
@ -263,14 +289,52 @@ export class ReplaceInVectorIndexOperation implements Operation {
// One commit generation for the whole replace (both branches + rollback).
const generation = this.generationFn?.()
// THE ZERO-NORM LAW (see AddToVectorIndexOperation's matching JSDoc): a
// real all-zero replacement vector must never land in the index — refuse
// loudly, canonical write unaffected. The row must not be left stale
// either: if it was genuinely indexed under `oldVector`, remove it
// rather than pretend the old vector still describes the row. A
// length-0 `newVector` (the unrelated "unvectored" shape) is handled the
// same way, silently — no caller today reaches this with an empty
// replacement (update() rejects a dimension-mismatched empty vector),
// but the seam stays consistent in case one ever legitimately does.
if (isZeroNormVector(this.newVector) || this.newVector.length === 0) {
const wasIndexed = this.oldVector.length > 0 && !isZeroNormVector(this.oldVector)
if (isZeroNormVector(this.newVector)) {
prodLog.warn(
`[vector-index] refusing to replace with a zero-norm vector for entity ${this.id}` +
`a zero-norm vector is not a vector and never crosses an engine boundary ` +
`(the canonical write is unaffected; the row is removed from the vector index instead)`
)
}
if (wasIndexed) {
await this.index.removeItem(this.id, generation)
}
return async () => {
// Restore the declared before-state.
if (wasIndexed) {
await this.index.addItem({ id: this.id, vector: this.oldVector }, generation)
}
}
}
if (typeof index.updateItem === 'function') {
// Atomic path: one in-place call, the row never leaves the index.
await index.updateItem({ id: this.id, vector: this.newVector }, generation)
return async () => {
// Restore the declared before-state in place (see class JSDoc for
// the item-did-not-exist posture).
await index.updateItem!({ id: this.id, vector: this.oldVector }, generation)
// the item-did-not-exist posture). A length-0 oldVector means the row
// was never actually indexed before this op ran (a length-0 vector is
// never a legal index member — see EmptyVectorIndexError) — there is
// no in-place "restore to empty" for the provider to perform, so
// rollback removes the row instead, leaving the same "not indexed"
// state the row was in before execute().
if (this.oldVector.length > 0) {
await index.updateItem!({ id: this.id, vector: this.oldVector }, generation)
} else {
await this.index.removeItem(this.id, generation)
}
}
}
@ -281,9 +345,14 @@ export class ReplaceInVectorIndexOperation implements Operation {
return async () => {
// updateItem-style restore via the same adjacent pair, back to the
// declared before-state.
// declared before-state. Same length-0 carve-out as the updateItem
// path above: an empty oldVector was never a legal index member, so
// rollback just leaves the row removed rather than attempting an
// illegal empty re-add.
await this.index.removeItem(this.id, generation)
await this.index.addItem({ id: this.id, vector: this.oldVector }, generation)
if (this.oldVector.length > 0) {
await this.index.addItem({ id: this.id, vector: this.oldVector }, generation)
}
}
}
}

Some files were not shown because too many files have changed in this diff Show more