Origin moved to the private forge; the public repo is now a mirror with an
unknown sync cadence. The release flow creates the GitHub release directly,
and a missing tag there would be silently created at the default-branch
head - the wrong commit. The script now pushes branch+tag to the mirror
itself and hard-stops if the tag's commit on the mirror differs from local,
before anything irreversible (npm publish) happens.
Reconciles the vector-index rename to the ruled three-layer naming: the
provider contract's identity field is now a REQUIRED readonly name (was
optional providerId), self-reported and truthful, rendered as
[vector-index:<name>] where the index identifies itself and stamped into
the op-name strings journals already parse (AddToVectorIndex(<name>)).
The built-in JS engine names itself hnsw-js. A runtime provider instance
compiled against the previous optional contract is tolerated - never
crashed on, never silently mislabeled: it stamps unknown-provider and
emits one loud warning naming the missing field. Graph index operations
keep their static names (they never interpolate provider identity), and
no public API exports a provider-routed hnsw-carrying name, so no
deprecation shim is required.
Three pieces addressing the cold-restart-write incident where a production
deployment's first writes after every restart (33-35s each on a cold page
cache) blew the op-count-scaled transact budget mid-batch: every write is
itself a multi-op transaction, so one cold operation consumed the whole
budget, the gate before the next operation tripped, and the write rolled
back atomically - refused, retried, and refused again until the page cache
warmed passively.
- The budget's start-gating contract is now explicit and pinned: it gates
STARTING the next operation, never rolling back completed work for
elapsed time (the shipped schedule since 8.7.0, now stated in contract
JSDoc, guarded by code for operation 0, and enforced by regression
tests). The 30s floor is configurable via transactionBudgetFloorMs for
stores whose cold operations legitimately run long.
- New brain.warm() eagerly loads the vector index, metadata index, and
graph adjacency so first operations after a cold restart run at
steady-state cost. Returns a WarmReport with an honest per-surface
outcome (warmed / probed / unavailable) - never reports a probe as a
warm. warmOnOpen: true runs it during init(). New optional provider hook
warm() on the vector and graph plugin contracts.
- Vector-index transaction op classes renamed from the backend-specific
AddToHNSWOperation / RemoveFromHNSWOperation to backend-neutral
AddToVectorIndexOperation / RemoveFromVectorIndexOperation, stamping the
active backend into the emitted op-name string (AddToVectorIndex(js-hnsw)
vs a native provider's own identity) so journals never misdirect an
operator toward an index that isn't running.
First edition of the per-release performance-envelope contract: every
number measured against the built dist on stated hardware, never
projected. Sub-0.1ms get/related (adjacency O(degree), scale-flat),
1-9ms indexed metadata finds, ~178ms semantic (query embedding
dominates), ~167ms durability-priced single-op writes flat across
scale, 8-45ms steady-state flush independent of history backlog
(the 8.9.0 change). Two weak spots stated honestly: addMany commits
per-item today (batched chunk commits belong to the unified-commit
roadmap), and pure-JS warm open grows with corpus (4.9s at 10k) —
the native accelerator's reason to exist. Refresh rule: any release
touching a measured path re-measures in the same release.
clearInterval() stops future heartbeat ticks but not one already in
flight: a straggler tick past its ownership guards could land its
atomic lock rewrite AFTER releaseWriterLock()'s unlink, re-creating
the lock file as a phantom that blocks the next writer until the
stale TTL expires (~60s) — the pool-eviction reopen case. Found as an
ENOENT heartbeat warning during benchmark teardown; the quiet variant
is the harmful one.
releaseWriterLock() now awaits the in-flight tick (tracked per tick,
self-clearing) before reading/unlinking, so a straggler's write always
lands BEFORE the unlink and gets removed with everything else. The
heartbeat's ENOENT is also now benign-by-contract (lock or directory
removed under us — the next acquire recreates it); other errors stay
loud.
Pin: straggler-past-guards simulation — lock file absent after close,
directory immediately claimable (fails on the undrained code).
flush() is durability work: it must cost what the current window's
deltas cost, never what the history backlog costs. Under adaptive
retention the byte budget derives from free memory, so bulk-load
pressure shrank the budget exactly at peak write volume and flush paid
actual reclaim inline — a production deployment measured single writes
blocked 25-191s behind reclaim-on-flush.
- flush() no longer calls autoCompactHistory(); close() is THE
auto-compaction site (already ran there; now alone).
- Every auto pass is time-bounded (CLOSE_COMPACTION_BUDGET_MS = 5s):
reclamation is oldest-first, so an early stop is a consistent prefix
and the next pass resumes. Explicit compactHistory() gains an
optional timeBudgetMs for caller-chosen maintenance windows.
- Documented trade stated where operators read: a long-lived writer
that never closes accumulates history until its next explicit
compactHistory() — predictable writes, explicit maintenance.
Pins: flush-never-reclaims + close-reclaims-durably (db-mvcc), bounded
pass stops-then-resumes as a consistent prefix (generationStore unit).
Four fixes from a consumer conformance report, one root disease — two
field-resolution regimes where there must be one:
- The delete/update aggregation hooks fed the engine a partial entity
view (type/service/data/metadata only), so a reserved-field groupBy
(subtype, visibility, ...) resolved to a nonexistent group on the way
down: counts drifted upward forever after deletes, and updates moving
an entity between reserved-field groups double-counted. The hooks now
pass the full-fidelity view via entityForAggFromRawRecord (every
reserved field top-level, mirroring the add path); the update sites
pass the full get() view instead of a hand-rolled subset.
- Aggregation source.where resolved fields only against the custom
metadata bag, so where on a reserved field silently matched nothing.
The matcher now resolves each filtered field through
resolveEntityField — the same single source of truth groupBy uses.
- removeMany() with no usable selector (bare array passed positionally,
empty params, ids: []) resolved successfully having deleted nothing.
All three now throw; the two legacy tests that pinned the silent
no-op as 'graceful' now pin the refusal.
- find() where keys accept both spellings: a metadata.-prefixed key
falls back to its flattened spelling when the prefixed one is not
indexed (metadata is flattened at index time). A literal nested custom
key named metadata still wins when indexed as spelled.
Five regression pins in aggregate-reserved-fields.test.ts (4 of 5 vary
red on the unfixed code).
scripts/push-docs.js collects docs/**/*.md with public:true frontmatter
and POSTs them (batches of 10, idempotent per slug) to the docs ingest
door after npm publish — release.sh step 12. Absent secret = loud skip
(publish already happened; the serving side can interim-sync); a failed
push exits non-zero so the docs site never silently trails npm. The
combined /docs landing index is deliberately NOT pushed per-repo — it
spans both engine corpora and is authored on the serving side.
Under default adaptive retention, every flush() recomputed total history
bytes by walking EVERY committed generation's delta — O(all generations)
with disk re-reads past the 4096-entry delta-cache bound. On a production
brain with 70,000+ accumulated generations this turned every write into a
full-tail scan (60-100s writes, escalating with history growth), even
though the free-RAM budget never tripped and nothing was ever reclaimed
(SELF-GENERATIONS-GROWTH).
historyBytes() now maintains a running total: seeded by one walk on first
use, then updated incrementally at both commit paths (+bytes) and the
compaction reclaim loop (−bytes), dropped on reopenAfterRestore. The
adaptive retention check on every flush is O(1). Invariant regression-
pinned: running total ≡ fresh walk through transact commits, single-op
group commits, and compaction.
New brain.historyStats() (exported HistoryStats): read-only generation
count / bytes / generation+timestamp range / horizon / retention mode /
effective budget — the one-call per-brain fleet audit for retention
exposure.
The post-import background deduplication pass (a merge-DELETE writer,
debounced ~5 minutes after import) had four lifecycle defects:
- enableDeduplication: false did not gate the background schedule — an
import that explicitly opted out could still have entities auto-removed
minutes later. The flag now gates both the inline and background passes.
- Each import() constructed its own coordinator-owned deduplicator, so the
debounce never spanned imports (N imports = N delete timers). The brain
now owns a single lazy instance (getBackgroundDeduplicator).
- close() never cancelled pending dedup; a delete pass could fire against
a closed brain. close() now cancels it first.
- The 5-minute timer held the process open (exit-hang class); now unref'd.
Four regression tests pin the contract (background-dedup-lifecycle);
import guides document that false disables both passes.
- New src/utils/osLimits.ts: reads RLIMIT_NOFILE (soft/hard, from
/proc/self/limits) and vm.max_map_count at open — once per process,
Linux-only, measurement-only — and warns loudly when either sits below the
pool-scale floors (soft NOFILE < 65536, max_map_count < 262144), with the
exact raise commands. On stock defaults the failure otherwise arrives as
EMFILE or a failed mmap deep inside an index open, long after the cause
stopped being visible. An unreadable limit produces NO warning — no
measurement, no claim — so non-Linux platforms stay silent.
- Exported for ops doors: checkOsLimits() returns the full OsLimitsReport;
floors exported as constants. Wired fire-and-forget in performInit after
storage init; the check can never affect open.
- Unit tests pin the parser (incl. 'unlimited'), the floor thresholds, the
null-never-warns rule, and the off-Linux silent path.
- acquireWriterLock now CLAIMS with an atomic create-exclusive write (O_EXCL)
inside a bounded retry loop: two processes racing an absent lock can never
both succeed (the old read-then-tmp-rename flow let the loser keep running
unlocked, silently). An EEXIST loser re-evaluates and either throws loudly
with the winner's details or performs a verified stale-takeover (re-read
before unlink so a lock that changed hands mid-deliberation is never
clobbered). Exhausted contention fails loudly instead of degrading into a
lockless open.
- BRAINY_WRITER_LOCKED passes through init() unwrapped: the error documents a
machine-readable contract (err.code + err.lockInfo with the holder's
pid/host/heartbeat), but init's blanket error wrapping stripped both,
leaving consumers a message to regex against.
- Two contract tests added: stale-foreign takeover installs OUR lock via the
atomic claim; the conflict error carries code + lockInfo at the public
init() surface.
- The transact apply budget scales with the batch — max(30s, opCount x 2s) —
or is exactly the caller's new TransactOptions.timeoutMs. A flat 30s cap
silently limited honest bulk work to ~15 operations on network-attached
storage (~2s/op measured in a production import) while looking generous for
small batches. Internal batch paths (removeMany chunks) get the same
scaling via the shared transactTimeoutBudget helper.
- TransactionTimeoutError now reports the operation it stopped at as i/N with
the operation's name, elapsed vs budgeted time, and states the batch rolled
back atomically and is retryable; context carries the fields
programmatically.
- The transact operational envelope is documented (optimistic-concurrency
guide): budget math, chunking with ifAbsent idempotency, when to reach for
addMany vs transact, and the precompute pattern (embedBatch + per-op
vector) that keeps inference out of the commit path.
- Embedding claims made honest per an end-to-end probe of the built dist:
batch and single embedding paths produce bit-identical vectors and a
vector-supplied add is fully searchable, but batch throughput on the
default WASM engine measures comparable to sequential (~160ms/text) — the
5-10x speedup claim in the addMany JSDoc is replaced with the measured
reality and the actual win (inference outside the budgeted write path).
- New public API: walks every canonical relationship record, queries the same
read path applications use (related() with all visibility tiers included),
and classifies every discrepancy into its failure family: missingFromReads
(records the read path omits — stale adjacency), danglingEndpoints
(endpoint entity gone — the partial-delete scar class), readOnlyVerbIds
(read-path edges with no canonical record — ghosts). Design-hidden
internal/system edges are counted separately so intentional hiding is never
misclassified as loss. Counts exact; example lists capped with an explicit
truncatedExamples flag; loud narration on incoherence; mutates nothing.
- Classification core lives in src/graph/graphAudit.ts behind injected seams
(canonical walks + the end-to-end read) so the discrepancy logic is testable
in isolation; brainy wires the seams to storage walks and related().
- getNounIdsWithPagination now refuses an undecodable resume cursor loudly —
the third and final pagination walk brought under the 8.5.2 cursor contract.
- Types exported: GraphAuditReport, GraphAuditDiscrepancy. Docs: the
inspection guide gains the audit -> repair -> audit verification ritual.
- Aggregation backfill walks build into a STAGING map and swap in atomically
on completion. A mid-walk failure drops the staging map — the previous live
state keeps serving, the aggregate stays flagged pending, and the storage
error surfaces to the failing query. Previously the walk wiped live state
before a scan that could throw, never cleared the pending flag on failure,
and re-ran a full walk on every subsequent query: a silent wipe/walk/throw
loop at the caller's retry rate.
- Failed walks are latched: retries within a 30s cooldown rethrow the recorded
error instantly instead of re-walking, so a tight caller-side retry loop
costs one loud error per query, never a full store walk per query.
- Persisted aggregation state is stamped with the store's committed generation
at flush; reopen adoption requires stamp equality. Stale state (unclean
shutdown) or over-counting state (a log truncation on a copied store pulled
the watermark back) triggers exactly one loud rescan, never a silent adopt.
- The backfill/adoption path narrates: adoption decisions, walk start/finish
with counts and duration, and failures all log by default.
- getNouns/getVerbs refuse a supplied-but-undecodable pagination cursor with a
loud error instead of silently restarting the walk at offset 0 (which
re-served page 1 forever to any while(hasMore) caller).
- The graph cold-load verb walk aborts loudly on a missing or non-advancing
cursor with hasMore=true.
- A versioned index provider whose generation is AHEAD of the committed
watermark (torn copy / crash-recovery truncation) is now named loudly at
open, alongside the existing behind-direction message.
- AggregationIndex: defineAggregate before init no longer forces a backfill.
init reconciles instead of clobbering: the app definition wins, persisted
state is adopted on hash match, a write landing pre-adoption forces an exact
rescan, and a successful state load clears the backfill flag. New ready()
settles every adoption decision before query paths consult backfill state.
- brainy: backfills are single-flight and batched. Concurrent queries share one
store walk and every pending aggregate fills from that same walk; the old
behavior let each concurrent query wipe the others' partial state and start
its own full walk, so a store under steady aggregate traffic never converged.
queryAggregate also waits for persisted definitions before deciding an
aggregate does not exist.
- paramValidation: recordQuery is telemetry-only. The duration-based cap
ratchet (x0.8 per recorded query while lifetime-average exceeded 1s, floored
at 1000 - below the documented 10000 auto floor, reported under a stale
basis label) is removed; the cap comes from its construction-time basis or
explicit overrides alone.
- storage: __aggregation_* and singleton system keys (brainy:entityIdMapper)
are recognized before the unknown-key warning fires; routing is unchanged.
- docs: find-limits cap-immutability note, aggregation reopen/adoption
semantics, RELEASES.md 8.5.1 entry.
- New public guide (guides/external-backups): the sparse-mmap reality
for external backup tooling — why a brain directory can show 100+ GB
apparent size on a small disk, which files are sparse, tar czSf /
rsync --sparse / cp --sparse=always, live-store caveats, and what
persist()/restore() already handle natively.
- New public concept (concepts/generation-fact-log): what facts are
(after-image commit records, body-less tombstones), the crash-safety
model (checksummed frames, reconcile-to-committed, absent = never
committed), the scanFacts()/factSegmentPaths() surfaces with the
telemetry shape, family stamps + open-time coherence, and the
storage capability seam for plugin authors.
- Cross-link from the snapshots guide; sparse notes added to the
public persist()/restore() JSDoc (the operator-facing sites).
setTimeout can fire a few ms early under load (timer coalescing) — a
10ms sleep measured 9ms and failed the >=10 assertion, tripping a
release gate. Sleep 25ms, assert >=20: the test verifies time is
MEASURED, not the OS timer's precision.
- storage.committedGeneration(): the committed watermark exposed on the
fact-scan capability, so a provider compares its stamp's
sourceGeneration against the store's truth without parsing the
private manifest format (the capability source now carries both the
live fact log and the committed closure).
- Pinned contract tests: fsync-before-ack (transact passes today; the
single-op path is pinned via it.fails as the documented target —
group commit must become LATENCY batching, ack waiting on the shared
fsync, never durability skipping; the pin flips red when that lands
so there is no cliff to discover) and scan-stability-under-rotation
(a scan snapshot yields exactly its facts — no gaps, duplicates, or
bleed-in — while segments rotate beneath it; the reclaim-during-scan
variant lands with fact compaction, which does not exist yet).
- FactLog accepts a rotateBytes option so tests exercise real rotation.
Three additive surfaces for native index providers, which hold only
`storage` and must never construct their own fact-log reader (the log's
open path is writer-side — it reconciles by truncating/rewriting):
- Fact-scan capability on the storage adapter (the getBinaryBlobPath
pattern): the host brain wires a closure over its LIVE fact log at
init; providers call storage.scanFacts()/factLogHeadGeneration()/
factSegmentPaths() and fall back to the enumeration walk on null.
Restore/reopen swaps the underlying log transparently.
- The family-stamp trio (readFamilyStamp/writeFamilyStamp/
verifyFamilyStamp + types) exported from @soulcraft/brainy/internals,
so 'one verifier' is literally one function shared with the native
side, never two synchronized copies.
- Rollup invariants widened to number|string: content fingerprints
(e.g. a per-tree SHA-256) are valid invariant values; strict equality
either way, and a type mismatch reads as incoherence, never a pass.
The canonical entity tree now carries a FAMILY STAMP
(_system/family-stamps/entity-tree.json, JSON — forensics stay
terminal-readable): which committed generation the tree reflects
(sourceGeneration) plus the rollup invariants (entity/relationship
counts) that verify a millions-of-files projection whole where per-file
checks cannot scale. Written at flush/close boundaries; open-time
coherence is a COMPARISON, not a walk:
- coherent / absent (legacy store) → silent
- behind → benign for the tree (it is written BY the commit; only the
stamp is stale after a crash between commit and flush) — refreshes at
the next flush
- incoherent (counts diverged at equal generation, or a stamp AHEAD of
the log head) → loud, names the failing invariant; repairIndex()'s
unconditional recount heals and RE-STAMPS so repair leaves a coherent
stamp behind
- a fault reading the stamp is UNVERIFIABLE — never conflated with
absence
One verifier (verifyFamilyStamp, exported) reads both member modes:
enumerated (exact byte size per member, bounded families) and rollup
(invariants, unbounded families). New exports: readFamilyStamp,
verifyFamilyStamp, ENTITY_TREE_STAMP_PATH, FamilyStamp, StampMembers,
StampVerdict.
Every committed generation now also appends a FACT — an after-image
commit record (what each touched entity/relationship became, or a
body-less tombstone for a removal) — to an append-only, crc32c-framed
segment log under _generations/facts/. The before-image history and the
canonical tree remain authoritative; the fact log gives consumers ONE
sequential, self-verifying stream (index heals, incremental replays)
in place of a per-entity directory walk.
- Wire format: positional msgpack facts [generation, timestamp, ops,
meta, blobHashes]; op = [kind u8, id bin16, record | nil tombstone];
32-byte segment header (magic, formatVersion, firstGeneration,
zeroed+verified reserved); length+crc32c frame per fact; zero-padded
segment names so lexicographic order == generation order; JSON
manifest with an atomic rename flip, manifest-first rotation.
- Commit protocol: facts append+fsync BEFORE the commit point inside
the existing durability window, so a crash can only leave the log
AHEAD of committed truth — open() truncates back (torn tails detected
by CRC). Absent generation = never committed; a scan can never see an
uncommitted fact. transact() facts are durable-on-return; single-op
facts ride the group-commit flush exactly like buffered history. A
fact-append failure fails the write, loudly — a silent gap would be a
lie a later replay discovers.
- New public surface: brain.scanFacts() (sequential batches with heal
telemetry: head/segments/approx up front, per-batch generation range
+ bytes + segment id, loud abort on gaps, summary cross-check) and
brain.factSegmentPaths() (immutable sealed segments for zero-copy
consumers; the mutable tail excluded). Exported types CommitFact,
FactOp, FactScanBatch, FactScanHandle.
- Storage: optional binary raw-byte primitives (appendRawBytes,
readRawBytes, writeRawBytes, rawByteSize) on StorageAdapter —
feature-detected; filesystem + memory adapters implement them; an
adapter without them hosts no fact log. Fact segments are byte-copied
(never hard-linked) into snapshots. The _generations/facts/ namespace
is registered as a protected family (rebuildable: false): no sweeper
or GC may delete under it.
- New crc32c (Castagnoli) utility with RFC known-answer tests.
Ports the fresh-brain probe that closed the type+subtype lens-drop
investigation into the permanent suite: a 7-pair corpus seeded through the
real write API (never restored bytes), every lens checked id-for-id against
an unfiltered canonical scan, warm AND after a cold reopen, plus the
type+subtype update-flip leg. Guards the under-inclusion class the egress
integrity guard structurally cannot catch.
A cross-directory rename added the new parent's Contains edge but skipped
removing the old one (a 'not critical' shortcut), leaving the moved entity
a child of BOTH directories: readdir(oldDir) kept listing it, re-creating
the old path showed the name twice, and tree-walking consumers saw the
file in two places. Reported live by a production deployment (37 stale
containment ghosts; blocks tree-sync integrations).
- rename() now removes the old parent's vfs containment edge(s) by edge
id resolved from the graph's own adjacency (the removal law: removal
never requires reading the removed thing), and a move to the root gets
its containment edge (previously skipped -> orphaned from readdir('/')).
- New vfs.repairContainment(): reconciles every VFS entity's containment
edges against canonical metadata.path — removes stale/duplicate vfs
edges, restores missing ones, never touches user knowledge edges. Wired
into repairIndex() so the one operator ritual heals existing ghosts.
- Regression: tests/integration/vfs-rename-containment.test.ts (7).
Persisted entity totals inflated permanently: a delete's count decrement
was sourced from re-reading the record being removed, so a null read
(replace race, or a ghost left by a pre-8.3.1 partial delete) silently
skipped the decrement while the paired add had counted — and
Math.max(persistedTotal, scanned) meant no disk cleanup could ever
lower the reported total. Reported by a production deployment with a
write→delete→re-create repro minting drift within hours.
- The caller's pre-delete read now rides through the entire delete path
(remove()/removeMany()/plan → DeleteNoun/DeleteVerb operations →
deleteNoun/deleteVerb → deleteNounMetadata/deleteVerbMetadata): a
null internal read falls back to the known prior record instead of
skipping the decrement. StorageAdapter.deleteNoun/deleteVerb gain an
optional priorMetadata parameter (additive).
- rebuildTypeCounts() rebuilt only the type-statistics arrays and
computed the total just to LOG it — the persisted scalar
(counts.json) survived every rebuild. One canonical walk now rebuilds
every counter rollup (scalar totals + per-type maps + type
statistics) and persists them; repairIndex() runs the recount
unconditionally, since counters can be inflated over clean shelves.
Two production-reported spine fixes:
- Canonical delete is a FULL removal: remove()/removeMany() deleted the
metadata leg but never the canonical vectors.json leg or the <id>/
directory, leaving ghost rows that inflated enumerated counts forever,
read as damage scars, and caused duplicate readdir entries for
re-created VFS paths (the stale-unpost path). The delete operation now
routes through storage.deleteNoun — both legs + the entity container —
with a full two-leg before-image rollback; deleteVerb symmetric; the
blind 'no metadata file' catch that masked real faults is gone. The
generation log still holds the delete's before-image, so asOf() history
is unchanged. repairIndex() gains a conservative, loud orphan-prune
sweep (containers with no metadata content leg) + count recompute for
stores damaged by earlier versions.
- Family-scoped migration gate: every read blocked on the whole-brain
migration lock even when the migrating index was irrelevant. The gate
now scopes to the index families a read actually consults — canonical
reads never wait; find() waits only on its query shape's families;
graph traversals wait only on the graph family. Writes keep the
conservative whole-brain wait. A read that needs the migrating family
still blocks (bounded) with the retryable MigrationInProgressError.
Consumer summary of the 8.3.0 minor: 16-way parallel canonical enumeration +
id-only getNounIdsWithPagination (index-heal speedup, standalone); the ADR-004
cross-layer integrity contract (validateInvariants delegation + repairIndex
native rebuild); and the registered-blob family contract (declared index blobs
undeletable). The two contract pieces are inert until a native provider
implements the matching hooks.
