open-brainy

Archived

Author	SHA1	Message	Date
David Snelling	51faefb084	feat(open): name the two steps that hold the vfs-bootstrap phase Some checks failed CI / Node 24 (push) Has been cancelled CI / Node 22 (push) Has been cancelled CI / Integration + conformance (Node 22) (push) Has been cancelled CI / Bun (latest) (push) Has been cancelled MEASURED on the personal-brain copy (bxl9000, gate-locked): the vfs-bootstrap phase costs 37.8s on main and 38.0s on this branch — unchanged — and NO "vfs.init" step line was emitted at all, meaning the VFS's own init fell under the 2s narration threshold. The phase is therefore almost entirely NOT the VFS, and the old-root sweep this branch moved to the background was never what made it expensive. What else lives in that span is now named: the log-authority artifact read, the adoption ORACLE (which verifies the log against canonical before flipping a brain to durable-at-ack), the legacy pending-embed sidecar bridge, and the pending-embed recovery fold. One of those holds ~38 seconds of every open of this store and the next measurement will say which, by name, instead of leaving a phase label to be guessed at.	2026-08-28 11:56:26 -07:00
David Snelling	ad7f885223	fix(storage): a dead flush watch falls back to the 500ms poll, not the 30s sweep Some checks failed CI / Integration + conformance (Node 22) (push) Has been cancelled CI / Node 24 (push) Has been cancelled CI / Node 22 (push) Has been cancelled CI / Bun (latest) (push) Has been cancelled The safety sweep is armed alongside the watch, and startFlushRequestPolling() declines to arm over an existing interval — so when a watch died mid-life the fallback did nothing and the store quietly answered flush requests on a 30s cadence instead of the 500ms one the door promises. The sweep is cleared first. A degrade nobody asked for is still a degrade.	2026-08-28 11:31:57 -07:00
David Snelling	f52053a3d3	fix(storage): the flush watcher cannot arm twice in its async window Some checks are pending CI / Node 22 (push) Waiting to run CI / Node 24 (push) Waiting to run CI / Integration + conformance (Node 22) (push) Waiting to run CI / Bun (latest) (push) Waiting to run Arming is asynchronous — the request directory is created before it can be watched — so during that window neither the watcher nor the sweep interval exists yet and the guard let a second call through, leaving two watchers and two sweeps for the life of the store. The callback is the flag that covers the window.	2026-08-28 11:30:00 -07:00
David Snelling	258176e6e6	perf(idle): the flush-request watch is event-driven; the heartbeat is observability Some checks are pending CI / Node 22 (push) Waiting to run CI / Node 24 (push) Waiting to run CI / Integration + conformance (Node 22) (push) Waiting to run CI / Bun (latest) (push) Waiting to run Three idle-burn items from the steady-state audit, and one correction. THE FLUSH-REQUEST WATCH (the strongest of them). It readdir'd the request directory every 500 ms, per brain, for the life of every writer — armed on every non-reader brain whether or not any inspector process existed. On a process holding 21 brains that is 42 directory reads per second on a completely idle service, plus a stale-request GC on every one of them. It now uses fs.watch, so the arrival itself wakes it and a request is seen SOONER than the poll saw it. Two concessions ride along, both stated in the code: a 30s safety sweep (fs.watch drops events on some network and fuse filesystems, and the GC needs a tick of its own — 0.7 reads/s across 21 brains where the poll cost 42), and a fall back to the original 500 ms poll, narrated, on a filesystem that cannot watch at all, because an inspector whose request is never seen waits forever. THE WRITER HEARTBEAT goes 10s → 60s. It is observability ONLY — staleness is decided by pid liveness and the fence compares pid + hostname, so no decision anywhere reads the timestamp — and at 10s it was a lock-file write every ten seconds per brain forever, for a value nothing computes with. An operator still sees a heartbeat inside the minute. THE HEALTH NARRATION dedupes by CONTENT, not by the provider's generation counter. That counter bumps on every ledger mutation and rebuild boundary, so a provider bumping it on routine work re-emitted the same unchanged line on every read, while one that never bumped could suppress a line whose reasons had genuinely changed. The generation is still reported; it no longer decides whether the line is worth saying. CORRECTION, and it is against my own earlier claim: the idle-flush commit read the field's "21 brains, no writes, a flush every ~35s, 1.26 cores" as caused by the flush path. That does not follow — this engine's cadence is write-driven (every trigger runs through noteWriteForPersistence, which only a committed write calls), so something was CALLING flush() on those brains and the caller is still unidentified. The clean-flush gate makes such a call free; it does not account for it. The code comments and the idle lane now say exactly that. Pins: tests/integration/flush-watcher-event-driven.test.ts — an idle writer makes at most one request-directory read in 8 seconds (the old poll made ~16), and a dropped request is still acked well inside the safety sweep.	2026-08-28 11:25:47 -07:00
David Snelling	068842cf7f	perf(open): answer "are there any entities?" with one directory read Some checks failed CI / Node 22 (push) Has been cancelled CI / Node 24 (push) Has been cancelled CI / Integration + conformance (Node 22) (push) Has been cancelled CI / Bun (latest) (push) Has been cancelled The 7.x-to-8.0 layout probe runs on the open path of every store that does not yet carry its completion marker — a restore, a store built by an older release — and asked whether the canonical tree holds anything by LISTING it: a recursive walk of every file in every entity directory, to learn a boolean. It now asks the one-level door added for generation discovery, falling back to the listing on an adapter that lacks it. Also files a defect found while ratifying the operator set: the VFS builds its path-prefix filter as `$startsWith`, an operator no engine spelling accepts, so vfs.searchFiles({ path }) throws INVALID_QUERY on every call that passes a path. Pre-existing, unrelated to the operator work, and left as a filing — a path-prefix search needs a design answer, not a spelling correction.	2026-08-28 11:13:24 -07:00
David Snelling	9ada83c58b	perf(generations): discover generations by directory name, not by walking the log Some checks are pending CI / Node 22 (push) Waiting to run CI / Node 24 (push) Waiting to run CI / Integration + conformance (Node 22) (push) Waiting to run CI / Bun (latest) (push) Waiting to run MEASURED on a copy of a real store (14,056 nouns / 72,679 verbs, an 11 GB generation history): the generation-store phase cost 55,538 ms of a WARM REOPEN after a clean close — with the fold correctly skipped, so nothing in that phase's name explained it. This is what it was doing. Discovering which generations exist on disk called listRawObjects('_generations'), which RECURSES the whole tree and returns every file in every generation directory — to extract a set of integers that the top-level directory NAMES already spell out. The cost scales with the entire history, is paid on every open, warm or cold, and grows for the life of the store. A one-level door — listRawPrefixes(prefix), the immediate child directory names — is added to the storage seam. The filesystem adapter answers it with a single readdir; BaseStorage derives it from the recursive listing, so an adapter without a cheap implementation is never wrong, only never faster; and the generation store falls back to the old listing when the door is absent. One behavioural difference, stated: an EMPTY generation directory is now discovered where the file listing could not see it. Above the committed watermark that is a crash scar, and recovery already has an explicit branch for it ("indeterminate partial dir" — dropped, narrated). Below it, it becomes a resolvable generation holding no records, which is what an empty generation means. Suites: the durability kill matrix (15), db-mvcc (30), history repacking (4), rollback trapdoor (3), entity-tree stamp (4) and the full unit suite (2,105) all green.	2026-08-28 11:09:05 -07:00
David Snelling	0304350a41	fix(flush): clear() and repairIndex() set the dirty witness themselves Some checks are pending CI / Node 22 (push) Waiting to run CI / Node 24 (push) Waiting to run CI / Integration + conformance (Node 22) (push) Waiting to run CI / Bun (latest) (push) Waiting to run Both mutate durable state outside the two commit paths, so neither was seen by the flush witness added with the idle-flush law. A clear() followed by a flush() would have found the brain "clean" and skipped the entity-tree stamp, leaving a stamp describing the population the clear had just removed — a false divergence warning at the next open. Closing the gap where it is, rather than widening the witness to guess.	2026-08-28 11:06:06 -07:00
David Snelling	bc1555feb8	feat(open): the open names the STEP that cost the time, not just the phase Some checks are pending CI / Node 22 (push) Waiting to run CI / Node 24 (push) Waiting to run CI / Integration + conformance (Node 22) (push) Waiting to run CI / Bun (latest) (push) Waiting to run A phase that costs a minute and names only itself tells an operator where to look but not what to look at. MEASURED on a real 14,056-noun / 72,679-verb store, the warm reopen's generation-store phase cost 55,538 ms with nothing inside it named — the fold was skipped (the close was clean), so the cost was somewhere else entirely and the breakdown could not say where. Six steps inside the open now report their own wall with their own cause when they exceed the phase threshold: the generation store's open (manifest, committed ranges, fact log, packed tier, crash replay), the entity-tree stamp verification, the brain-format read, the pre-upgrade backup, the derived-index gate, and the VFS init. Silent under the threshold, so a fast open says nothing extra. Same always-visible channel as the phase lines.	2026-08-28 11:02:57 -07:00
David Snelling	a51e9654fc	perf(vfs): the old-root sweep runs once per store, not once per open Some checks are pending CI / Node 22 (push) Waiting to run CI / Node 24 (push) Waiting to run CI / Integration + conformance (Node 22) (push) Waiting to run CI / Bun (latest) (push) Waiting to run MEASURED on a 14,056-noun / 72,679-verb store (bxl9000, a copy of a real brain): the vfs-bootstrap phase cost 43,021 ms of a cold open and 52,696 ms of a WARM REOPEN. What dominates it is a migration sweep — a filtered find() over the whole store hunting for root directories created before the fixed root id existed. A store either carries such duplicates or never will, and the sweep ran on every open, forever, in the foreground. It is now caused by the store's state instead of by the open count: a durable marker under _system/ records that the sweep has run, and a store carrying it never sweeps again. A store without one sweeps in the BACKGROUND — the sweep only removes duplicate roots, nothing serves from them, and it was already declared non-critical — narrated at both ends, with whenRootSweepSettled() for anyone who needs to observe rather than race it. An adapter with no raw-object door keeps the old behaviour: correctness over cost, never a silent skip. Pins: tests/integration/vfs-root-sweep-once.test.ts — the sweep runs on the first open and never on the second or third; a sweep slowed to 4s does not delay the open.	2026-08-28 11:01:43 -07:00
David Snelling	c66080acca	chore: keep the generated neural stamps at main's values Some checks are pending CI / Node 22 (push) Waiting to run CI / Node 24 (push) Waiting to run CI / Integration + conformance (Node 22) (push) Waiting to run CI / Bun (latest) (push) Waiting to run The build regenerates these from the git commit time; a local rebuild moved only the stamp. Restored so the branch carries no incidental churn.	2026-08-28 10:58:00 -07:00
David Snelling	cd4fe799e7	feat(contract): declare contract 1, serve three operators, refuse four by name Some checks are pending CI / Node 22 (push) Waiting to run CI / Node 24 (push) Waiting to run CI / Integration + conformance (Node 22) (push) Waiting to run CI / Bun (latest) (push) Waiting to run Open Brainy's side of the API contract the accelerated engine published. DECLARED: package.json carries "brainyContract": 1 and the engine states its own via contractVersion() / BRAINY_CONTRACT_VERSION — two engines compare an integer instead of probing prototypes, and a tool reads the package field without importing the engine. Pinned so the two can never drift apart. SERVED: hasAll, noneOf and excludes now work on the index path. The defect underneath was worse than the reported divergence — the metadata index's operator switch had NO DEFAULT CASE, so any operator without a case left the field's match set at its initial [] and find() returned an empty page. Documented, validator-accepted, matcher-implemented operators answering silently wrong. hasAll intersects each element's posting set (an empty operand is vacuously true of every row that has the field), noneOf complements their union, excludes complements contains. REFUSED BY NAME: startsWith, endsWith, matches and length raise INVALID_QUERY naming the operator, the field and the reason. An equality/range posting index cannot evaluate a substring, a pattern or an array length without reading every row — which is the cost this path exists to avoid — so it refuses rather than answering an empty page. Both engines now agree on all 25 tokens and contract 1 has no remaining operator divergence. This is a visible change for a consumer calling those four through find({ where }): an empty page becomes a typed refusal. EMITTED: scripts/emit-contract-manifest.mjs generates docs/api-contract.json from the BUILT surface — prototype doors, exported error classes, the operator sets read out of their single definitions, the field-addressing vocabulary, the health verdicts. Nothing hand-maintained, so a diff between two manifests is a diff between two engines. `--check` fails on a stale manifest, which makes the announce-every-addition duty mechanical rather than remembered. RATIFIED in docs/contract-1-ratification.md: the 41-of-57 required split with the promise spelled out (a refusal is part of a door; deprecation is not removal), the serving-withholding list confirmed exhaustive and identical, the minor/major rule adopted with the announcement duty, the 30 storage seam methods committed as supported surface until Stage 2, and a finding filed against the spec — is / isNot / greaterEqual / lessEqual are listed there as served aliases and have never existed in this engine, which throws INVALID_QUERY on all four.	2026-08-28 10:57:43 -07:00
David Snelling	096b1b102e	fix(open): a provider rebuilding itself is a third state, not a CRITICAL Some checks are pending CI / Node 22 (push) Waiting to run CI / Node 24 (push) Waiting to run CI / Integration + conformance (Node 22) (push) Waiting to run CI / Bun (latest) (push) Waiting to run Follower to the self-rebuild deference. The open gate's consistency check — "metadata index has 0 entries but storage has N entities" → CRITICAL + a forced second rebuild — knew two states, migrating and not. A provider whose rebuild() returns once the rebuild is OWNED AND RUNNING ONLINE (its doors refusing by name while other families serve) legitimately reports 0 entries there, so every first contact printed a false CRITICAL and kicked a redundant second rebuild. The exemption rides the rebuild-progress hook, NOT isMigrating() — widening that would hold every write and 503 the whole brain through the migration snapshot, which is worse than the false alarm. The check's real class is untouched: a provider reporting 0 entries with no rebuild in progress still trips it. The crash-recovery rebuild kick gets the same deference: a provider already rebuilding itself from canonical is doing exactly that work, and the fold ran in the generation store's open before any provider existed, so what it is reading is the repaired canonical. Pin: a provider stub reporting a rebuild and 0 entries opens with no CRITICAL line and no second rebuild; the vacuous-stub case fails loudly.	2026-08-28 10:50:26 -07:00
David Snelling	148ae92413	feat(open): open never waits for a provider that is rebuilding itself Some checks are pending CI / Node 22 (push) Waiting to run CI / Node 24 (push) Waiting to run CI / Integration + conformance (Node 22) (push) Waiting to run CI / Bun (latest) (push) Waiting to run MEASURED on a production store: a metadata provider that had to rebuild made init() pay the ENTIRE rebuild on the foreground — 641 seconds — with every other family idle behind it. The cause is a missing distinction: a provider reporting serving:false because it is BUSY BUILDING ITSELF and one reporting serving:false because it is BROKEN looked identical through healthReport(), and both were answered the same way — call rebuild(), and wait for it. The contract that tells them apart is one optional, synchronous, O(1) hook: `rebuildInProgress(): ProviderRebuildProgress \| null`, reporting a phase name and whatever the provider actually measures (done/total/startedAt) — never an estimate dressed as a fact. A provider without the hook behaves exactly as before. With it, a provider owns its own rebuild: - the open gate neither starts a second rebuild nor waits for the provider's, and narrates that it is not waiting and what will refuse meanwhile; - init() returns and every other family serves; - that family's doors refuse BY NAME, carrying the provider's own progress, and say plainly that the door opens by itself and no action is needed — distinct from a broken index, which names repairIndex(); - the epoch stamp does not advance while any family is still being built. Nothing is ever served empty: a not-serving family refuses, as it already did. Pins: tests/integration/open-does-not-wait-for-a-rebuilding-provider.test.ts — init() returns in milliseconds against a provider claiming a 6s rebuild, brainy starts no rebuild of its own, a filtered read refuses naming the phase and the 4,096/14,056 progress, and the door answers once the provider reports serving. The pin fails loudly rather than vacuously if its stub never installs.	2026-08-28 10:48:52 -07:00
David Snelling	ebf553ee51	perf(flush): an idle brain does no work — no periodic flush without a write Some checks are pending CI / Node 22 (push) Waiting to run CI / Node 24 (push) Waiting to run CI / Integration + conformance (Node 22) (push) Waiting to run CI / Bun (latest) (push) Waiting to run MEASURED on a production process holding 21 brains: with no writes for ten minutes it printed "All indexes flushed to disk in 216-601ms" per brain every ~35 seconds and idled at 1.26 cores. Every one of those flushes re-persisted state identical to what was already on disk — the provider flushes, the watermark stamps, the generation counter, the entity-tree stamp — because flush() never asked whether anything had changed. - flush() over a clean brain is now O(1) and silent: a dirty witness is set by every committed write (both commit paths end at noteWriteForPersistence, and the deferred-embed worker lands through the single-op path) and cleared by a flush that runs. A write landing DURING a flush sets it again, so no write's work is ever skipped — it is done by the next flush. Set before the policy check, so a `'manual'` consumer's explicit flush is never a no-op it didn't ask for. - An explicit flush now tells the cadence it happened. It didn't, so the very next write saw "30s since the last flush" and kicked a background flush with nothing to do, and the idle timer fired two seconds later over writes the explicit flush had already persisted. - The graph adjacency index's auto-flush asks before it acts: two O(1) reads of the LSM MemTables, and a tick over a quiet index returns without calling into the trees at all. assessProviderHealth is NOT timer-driven — it is a synchronous O(1) read of a provider's own healthReport(), called on the read gate, so it costs nothing on an idle brain. No change needed there. Pins: tests/integration/idle-costs-nothing.test.ts — 90 idle seconds produce zero flushes, zero provider calls and zero log lines; three explicit flushes over a clean brain call no provider; one write earns exactly one flush.	2026-08-28 10:44:38 -07:00

1 changed files with 0 additions and 7 deletions

									
										7

CONTRIBUTING.md
									
										View file
										
				@ -57,13 +57,6 @@ see `package.json` for `test:integration`, `test:coverage`, and friends.

				  description states a number, cite the benchmark that produced it (see

				  [docs/performance-envelopes.md](docs/performance-envelopes.md) for the

				  pattern). Don't state an estimate as if it were measured.

				- **Measurements carry numbers, not provenance.** Public commit messages and

				  docs give the SHAPE a number was taken at and never where it was taken: no

				  hostnames, no store or deployment identities, no operational anecdotes about

				  someone's running system. "A 14,056-noun / 72,679-verb production-shaped

				  store, measured solo under an exclusive lock" tells a reader everything the

				  number depends on; the machine it ran on and whose data it was tell them

				  nothing except where somebody's infrastructure lives.

				## License

Compare commits

14 commits

c8e05189e3 ... 51faefb084

7

CONTRIBUTING.md

View file

Compare commits

14 commits c8e05189e3 ... 51faefb084

7 CONTRIBUTING.md Unescape Escape View file

14 commits

c8e05189e3 ... 51faefb084

7

CONTRIBUTING.md

View file