diff --git a/.forgejo/workflows/publish-source.yml b/.forgejo/workflows/publish-source.yml index 6bd42b2a..58cb1d30 100644 --- a/.forgejo/workflows/publish-source.yml +++ b/.forgejo/workflows/publish-source.yml @@ -12,11 +12,6 @@ on: push: tags: - 'v*' - workflow_dispatch: - inputs: - ref_reason: - description: 'why this manual run (e.g. tag event dropped)' - required: false jobs: publish: diff --git a/CHANGELOG.md b/CHANGELOG.md index c4f89332..62d81cfb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,28 +2,6 @@ All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines. - -### [10.4.13](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.12...v10.4.13) (2026-09-03) - -- A shutdown that holds its listener until the exit decision, and a test suite that closes every brain it opens -- fix(shutdown): the engine's signal handler keeps its listener registered until the exit decision is made — closing the last live instance no longer deregisters the handler mid-run, so a second signal delivery during a clean shutdown can never kill the process after the work is done (a2ea21b3) -- fix(release): the release wall entry commits under an explicit git identity read from the developer's checkout; a host with no identity refuses by name instead of failing inside git (aac853d3) -- test(hygiene): every brain a test file creates is closed by that file — 40 files fixed, the leaks that let a stray cadence narrate into later files are gone; brains whose init() was expected to fail are closed too (6eb5e448) - -### [10.4.12](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.11...v10.4.12) (2026-09-03) - -- Mixed-kind fields index exactly, arrays to 256, a drained loop is not a shutdown, and finds project from the column store -- fix(index): a metadata field holds every value kind it was written with — one posting column per (field, kind); an equality filter reads the query value's own kind, a range routes by its bounds; nothing is refused and nothing is silently dropped; an index written by the old shape opens unchanged (a128f0ed) -- fix(metadata): metadata arrays index up to 256 elements; a longer array refuses at write time by name (MetadataArrayTooLargeError) — a vector parked in metadata now throws; move it to `vector` (e435da78) -- fix(shutdown): beforeExit runs a non-closing flush only — a script that never calls close() exits with the writer lock on disk and no clean-shutdown marker, and the next open evicts the stale lock and folds the log, bounded; SIGTERM and SIGINT are unchanged (6baa4d7f) -- feat(find): field projection — find({fields}) and get({fields}) resolve scalars from the column store on every leg, including vector-leg finds; absent fields stay absent (ad0f493f) -- fix(find): orderBy is the order on every find path, not only the metadata-only one (5e720d17) -- fix(metadata): the legacy sparse range path orders values, or refuses by name — never ranks by hash (a7eb7f52) -- fix(close): a read-only brain writes nothing under `_system/` (f27a7776) -- fix(contract): the flush gate's internals are private, not doors (72c8ee6a) -- test(hygiene): the triple-intelligence correctness cases sit in the gate; the idle and connected-find pins name the brain they measure (28083981) -- ci(release): the rail writes its own wall entry into the shared releases repo — never hand-written again (adcb883e) - ### [10.4.11](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.9...v10.4.11) (2026-09-02) - ci: superseded pushes cancel their own runs (concurrency per ref) (6053f6d4) diff --git a/README.md b/README.md index fbf129ac..762c9ec3 100644 --- a/README.md +++ b/README.md @@ -4,11 +4,6 @@

Brainy

-> **Frozen at 10.4.13 (2026-09-03).** This repository is the reference implementation of the Brainy store format and API, -> published under the MIT license. Version 10.4.13 is its last release; the repository is read-only from here. The engine -> continues as `@soulcraft/brainy`, which bundles this layer as owned code; every published version of this package stays -> available on The Source. Use this repository to read a Brainy store independently or to verify the conformance contract. -

Three database paradigms. One API. Zero configuration.
The in-process knowledge database for TypeScript — vector search, graph traversal,
diff --git a/RELEASES.md b/RELEASES.md index 64e64873..e8833b80 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -1,15 +1,5 @@ # @soulcraft/brainy — Release Notes for Consumers -> **Frozen at 10.4.13 (2026-09-03).** 10.4.13 is the last release of `@soulcraftlabs/brainy`; this repository is read-only from here. -> Release notes for the product engine continue on its own wall. - -Machine-readable release notes are published at -https://source.soulcraft.com/soulcraftlabs/releases/raw/branch/main/open-brainy.json -(this engine) and -https://source.soulcraft.com/soulcraftlabs/releases/raw/branch/main/brainy.json -(the product engine) — read by HQ's `/hq/releases` door, and the source of -truth ahead of this file. - This file is the **quick reference for downstream sessions** tracking Brainy changes. Full auto-generated changelog: `CHANGELOG.md` · Releases: https://source.soulcraft.com/soulcraftlabs/open-brainy/releases diff --git a/docs/FIND_SYSTEM.md b/docs/FIND_SYSTEM.md index 6aa33515..77fbbd79 100644 --- a/docs/FIND_SYSTEM.md +++ b/docs/FIND_SYSTEM.md @@ -369,71 +369,6 @@ return results.slice(offset, offset + limit) // → Auto-correction: Use most likely alternative based on affinity data ``` -## Field Projection (`fields`) - -`find()` and `get()` accept a `fields` list. Without it they return the whole -record; with it they return only the fields you name — and, where the index can -supply them, without opening the canonical record at all. - -```ts -// A list page: two user fields and one engine scalar. No document bodies. -await brain.find({ - where: { kind: 'post' }, - fields: ['title', 'slug', 'system.createdAt'], - limit: 50 -}) - -await brain.get(id, { fields: ['title'] }) -``` - -### Why it exists - -A list view that renders a title and a date does not need the body, but without -a projection every row hydrates its full record and throws almost all of it -away. On a posts list that is the dominant cost of the query. - -### The rules - -| | | -|---|---| -| **`fields` absent** | The full record, byte-identical to before. Nothing changes. | -| **Field names** | The one addressing law: a bare name is user metadata (`'title'`), `system.*` is an engine scalar (`'system.createdAt'`). | -| **A field the row lacks** | Simply **absent** from the result. Never an error. | -| **Identity** | Every row keeps its `id` (and `score` on `find`) regardless — a row you cannot identify is not a row. | -| **Where values come from** | The **column store**, which holds raw values. Never the sparse index, which buckets timestamps for range queries. | -| **A field the column cannot serve** | The canonical record is read for that field only. Correct, just not free. | - -### Missing fields are absent, not errors - -This is deliberate and differs from `orderBy`, which throws -`UnresolvableFieldError` for an unknown field. A typo in `orderBy` silently -changes the ordering, so it must be loud. A projection asks "give me these if -you have them", and an optional field must not turn a list into a failure — so -`fields` uses the permissive path. - -### Cost - -When every named field is column-served, a projected page performs **zero** -canonical reads. When one is not, only that read happens and the rest still come -from the index. Both are pinned by counting reads rather than timing them, in -`tests/integration/find-fields-projection.test.ts`. - -### `related()` takes no `fields` - -A `Relation` carries `from` and `to` as **ids** and hydrates no entity record, -so there is nothing for a projection to trim. Projecting the endpoints would be -a new capability rather than a projection of an existing one. - -### For engine implementers - -Projection is served through an optional provider door, -`getScalarsForIds(ids, fields)` on `MetadataIndexProvider`. The contract is in -`src/plugin.ts`; the short version is **return only what you can serve exactly, -and say what you served**. The caller diffs the answer against the request and -reads records for the remainder, so omission costs a read while a wrong value is -a wrong answer nobody can see. An engine without the door still works — every -field falls back to the record. - ## Performance Characteristics ### Query Performance by Type diff --git a/docs/architecture/data-storage-architecture.md b/docs/architecture/data-storage-architecture.md index 12398747..83b9e23a 100644 --- a/docs/architecture/data-storage-architecture.md +++ b/docs/architecture/data-storage-architecture.md @@ -217,40 +217,6 @@ membership queries at scale: `__words__` for tokenized text…). - `_blobs/_column_index/{field}/L0-NNNNNN.bin` — the actual level-0 run segments, stored through the shared `_blobs/.bin` binary convention. -- `_column_index/{field}/k/{kind}/…` — the same two files again, for a - **second value kind** on the same field (see below). Absent for a field that - holds one kind, which is nearly all of them. - -### One posting column per (field, kind) - -A field is not obliged to hold one type of value. `category` may carry -`'electronics'` on some rows and `5` on others, and both are real values of -that field. A segment, though, has one encoding — i64, f64, UTF-8, or boolean -— so a field that holds several kinds gets **one column per kind**: - -- The first kind a field ever sees owns the plain `_column_index/{field}/` - layout above. A single-kind field is therefore byte-identical to what earlier - versions wrote, and an index written before typed postings opens unchanged. -- Every later kind gets its own column beside it at - `_column_index/{field}/k/{kind}/`, where `{kind}` is `number`, `string` or - `boolean`. - -What that buys at query time: - -| | | -|---|---| -| **Equality** | Answered from the column matching the **query value's own kind**. `where {category: 5}` reads the number postings; `where {category: '5'}` reads the string postings. Neither borrows the other's rows — a row written with the number `5` is not a row whose category is the text `'5'`. | -| **A kind the field never held** | Matches nothing. That is the true answer, not a coerced one. | -| **Ranges** | Routed by the kind of the bounds: numeric bounds read the numeric postings and ignore the field's strings. An **unbounded** range is the "has any value here" probe behind `exists`, and reads every kind. | -| **`orderBy`** | A number and a string have no order between them, so a mixed field orders by kind first (number, string, boolean) and by value within a kind. A single-kind field sorts exactly as it always did. | -| **Numbers** | One kind, one column: an integer column is written as i64 and widens to f64 the first time a non-integer arrives, so `4.5` is stored as itself rather than rounded. | - -`null` and `undefined` are not kinds and are never posted; their absence is -what the `exists` / `missing` operators read. - -Older readers are unaffected by the additional columns: they see the field's -primary column exactly where it has always been, and a `k/{kind}` directory is -simply a name they never query. Sparse per-field indexes, roaring-bitmap chunks, and zone-map/bloom segments additionally live as bucketed keys under `_system/idx/` (see §3). Which path diff --git a/docs/concepts/multi-process.md b/docs/concepts/multi-process.md index d698eee8..8fda315f 100644 --- a/docs/concepts/multi-process.md +++ b/docs/concepts/multi-process.md @@ -95,15 +95,8 @@ The heartbeat interval rewrites the lock file every 10 seconds. The timer is unref'd, so it does not keep the event loop alive on its own. On normal shutdown the writer releases the lock in `close()`. The shutdown -hooks Brainy registers for `SIGTERM` and `SIGINT` close every live brain by -that same `close()`, so a container restart doesn't strand the directory. - -`beforeExit` is not one of them. Node emits it whenever the event loop has -no ref'd work left — a state a healthy script reaches routinely, because -Brainy's own idle and cadence timers are unref'd — and a drained event loop -is not a shutdown. That hook only persists derived state with a non-closing -`flush()`: it closes nothing, releases no lock, and leaves every brain open -and usable. If you want a shutdown, call `close()` or send `SIGTERM`. +hooks Brainy registers for `SIGTERM`, `SIGINT`, and `beforeExit` also +release the lock so a container restart doesn't strand the directory. ## How to inspect a live writer diff --git a/package-lock.json b/package-lock.json index bd12e46c..3e3bf96d 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@soulcraftlabs/brainy", - "version": "10.4.13", + "version": "10.4.11", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@soulcraftlabs/brainy", - "version": "10.4.13", + "version": "10.4.11", "license": "MIT", "dependencies": { "@msgpack/msgpack": "^3.1.2", diff --git a/package.json b/package.json index a3bd0483..8676f8b7 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@soulcraftlabs/brainy", - "version": "10.4.13", + "version": "10.4.11", "brainyContract": 1, "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.", "main": "dist/index.js", diff --git a/scripts/release.sh b/scripts/release.sh index a9a1f6e9..1a4fe575 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -154,26 +154,13 @@ else fi # Create new changelog entry -RELEASE_DATE=$(date +%Y-%m-%d) -CHANGELOG_ENTRY="### [${NEW_VERSION}](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) (${RELEASE_DATE}) +CHANGELOG_ENTRY="### [${NEW_VERSION}](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d)) ${COMMITS} " -# A CURATED entry wins over the generated one. When a release is cut from a -# lineage that diverged from the previous tag (a candidate branch carrying -# main's history), `git log ..HEAD` lists every commit the tag never -# saw — old notes, already-shipped fixes under new hashes, merge commits — and a -# wall entry derived from it would misreport the release. If CHANGELOG.md -# already carries a `### [NEW_VERSION]` heading, it was written on purpose: -# keep it, and skip the generated prepend entirely. -CURATED_ENTRY=false -if grep -qE "^### \[${NEW_VERSION}\]" CHANGELOG.md 2>/dev/null; then - CURATED_ENTRY=true - echo -e "${YELLOW}CHANGELOG already carries a curated ### [${NEW_VERSION}] entry — keeping it, not generating one from commits${NC}" -fi # Prepend to CHANGELOG.md after header -if [ "$CURATED_ENTRY" = false ] && [ -f "CHANGELOG.md" ]; then +if [ -f "CHANGELOG.md" ]; then # Read header (first 4 lines) HEADER=$(head -n 4 CHANGELOG.md) # Read rest of file @@ -187,19 +174,6 @@ if [ "$CURATED_ENTRY" = false ] && [ -f "CHANGELOG.md" ]; then fi echo -e "${GREEN}✅ CHANGELOG updated${NC}\n" -# Step 6b: Update the releases wall entry — mechanical, derived from the -# CHANGELOG entry just composed. The fleet's HQ page reads open-brainy.json -# from the one shared releases repo, soulcraftlabs/releases on The Source — -# this used to be hand-written after every release (David: never again — -# make it a step of the rail, landed in the one shared home; this repo no -# longer hosts its own copy). This step clones/fetches that repo into a -# local cache, prepends the entry, and pushes it directly — a real -# cross-repo push, refusing loudly (never skipping) on any -# clone/validation/commit/push failure. -echo -e "${BLUE}5️⃣▸ Updating the releases wall...${NC}" -node scripts/wall-entry.mjs --product open-brainy --version "${NEW_VERSION}" --date "${RELEASE_DATE}" --from-changelog CHANGELOG.md -echo -e "${GREEN}✅ Releases wall updated${NC}\n" - # Step 7: Create release commit echo -e "${BLUE}6️⃣ Creating release commit...${NC}" git add package.json package-lock.json CHANGELOG.md @@ -263,7 +237,7 @@ fi # and RELEASES.md are the record; this just gives The Source's UI a release page). echo -e "${BLUE}🔟 Creating release page on The Source...${NC}" if [ -n "${FORGEJO_RELEASE_TOKEN:-}" ]; then - if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraftlabs/open-brainy/releases" \ + if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraft/brainy/releases" \ -H "Authorization: token ${FORGEJO_RELEASE_TOKEN}" -H "Content-Type: application/json" \ -d "{\"tag_name\":\"v${NEW_VERSION}\",\"name\":\"v${NEW_VERSION}\",\"prerelease\":${PRERELEASE}}" >/dev/null; then echo -e "${GREEN}✅ Release page created on The Source${NC}\n" diff --git a/scripts/wall-entry.mjs b/scripts/wall-entry.mjs deleted file mode 100644 index 5079cf86..00000000 --- a/scripts/wall-entry.mjs +++ /dev/null @@ -1,539 +0,0 @@ -#!/usr/bin/env node -/** - * @module scripts/wall-entry - * @description The releases-wall entry, made mechanical. The fleet's HQ page - * reads one public JSON per product from the ONE releases repo on The Source - * (soulcraftlabs/releases, files .json at its root — shape - * {product, entries:[{version, date, headline, items, url, thumb?}]}), at - * https://source.soulcraft.com/soulcraftlabs/releases/raw/branch/main/.json. - * Those entries were hand-written after every release, then briefly written - * into this repo's own releases/.json; this script is the one door - * that composes an entry and lands it in the shared repo, so it is never - * hand-written and never forked across repos again. - * - * Two modes: - * - * 1. Generate + publish (default): - * node wall-entry.mjs --product

--version --date \ - * --from-changelog - * Derives an entry from the CHANGELOG.md entry for (headline = the - * entry's first bullet, items = every bullet, trimmed of its trailing - * commit hash), then: - * - clones (or, if a cached clone already exists, fetches and resets) - * the releases repo into a local cache directory, - * - prepends the entry to /

.json, newest first — replacing - * any existing entry for the same version so a re-run is idempotent, - * - validates the file's shape before and after, - * - commits the change as "chore(wall):

" and pushes main. - * A failure at any step (clone, validation, commit, push, a - * non-fast-forward remote) exits non-zero naming the cure. Nothing is - * ever skipped — the wall either lands correctly or the release fails. - * - * 2. Dry run: - * node wall-entry.mjs --dry-run --product

--version \ - * --date --from-changelog - * Derives the entry exactly as above and prints it, along with the file - * it would be written to, but touches no clone and no remote — usable - * from a fresh checkout with no cache and no network. - * - * 3. Validate only (--check): - * node wall-entry.mjs --check --file - * Validates an arbitrary wall file's exact key set (top-level and - * per-entry), field types, and strict-descending semver ordering with - * no duplicates. Read-only; never writes. Exit 0 = clean, exit 1 = - * named violations printed to stderr. - * - * The remote and the local cache directory are each overridable - * (--remote / --cache-dir, or WALL_ENTRY_RELEASES_REMOTE / - * WALL_ENTRY_RELEASES_CACHE_DIR) so tests can point at a throwaway local - * bare repo and a throwaway cache directory — never the real remote or the - * real developer cache. - * - * No dependencies beyond the system `git` binary — CHANGELOG parsing, - * semver comparison, and JSON shape checking are all hand-rolled below. - */ - -import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs' -import { execFileSync } from 'node:child_process' -import { homedir } from 'node:os' -import { dirname, join } from 'node:path' - -const DEFAULT_REMOTE = 'git@source.soulcraft.com:soulcraftlabs/releases.git' - -/** @returns {string} */ -function defaultCacheDir() { - const base = process.env.XDG_CACHE_HOME || join(homedir(), '.cache') - return join(base, 'soulcraft-releases') -} - -// Required on every entry; "thumb" is optional (may be absent, or present as -// string | null) — matching the HQ contract's {..., thumb?}. -const ENTRY_REQUIRED_KEYS = ['version', 'date', 'headline', 'items', 'url'] -const ENTRY_OPTIONAL_KEYS = ['thumb'] -const ENTRY_ALLOWED_KEYS = [...ENTRY_REQUIRED_KEYS, ...ENTRY_OPTIONAL_KEYS] -const FILE_KEYS = ['product', 'entries'] - -// The public permalink pattern, by product. Every entry MUST carry an https -// permalink: HQ's parser rejects a wall whose entries carry url: null (the -// whole feed became unreadable on 2026-09-02). A product whose forge repo is -// private links its PUBLIC package page on The Source instead of a release -// page that would 404 for HQ's readers. -const RELEASE_URL_PATTERNS = { - 'open-brainy': (version) => `https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v${version}`, - 'brainy': (version) => `https://source.soulcraft.com/soulcraft/-/packages/npm/@soulcraft%2Fbrainy/${version}`, -} - -/** - * Parse argv into a flag map. `--flag value` sets a string; `--flag` alone - * (end of argv, or followed by another `--flag`) sets boolean true. - * @param {string[]} argv - * @returns {Record} - */ -function parseArgs(argv) { - /** @type {Record} */ - const args = {} - for (let i = 0; i < argv.length; i++) { - const a = argv[i] - if (!a.startsWith('--')) continue - const key = a.slice(2) - const next = argv[i + 1] - if (next === undefined || next.startsWith('--')) { - args[key] = true - } else { - args[key] = next - i++ - } - } - return args -} - -/** - * Print a loud, named error and exit 1. Every refusal in this script goes - * through here so the failure mode is always the same shape: "wall-entry: ". - * @param {string} message - * @returns {never} - */ -function fail(message) { - console.error(`wall-entry: ${message}`) - process.exit(1) -} - -/** - * @param {string} version - * @returns {{major: number, minor: number, patch: number, pre: string | null} | null} - */ -function parseSemver(version) { - const m = /^(\d+)\.(\d+)\.(\d+)(?:-([0-9A-Za-z.-]+))?$/.exec(version) - if (!m) return null - return { major: Number(m[1]), minor: Number(m[2]), patch: Number(m[3]), pre: m[4] ?? null } -} - -/** - * @param {string} a - * @param {string} b - * @returns {number} positive if a > b, negative if a < b, 0 if equal. - */ -function compareSemver(a, b) { - const pa = parseSemver(a) - const pb = parseSemver(b) - if (!pa || !pb) throw new Error(`cannot compare non-semver versions "${a}" vs "${b}"`) - if (pa.major !== pb.major) return pa.major - pb.major - if (pa.minor !== pb.minor) return pa.minor - pb.minor - if (pa.patch !== pb.patch) return pa.patch - pb.patch - if (pa.pre === pb.pre) return 0 - if (pa.pre === null) return 1 // a release outranks any prerelease of the same core version - if (pb.pre === null) return -1 - return pa.pre < pb.pre ? -1 : pa.pre > pb.pre ? 1 : 0 -} - -/** - * Validate a wall file's full shape: top-level keys ("product", "entries" — - * no more, no less), per-entry keys and field types ("thumb" optional), and - * strict-descending semver ordering with no duplicates. Collects every - * violation instead of failing on the first, so a caller reports the whole - * picture in one pass. - * @param {unknown} data - * @returns {string[]} Violation messages; empty means the file is clean. - */ -function validateShape(data) { - /** @type {string[]} */ - const errors = [] - - if (typeof data !== 'object' || data === null || Array.isArray(data)) { - return ['top level: expected a JSON object'] - } - const obj = /** @type {Record} */ (data) - - const topKeys = Object.keys(obj) - const missingTop = FILE_KEYS.filter((k) => !(k in obj)) - const extraTop = topKeys.filter((k) => !FILE_KEYS.includes(k)) - if (missingTop.length) errors.push(`top level: missing key(s) ${missingTop.join(', ')}`) - if (extraTop.length) errors.push(`top level: unexpected key(s) ${extraTop.join(', ')}`) - - if (typeof obj.product !== 'string' || obj.product.trim() === '') { - errors.push('top level: "product" must be a non-empty string') - } - if (!Array.isArray(obj.entries)) { - errors.push('top level: "entries" must be an array') - return errors // nothing further to check without an array - } - - const entries = /** @type {unknown[]} */ (obj.entries) - entries.forEach((rawEntry, i) => { - const label = `entries[${i}]` - if (typeof rawEntry !== 'object' || rawEntry === null || Array.isArray(rawEntry)) { - errors.push(`${label}: expected an object`) - return - } - const entry = /** @type {Record} */ (rawEntry) - const keys = Object.keys(entry) - const missing = ENTRY_REQUIRED_KEYS.filter((k) => !(k in entry)) - const extra = keys.filter((k) => !ENTRY_ALLOWED_KEYS.includes(k)) - if (missing.length) errors.push(`${label}: missing key(s) ${missing.join(', ')}`) - if (extra.length) errors.push(`${label}: unexpected key(s) ${extra.join(', ')}`) - - if (typeof entry.version !== 'string' || !parseSemver(entry.version)) { - errors.push(`${label}: "version" must be a semver string (got ${JSON.stringify(entry.version)})`) - } - if (typeof entry.date !== 'string' || !/^\d{4}-\d{2}-\d{2}$/.test(entry.date) || Number.isNaN(Date.parse(entry.date))) { - errors.push(`${label}: "date" must be a YYYY-MM-DD string (got ${JSON.stringify(entry.date)})`) - } - if (typeof entry.headline !== 'string' || entry.headline.trim() === '') { - errors.push(`${label}: "headline" must be a non-empty string`) - } - if (!Array.isArray(entry.items) || entry.items.length === 0 || entry.items.some((it) => typeof it !== 'string' || it.trim() === '')) { - errors.push(`${label}: "items" must be a non-empty array of non-empty strings`) - } - if (typeof entry.url !== 'string' || !/^https:\/\/\S+$/.test(entry.url)) { - errors.push(`${label}: "url" must be an https permalink — never null; HQ's parser rejects the whole feed`) - } - if ('thumb' in entry && !(entry.thumb === null || typeof entry.thumb === 'string')) { - errors.push(`${label}: "thumb" must be a string or null when present`) - } - }) - - // Ordering: newest first, strictly descending, no duplicate versions — - // checked only over entries whose version parsed (a bad version is - // already reported above; comparing it too would just be noise). - const versioned = entries - .map((e, i) => ({ i, version: /** @type {any} */ (e)?.version })) - .filter((e) => typeof e.version === 'string' && parseSemver(e.version)) - for (let i = 0; i < versioned.length - 1; i++) { - const a = versioned[i] - const b = versioned[i + 1] - const cmp = compareSemver(a.version, b.version) - if (cmp === 0) { - errors.push(`entries[${a.i}] and entries[${b.i}]: duplicate version ${a.version}`) - } else if (cmp < 0) { - errors.push(`entries[${a.i}] (${a.version}) sits above entries[${b.i}] (${b.version}) — not newest-first`) - } - } - - return errors -} - -/** - * Extract one version's entry body from a standard-version-style CHANGELOG.md - * (headings `### [version](url) (date)`, followed by `- bullet (hash)` lines - * until the next heading or EOF). - * @param {string} changelog - * @param {string} version - * @returns {string[]} Bullet lines, trimmed of their leading "- " and - * trailing " (hash)". - */ -function extractChangelogBullets(changelog, version) { - const lines = changelog.split('\n') - const headingRe = /^### \[([^\]]+)\]\(.*\)\s*\(\d{4}-\d{2}-\d{2}\)\s*$/ - let start = -1 - for (let i = 0; i < lines.length; i++) { - const m = headingRe.exec(lines[i]) - if (m && m[1] === version) { - start = i + 1 - break - } - } - if (start === -1) { - fail( - `version ${version} has no CHANGELOG entry yet — run this after the CHANGELOG step composes "### [${version}]", not before`, - ) - } - /** @type {string[]} */ - const bullets = [] - for (let i = start; i < lines.length; i++) { - if (headingRe.test(lines[i])) break // next entry starts - const bulletMatch = /^- (.+?)(?:\s\(([0-9a-f]{6,40})\))?$/.exec(lines[i].trim()) - if (lines[i].trim().startsWith('- ') && bulletMatch) { - const text = bulletMatch[1].trim() - if (text) bullets.push(text) - } - } - if (bullets.length === 0) { - fail(`version ${version}'s CHANGELOG entry has no bullets to derive a headline/items from`) - } - return bullets -} - -/** - * Derive a wall entry from a CHANGELOG.md. - * @param {{product: string, version: string, date: string, changelogPath: string, url?: string, thumb?: string | null}} opts - * @returns {{version: string, date: string, headline: string, items: string[], url: string, thumb: string | null}} - */ -function deriveEntry({ product, version, date, changelogPath, url, thumb }) { - if (!parseSemver(version)) fail(`--version "${version}" is not a semver string`) - if (!/^\d{4}-\d{2}-\d{2}$/.test(date) || Number.isNaN(Date.parse(date))) { - fail(`--date "${date}" is not a YYYY-MM-DD date`) - } - if (!existsSync(changelogPath)) fail(`--from-changelog "${changelogPath}" does not exist`) - - const changelog = readFileSync(changelogPath, 'utf8') - const items = extractChangelogBullets(changelog, version) - const headline = items[0] - - const pattern = RELEASE_URL_PATTERNS[product] - if (url === undefined && pattern === undefined) { - throw new Error(`wall-entry: no permalink pattern for product "${product}" — add one to RELEASE_URL_PATTERNS or pass --url; entries never carry url: null`) - } - const resolvedUrl = url !== undefined ? url : pattern(version) - const resolvedThumb = thumb !== undefined ? thumb : null - - return { version, date, headline, items, url: resolvedUrl, thumb: resolvedThumb } -} - -/** - * Load and shape-validate a wall file. - * @param {string} filePath - * @returns {Record} - */ -function loadWallFile(filePath) { - if (!existsSync(filePath)) fail(`"${filePath}" does not exist`) - /** @type {unknown} */ - let data - try { - data = JSON.parse(readFileSync(filePath, 'utf8')) - } catch (err) { - fail(`"${filePath}" is not valid JSON: ${/** @type {Error} */ (err).message}`) - } - const errors = validateShape(data) - if (errors.length) { - fail(`"${filePath}" fails shape validation —\n ${errors.join('\n ')}`) - } - return /** @type {Record} */ (data) -} - -/** - * Run a git command, throwing an Error whose message is git's own stderr - * (trimmed) on failure — every caller wraps this to name the cure. - * @param {string[]} args - * @param {string} cwd - * @returns {string} stdout, trimmed. - */ -function git(args, cwd) { - try { - return execFileSync('git', args, { cwd, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }).trim() - } catch (err) { - const stderr = /** @type {any} */ (err).stderr - const message = (typeof stderr === 'string' && stderr.trim()) || /** @type {Error} */ (err).message - throw new Error(message) - } -} - -/** - * Resolve the git identity for the wall commit from the repository the rail - * is actually running in — the developer's own checkout (`process.cwd()`; - * `release.sh` invokes this script from the repo root with no `cd`), via - * git's normal config precedence (repo-local, then global, then system). - * Never guessed and never left to git's own "who are you?" prompt: a host - * with no configured identity anywhere (a bare CI box, say) must refuse - * loudly rather than have git manufacture a placeholder identity or hang. - * @returns {{name: string, email: string}} - */ -function resolveWallCommitIdentity() { - const repo = process.cwd() - let name = '' - let email = '' - try { - name = git(['config', 'user.name'], repo) - } catch { - name = '' - } - try { - email = git(['config', 'user.email'], repo) - } catch { - email = '' - } - if (!name || !email) { - fail('no git identity for the wall commit — set user.name/user.email') - } - return { name, email } -} - -/** - * Ensure a clean, up-to-date local clone of the releases repo at - * `cacheDir`, checked out on `main` — cloning fresh if `cacheDir` has no - * `.git`, otherwise fetching and hard-resetting onto `origin/main` (so a - * stray local commit or edit left by a previous failed run can never leak - * into the next one). - * @param {string} remote - * @param {string} cacheDir - */ -function ensureReleasesClone(remote, cacheDir) { - if (existsSync(join(cacheDir, '.git'))) { - try { - git(['remote', 'set-url', 'origin', remote], cacheDir) - git(['fetch', '--prune', 'origin'], cacheDir) - git(['checkout', 'main'], cacheDir) - git(['reset', '--hard', 'origin/main'], cacheDir) - git(['clean', '-fd'], cacheDir) - } catch (err) { - fail( - `cannot refresh the cached releases checkout at "${cacheDir}" from "${remote}" — ${/** @type {Error} */ (err).message}\n` + - ` cure: delete "${cacheDir}" and re-run so it re-clones from scratch, or confirm SSH access with "ssh -T git@source.soulcraft.com"`, - ) - } - return - } - - mkdirSync(dirname(cacheDir), { recursive: true }) - try { - git(['clone', remote, cacheDir], dirname(cacheDir)) - } catch (err) { - fail( - `cannot clone "${remote}" — ${/** @type {Error} */ (err).message}\n` + - ` cure: confirm SSH access with "ssh -T git@source.soulcraft.com" and that the soulcraftlabs/releases repo exists yet`, - ) - } - try { - git(['checkout', 'main'], cacheDir) - } catch (err) { - fail( - `cloned "${remote}" into "${cacheDir}" but could not check out "main" — ${/** @type {Error} */ (err).message}\n` + - ` cure: confirm the releases repo's default branch is named "main"`, - ) - } -} - -/** - * Prepend `entry` to the wall at `/.json`, replacing any - * existing entry for the same version (idempotent re-runs), validating - * before and after, committing, and pushing — or refusing loudly, naming - * the cure, at whichever step fails. - * @param {{version: string, date: string, headline: string, items: string[], url: string, thumb: string | null}} entry - * @param {string} product - * @param {string} remote - * @param {string} cacheDir - */ -function publishEntry(entry, product, remote, cacheDir) { - ensureReleasesClone(remote, cacheDir) - - const filePath = join(cacheDir, `${product}.json`) - if (!existsSync(filePath)) { - fail( - `"${filePath}" does not exist in the releases repo — cure: seed "${product}.json" at the repo root first (it must exist before any release rail can prepend to it)`, - ) - } - const wall = loadWallFile(filePath) - - if (wall.product !== product) { - fail(`"${filePath}" has product "${wall.product}", but --product "${product}" was given — refusing a cross-product write`) - } - - const replacing = wall.entries.some((e) => e.version === entry.version) - wall.entries = [entry, ...wall.entries.filter((e) => e.version !== entry.version)] - - const postErrors = validateShape(wall) - if (postErrors.length) { - fail(`the entry for ${entry.version} would leave "${filePath}" invalid —\n ${postErrors.join('\n ')}`) - } - - writeFileSync(filePath, JSON.stringify(wall, null, 2) + '\n', 'utf8') - - const status = git(['status', '--porcelain', '--', `${product}.json`], cacheDir) - if (status === '') { - console.log(`wall-entry: "${product}.json" already carries an identical entry for ${entry.version} — nothing to commit or push`) - return - } - - const identity = resolveWallCommitIdentity() - - try { - git(['add', `${product}.json`], cacheDir) - git( - ['-c', `user.name=${identity.name}`, '-c', `user.email=${identity.email}`, 'commit', '-m', `chore(wall): ${product} ${entry.version}`], - cacheDir, - ) - } catch (err) { - fail(`cannot commit the wall entry in "${cacheDir}" — ${/** @type {Error} */ (err).message}\n cure: inspect "${cacheDir}" by hand and re-run once its git state is clean`) - } - - try { - git(['push', 'origin', 'main'], cacheDir) - } catch (err) { - fail( - `push to "${remote}" failed (likely a non-fast-forward — another release landed on main first) — ${/** @type {Error} */ (err).message}\n` + - ` cure: re-run this release step; it re-fetches and resets onto the latest origin/main before retrying`, - ) - } - - const sha = git(['rev-parse', 'HEAD'], cacheDir) - console.log( - `wall-entry: ${replacing ? 'replaced' : 'wrote'} v${entry.version} in "${product}.json" (${wall.entries.length} entries, newest first) — pushed ${sha} to ${remote} main`, - ) -} - -function main() { - const args = parseArgs(process.argv.slice(2)) - - if (args.check) { - const filePath = /** @type {string | undefined} */ (args.file) - if (!filePath) fail('--check needs --file ') - const wall = loadWallFile(/** @type {string} */ (filePath)) - console.log(`wall-entry --check: "${filePath}" OK — product "${wall.product}", ${wall.entries.length} entries, newest-first, no duplicates`) - process.exit(0) - } - - // Generate mode (default, also covers --dry-run): --product, --version, - // --date, --from-changelog required. - const product = /** @type {string | undefined} */ (args.product) - const version = /** @type {string | undefined} */ (args.version) - const date = /** @type {string | undefined} */ (args.date) - const fromChangelog = /** @type {string | undefined} */ (args['from-changelog']) - - const missing = [] - if (!product) missing.push('--product') - if (!version) missing.push('--version') - if (!date) missing.push('--date') - if (!fromChangelog) missing.push('--from-changelog') - if (missing.length) { - fail( - `missing required flag(s): ${missing.join(', ')}\n` + - 'Usage:\n' + - ' wall-entry.mjs --product

--version --date --from-changelog [--dry-run]\n' + - ' wall-entry.mjs --check --file ', - ) - } - - const urlArg = args.url === true ? undefined : /** @type {string | undefined} */ (args.url) - const thumbArg = args.thumb === true ? undefined : /** @type {string | undefined} */ (args.thumb) - - const entry = deriveEntry({ - product: /** @type {string} */ (product), - version: /** @type {string} */ (version), - date: /** @type {string} */ (date), - changelogPath: /** @type {string} */ (fromChangelog), - url: urlArg, - thumb: thumbArg, - }) - - const remote = /** @type {string} */ (args.remote ?? process.env.WALL_ENTRY_RELEASES_REMOTE ?? DEFAULT_REMOTE) - const cacheDir = /** @type {string} */ (args['cache-dir'] ?? process.env.WALL_ENTRY_RELEASES_CACHE_DIR ?? defaultCacheDir()) - - if (args['dry-run']) { - console.log(`wall-entry --dry-run: would write to "${join(cacheDir, `${product}.json`)}" in ${remote} (main), pushed as "chore(wall): ${product} ${version}"`) - console.log(JSON.stringify(entry, null, 2)) - process.exit(0) - } - - publishEntry(entry, /** @type {string} */ (product), remote, cacheDir) -} - -main() diff --git a/src/brainy.ts b/src/brainy.ts index fc08f291..fe672305 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -531,36 +531,6 @@ export class Brainy implements BrainyInterface { private static sigintListener?: () => void private static beforeExitListener?: () => void - /** True while the `beforeExit` pass is running its flushes. Node re-emits - * 'beforeExit' after every loop drain and that pass schedules async work, so - * a second emit can arrive on top of the first; it returns instead of - * stacking a parallel pass. NOT a one-shot: every genuine drain still gets a - * flush. See {@link registerShutdownHooks}. */ - private static beforeExitFlushInFlight = false - - /** Whether the drained-event-loop notice has been printed for this - * registration cycle. Printed ONCE — `console.log` to a pipe is itself - * event-loop work, so narrating on every emit would keep the loop turning - * and narrate forever. Reset by {@link deregisterShutdownHooksIfIdle}. */ - private static beforeExitNarrated = false - - /** True for the entire duration of ONE `closeOnShutdown()` run (the - * signal-path handler in {@link registerShutdownHooks}) — from before it - * starts closing instances until after it has decided whether to exit. - * THE RACE THIS CLOSES: closing the LAST live instance calls - * `close()` → `deregisterShutdownHooksIfIdle()` synchronously, which - * removes `Brainy.sigtermListener` from `process` — while `closeOnShutdown` - * (that very listener's OWN still-running invocation) hasn't yet reached - * `exitIfSoleShutdownOwner()`'s `process.exit(0)`. In that window Node has - * NO registered SIGTERM listener, so a second/concurrent delivery of the - * same signal (a raced re-send, common on a loaded host) falls through to - * Node's default disposition and kills the process outright — the - * clean-shutdown work already finished, but the process never reports the - * 0 it earned. `deregisterShutdownHooksIfIdle()` checks this flag and - * defers; `closeOnShutdown()`'s `finally` re-runs the deregistration check - * once it is done, so the listener never actually leaks past its use. */ - private static shutdownSignalHandlerActive = false - /** Poll cadence (ms) for the migration LOCK when a provider exposes no * event-driven `whenMigrationComplete()` signal. See {@link awaitMigrationLock}. */ private static readonly MIGRATION_POLL_INTERVAL_MS = 250 @@ -2160,11 +2130,9 @@ export class Brainy implements BrainyInterface { * Critical for Cloud Run, Fargate, Lambda, and other containerized deployments. * * Handles: - * - SIGTERM: Graceful termination (Cloud Run, Fargate, Lambda) — CLOSES. - * - SIGINT: Ctrl+C (development/local testing) — CLOSES. - * - beforeExit: the event loop drained — FLUSHES, and closes NOTHING. A - * drained loop is not a shutdown; see {@link flushOnDrainedEventLoop}'s - * contract below. + * - SIGTERM: Graceful termination (Cloud Run, Fargate, Lambda) + * - SIGINT: Ctrl+C (development/local testing) + * - beforeExit: Node.js cleanup hook (fallback) * * NOTE: Registers globally (once for all instances) to avoid MaxListenersExceededWarning */ @@ -2213,174 +2181,51 @@ export class Brainy implements BrainyInterface { */ const closeOnShutdown = async () => { console.log('Shutdown signal received - flushing pending data...') - // HOLD THE LISTENER FOR THE WHOLE RUN. Closing the LAST live instance - // below calls close() → deregisterShutdownHooksIfIdle(), which removes - // Brainy's own SIGTERM/SIGINT listeners from `process` — synchronously, - // before THIS invocation has reached exitIfSoleShutdownOwner()'s - // process.exit(0). Left alone, that opens a window with no registered - // listener for the signal at all, so a second/concurrent delivery of - // the same signal (a raced re-send — not rare on a loaded host) falls - // through to Node's default disposition and kills the process outright - // AFTER the clean-shutdown work already finished, reporting a signal - // kill instead of the 0 the shutdown earned. Setting this flag makes - // deregisterShutdownHooksIfIdle() defer; the `finally` below re-checks - // it once this run is fully done — closeOnShutdown, not a nested - // close(), owns exactly when the listener actually comes off. - Brainy.shutdownSignalHandlerActive = true - try { - // DEFER ONE MACROTASK. A host application registers its own listener - // on the same signal, and Node runs listeners in registration order — - // ours is usually first, because the brain was opened before the - // host wired its shutdown. Yielding once lets every other listener - // for this signal run its synchronous prologue, so a host that calls - // close() gets to be the owner. It is only a courtesy, never the - // safety: close()'s own single-flight gate is what makes a lost race - // harmless. - await new Promise((resolve) => setImmediate(resolve)) + // DEFER ONE MACROTASK. A host application registers its own listener on + // the same signal, and Node runs listeners in registration order — ours + // is usually first, because the brain was opened before the host wired + // its shutdown. Yielding once lets every other listener for this signal + // run its synchronous prologue, so a host that calls close() gets to be + // the owner. It is only a courtesy, never the safety: close()'s own + // single-flight gate is what makes a lost race harmless. + await new Promise((resolve) => setImmediate(resolve)) - let closedCount = 0 - let deferredCount = 0 - let failedCount = 0 - // Snapshot: close() splices Brainy.instances while we iterate. - for (const instance of [...Brainy.instances]) { - if (!instance.initialized) continue - // SOMEONE ELSE OWNS THIS ONE. Not a flush, not a lock release, not a - // component close — nothing. Touching a brain whose close is running - // is the whole defect this handler was rewritten for. - if (instance.closed || instance._closeInFlight !== null) { - deferredCount++ - continue - } - try { - // Law 1: this try/catch is the isolation — the loop continues. - await instance.close() - closedCount++ - } catch (error) { - failedCount++ - console.error('Failed to close one Brainy instance on shutdown:', error) - } + let closedCount = 0 + let deferredCount = 0 + let failedCount = 0 + // Snapshot: close() splices Brainy.instances while we iterate. + for (const instance of [...Brainy.instances]) { + if (!instance.initialized) continue + // SOMEONE ELSE OWNS THIS ONE. Not a flush, not a lock release, not a + // component close — nothing. Touching a brain whose close is running + // is the whole defect this handler was rewritten for. + if (instance.closed || instance._closeInFlight !== null) { + deferredCount++ + continue } - if (closedCount > 0) { - console.log(`Flushed successfully (${closedCount} instance${closedCount > 1 ? 's' : ''})`) + try { + // Law 1: this try/catch is the isolation — the loop continues. + await instance.close() + closedCount++ + } catch (error) { + failedCount++ + console.error('Failed to close one Brainy instance on shutdown:', error) } - if (deferredCount > 0) { - console.log( - `${deferredCount} Brainy instance${deferredCount > 1 ? 's are' : ' is'} already ` + - `closing — left to the caller that owns that close.` - ) - } - if (failedCount > 0) { - console.error( - `${failedCount} Brainy instance${failedCount > 1 ? 's' : ''} did not complete shutdown — ` + - `their writer locks were released, but their next open will run crash recovery.` - ) - } - } finally { - // Release the hold and run the deferred check ourselves — the last - // close() above may have found the flag set and skipped its own - // deregistration, so nobody else will do this if we don't. - Brainy.shutdownSignalHandlerActive = false - Brainy.deregisterShutdownHooksIfIdle() } - } - - /** - * THE DRAINED-EVENT-LOOP PATH. A DRAINED LOOP IS NOT A SHUTDOWN. - * - * Node emits `'beforeExit'` whenever the event loop has no REF'd work - * left — NOT when the process is ending, and with no signal involved. A - * perfectly healthy script reaches that state routinely: this engine - * unref's its idle and cadence timers ("an idle brain costs nothing"), so - * a script awaiting anything those timers drive is, for that instant, - * a process with no ref'd work and an open brain. - * - * MEASURED on the 11.1 rehearsal lane against a copy of a real store: the - * `beforeExit` listener was wired to the SIGNAL path, so after the heal - * phase the log printed `Shutdown signal received - flushing pending - * data...` and `Flushed successfully (1 instance)` with NO signal ever - * sent, and the script's very next `add()` threw `Brainy instance is not - * initialized: it was closed via close(). Create a new instance.` The - * engine had closed a live brain out from under a running script. - * - * SO, THE LAW: this path NEVER closes, deregisters, tears down or - * force-exits anything, and never releases a writer lock. It runs - * `flush()` — the engine's own non-closing durability door — on each live - * brain, and leaves every one of them open and usable. - * - * WHY flush() AND NOT NOTHING. Each claim checked against the code it - * names: - * 1. IT CANNOT CLOSE ANYTHING. `flush()` → `_flushSteps()` persists - * DERIVED state only: the count ledger, the metadata/graph/vector - * projections, the generation counter, aggregation state, the - * entity-tree stamp. It closes no component, deactivates no plugin, - * touches neither `initialized` nor `closed`, and never calls - * `releaseWriterLock()` — the clean-shutdown marker is written by - * `generationStore.close()` alone, reached only from `close()`. - * 2. IT CANNOT RACE A LATER WRITE INTO CORRUPTION. A background flush - * concurrent with live writes is the engine's ORDINARY steady state: - * `noteWriteForPersistence()` kicks exactly this call off an unref'd - * timer on every busy brain. `flush()` is single-flight with one queued - * follow-up, and a write landing mid-flush re-sets the dirty witness, - * so its work is never lost — it belongs to the next flush. - * 3. IT CANNOT SPIN. `flush()` on a clean brain returns without touching a - * provider or scheduling I/O, so the second emit does no event-loop - * work and the process exits. That is also why the listener is NOT - * self-deregistered any more: a one-shot listener spent on a spurious - * mid-script drain leaves the genuine end-of-script drain with nothing. - * 4. A FAILED FLUSH IS SURVIVABLE AND LOUD. The write path is durable at - * ack via the fact log; derived state is rebuildable. A throw is - * reported per instance and the loop continues — exactly how - * `kickBackgroundFlush()` already treats the same failure. - * - * The one thing lost against a closing handler is the clean-shutdown - * marker for a script that opens a brain and never closes it: its next - * open folds the log. That is the correct trade — a missing marker costs - * a recovery fold, closing a live brain costs the caller its brain — and - * the narration below names the cure. - */ - const flushOnDrainedEventLoop = async () => { - // A second emit can land on top of the first (this pass schedules async - // work, the loop turns, the loop drains again). One pass at a time. - if (Brainy.beforeExitFlushInFlight) return - - // Step aside for anyone whose close is running or done — the same - // ownership rule the signal path follows. - const live = [...Brainy.instances].filter( - (instance) => instance.initialized && !instance.closed && instance._closeInFlight === null - ) - if (live.length === 0) return - - // ONCE per registration cycle: a `console.log` to a pipe is itself - // event-loop work, so narrating on every emit would keep the loop - // turning and narrate forever. - if (!Brainy.beforeExitNarrated) { - Brainy.beforeExitNarrated = true + if (closedCount > 0) { + console.log(`Flushed successfully (${closedCount} instance${closedCount > 1 ? 's' : ''})`) + } + if (deferredCount > 0) { console.log( - `[Brainy] event loop drained with ${live.length} brain${live.length > 1 ? 's' : ''} ` + - `open — persisting derived state; NOTHING was closed. A drained loop is not a ` + - `shutdown: call close() (or send SIGTERM) when you mean one.` + `${deferredCount} Brainy instance${deferredCount > 1 ? 's are' : ' is'} already ` + + `closing — left to the caller that owns that close.` ) } - - Brainy.beforeExitFlushInFlight = true - try { - for (const instance of live) { - try { - await instance.flush() - } catch (error) { - // Per-instance isolation, and never fatal: canonical data is - // durable at ack, so a failed derived-state flush costs the next - // open a rebuild — it must not cost this one its brain. - console.error( - '[Brainy] flush on a drained event loop failed for one open brain ' + - '(the brain stays open and usable; derived-state persistence retries at the ' + - 'next flush, and canonical data is unaffected):', - error - ) - } - } - } finally { - Brainy.beforeExitFlushInFlight = false + if (failedCount > 0) { + console.error( + `${failedCount} Brainy instance${failedCount > 1 ? 's' : ''} did not complete shutdown — ` + + `their writer locks were released, but their next open will run crash recovery.` + ) } } @@ -2409,14 +2254,6 @@ export class Brainy implements BrainyInterface { * last brain deregisters Brainy's own listeners — so a host application's * single remaining listener would look like `<= 1` and get force-exited * out of its own graceful shutdown, precisely the failure above. - * - * SIGNALS ONLY — NEVER `beforeExit`. The reasoning above is entirely about - * a signal Brainy has suppressed Node's default terminate behaviour for. - * `beforeExit` suppresses nothing: Node exits by itself once the loop is - * genuinely done, and the script that is still running when it fires is - * not shutting down at all. Calling this from that path would end a live - * script at exit code 0 mid-work. It is called from the two signal - * listeners below and from nowhere else. */ const exitIfSoleShutdownOwner = (ownersWhenSignalled: number): void => { if (ownersWhenSignalled <= 1) { @@ -2433,7 +2270,18 @@ export class Brainy implements BrainyInterface { await closeOnShutdown() exitIfSoleShutdownOwner(owners) } - Brainy.beforeExitListener = flushOnDrainedEventLoop + Brainy.beforeExitListener = async () => { + // Self-deregister FIRST: Node re-emits 'beforeExit' after every event- + // loop drain, and this flush schedules new async work — with the + // listener still attached, a script that never calls close() would spin + // flush → drain → flush forever and never exit. One flush, then the + // next drain finds no listener and the process exits. + if (Brainy.beforeExitListener) { + process.off('beforeExit', Brainy.beforeExitListener) + Brainy.beforeExitListener = undefined + } + await closeOnShutdown() + } process.on('SIGTERM', Brainy.sigtermListener) process.on('SIGINT', Brainy.sigintListener) process.on('beforeExit', Brainy.beforeExitListener) @@ -2444,17 +2292,9 @@ export class Brainy implements BrainyInterface { * script that closed every brain exits on its own — a library must never * keep its host process alive. Re-initializing later re-registers them * (the `shutdownHooksRegisteredGlobally` flag resets here). - * - * Deferred (not skipped — {@link closeOnShutdown}'s `finally` always - * re-checks) while a signal-path shutdown is actively running: that - * handler's OWN still-in-flight invocation is `Brainy.sigtermListener`, and - * removing it out from under itself — which closing the LAST instance here - * would otherwise do, synchronously, mid-run — would leave `process` with - * no listener for the signal for the remainder of that run. See - * {@link shutdownSignalHandlerActive}'s doc for the exact race this closes. */ private static deregisterShutdownHooksIfIdle(): void { - if (Brainy.instances.length > 0 || !Brainy.shutdownHooksRegisteredGlobally || Brainy.shutdownSignalHandlerActive) { + if (Brainy.instances.length > 0 || !Brainy.shutdownHooksRegisteredGlobally) { return } if (Brainy.sigtermListener) process.off('SIGTERM', Brainy.sigtermListener) @@ -2463,11 +2303,6 @@ export class Brainy implements BrainyInterface { Brainy.sigtermListener = undefined Brainy.sigintListener = undefined Brainy.beforeExitListener = undefined - // A later re-init is a fresh cycle: it may narrate its own drained-loop - // notice, and no pass of the previous cycle can still be running (the last - // close() drained the flush chain). - Brainy.beforeExitNarrated = false - Brainy.beforeExitFlushInFlight = false Brainy.shutdownHooksRegisteredGlobally = false } @@ -4241,16 +4076,6 @@ export class Brainy implements BrainyInterface { } // Route to metadata-only or full entity based on options - // A PROJECTED get goes through the same seam every list page uses, so a - // detail read of two scalars costs an index read rather than a record read. - // It is checked before `includeVectors` because the two are incompatible by - // construction: a projection returns the named fields, and a vector is not - // one of them unless it was named. - if (options?.fields !== undefined && options.fields.length > 0) { - const page = await this.#hydratePage([id], options.fields) - return page.get(id) ?? null - } - const includeVectors = options?.includeVectors ?? false // Default: metadata-only (fast) if (includeVectors) { @@ -4297,170 +4122,6 @@ export class Brainy implements BrainyInterface { * const children = childIds.map(id => childrenMap.get(id)).filter(Boolean) * ``` */ - /** - * **The projection seam** — hydrate a page of ids under an optional `fields` - * projection, opening the canonical record only when the index cannot serve - * what was asked for. - * - * Without a projection this is exactly `batchGet`, byte for byte: the whole - * point is that `fields` absent changes nothing. - * - * With one, the order is: ask the index for the named scalars in a single - * batched door; see which requested fields it actually served; and read - * records ONLY if something is still missing — and only to fill those fields. - * A page whose every requested field is index-served performs zero canonical - * reads, which is the whole reason the door exists. - * - * `guardFields` are fetched ALONGSIDE the projection and trimmed off before - * the caller sees them. find()'s index-integrity guard re-validates every row - * against its own predicate, and it reads the entity to do so — so a row - * projected down to `title` would fail a `where: { kind }` it genuinely - * matches, and the whole page would vanish. The fields a filter names are - * fields the index can serve by definition, so carrying them costs nothing - * and keeps the guard honest. - * - * A field nothing can supply is simply absent from the row. That is the - * permissive law: a projection asks "these, if you have them", and an - * optional field must not turn a list into an exception. It deliberately does - * NOT route through the strict address resolver, which throws - * `UnresolvableFieldError` for an unknown key — that strictness is right for - * `orderBy`, where a typo silently changes the order, and wrong here, where - * the honest answer is "this row does not have that". - * - * @param ids - Canonical ids for the page. - * @param fields - The projection, or undefined for the full record. - * @returns `id → entity`, projected when `fields` was given. - */ - /** - * The index keys find()'s integrity guard reads when it re-validates a row. - * - * The guard calls `entityMatchesFind(entity, params)`, so a projected entity - * must still carry whatever the params constrain — otherwise a row that - * genuinely matches is dropped for lacking the evidence. These are fetched - * with the projection and trimmed off before the caller sees them. - * - * @param params - The find params. - * @returns Index keys to carry through hydration. - */ - #guardFieldsFor(params: FindParams): string[] { - const keys: string[] = [] - if (params.where && typeof params.where === 'object') { - // Top-level where keys only: nested `anyOf`/`allOf` branches are carried - // by their own keys when the guard walks them, and a filter whose - // evidence is missing keeps the row (the guard's own catch) rather than - // dropping it. - for (const key of Object.keys(params.where as Record)) { - if (key === 'anyOf' || key === 'allOf' || key === 'not') continue - keys.push(key) - } - } - if (params.type !== undefined) keys.push('system.type') - if (params.subtype !== undefined) keys.push('system.subtype') - if (params.service !== undefined) keys.push('system.service') - if (params.excludeVFS === true) keys.push('vfsType', 'isVFSEntity') - return keys - } - - async #hydratePage( - ids: string[], - fields?: readonly string[], - guardFields: readonly string[] = [] - ): Promise>> { - if (fields === undefined || fields.length === 0) return this.batchGet(ids) - - const wanted = [...new Set([...fields, ...guardFields])] - const provider = this.metadataIndex as unknown as MetadataIndexProvider - let served = new Map>() - if (typeof provider.getScalarsForIds === 'function') { - served = await provider.getScalarsForIds(ids, wanted) - } - - // Which ids still owe a field? Only those cost a record read, and a page - // that owes nothing costs none at all. - const owing: string[] = [] - for (const id of ids) { - const row = served.get(id) - if (row === undefined || wanted.some((f) => !(f in row))) owing.push(id) - } - - // The records are read for the OWED fields only; everything the index - // already served is used as-is, so a body field pulls its own record and - // no more than that. - const records = owing.length > 0 ? await this.batchGet(owing) : new Map>() - - const out = new Map>() - for (const id of ids) { - const fromIndex = served.get(id) - const record = records.get(id) - // An id neither the index nor storage knows is not a row. - if (fromIndex === undefined && record === undefined) continue - out.set(id, this.#projectEntity(id, wanted, fromIndex, record)) - } - return out - } - - /** - * Build one projected entity: `id`, plus exactly the requested fields that - * something could supply. - * - * Values come from the index first and the record second, and they must agree - * — the index only reports what it can serve exactly, so a field it served is - * the record's value. A field neither has is omitted rather than set to - * `undefined`: absent and present-and-undefined are different answers, and a - * caller checking `'slug' in row.metadata` deserves the true one. - * - * @param id - The entity id, always present on the result. - * @param fields - The requested index keys. - * @param fromIndex - What the index served for this id, if anything. - * @param record - The canonical entity, if one had to be read. - * @returns The projected entity. - */ - #projectEntity( - id: string, - fields: readonly string[], - fromIndex: Record | undefined, - record: Entity | undefined - ): Entity { - const projected: Record = { id } - const metadata: Record = {} - let sawMetadata = false - - for (const field of fields) { - let value: unknown - let found = false - if (fromIndex !== undefined && field in fromIndex) { - value = fromIndex[field] - found = true - } else if (record !== undefined) { - if (field.startsWith('system.')) { - const inner = field.slice('system.'.length) - const bag = record as unknown as Record - if (inner in bag && bag[inner] !== undefined) { - value = bag[inner] - found = true - } - } else { - const bag = (record.metadata ?? {}) as Record - if (field in bag) { - value = bag[field] - found = true - } - } - } - if (!found) continue - - if (field.startsWith('system.')) { - projected[field.slice('system.'.length)] = value - } else { - metadata[field] = value - sawMetadata = true - } - } - - if (sawMetadata) projected.metadata = metadata - return projected as unknown as Entity - } - async batchGet(ids: string[], options?: GetOptions): Promise>> { // Canonical read (see get): resolves by id from storage, no derived index. await this.ensureInitialized({ needs: [] }) @@ -8259,7 +7920,7 @@ export class Brainy implements BrainyInterface { // Batch-load entities for 10x faster cloud storage performance // GCS: 10 entities = 1×50ms vs 10×50ms = 500ms (10x faster) - const entitiesMap = await this.#hydratePage(pageIds, params.fields, this.#guardFieldsFor(params)) + const entitiesMap = await this.batchGet(pageIds) for (const id of pageIds) { const entity = entitiesMap.get(id) if (entity) { @@ -8296,7 +7957,7 @@ export class Brainy implements BrainyInterface { if (hiddenIds.size > 0) allUuids = allUuids.filter((id) => !hiddenIds.has(id)) const pageIds = allUuids.slice(offset, offset + limit) - const entitiesMap = await this.#hydratePage(pageIds, params.fields, this.#guardFieldsFor(params)) + const entitiesMap = await this.batchGet(pageIds) for (const id of pageIds) { const entity = entitiesMap.get(id) if (entity) { @@ -8324,7 +7985,7 @@ export class Brainy implements BrainyInterface { const pageIds = filteredIds.slice(offset, offset + limit) // Batch-load entities for 10x faster cloud storage performance - const entitiesMap = await this.#hydratePage(pageIds, params.fields, this.#guardFieldsFor(params)) + const entitiesMap = await this.batchGet(pageIds) for (const id of pageIds) { const entity = entitiesMap.get(id) if (entity) { @@ -8575,7 +8236,7 @@ export class Brainy implements BrainyInterface { // Batch-load entities for current page - O(page_size) instead of O(total_results) // GCS: 10 entities = 1×50ms vs 10×50ms = 500ms (10x faster) - const entitiesMap = await this.#hydratePage(pageIds, params.fields, this.#guardFieldsFor(params)) + const entitiesMap = await this.batchGet(pageIds) for (const id of pageIds) { const entity = entitiesMap.get(id) if (entity) { @@ -8603,7 +8264,7 @@ export class Brainy implements BrainyInterface { // Batch-load entities for paginated results (10x faster on GCS) const sortedResults: Result[] = [] - const entitiesMap = await this.#hydratePage(pageIds, params.fields, this.#guardFieldsFor(params)) + const entitiesMap = await this.batchGet(pageIds) for (const id of pageIds) { const entity = entitiesMap.get(id) if (entity) { @@ -8708,28 +8369,6 @@ export class Brainy implements BrainyInterface { }) } - // PROJECTION TRIM — applied once, here, AFTER the integrity guard, so every - // find() path is trimmed uniformly and the guard still saw the evidence it - // needs. Hydration carried the guard's fields alongside the projection; - // this removes them, leaving exactly what the caller named. - // - // Rows that reached here from a path the seam does not hydrate (a vector or - // text leg builds its own entities) are trimmed from what they already - // hold, so the ANSWER is the same everywhere — only the cost differs, and - // only on the paths that still read a record. - if (params.fields !== undefined && params.fields.length > 0 && result.length > 0) { - const named = [...new Set(params.fields)] - result = result.map((r) => { - const projected = this.#projectEntity( - r.id, - named, - undefined, - r.entity as unknown as Entity - ) - return { ...r, entity: projected } as typeof r - }) - } - // includeVectors — opt-in vector hydration. Default (false) keeps the perf // contract: every result path above builds entities via the metadata-only // fast path, so `entity.vector` is the empty stub. When requested, fetch the @@ -17112,7 +16751,7 @@ export class Brainy implements BrainyInterface { ordered = valued.map((v) => v.id) } const pageIds = ordered.slice(offset, offset + limit) - const entitiesMap = await this.#hydratePage(pageIds, params.fields, this.#guardFieldsFor(params)) + const entitiesMap = await this.batchGet(pageIds) const results: Result[] = [] for (const id of pageIds) { const entity = entitiesMap.get(id) diff --git a/src/errors/brainyError.ts b/src/errors/brainyError.ts index 2fbdbe8d..4301d3f7 100644 --- a/src/errors/brainyError.ts +++ b/src/errors/brainyError.ts @@ -412,23 +412,18 @@ export class MigrationInProgressError extends BrainyError { * embedding parked in the metadata bag would mint 384 postings for one row. * The bound exists to keep that out of the index. * - * 256 is hardcoded on purpose (the zero-config law: no knob). It sits far above + * 64 is hardcoded on purpose (the zero-config law: no knob). It sits far above * every legitimate multi-value field the engine has seen — tags, authors, - * categories, labels, keyword lists, participant lists — and still below the - * narrowest embedding this engine will ever meet (384 dimensions, the smallest - * model it ships), so the two populations do not overlap and no caller has to - * tune it. A vector parked in metadata is refused; a long keyword list is not. + * categories, labels, participant lists — and far below any real embedding + * width, so the two populations do not overlap and no caller has to tune it. * * It replaces a limit of 10 that was applied SILENTLY: a row whose `tags` array * held eleven entries had that field skipped entirely and dropped out of every * filtered search on it, with no error, no warning and no way to tell the * difference from "no row matches". A rule this consequential is a law with a * name and a refusal, not a `continue`. - * - * This is the ONE place the number lives. Every message, warning, doc line and - * pin derives it from here — never a literal. */ -export const MAX_INDEXED_ARRAY_LENGTH = 256 +export const MAX_INDEXED_ARRAY_LENGTH = 64 /** * A metadata field carries an array longer than {@link MAX_INDEXED_ARRAY_LENGTH}. diff --git a/src/indexes/columnStore/ColumnStore.ts b/src/indexes/columnStore/ColumnStore.ts index 6bff86d4..4fe45bff 100644 --- a/src/indexes/columnStore/ColumnStore.ts +++ b/src/indexes/columnStore/ColumnStore.ts @@ -23,10 +23,7 @@ import type { ColumnStoreProvider, SegmentMeta } from './types.js' import { ValueType, DEFAULT_FLUSH_THRESHOLD, - FLAG_MULTI_VALUE, - POSTING_KINDS, - KIND_PATH_SEGMENT, - type PostingKind + FLAG_MULTI_VALUE } from './types.js' import { ColumnTailBuffer } from './ColumnTailBuffer.js' import { ColumnManifest } from './ColumnManifest.js' @@ -55,89 +52,10 @@ interface HeapEntry { value: number | string entityIntId: number cursorIndex: number - /** - * Rank of the posting kind this entry came from, from {@link POSTING_KINDS}. - * A mixed-kind field has no natural total order, so the merge orders by kind - * first and by value within a kind. - */ - kindRank: number /** Iterator for the cursor — call next() to advance */ iterator: Generator } -/** - * One physical posting column: a (field, kind) pair and the key every internal - * map and every storage path uses for it. - */ -interface KindColumn { - /** The field as the query language names it. */ - field: string - /** The kind of value this column holds. */ - kind: PostingKind - /** - * Internal map / storage key. The field's PRIMARY kind uses the bare field - * name — the historical layout — and every other kind uses - * `//`. - */ - key: string -} - -/** - * The KIND a value indexes under — its JavaScript `typeof` class, not its - * storage encoding. - * - * Anything that is not a number, string or boolean indexes as a string, which - * is the `String(value)` treatment those values already received. `null` and - * `undefined` never reach here: `addEntity` skips them, and their absence is - * what the `exists` / `missing` operators read. - * - * @param value - The value about to be indexed or queried - * @returns The posting kind that owns this value - */ -function kindOfValue(value: unknown): PostingKind { - const t = typeof value - if (t === 'number') return 'number' - if (t === 'boolean') return 'boolean' - return 'string' -} - -/** - * The segment encoding a fresh column of this kind starts with. - * - * Only the number kind has a choice: an integer column starts as i64 and - * widens to f64 the first time a non-integer arrives - * ({@link ColumnTailBuffer.promoteToFloat}). - */ -function initialValueTypeFor(kind: PostingKind, firstValue: unknown): ValueType { - switch (kind) { - case 'boolean': - return ValueType.Boolean - case 'string': - return ValueType.String - case 'number': - return Number.isInteger(firstValue) ? ValueType.Number : ValueType.Float - } -} - -/** - * The kind a column of this encoding holds — the inverse of - * {@link initialValueTypeFor}, used to read a kind back off a manifest written - * before typed postings existed. - */ -function kindOfValueType(valueType: ValueType): PostingKind { - switch (valueType) { - case ValueType.Boolean: - return 'boolean' - case ValueType.String: - return 'string' - case ValueType.Number: - case ValueType.Float: - return 'number' - default: - throw new Error(`Unknown ValueType: ${valueType}`) - } -} - /** * Unified column store coordinator. * @@ -203,19 +121,9 @@ export class ColumnStore implements ColumnStoreProvider { */ private deletedEntities: Map = new Map() - /** Segment encoding per COLUMN key (not per field — a field has one per kind). */ + /** Known field value types (inferred from first write). */ private fieldTypes: Map = new Map() - /** - * Every posting column a field owns: field → kind → column key. - * - * This is the map that ends the first-writer type freeze. A field's first - * kind takes the bare field name as its column key, keeping the historical - * on-disk layout; each later kind takes its own column beside it. Nothing is - * coerced across kinds and nothing is dropped for being the wrong type. - */ - private fieldColumns: Map> = new Map() - /** Whether init() has completed. */ private initialized = false @@ -232,128 +140,6 @@ export class ColumnStore implements ColumnStoreProvider { this.l0CompactionTrigger = config?.l0CompactionTrigger ?? 4 } - // ========================================================================= - // Posting columns: (field, kind) → one physical column - // ========================================================================= - - /** - * Storage / map key for a (field, kind) column. - * - * `primary` is the kind that owns the bare field name. It is whichever kind - * the field saw first, which for an index written before typed postings is - * simply the kind of its single manifest — so the historical layout is - * preserved rather than migrated. - */ - private static columnKeyFor(field: string, kind: PostingKind, primary: PostingKind | null): string { - return primary === null || kind === primary - ? field - : `${field}/${KIND_PATH_SEGMENT}/${kind}` - } - - /** - * Split a discovered manifest path back into its (field, kind) column, or - * `null` when the path names a field's primary column rather than a kind - * column. `/k/` is the only shape that reads as a kind column, - * and only for a `` this version knows. - */ - private static parseKindColumnKey(key: string): { field: string; kind: PostingKind } | null { - const marker = `/${KIND_PATH_SEGMENT}/` - const at = key.lastIndexOf(marker) - if (at <= 0) return null - const kind = key.slice(at + marker.length) - if (!POSTING_KINDS.includes(kind as PostingKind)) return null - return { field: key.slice(0, at), kind: kind as PostingKind } - } - - /** Record a discovered or freshly created column against its field. */ - private registerColumn(field: string, kind: PostingKind, key: string): void { - let byKind = this.fieldColumns.get(field) - if (!byKind) { - byKind = new Map() - this.fieldColumns.set(field, byKind) - } - const existing = byKind.get(kind) - if (existing !== undefined && existing !== key) { - // Two columns claiming one (field, kind) means the layout on disk is not - // one this writer could have produced. Serving it would silently answer - // from half the postings, so say which two and stop. - throw new Error( - `ColumnStore: field '${field}' has two '${kind}' posting columns on ` + - `disk ('${existing}' and '${key}'). The column index layout is ` + - `inconsistent — rebuild/repair the metadata index rather than ` + - `serving from one half of it.` - ) - } - byKind.set(kind, key) - } - - /** The column key for this (field, kind), or `null` if the field has no such kind. */ - private columnKey(field: string, kind: PostingKind): string | null { - return this.fieldColumns.get(field)?.get(kind) ?? null - } - - /** - * The column key for this (field, kind), creating the registration if the - * field has not seen this kind before. Write path only. - */ - private ensureColumnKey(field: string, kind: PostingKind): string { - const byKind = this.fieldColumns.get(field) - const existing = byKind?.get(kind) - if (existing !== undefined) return existing - - // The primary kind is the one already holding the bare field name, if any. - let primary: PostingKind | null = null - if (byKind) { - for (const [k, key] of byKind) { - if (key === field) { primary = k; break } - } - } - const key = ColumnStore.columnKeyFor(field, kind, primary) - this.registerColumn(field, kind, key) - return key - } - - /** - * Every posting column this field owns, in {@link POSTING_KINDS} order. - * - * Read doors that are not about one particular value — an unbounded range - * used as an "any value present" probe, distinct values, sorting — fan out - * over all of them. - */ - private columnsForField(field: string): KindColumn[] { - const byKind = this.fieldColumns.get(field) - if (!byKind) return [] - const out: KindColumn[] = [] - for (const kind of POSTING_KINDS) { - const key = byKind.get(kind) - if (key !== undefined) out.push({ field, kind, key }) - } - return out - } - - /** - * Which value kinds this field actually holds, in {@link POSTING_KINDS} - * order — the honest answer to "what type is this field?". - * - * A field that carries both `'electronics'` and `5` reports - * `['number', 'string']`, not whichever of them was written first. - * - * @param field - Field name - * @returns Every kind with at least one posting, or `[]` for an unknown field - */ - getFieldKinds(field: string): PostingKind[] { - return this.columnsForField(field) - .filter((c) => this.columnHasData(c.key)) - .map((c) => c.kind) - } - - /** Does this physical column hold any postings (persisted or buffered)? */ - private columnHasData(key: string): boolean { - const manifest = this.manifests.get(key) - const buffer = this.tailBuffers.get(key) - return (manifest !== undefined && !manifest.isEmpty()) || (buffer !== undefined && buffer.size > 0) - } - /** * Initialize the column store: discover existing field manifests. */ @@ -371,23 +157,11 @@ export class ColumnStore implements ColumnStoreProvider { }).listObjectsUnderPath(this.basePath + '/') for (const path of paths) { if (path.endsWith('/MANIFEST.json')) { - // The discovered name is a COLUMN key: either a bare field (that - // field's primary kind, which is every column an index written - // before typed postings has) or `/k/` for a second - // kind that arrived on a field later. - const columnKey = path.replace(this.basePath + '/', '').replace('/MANIFEST.json', '') - const manifest = new ColumnManifest(columnKey, this.basePath) + const fieldName = path.replace(this.basePath + '/', '').replace('/MANIFEST.json', '') + const manifest = new ColumnManifest(fieldName, this.basePath) await manifest.load(storage) - this.manifests.set(columnKey, manifest) - this.fieldTypes.set(columnKey, manifest.valueType) - - const parsed = ColumnStore.parseKindColumnKey(columnKey) - if (parsed) { - this.registerColumn(parsed.field, parsed.kind, columnKey) - } else { - this.registerColumn(columnKey, kindOfValueType(manifest.valueType), columnKey) - } - const fieldName = columnKey + this.manifests.set(fieldName, manifest) + this.fieldTypes.set(fieldName, manifest.valueType) // Load global deleted bitmap if it exists. Raw blob preferred // (2.4.0 #4 cortex-shared format); legacy envelope fallback for @@ -490,43 +264,26 @@ export class ColumnStore implements ColumnStoreProvider { /** * Point filter: find entities where field equals value. * - * The QUERY VALUE'S OWN KIND picks the posting column, and only that column - * is read. `where {category: 5}` answers from the number postings and - * `where {category: '5'}` from the string postings — neither borrows the - * other's rows, because a row written with the number `5` is not a row whose - * category is the text `'5'`. - * - * A field that has never seen this kind matches nothing, which is the true - * answer rather than a coerced one. - * - * Searches all segments + tail buffer of that column, returns the union as a - * roaring bitmap. Excludes globally deleted entities. + * Searches all segments + tail buffer, returns union as roaring bitmap. + * Excludes globally deleted entities. */ async filter(field: string, value: unknown): Promise { const result = new RoaringBitmap32() - const columnKey = this.columnKey(field, kindOfValue(value)) - if (columnKey === null) return result - - // The query value takes the column's encoding — a boolean queried against - // a boolean column has to become the 1/0 the column stores. - const encoded = this.normalizeValue(value, this.fieldTypes.get(columnKey) ?? ValueType.String) - if (encoded === undefined) return result - - const deleted = this.deletedEntities.get(columnKey) + const deleted = this.deletedEntities.get(field) // Search segments - const cursors = await this.getSegmentCursors(columnKey) + const cursors = await this.getSegmentCursors(field) for (const cursor of cursors) { - const ids = cursor.getEntityIdsForValue(encoded) + const ids = cursor.getEntityIdsForValue(value as number | string) for (const id of ids) { if (!deleted || !deleted.has(id)) result.add(id) } } // Search tail buffer - const tailCursor = this.getTailBufferCursor(columnKey) + const tailCursor = this.getTailBufferCursor(field) if (tailCursor) { - const ids = tailCursor.getEntityIdsForValue(encoded) + const ids = tailCursor.getEntityIdsForValue(value as number | string) for (const id of ids) { if (!deleted || !deleted.has(id)) result.add(id) } @@ -535,62 +292,6 @@ export class ColumnStore implements ColumnStoreProvider { return result } - /** - * Read this column's value for each of `entityIntIds` — the per-id read - * behind `find({ fields })`. - * - * Every other read door here answers "which entities have this value". A - * projection asks the opposite — "what value does this entity have" — and - * without it a projection has to go to the canonical record for a field the - * column is already holding. - * - * The column is walked ONCE and the wanted ids are picked out as they pass, - * so the cost is O(column) per field rather than O(ids x column). Later - * sources win: the tail buffer holds writes newer than any segment, and - * within the segments a later one supersedes an earlier, exactly as `filter` - * treats them. - * - * Values are EXACT — this store keeps raw values, not the bucketed form the - * sparse index uses for range queries — which is what makes it safe to - * project from. Deleted entities are skipped; an id with no value in this - * column is simply absent from the result. - * - * @param field - Field name to read. - * @param entityIntIds - Entity integer ids to read values for. - * @returns `entityIntId -> value` for the ids this column holds. - */ - async valuesForIds( - field: string, - entityIntIds: Iterable - ): Promise> { - const wanted = new Set(entityIntIds) - const out = new Map() - if (wanted.size === 0 || !this.hasField(field)) return out - - // Every kind the field holds is read, in POSTING_KINDS order — a value an - // entity wrote as a string is still that entity's value for this field. - for (const column of this.columnsForField(field)) { - const deleted = this.deletedEntities.get(column.key) - const take = (entry: { value: number | string; entityIntId: number }): void => { - if (!wanted.has(entry.entityIntId)) return - if (deleted && deleted.has(entry.entityIntId)) return - out.set(entry.entityIntId, entry.value) - } - - // Segments oldest -> newest, then the tail: a later write overwrites an - // earlier one for the same id. - const cursors = await this.getSegmentCursors(column.key) - for (const cursor of cursors) { - for (const entry of cursor.iterateForward()) take(entry) - } - const tailCursor = this.getTailBufferCursor(column.key) - if (tailCursor) { - for (const entry of tailCursor.iterateForward()) take(entry) - } - } - return out - } - /** * Range filter: find entities where field is within the bounds. * @@ -610,59 +311,41 @@ export class ColumnStore implements ColumnStoreProvider { includeMax: boolean = true ): Promise { const result = new RoaringBitmap32() + const cursors = await this.getSegmentCursors(field) const hasMin = min !== undefined && min !== null const hasMax = max !== undefined && max !== null - // The BOUNDS pick the column: numeric bounds read the numeric postings, - // string bounds the string postings. An unbounded call is not a range at - // all — it is the "has any value here" probe behind `exists` — so it fans - // out over every kind the field holds. - const columns: KindColumn[] = hasMin - ? this.columnsForKind(field, kindOfValue(min)) - : hasMax - ? this.columnsForKind(field, kindOfValue(max)) - : this.columnsForField(field) + for (const cursor of cursors) { + const lo = hasMin ? min as number | string : cursor.minValue + const hi = hasMax ? max as number | string : cursor.maxValue + if (lo === undefined || hi === undefined) continue + // Exclusivity applies only to an explicitly provided bound. A bound taken + // from the segment's own min/max is a real stored value and must stay + // inclusive, or the segment's boundary entities would be wrongly dropped. + const ids = cursor.getEntityIdsInRange( + lo, + hi, + hasMin ? includeMin : true, + hasMax ? includeMax : true + ) + for (const id of ids) result.add(id) + } - for (const column of columns) { - const cursors = await this.getSegmentCursors(column.key) - for (const cursor of cursors) { - const lo = hasMin ? min as number | string : cursor.minValue - const hi = hasMax ? max as number | string : cursor.maxValue - if (lo === undefined || hi === undefined) continue - // Exclusivity applies only to an explicitly provided bound. A bound taken - // from the segment's own min/max is a real stored value and must stay - // inclusive, or the segment's boundary entities would be wrongly dropped. - const ids = cursor.getEntityIdsInRange( - lo, - hi, - hasMin ? includeMin : true, - hasMax ? includeMax : true - ) - for (const id of ids) result.add(id) - } - - // Tail buffer range: linear scan (tail is small) - const tailCursor = this.getTailBufferCursor(column.key) - if (tailCursor) { - for (const entry of tailCursor.iterateForward()) { - const v = entry.value as any - const loOk = !hasMin || (includeMin ? v >= (min as any) : v > (min as any)) - const hiOk = !hasMax || (includeMax ? v <= (max as any) : v < (max as any)) - if (loOk && hiOk) result.add(entry.entityIntId) - } + // Tail buffer range: linear scan (tail is small) + const tailCursor = this.getTailBufferCursor(field) + if (tailCursor) { + for (const entry of tailCursor.iterateForward()) { + const v = entry.value as any + const loOk = !hasMin || (includeMin ? v >= (min as any) : v > (min as any)) + const hiOk = !hasMax || (includeMax ? v <= (max as any) : v < (max as any)) + if (loOk && hiOk) result.add(entry.entityIntId) } } return result } - /** The single column for this (field, kind), as a list, or empty if absent. */ - private columnsForKind(field: string, kind: PostingKind): KindColumn[] { - const key = this.columnKey(field, kind) - return key === null ? [] : [{ field, kind, key }] - } - /** * Sort top-K: return K entity int IDs in sorted order (u64-safe BigInt). * @@ -693,21 +376,18 @@ export class ColumnStore implements ColumnStoreProvider { */ async getFilterValues(field: string): Promise { const valueSet = new Set() + const cursors = await this.getSegmentCursors(field) - for (const column of this.columnsForField(field)) { - const cursors = await this.getSegmentCursors(column.key) - - for (const cursor of cursors) { - for (const entry of cursor.iterateForward()) { - valueSet.add(String(entry.value)) - } + for (const cursor of cursors) { + for (const entry of cursor.iterateForward()) { + valueSet.add(String(entry.value)) } + } - const tailCursor = this.getTailBufferCursor(column.key) - if (tailCursor) { - for (const entry of tailCursor.iterateForward()) { - valueSet.add(String(entry.value)) - } + const tailCursor = this.getTailBufferCursor(field) + if (tailCursor) { + for (const entry of tailCursor.iterateForward()) { + valueSet.add(String(entry.value)) } } @@ -718,7 +398,9 @@ export class ColumnStore implements ColumnStoreProvider { * Check if a field has any indexed data. */ hasField(field: string): boolean { - return this.columnsForField(field).some((c) => this.columnHasData(c.key)) + const manifest = this.manifests.get(field) + const buffer = this.tailBuffers.get(field) + return (manifest !== undefined && !manifest.isEmpty()) || (buffer !== undefined && buffer.size > 0) } /** @@ -728,11 +410,12 @@ export class ColumnStore implements ColumnStoreProvider { * store will actually serve queries from. */ getIndexedFields(): string[] { - // Names FIELDS, not columns: a field carrying two kinds is one name here, - // the same name a caller queries with. const fields = new Set() - for (const [field] of this.fieldColumns) { - if (this.hasField(field)) fields.add(field) + for (const [field, manifest] of this.manifests) { + if (!manifest.isEmpty()) fields.add(field) + } + for (const [field, buffer] of this.tailBuffers) { + if (buffer.size > 0) fields.add(field) } return Array.from(fields).sort() } @@ -747,16 +430,12 @@ export class ColumnStore implements ColumnStoreProvider { getFieldSizeSummary(): Array<{ field: string; segmentCount: number; tailSize: number }> { const summary: Array<{ field: string; segmentCount: number; tailSize: number }> = [] for (const field of this.getIndexedFields()) { - // Summed across the field's kind columns — the caller asked about a - // field, and a field's size is all of the postings under its name. - let segmentCount = 0 - let tailSize = 0 - for (const column of this.columnsForField(field)) { - const manifest = this.manifests.get(column.key) - const buffer = this.tailBuffers.get(column.key) - if (manifest && !manifest.isEmpty()) segmentCount += manifest.getAllSegments().length - if (buffer) tailSize += buffer.size - } + const manifest = this.manifests.get(field) + const buffer = this.tailBuffers.get(field) + const segmentCount = manifest && !manifest.isEmpty() + ? manifest.getAllSegments().length + : 0 + const tailSize = buffer ? buffer.size : 0 summary.push({ field, segmentCount, tailSize }) } return summary @@ -784,8 +463,6 @@ export class ColumnStore implements ColumnStoreProvider { this.segmentCache.clear() this.manifests.clear() this.deletedEntities.clear() - this.fieldColumns.clear() - this.fieldTypes.clear() this.initialized = false } @@ -794,64 +471,32 @@ export class ColumnStore implements ColumnStoreProvider { // ========================================================================= /** - * Push a single value to the posting column for its (field, KIND). - * - * The value's own kind picks the column — a string goes to the field's - * string postings, a number to its number postings — so a field carrying - * `'electronics'` and `5` keeps both, each answerable by an equality filter - * of its own kind. Under the first-writer type freeze this method replaced, - * the first value's type became the field's type and every later value of - * another kind was coerced to it or, when coercion failed, dropped with no - * error at all. - * - * Creates the column's buffer and manifest on its first value. + * Push a single value to a field's tail buffer. + * Creates the buffer and manifest if first write to this field. + * Infers ValueType from the first value seen. */ private pushToBuffer(field: string, value: unknown, entityIntId: number, isMultiValue: boolean): void { - const kind = kindOfValue(value) - const columnKey = this.ensureColumnKey(field, kind) - - let buffer = this.tailBuffers.get(columnKey) + let buffer = this.tailBuffers.get(field) if (!buffer) { - // A reopened column takes its encoding from its manifest — an integer - // column that widened to f64 in an earlier session stays widened. - const valueType = - this.manifests.get(columnKey)?.valueType ?? initialValueTypeFor(kind, value) - buffer = new ColumnTailBuffer(columnKey, valueType, this.flushThreshold) - this.tailBuffers.set(columnKey, buffer) - this.fieldTypes.set(columnKey, valueType) + const valueType = this.inferValueType(value) + buffer = new ColumnTailBuffer(field, valueType, this.flushThreshold) + this.tailBuffers.set(field, buffer) + this.fieldTypes.set(field, valueType) // Ensure manifest exists - if (!this.manifests.has(columnKey)) { - const manifest = new ColumnManifest(columnKey, this.basePath) + if (!this.manifests.has(field)) { + const manifest = new ColumnManifest(field, this.basePath) manifest.valueType = valueType manifest.multiValue = isMultiValue - this.manifests.set(columnKey, manifest) + this.manifests.set(field, manifest) } } - // An integer column widens the first time a non-integer number arrives, so - // the value is stored as itself instead of rounded to the nearest integer. - if (kind === 'number' && buffer.valueType === ValueType.Number && !Number.isInteger(value)) { - buffer.promoteToFloat() - this.fieldTypes.set(columnKey, ValueType.Float) - const manifest = this.manifests.get(columnKey) - if (manifest) manifest.valueType = ValueType.Float - } - + // Normalize value to the column type const normalizedValue = this.normalizeValue(value, buffer.valueType) - if (normalizedValue === undefined) { - // Unreachable by construction: the column was chosen BY this value's - // kind, so the encoding always accepts it. Reaching here would mean a - // value had been silently dropped from the index — the exact failure - // typed postings exist to end — so it is an error, never a skip. - throw new Error( - `ColumnStore: field '${field}' rejected a ${kind} value for its own ` + - `${ValueType[buffer.valueType]} posting column. The value would have ` + - `been dropped from the index while the row stayed readable by id — ` + - `this is a kind-routing bug, not a value the caller may ignore.` - ) + if (normalizedValue !== undefined) { + buffer.add(normalizedValue, entityIntId) } - buffer.add(normalizedValue, entityIntId) } /** @@ -980,15 +625,8 @@ export class ColumnStore implements ColumnStoreProvider { /** Torn-segment quarantine entries for a field (observability + heal input). */ quarantinedSegments(field: string): Array<{ segment: string; error: string; hits: number }> { const out: Array<{ segment: string; error: string; hits: number }> = [] - // Across every kind column of the field — a torn segment in the string - // postings is this field's torn segment as much as one in the numbers. - for (const column of this.columnsForField(field)) { - const prefix = `${column.key}:` - for (const [key, q] of this.segmentQuarantine) { - if (key.startsWith(prefix)) { - out.push({ segment: key.slice(prefix.length), error: q.error, hits: q.hits }) - } - } + for (const [key, q] of this.segmentQuarantine) { + if (key.startsWith(`${field}:`)) out.push({ segment: key.slice(field.length + 1), error: q.error, hits: q.hits }) } return out } @@ -1160,22 +798,17 @@ export class ColumnStore implements ColumnStoreProvider { k: number, filterBitmap: RoaringBitmap32 | null ): Promise { - // Collect cursors across EVERY kind the field holds. A single-kind field — - // nearly all of them — merges exactly the cursors it always did. + // Collect all cursors (segments + tail buffer) + const segCursors = await this.getSegmentCursors(field) + const tailCursor = this.getTailBufferCursor(field) + + // Create iterators for each cursor in the specified direction const iterators: Generator[] = [] - const iteratorKindRank: number[] = [] - for (const column of this.columnsForField(field)) { - const kindRank = POSTING_KINDS.indexOf(column.kind) - const segCursors = await this.getSegmentCursors(column.key) - for (const cursor of segCursors) { - iterators.push(order === 'asc' ? cursor.iterateForward() : cursor.iterateBackward()) - iteratorKindRank.push(kindRank) - } - const tailCursor = this.getTailBufferCursor(column.key) - if (tailCursor) { - iterators.push(order === 'asc' ? tailCursor.iterateForward() : tailCursor.iterateBackward()) - iteratorKindRank.push(kindRank) - } + for (const cursor of segCursors) { + iterators.push(order === 'asc' ? cursor.iterateForward() : cursor.iterateBackward()) + } + if (tailCursor) { + iterators.push(order === 'asc' ? tailCursor.iterateForward() : tailCursor.iterateBackward()) } if (iterators.length === 0) return [] @@ -1189,21 +822,16 @@ export class ColumnStore implements ColumnStoreProvider { value: next.value.value, entityIntId: next.value.entityIntId, cursorIndex: i, - kindRank: iteratorKindRank[i], iterator: iterators[i] }) } } - // Heapify. A number and a string have no ordering between them, so a - // mixed-kind field orders by KIND first (POSTING_KINDS order) and by value - // within a kind — one defined total order instead of a comparison whose - // answer depends on which value happened to be on the left. + // Heapify + const isString = (this.fieldTypes.get(field) ?? ValueType.Number) === ValueType.String const compare = (a: HeapEntry, b: HeapEntry): number => { let cmp: number - if (a.kindRank !== b.kindRank) { - cmp = a.kindRank - b.kindRank - } else if (POSTING_KINDS[a.kindRank] === 'string') { + if (isString) { cmp = compareCodePoints(String(a.value), String(b.value)) } else { cmp = (a.value as number) - (b.value as number) @@ -1235,7 +863,6 @@ export class ColumnStore implements ColumnStoreProvider { value: next.value.value, entityIntId: next.value.entityIntId, cursorIndex: top.cursorIndex, - kindRank: top.kindRank, iterator: top.iterator } } @@ -1243,11 +870,8 @@ export class ColumnStore implements ColumnStoreProvider { this.heapDown(heap, 0, compare) } - // Apply global deleted check, filter, and dedup. The deleted bitmap is - // per COLUMN, and the entry came from the column its kind names. - const deleted = this.deletedEntities.get( - this.columnKey(field, POSTING_KINDS[top.kindRank]) ?? field - ) + // Apply global deleted check, filter, and dedup + const deleted = this.deletedEntities.get(field) if (deleted && deleted.has(top.entityIntId)) continue if (seen.has(top.entityIntId)) continue if (filterBitmap && !filterBitmap.has(top.entityIntId)) continue @@ -1289,31 +913,35 @@ export class ColumnStore implements ColumnStoreProvider { } /** - * Encode a value for the column its own kind selected. - * - * This does NOT convert between kinds. It used to: a string reaching a - * numeric column was run through `Number(value)`, and a number reaching a - * numeric column was run through `Math.round`, so `'electronics'` became - * `NaN` and vanished while `4.5` became `5` and answered the wrong query. - * Kind routing removes the need for either — the only work left is picking - * the encoding the column already committed to. - * - * @returns The encoded value, or `undefined` if the value does not belong in - * this column at all — which the caller treats as a routing bug and - * raises, never as a value to skip. + * Infer ValueType from a JavaScript value. + */ + private inferValueType(value: unknown): ValueType { + if (typeof value === 'boolean') return ValueType.Boolean + if (typeof value === 'number') { + return Number.isInteger(value) ? ValueType.Number : ValueType.Float + } + return ValueType.String + } + + /** + * Normalize a JavaScript value to the column's ValueType. */ private normalizeValue(value: unknown, type: ValueType): number | string | undefined { switch (type) { case ValueType.Number: - // Integer column. Non-integers widen it to Float before reaching here. - return typeof value === 'number' && Number.isInteger(value) ? value : undefined + if (typeof value === 'number') return Math.round(value) + if (typeof value === 'string') { const n = Number(value); return isNaN(n) ? undefined : Math.round(n) } + if (typeof value === 'boolean') return value ? 1 : 0 + return undefined case ValueType.Float: - return typeof value === 'number' ? value : undefined + if (typeof value === 'number') return value + if (typeof value === 'string') { const n = Number(value); return isNaN(n) ? undefined : n } + return undefined case ValueType.Boolean: - return typeof value === 'boolean' ? (value ? 1 : 0) : undefined + if (typeof value === 'boolean') return value ? 1 : 0 + if (typeof value === 'number') return value ? 1 : 0 + return undefined case ValueType.String: - // The string kind is also where objects and bigints land, exactly as - // they always did. return String(value) default: return undefined diff --git a/src/indexes/columnStore/ColumnTailBuffer.ts b/src/indexes/columnStore/ColumnTailBuffer.ts index e730f884..c5874ac2 100644 --- a/src/indexes/columnStore/ColumnTailBuffer.ts +++ b/src/indexes/columnStore/ColumnTailBuffer.ts @@ -55,12 +55,8 @@ export class ColumnTailBuffer { /** Field name this buffer is for. */ readonly fieldName: string - /** - * Value type determines sort comparator and segment encoding. - * - * Widened in place by {@link promoteToFloat} — never otherwise reassigned. - */ - valueType: ValueType + /** Value type determines sort comparator. */ + readonly valueType: ValueType /** Flush threshold. */ readonly threshold: number @@ -85,38 +81,6 @@ export class ColumnTailBuffer { this.threshold = threshold } - /** - * Widen an integer column to floating point, losslessly and in place. - * - * The number posting kind holds every JavaScript number, but a segment picks - * ONE encoding: i64 for integers, f64 for the rest. A column that has only - * ever seen integers is written as i64; the first non-integer to arrive - * widens it here, so that value is stored as itself instead of being rounded - * to the nearest integer with no error — the rounding that made `4.5` and - * `5.5` both answer `where {score: 5}` and neither answer its own value. - * - * Widening is lossless in both directions it has to be: every value already - * buffered is an integer, and every integer is exactly representable as f64. - * Segments already on disk keep their own i64 encoding in their own headers - * and keep decoding by it — only segments written from here on are f64. - * - * @throws Error if called on a column that is not an integer column — the - * only legal widening is Number → Float, and any other request is a bug in - * the caller's kind routing rather than something to absorb quietly. - */ - promoteToFloat(): void { - if (this.valueType === ValueType.Float) return - if (this.valueType !== ValueType.Number) { - throw new Error( - `ColumnTailBuffer '${this.fieldName}': cannot widen a ` + - `${ValueType[this.valueType]} column to Float — only an integer ` + - `(Number) column widens, and this call means a value reached the ` + - `wrong kind's column` - ) - } - this.valueType = ValueType.Float - } - /** * Add a (value, entityIntId) entry to the buffer. * diff --git a/src/indexes/columnStore/types.ts b/src/indexes/columnStore/types.ts index ee949bd0..71dd99a0 100644 --- a/src/indexes/columnStore/types.ts +++ b/src/indexes/columnStore/types.ts @@ -58,53 +58,6 @@ export enum ValueType { Boolean = 3 } -/** - * The KIND of a value, as the query language sees it. - * - * A kind is a JavaScript `typeof` class, not a storage encoding: `5` and `5.5` - * are one kind (`'number'`) held in one posting column, even though they need - * different segment encodings (i64 vs f64 — see {@link ValueType}). - * - * A field holds ONE POSTING COLUMN PER KIND, so `category` may carry string - * values and number values at the same time and answer equality on each. This - * replaces the first-writer type freeze, under which the first value's type - * became the field's type and every later value of another kind was coerced — - * or, when coercion failed (`Number('electronics')`), dropped from the index - * with no error: the row stayed readable by id and by vector but vanished from - * every equality filter on that field. - * - * Kinds do not coerce into one another at query time either: `where {c: 5}` - * matches rows written with the NUMBER `5`, and `where {c: '5'}` matches rows - * written with the STRING `'5'`. Neither ever matches the other. - * - * Values that are none of these three (objects, bigints) index as strings — - * the same `String(value)` treatment they received before. - */ -export type PostingKind = 'number' | 'string' | 'boolean' - -/** - * Every posting kind, in the order that defines cross-kind sort position. - * - * A mixed-kind field has no natural total order — a number does not compare - * with a string — so `sortTopK` orders by KIND first (numbers, then strings, - * then booleans) and by value within a kind. A single-kind field, which is - * nearly every field, sorts exactly as it always did. - */ -export const POSTING_KINDS: readonly PostingKind[] = ['number', 'string', 'boolean'] - -/** - * Path segment marking a field's NON-PRIMARY kind columns on disk. - * - * The first kind a field ever sees keeps the historical layout — - * `//MANIFEST.json` and `//L0-NNNNNN` — so every - * index written before typed postings opens unchanged, and the byte-for-byte - * interchange with the native column store is untouched for the single-kind - * fields that are nearly all of them. A second kind arriving on the same field - * gets its own column at `//k//…` rather than overwriting or - * being coerced into the first. - */ -export const KIND_PATH_SEGMENT = 'k' - // --------------------------------------------------------------------------- // Segment header and footer // --------------------------------------------------------------------------- @@ -314,19 +267,6 @@ export interface ColumnStoreProvider { */ hasField(field: string): boolean - /** - * Which value KINDS this field actually holds, in {@link POSTING_KINDS} - * order — the honest answer to "what type is this field?" for a field that - * carries more than one. - * - * OPTIONAL so an implementation written against the pre-typed-postings - * contract still satisfies this interface; feature-detect before calling. - * - * @param field - Field name - * @returns Every kind with at least one posting, or `[]` for an unknown field - */ - getFieldKinds?(field: string): PostingKind[] - /** * Flush all in-memory tail buffers to L0 segments on disk. * Saves all manifests. diff --git a/src/neural/embeddedPatterns.ts b/src/neural/embeddedPatterns.ts index 92e3057a..4f4339f4 100644 --- a/src/neural/embeddedPatterns.ts +++ b/src/neural/embeddedPatterns.ts @@ -2,7 +2,7 @@ * 🧠 BRAINY EMBEDDED PATTERNS * * AUTO-GENERATED - DO NOT EDIT - * Generated: 2026-08-27T09:18:45-07:00 + * Generated: 2025-09-29T10:10:00-07:00 * Patterns: 220 * Coverage: 94-98% of all queries * diff --git a/src/plugin.ts b/src/plugin.ts index 23a8c883..64abfe26 100644 --- a/src/plugin.ts +++ b/src/plugin.ts @@ -495,45 +495,6 @@ export interface MetadataIndexProvider { query: string, ids: readonly string[] ): Promise> - /** - * @description OPTIONAL: read named SCALAR fields for many ids at once, from - * the index's own value storage, WITHOUT touching the canonical record. - * - * This is the door behind `find/get/related({ fields })`. A list view that - * needs a title and a slug currently hydrates the whole record for every row - * — document bodies included — and then discards almost all of it. Serving - * the named scalars from the index turns that into an index read. - * - * ## The contract, and the one rule that makes it safe - * - * **Return only what you can serve EXACTLY, and say what you served.** The - * answer is a per-id map of the fields this index actually resolved; the - * caller diffs it against what was requested and reads the canonical record - * for the remainder. An implementation must therefore OMIT a field rather - * than approximate it — and omission costs only a record read, while a wrong - * value is a wrong answer nobody can see. - * - * That rule is not hypothetical. This engine's own index buckets - * `system.createdAt` and `system.updatedAt` to the minute for range queries, - * so it cannot serve them exactly and omits them. An engine whose column - * store holds raw values can serve the same fields — so the two answer - * differently in COST and identically in CONTENT, which is the only - * difference a projection door is allowed to have. - * - * A field absent from an entity is simply absent from that entity's map. It - * is never an error, and never a `null` standing in for one: absent and - * present-and-null are different answers. - * - * @param ids - Canonical entity ids to read. - * @param fields - Index KEYS (bare = user metadata, `system.*` = engine - * scalar), already address-resolved by the caller. - * @returns `id → { field: value }` for the fields this index served exactly. - * Ids with nothing to serve may be omitted entirely. - */ - getScalarsForIds?( - ids: readonly string[], - fields: readonly string[] - ): Promise>> getSortedIdsForFilter(filter: any, orderBy: string, order?: 'asc' | 'desc', topK?: number): Promise getFilterValues(field: string): Promise getFilterFields(): Promise diff --git a/src/types/brainy.types.ts b/src/types/brainy.types.ts index b99f0261..a0d55c1e 100644 --- a/src/types/brainy.types.ts +++ b/src/types/brainy.types.ts @@ -561,33 +561,6 @@ export interface UpdateRelationParams { * refusal with the fix in hand beats a silent behavior flip. */ export interface FindParams { - /** - * **Field projection** — return only these fields on each row, instead of the - * whole record. - * - * A list view that shows a title and a slug does not need the document body, - * yet without a projection every row hydrates its full record and throws - * almost all of it away. Naming the fields lets them be served from the index - * itself: a scalar the index holds exactly is read from the index, and the - * canonical record is opened ONLY when a requested field cannot be. - * - * Field names follow the one addressing law: a bare name is the user's - * metadata (`'title'`), and `system.*` is an engine scalar - * (`'system.createdAt'`). - * - * - **Absent** ⇒ the full record, exactly as before. - * - A requested field the entity does not carry is simply **absent** from the - * row. It is never an error — a projection asks "give me these if you have - * them", so an optional field must not turn a list into a failure. - * - Every returned row carries `id` (and, on `find`, `score`) regardless: a - * row you cannot identify is not a row. - * - * @example - * // A list page: two user fields and one engine scalar, no document bodies. - * await brain.find({ where: { kind: 'post' }, fields: ['title', 'slug', 'system.createdAt'], limit: 50 }) - */ - fields?: readonly string[] - // Vector Intelligence /** Natural language or semantic search query (embedded and matched via HNSW + text index) */ query?: string @@ -816,12 +789,6 @@ export interface SimilarParams { * Added string ID shorthand syntax */ export interface RelatedParams { - // NOTE: `fields` is deliberately NOT offered here. A Relation carries `from` - // and `to` as IDS and hydrates no entity record, so there is nothing for a - // projection to trim — the param would be decorative. Projecting the - // ENDPOINTS would be a new capability (related() hydrating entities), not a - // projection of an existing one, and it belongs in its own decision. - /** * Filter by source entity ID * @@ -1447,33 +1414,6 @@ export interface ImportResult { * */ export interface GetOptions { - /** - * **Field projection** — return only these fields on each row, instead of the - * whole record. - * - * A list view that shows a title and a slug does not need the document body, - * yet without a projection every row hydrates its full record and throws - * almost all of it away. Naming the fields lets them be served from the index - * itself: a scalar the index holds exactly is read from the index, and the - * canonical record is opened ONLY when a requested field cannot be. - * - * Field names follow the one addressing law: a bare name is the user's - * metadata (`'title'`), and `system.*` is an engine scalar - * (`'system.createdAt'`). - * - * - **Absent** ⇒ the full record, exactly as before. - * - A requested field the entity does not carry is simply **absent** from the - * row. It is never an error — a projection asks "give me these if you have - * them", so an optional field must not turn a list into a failure. - * - Every returned row carries `id` (and, on `find`, `score`) regardless: a - * row you cannot identify is not a row. - * - * @example - * // A list page: two user fields and one engine scalar, no document bodies. - * await brain.find({ where: { kind: 'post' }, fields: ['title', 'slug', 'system.createdAt'], limit: 50 }) - */ - fields?: readonly string[] - /** * Include 384-dimensional vector embeddings in the response * diff --git a/src/utils/fieldTypeInference.ts b/src/utils/fieldTypeInference.ts index 0f085f8c..36a415b2 100644 --- a/src/utils/fieldTypeInference.ts +++ b/src/utils/fieldTypeInference.ts @@ -55,30 +55,8 @@ export enum FieldType { */ export interface FieldTypeInfo { field: string - /** - * The DOMINANT reading of the field — one type, the most specific one every - * sampled value satisfies. - * - * A field is not obliged to hold one kind, so this is not the whole answer - * for a field that holds several. Read {@link kinds} beside it: a field - * carrying `'electronics'` and `5` infers as STRING here and reports - * `['number', 'string']` there, and the metadata index keeps a separate - * posting column for each of them. - */ inferredType: FieldType confidence: number // 0-1 confidence score - /** - * Every value KIND observed in the sample, in the order - * number → string → boolean. More than one entry means a genuinely - * mixed field, and every one of those kinds is independently filterable. - * - * Kinds are JavaScript `typeof` classes, one level coarser than - * {@link FieldType}: a UUID and a category name are both `'string'`, and an - * integer and a timestamp are both `'number'`. - * - * Optional only for cached analyses written before this was reported. - */ - kinds?: Array<'number' | 'string' | 'boolean'> sampleSize: number // Number of values analyzed lastUpdated: number // Timestamp of last analysis detectionMethod: 'value' // Always 'value' (no fallbacks!) @@ -155,71 +133,14 @@ export class FieldTypeInference { } /** - * Analyze values to determine field type, and report every KIND the field - * actually holds alongside it. - * - * The classification below picks ONE type, because every one of its - * heuristics asks `samples.every(...)`: a field carrying `'electronics'` and - * `5` satisfies none of them and lands on STRING. That single answer is true - * as far as it goes — string is the dominant reading — but on its own it - * says nothing about the numbers also in the field, and a caller that treats - * it as the field's only type reproduces the first-writer freeze the index - * itself no longer has. {@link FieldTypeInfo.kinds} carries the rest. - */ - private async analyzeValues(field: string, values: any[]): Promise { - const info = await this.classifyValues(field, values) - info.kinds = FieldTypeInference.observedKinds(values) - if (info.kinds.length > 1 && info.metadata) { - info.metadata.format = `${info.metadata.format} (field also holds: ${info.kinds - .filter((k) => k !== FieldTypeInference.kindOfType(info.inferredType)) - .join(', ')})` - } - return info - } - - /** - * The distinct value kinds present in a sample, in a stable order. - * - * Kinds are JavaScript `typeof` classes — the same classes the metadata - * index keeps separate posting columns for — not the finer - * {@link FieldType} readings, which are interpretations layered on top of - * them (a UUID and a category name are both the `string` kind). - */ - private static observedKinds(values: any[]): Array<'number' | 'string' | 'boolean'> { - const order: Array<'number' | 'string' | 'boolean'> = ['number', 'string', 'boolean'] - const seen = new Set<'number' | 'string' | 'boolean'>() - for (const v of values) { - if (v === null || v === undefined) continue - const t = typeof v - seen.add(t === 'number' ? 'number' : t === 'boolean' ? 'boolean' : 'string') - } - return order.filter((k) => seen.has(k)) - } - - /** The value kind a {@link FieldType} reading is an interpretation of. */ - private static kindOfType(type: FieldType): 'number' | 'string' | 'boolean' { - switch (type) { - case FieldType.BOOLEAN: - return 'boolean' - case FieldType.INTEGER: - case FieldType.FLOAT: - case FieldType.TIMESTAMP_MS: - case FieldType.TIMESTAMP_S: - return 'number' - default: - return 'string' - } - } - - /** - * Classify values into a single field type. + * Analyze values to determine field type * * Uses DuckDB-inspired type detection order: * BOOLEAN → INTEGER → FLOAT → DATE → TIMESTAMP → UUID → STRING * * No fallbacks - pure value-based detection */ - private async classifyValues(field: string, values: any[]): Promise { + private async analyzeValues(field: string, values: any[]): Promise { // Filter null/undefined values const validValues = values.filter(v => v !== null && v !== undefined) diff --git a/src/utils/metadataIndex.ts b/src/utils/metadataIndex.ts index 1a882945..0a5aef75 100644 --- a/src/utils/metadataIndex.ts +++ b/src/utils/metadataIndex.ts @@ -2930,67 +2930,6 @@ export class MetadataIndexManager implements MetadataIndexProvider { return order === 'asc' ? comparison : -comparison } - /** - * Read named scalar fields for many ids from the COLUMN STORE, without - * touching the canonical record — the `find({ fields })` door. - * - * ## Why the column store and not the sparse index - * - * The column store keeps RAW values; the sparse index keeps a normalized, - * bucketed form built for range queries — `system.createdAt` is indexed at - * minute precision there. A projection served from the sparse index would - * hand back a value that differs from the record's, which is a wrong answer - * nobody can see. So this door reads the column store, and a field the - * column store does not hold is OMITTED rather than approximated. - * - * ## Why batched - * - * `getFieldValueForEntity` answers one (id, field) pair by walking the - * field's storage; called per row it re-walks the same column for every id. - * This walks each column ONCE and picks out every requested id as it passes: - * O(fields x column) instead of O(ids x fields x column). - * - * Omission is always safe — it costs the caller a record read. The caller - * diffs what it asked for against what came back and reads records for the - * remainder, so an index that can serve nothing is slow, never wrong. - * - * @param ids - Canonical entity ids. - * @param fields - Index keys (bare = user metadata, `system.*` = engine scalar). - * @returns `id -> { field: value }` for exactly the pairs this index served. - */ - async getScalarsForIds( - ids: readonly string[], - fields: readonly string[] - ): Promise>> { - const out = new Map>() - if (ids.length === 0 || fields.length === 0) return out - - // int -> id, so a column hit resolves back to the caller's id. An id the - // mapper does not know cannot be in any column, so it is simply absent. - const idByInt = new Map() - for (const id of ids) { - const intId = this.idMapper.getInt(id) - if (intId !== undefined) idByInt.set(intId, id) - } - if (idByInt.size === 0) return out - - for (const field of fields) { - if (!this.columnStore.hasField(field)) continue - const values = await this.columnStore.valuesForIds(field, idByInt.keys()) - for (const [intId, value] of values) { - const id = idByInt.get(intId) - if (id === undefined) continue - let row = out.get(id) - if (row === undefined) { - row = {} - out.set(id, row) - } - row[field] = value - } - } - return out - } - async getFieldValueForEntity(entityId: string, field: string): Promise { // `field` arrives as a FROZEN INDEX KEY (bare = user metadata; // 'system.' = engine scalar). Storage fallbacks read the matching diff --git a/tests/configs/vitest.perf.config.ts b/tests/configs/vitest.perf.config.ts index 6936a71c..6c0f2d1b 100644 --- a/tests/configs/vitest.perf.config.ts +++ b/tests/configs/vitest.perf.config.ts @@ -57,14 +57,7 @@ export default defineConfig({ // otherwise-correctness integration suite (self-skipped everywhere // else via BRAINY_PERF_LANE). Stays in the integration gate's // include too, so every OTHER test in the file keeps running there. - 'tests/integration/storage-batch-operations.test.ts', - // Same pattern: one wall-clock budget case (100-file write + readdir, - // 5.5s budget) inside an otherwise-correctness VFS unit suite - // (self-skipped everywhere else via BRAINY_PERF_LANE — see - // tests/vfs/vfs.unit.test.ts's 'Performance > should handle many - // files efficiently'). Stays in the unit gate's *.unit.test.ts match - // too, so every OTHER test in the file keeps running there. - 'tests/vfs/vfs.unit.test.ts' + 'tests/integration/storage-batch-operations.test.ts' ], reporters: process.env.CI ? ['dot'] : ['basic'], diff --git a/tests/integration/api-parameter-validation.test.ts b/tests/integration/api-parameter-validation.test.ts index da4aed14..4e25e781 100644 --- a/tests/integration/api-parameter-validation.test.ts +++ b/tests/integration/api-parameter-validation.test.ts @@ -34,10 +34,6 @@ describe('API Parameter Validation', () => { }) }) - afterAll(async () => { - await brain.close() - }) - it('should use "where" parameter for metadata filtering', async () => { const results = await brain.find({ where: { category: 'test-category' }, diff --git a/tests/integration/beforeexit-never-closes.test.ts b/tests/integration/beforeexit-never-closes.test.ts deleted file mode 100644 index b7a2f95b..00000000 --- a/tests/integration/beforeexit-never-closes.test.ts +++ /dev/null @@ -1,309 +0,0 @@ -/** - * @module tests/integration/beforeexit-never-closes - * @description A DRAINED EVENT LOOP IS NOT A SHUTDOWN. - * - * MEASURED on the 11.1 rehearsal lane, against a copy of a real store. The - * `beforeExit` listener had been wired to the SIGNAL path — the path whose job - * is to `close()` every live brain — so after the heal phase the log printed - * - * "Shutdown signal received - flushing pending data..." - * "Flushed successfully (1 instance)" - * - * with no signal ever sent, and the script's very next `add()` threw - * - * "Brainy instance is not initialized: it was closed via close(). - * Create a new instance." - * - * Node emits `'beforeExit'` whenever the event loop has no REF'd work left. - * That is not "the process is ending" — it is a state a perfectly healthy - * script reaches, because this engine unref's its idle and cadence timers - * ("an idle brain costs nothing"), so a script awaiting anything those timers - * drive is, for that instant, a process with no ref'd work and an open brain. - * The engine closed a live brain out from under a running script. - * - * The contract pinned here: - * (1) `'beforeExit'` firing while a brain is open closes NOTHING: the brain - * is still open, `add()` and `find()` still work, the writer lock is - * still held, and the process still exits 0 on its own afterwards. - * (2) The pass DOES persist derived state — a non-closing `flush()` ran — - * and it wrote no clean-shutdown marker and no clean-close record: those - * are `close()`'s word about itself, and no close happened. - * (3) The signal path is untouched: SIGTERM still closes through `close()` - * (pinned by tests/integration/shutdown-single-owner.test.ts, re-run - * with this change). - */ - -import { describe, it, expect, beforeEach, afterEach } from 'vitest' -import { mkdtempSync, rmSync, existsSync, readFileSync, writeFileSync } from 'node:fs' -import { spawn } from 'node:child_process' -import { tmpdir } from 'node:os' -import { join } from 'node:path' -import { Brainy } from '../../src/brainy.js' -import { NounType } from '../../src/types/graphTypes.js' - -const REPO_ROOT = process.cwd() -const TSX = join(REPO_ROOT, 'node_modules', '.bin', 'tsx') -const BRAINY_SRC = join(REPO_ROOT, 'src', 'brainy.ts') - -function makeTempDir(prefix: string): string { - return mkdtempSync(join(tmpdir(), prefix)) -} - -/** The writer lock itself — present for as long as this process owns the store. */ -const writerLockPath = (dir: string) => join(dir, 'locks', '_writer.lock') -/** The clean-close record — written by `releaseWriterLock()`, i.e. by close(). */ -const closeRecordPath = (dir: string) => join(dir, 'locks', '_writer.close') -/** - * The generation store's clean-shutdown marker — written by - * `generationStore.close()` alone, reached only from `close()`. (Raw objects - * are gzipped on disk, so both spellings are accepted.) - */ -const cleanShutdownWritten = (dir: string) => - existsSync(join(dir, '_system', 'clean-shutdown.json.gz')) || - existsSync(join(dir, '_system', 'clean-shutdown.json')) - -/** - * Write a child script and run it under tsx to completion, collecting stdout - * and stderr and the exit code. (A file, not `tsx -e`: the eval form compiles - * to CommonJS, which has no top-level await.) - */ -function runChild( - scriptDir: string, - body: string -): Promise<{ code: number | null; out: string }> { - const scriptPath = join(scriptDir, 'child-process.mts') - writeFileSync(scriptPath, body) - // The child is an ORDINARY consumer process, so it runs the real embedding - // pipeline: this suite's deterministic-embedder switch is inherited through - // the environment, and under it `find()` self-retrieval returns nothing — - // which would make the read half of this pin vacuous. (That property is the - // deterministic embedder's, not this change's: it reproduces in a plain - // script with no 'beforeExit' involved.) - const env = { ...process.env } - delete env.BRAINY_DETERMINISTIC_EMBEDDINGS - const child = spawn(TSX, [scriptPath], { - cwd: REPO_ROOT, - stdio: ['ignore', 'pipe', 'pipe'], - env - }) - let out = '' - child.stdout?.on('data', (d) => { out += String(d) }) - child.stderr?.on('data', (d) => { out += String(d) }) - return new Promise((resolvePromise) => { - child.on('exit', (code) => resolvePromise({ code, out })) - }) -} - -describe('beforeExit never closes a live brain', () => { - let dir: string - let scriptDir: string - let resultPath: string - - beforeEach(() => { - dir = makeTempDir('brainy-beforeexit-') - scriptDir = makeTempDir('brainy-beforeexit-script-') - resultPath = join(scriptDir, 'result.json') - }) - - afterEach(() => { - for (const d of [dir, scriptDir]) { - try { rmSync(d, { recursive: true, force: true }) } catch { /* ignore */ } - } - }) - - it('(1)+(2) a drained event loop flushes, closes nothing, and the script keeps working', async () => { - /** - * THE DRAIN, and why the script survives it. The script awaits a promise - * that only an UNREF'd timer will resolve — the shape every engine cadence - * timer has, and the reason a healthy script reaches a loop with no ref'd - * work. Node emits `'beforeExit'` there, with the brain wide open. - * - * The engine's listener runs first (registered by `init()`, before the - * script's). The script's own listener is both its witness — it records - * that the emit happened, and the flush count AT that moment — and its - * belt: it resolves the same promise, so the pin never depends on how many - * milliseconds the engine's pass happens to keep the loop turning. - * - * The brain is DIRTY at the drain (one add, after a settling flush), so - * the pass has real work to do and pin (2) is about a flush that ran, not - * a flush that was skipped as a no-op. - */ - const script = ` - import { writeFileSync as __writeFileSync, existsSync as __existsSync } from 'node:fs' - import { join as __join } from 'node:path' - import { Brainy } from ${JSON.stringify(BRAINY_SRC)} - - const DIR = ${JSON.stringify(dir)} - const brain = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: DIR } }) - await brain.init() - - // Count every flush that RUNS on this brain. An own property shadows the - // prototype for every caller, including the engine's own listeners. - let flushes = 0 - const flushImpl = brain.flush.bind(brain) - brain.flush = () => { flushes++; return flushImpl() } - // ...and every close ENTERED. This must still be 0 after the drain. - let closes = 0 - const closeImpl = brain.close.bind(brain) - brain.close = () => { closes++; return closeImpl() } - - await brain.add({ data: 'written before the drain', type: 'concept' }) - await brain.flush() // settle: clean brain - await new Promise((r) => setTimeout(r, 250)) // let the cadence quiet down - await brain.add({ data: 'the write the drain must persist', type: 'concept' }) - - const flushesBeforeDrain = flushes - let drains = 0 - let flushesAtDrain = -1 - const drained = new Promise((resolve) => { - const t = setTimeout(resolve, 5) - if (typeof t.unref === 'function') t.unref() - process.on('beforeExit', () => { - drains++ - if (flushesAtDrain === -1) flushesAtDrain = flushes - resolve() - }) - }) - await drained - - // GIVE THE ENGINE'S PASS ITS FULL TURN before judging it. The signal - // path this listener used to share defers one macrotask before it - // touches an instance, so a script that resumes on the same tick as the - // emit would race past the damage and see an open brain that is about to - // be closed underneath it. Wait it out (a ref'd timer — the drain has - // already happened), then look. - await new Promise((r) => setTimeout(r, 1000)) - - // ---- The script is still running. The brain must still be its brain. ---- - const stateAtResume = { - drains, - flushesBeforeDrain, - flushesAtDrain, - closes, - isClosed: brain.isClosed, - isClosing: brain.isClosing, - writerLockHeld: __existsSync(__join(DIR, 'locks', '_writer.lock')), - cleanCloseRecord: __existsSync(__join(DIR, 'locks', '_writer.close')), - cleanShutdownMarker: - __existsSync(__join(DIR, '_system', 'clean-shutdown.json.gz')) || - __existsSync(__join(DIR, '_system', 'clean-shutdown.json')) - } - - let addAfterDrain = null - let addError = null - try { - addAfterDrain = await brain.add({ data: 'written AFTER the drained event loop', type: 'concept' }) - } catch (error) { - addError = error instanceof Error ? error.message : String(error) - } - - let findHits = -1 - let findError = null - try { - const results = await brain.find('written AFTER the drained event loop') - findHits = results.length - } catch (error) { - findError = error instanceof Error ? error.message : String(error) - } - - __writeFileSync( - ${JSON.stringify(resultPath)}, - JSON.stringify({ ...stateAtResume, addAfterDrain, addError, findHits, findError, closesBeforeOurs: closes }) - ) - - // The script ends the way a script ends: it closes its own brain, and - // the process exits on its own because nothing is left holding the loop. - await brain.close() - ` - - const { code, out } = await runChild(scriptDir, script) - - expect(existsSync(resultPath), `child wrote no result file:\n${out}`).toBe(true) - const r = JSON.parse(readFileSync(resultPath, 'utf-8')) - - // The drain really happened — this test proves nothing otherwise. - expect(r.drains, `'beforeExit' never fired:\n${out}`).toBeGreaterThanOrEqual(1) - - // (1) NOTHING WAS CLOSED. This is the regression: under 10.4.11 the pass - // ran close() here and `addError` carried "it was closed via close()". - expect(r.addError, `add() after the drain failed:\n${out}`).toBeNull() - expect(r.findError, `find() after the drain failed:\n${out}`).toBeNull() - expect(r.closes, 'the engine closed the brain on a drained event loop').toBe(0) - expect(r.isClosed).toBe(false) - expect(r.isClosing).toBe(false) - expect(typeof r.addAfterDrain).toBe('string') - expect(r.findHits, `find() returned nothing:\n${out}`).toBeGreaterThanOrEqual(1) - - // (1) The writer lock was never given up — a drained loop is not a handover. - expect(r.writerLockHeld, 'the writer lock was released on a drained event loop').toBe(true) - - // (2) A flush RAN, and it wrote neither of close()'s markers. - expect( - r.flushesAtDrain, - `the drained-loop pass ran no flush (before=${r.flushesBeforeDrain}):\n${out}` - ).toBeGreaterThan(r.flushesBeforeDrain) - expect(r.cleanShutdownMarker, 'the drained-loop flush stamped a clean-shutdown marker').toBe(false) - expect(r.cleanCloseRecord, 'the drained-loop flush wrote a clean-close record').toBe(false) - expect(out).toMatch(/All indexes flushed to disk/) - - // The narration says what happened, and never claims a shutdown. - expect(out).toMatch(/event loop drained with 1 brain open/) - expect(out).toMatch(/NOTHING was closed\. A drained loop is not a shutdown/) - expect(out).not.toMatch(/Shutdown signal received/) - expect(out).not.toMatch(/Flushed successfully/) - expect(out).not.toMatch(/is not initialized/) - - // (1) And the process still exits 0 on its own once the script closes up. - expect(code, `child output:\n${out}`).toBe(0) - - // The store the script left behind is clean: it closed properly at the end. - expect(cleanShutdownWritten(dir), 'the script\'s own close() wrote no marker').toBe(true) - expect(existsSync(closeRecordPath(dir)), 'the script\'s own close() left no clean-close record').toBe(true) - expect(existsSync(writerLockPath(dir)), 'the writer lock outlived close()').toBe(false) - }, 300_000) - - it('(2) the pass is repeatable and idempotent: a second drain closes nothing either', async () => { - // In-process, so the assertions are on the object itself rather than on a - // report: 'beforeExit' is an ordinary event, and emitting it twice must - // leave the brain exactly as usable as it was. - const brain = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } }) - await brain.init() - - const flushed: Promise[] = [] - const flushImpl = brain.flush.bind(brain) - ;(brain as unknown as { flush: () => Promise }).flush = () => { - const p = flushImpl() - flushed.push(p) - return p - } - - await brain.add({ data: 'a write the drained loop must persist', type: NounType.Concept }) - - for (const pass of [1, 2]) { - const before = flushed.length - process.emit('beforeExit', 0) - await Promise.all(flushed.slice(before).map((p) => p.catch(() => {}))) - // Let the pass's own `finally` run (it settles a microtask after ours), - // so the next emit is not turned away by the in-flight guard. - await new Promise((r) => setTimeout(r, 50)) - - expect(brain.isClosed, `pass ${pass} closed the brain`).toBe(false) - expect(brain.isClosing, `pass ${pass} started a close`).toBe(false) - expect(existsSync(writerLockPath(dir)), `pass ${pass} released the writer lock`).toBe(true) - expect(existsSync(closeRecordPath(dir)), `pass ${pass} wrote a clean-close record`).toBe(false) - expect(cleanShutdownWritten(dir), `pass ${pass} stamped a clean-shutdown marker`).toBe(false) - - // Still a working brain, after every pass. - const id = await brain.add({ data: `still writable after drain ${pass}`, type: NounType.Concept }) - expect(id).toBeTruthy() - } - - // The first pass had a dirty brain and flushed it; the second found it - // clean and cost nothing. Either way, neither closed anything. - expect(flushed.length).toBeGreaterThanOrEqual(2) - - await brain.close() - expect(brain.isClosed).toBe(true) - expect(cleanShutdownWritten(dir)).toBe(true) - }, 300_000) -}) diff --git a/tests/integration/entity-confidence-weight.test.ts b/tests/integration/entity-confidence-weight.test.ts index 031d29f1..b5bb34c5 100644 --- a/tests/integration/entity-confidence-weight.test.ts +++ b/tests/integration/entity-confidence-weight.test.ts @@ -7,7 +7,7 @@ * - Backward compatibility preserved */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy } from '../../src/brainy.js' import { NounType } from '../../src/types/graphTypes.js' @@ -19,10 +19,6 @@ describe('Entity Confidence & Weight Exposure', () => { await brain.init() }) - afterEach(async () => { - await brain.close() - }) - describe('Entity interface', () => { it('should expose confidence when adding entity with confidence', async () => { const id = await brain.add({ diff --git a/tests/integration/find-connected-order.test.ts b/tests/integration/find-connected-order.test.ts index 3b7560e4..b04e7f99 100644 --- a/tests/integration/find-connected-order.test.ts +++ b/tests/integration/find-connected-order.test.ts @@ -67,13 +67,6 @@ describe('find({ connected }) is graph-first: neighbours → filter → page', ( }) afterAll(async () => { - // CLOSE IT. Dropping the reference does not close a brain — it only makes - // it unreachable from here. The instance stays open and registered, its - // unref'd cadence timer keeps running, and because the gate config runs the - // whole suite in ONE process (pool: 'forks', singleFork: true) it goes on - // narrating its flushes into every test file that runs after this one. - // A test that leaks a brain is a defect of the test. - await brain?.close() brain = null as any }) @@ -144,34 +137,13 @@ describe('find({ connected }) is graph-first: neighbours → filter → page', ( }) it('walks the vector leg over the neighbours only', async () => { - // The SAME query without the vector leg, first. Both legs draw from the - // one neighbour set, so this is the control: it says whether a short answer - // came from the adjacency/filter (both legs short) or from the vector walk - // alone (only the vector leg short). Cheap, and it turns a bare count - // mismatch into a named half — this case has gone red on the gate box - // while passing in isolation and beside its own predecessor, so the next - // red must arrive already carrying the half it belongs to. - const control = await brain.find({ - connected: { from: anchor, direction: 'out' }, - where: { kind: 'note' }, - limit: 5 - }) - const results = await brain.find({ vector: sharedVector, connected: { from: anchor, direction: 'out' }, where: { kind: 'note' }, limit: 5 }) - - expect( - results.length, - `the vector leg returned ${results.length} of a requested 5. The same query ` + - `WITHOUT the vector returned ${control.length}: if that is also short the ` + - `neighbour set or the filter is the cause, and if it is 5 the vector walk is — ` + - `note every row in this corpus carries an identical vector, so the walk is ` + - `ranking an exact tie.` - ).toBe(5) + expect(results).toHaveLength(5) for (const r of results) expect(neighbourIds.has(r.entity.id)).toBe(true) }) diff --git a/tests/integration/find-fields-projection.test.ts b/tests/integration/find-fields-projection.test.ts deleted file mode 100644 index 25ee416c..00000000 --- a/tests/integration/find-fields-projection.test.ts +++ /dev/null @@ -1,265 +0,0 @@ -/** - * @module tests/integration/find-fields-projection - * @description **Field projection** — `find/get({ fields })` returns only the - * named fields, and serves them from the index when it can. - * - * A list view that shows a title and a slug does not need the document body, - * yet without a projection every row hydrates its whole record and discards - * almost all of it. These pins hold the two halves of the fix: - * - * **The answer.** A projected row is a SUBSET of the full row — for every - * requested field, the projected value equals the value the same query returns - * unprojected. Absent `fields` is byte-identical to today. A requested field the - * entity does not carry is simply absent, never an error. `system.*` resolves to - * the engine scalar, a bare name to the user's metadata. - * - * **The cost.** When every requested field is index-served, the canonical - * record is never opened — asserted by counting reads, not by timing them, so - * it cannot flake into a false green. When one requested field is NOT - * index-served (a body field, or a bucketed timestamp), exactly the owing rows - * are read and the rest are still served from the index. - */ -import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest' -import { Brainy } from '../../src/brainy' -import { NounType } from '../../src/types/graphTypes' -import { generateTestVector } from '../helpers/test-factory' - -/** Rows carrying a title, a slug, and a large body nobody wants in a list. */ -const ROWS = 12 -const BODY = 'x'.repeat(4096) - -describe('find/get({ fields }) — projection', () => { - let brain: Brainy - const ids: string[] = [] - - beforeAll(async () => { - brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' } }) - await brain.init() - for (let i = 0; i < ROWS; i++) { - ids.push( - await brain.add({ - id: `post-${i}`, - data: `post ${i}`, - type: NounType.Thing, - metadata: { - kind: 'post', - title: `Title ${i}`, - slug: `slug-${i}`, - rank: i, - body: BODY, - // Only some rows carry this, so "missing is absent" is exercised - // by real data rather than by a name nothing ever had. - ...(i % 2 === 0 ? { featured: true } : {}) - }, - vector: generateTestVector() - }) - ) - } - // Persist so the column store holds the values a projection reads from. - await brain.flush() - }) - - afterAll(async () => { - await brain.close() - }) - - /** Count canonical record reads for one call. */ - const countingReads = async (body: () => Promise): Promise<{ out: R; reads: number }> => { - const spy = vi.spyOn(brain as any, 'batchGet') - try { - const out = await body() - const reads = spy.mock.calls.reduce( - (n, call) => n + ((call[0] as string[] | undefined)?.length ?? 0), - 0 - ) - return { out, reads } - } finally { - spy.mockRestore() - } - } - - it('absent fields is byte-identical to today', async () => { - const params = { where: { kind: 'post' }, limit: 5 } - const a = await brain.find({ ...params }) - const b = await brain.find({ ...params, fields: undefined }) - expect(JSON.stringify(b)).toBe(JSON.stringify(a)) - }) - - it('a projected row is a SUBSET of the full row, field for field', async () => { - const shapes: Array> = [ - { where: { kind: 'post' }, limit: 6 }, - { where: { kind: 'post' }, limit: 6, offset: 3 }, - { where: { kind: 'post' }, orderBy: 'rank', order: 'asc', limit: 6 }, - { where: { kind: 'post' }, orderBy: 'rank', order: 'desc', limit: 4 } - ] - for (const shape of shapes) { - const full = await brain.find(shape as never) - const projected = await brain.find({ ...shape, fields: ['title', 'slug'] } as never) - expect(projected.map((r) => r.id), JSON.stringify(shape)).toEqual(full.map((r) => r.id)) - for (let i = 0; i < full.length; i++) { - const fullMeta = (full[i].entity.metadata ?? {}) as Record - const projMeta = (projected[i].entity.metadata ?? {}) as Record - expect(projMeta.title, `${JSON.stringify(shape)} row ${i}`).toEqual(fullMeta.title) - expect(projMeta.slug).toEqual(fullMeta.slug) - } - } - }) - - it('returns ONLY the named fields — the body never rides along', async () => { - const rows = await brain.find({ where: { kind: 'post' }, fields: ['title'], limit: 4 }) - expect(rows).toHaveLength(4) - for (const r of rows) { - const meta = (r.entity.metadata ?? {}) as Record - expect(Object.keys(meta)).toEqual(['title']) - expect(meta.body).toBeUndefined() - // Identity always survives a projection: a row you cannot identify is - // not a row. - expect(typeof r.id).toBe('string') - expect(r.entity.id).toBe(r.id) - } - }) - - it('a missing field is simply ABSENT — never an error', async () => { - // `featured` exists on half the rows; `no-such-field` on none. Neither - // throws, and neither appears as an explicit undefined. - const rows = await brain.find({ - where: { kind: 'post' }, - fields: ['title', 'featured', 'no-such-field'], - limit: ROWS - }) - expect(rows.length).toBeGreaterThan(0) - let withFeatured = 0 - for (const r of rows) { - const meta = (r.entity.metadata ?? {}) as Record - expect('no-such-field' in meta).toBe(false) - if ('featured' in meta) withFeatured += 1 - } - // Real data, not a name nothing ever had: some rows carry it, some do not. - expect(withFeatured).toBeGreaterThan(0) - expect(withFeatured).toBeLessThan(rows.length) - }) - - it('a strict address resolver is NOT on this path', async () => { - // orderBy throws UnresolvableFieldError for an unknown user key, because a - // typo there silently changes the order. A projection must not inherit that - // strictness: the honest answer to "give me this if you have it" is silence. - await expect( - brain.find({ where: { kind: 'post' }, fields: ['definitely-not-a-field'], limit: 2 }) - ).resolves.toBeInstanceOf(Array) - }) - - it('system.* resolves to the engine scalar, a bare name to user metadata', async () => { - const full = await brain.find({ where: { kind: 'post' }, limit: 3 }) - const rows = await brain.find({ - where: { kind: 'post' }, - fields: ['system.createdAt', 'title'], - limit: 3 - }) - for (let i = 0; i < rows.length; i++) { - expect((rows[i].entity as any).createdAt).toEqual((full[i].entity as any).createdAt) - const meta = (rows[i].entity.metadata ?? {}) as Record - expect(meta.title).toEqual((full[i].entity.metadata as any).title) - // The engine scalar lands at the top level, not in the metadata bag — - // the two address spaces never shadow each other. - expect('system.createdAt' in meta).toBe(false) - expect('createdAt' in meta).toBe(false) - } - }) - - it('reads NO canonical record when every requested field is index-served', async () => { - // The cost pin, counted rather than timed. `title` and `slug` are ordinary - // indexed user fields, so the index can serve them exactly. - const { out, reads } = await countingReads(() => - brain.find({ where: { kind: 'post' }, fields: ['title', 'slug'], limit: ROWS }) - ) - expect(out.length).toBeGreaterThan(0) - expect(reads).toBe(0) - }) - - it('reads records only for the fields the column cannot serve', async () => { - // `system.data` is NOT a column the store holds (verified against - // getIndexedFields), so the record must be opened for it — while `title`, - // which the column does hold, still comes from the index. - const { out, reads } = await countingReads(() => - brain.find({ where: { kind: 'post' }, fields: ['title', 'system.data'], limit: 4 }) - ) - expect(out).toHaveLength(4) - expect(reads).toBe(4) - for (const r of out) { - const meta = (r.entity.metadata ?? {}) as Record - expect(Object.keys(meta)).toEqual(['title']) - expect(typeof (r.entity as any).data).toBe('string') - } - }) - - it('a large field the column DOES hold costs no record read', async () => { - // Worth pinning because it is the venue case: the body is column-served on - // this engine, so a list that projects around it pays nothing for it, and - // a list that projects it still pays no record read. - const { reads } = await countingReads(() => - brain.find({ where: { kind: 'post' }, fields: ['body'], limit: 4 }) - ) - expect(reads).toBe(0) - }) - - it('projects a vector-leg find too — the ANSWER is uniform, only the cost is not', async () => { - // The seam hydrates the metadata and graph page paths. A vector or text leg - // builds its own entities, so those rows are trimmed after the integrity - // guard instead. That difference is a COST difference, and this pin exists - // so it can never quietly become an ANSWER difference. - const rows = await brain.find({ query: 'post', fields: ['title'], limit: 3 }) - for (const r of rows) { - const meta = (r.entity.metadata ?? {}) as Record - expect(Object.keys(meta)).toEqual(['title']) - expect(meta.body).toBeUndefined() - expect(r.entity.id).toBe(r.id) - } - }) - - it('get({ fields }) projects a single row through the same seam', async () => { - const full = await brain.get(ids[0]) - const projected = await brain.get(ids[0], { fields: ['title', 'slug'] }) - expect(projected).not.toBeNull() - expect(projected!.id).toBe(full!.id) - const fullMeta = (full!.metadata ?? {}) as Record - const projMeta = (projected!.metadata ?? {}) as Record - expect(projMeta.title).toEqual(fullMeta.title) - expect(projMeta.slug).toEqual(fullMeta.slug) - expect(Object.keys(projMeta).sort()).toEqual(['slug', 'title']) - expect((projected as any).body).toBeUndefined() - }) - - it('get({ fields }) reads no record when the index serves the fields', async () => { - const { reads } = await countingReads(() => brain.get(ids[1], { fields: ['title'] })) - expect(reads).toBe(0) - }) - - it('the door serves EXACT values — the column, never the bucketed index', async () => { - // The sparse index buckets `system.createdAt` to the minute for range - // queries; the column store keeps raw ms. Serving a projection from the - // former would hand back a value that differs from the record's, so the - // door reads the column — and this pin is what proves which one it read. - const index = (brain as any).metadataIndex - const sample = ids.slice(0, 3) - const served = await index.getScalarsForIds(sample, ['title', 'system.createdAt']) - expect(served.size).toBe(sample.length) - for (const id of sample) { - const row = served.get(id)! - const record = await brain.get(id) - expect(row.title).toEqual((record!.metadata as any).title) - // Exact to the millisecond — a bucketed value would be rounded down to - // the minute and this would fail. - expect(row['system.createdAt']).toEqual((record as any).createdAt) - } - }) - - it('a field the column store does not hold is OMITTED, not approximated', async () => { - const index = (brain as any).metadataIndex - const served = await index.getScalarsForIds(ids.slice(0, 2), ['title', 'system.data']) - for (const [, row] of served) { - expect('title' in row).toBe(true) - // Omission is what makes the caller read the record for it. - expect('system.data' in row).toBe(false) - } - }) -}) diff --git a/tests/integration/find-hybrid-filter-before-hydrate.test.ts b/tests/integration/find-hybrid-filter-before-hydrate.test.ts index 7f326729..3e74f5d8 100644 --- a/tests/integration/find-hybrid-filter-before-hydrate.test.ts +++ b/tests/integration/find-hybrid-filter-before-hydrate.test.ts @@ -31,7 +31,7 @@ * never the legs. And the text leg is asked about the universe's ids only — * what it marshals is bounded by the universe, not by the store. */ -import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest' +import { describe, it, expect, beforeAll, vi } from 'vitest' import { Brainy } from '../../src/brainy' import { NounType, VerbType } from '../../src/types/graphTypes' import { rankIndicesByScore, reorderByIndices } from '../../src/utils/resultRanking' @@ -287,10 +287,6 @@ describe('hybrid find: filter before hydrate — the answer is unchanged', () => expect(typeof (brain as any).metadataIndex.getIdSetForFilter).not.toBe('function') }) - afterAll(async () => { - await brain.close() - }) - it('the fixture does not truncate the text leg — the universe covers every text match', async () => { const index = (brain as any).metadataIndex const textMatches = await index.getIdsForTextQuery(QUERY) @@ -557,10 +553,6 @@ describe('hybrid find: the text leg ranks inside the filter, not around it', () } }) - afterAll(async () => { - await brain.close() - }) - it('the old order let the filter consume the whole text leg', async () => { const index = (brain as any).metadataIndex const universe: string[] = await (brain as any).filterIdsBelted({ lane: 'alpha' }) diff --git a/tests/integration/find-near.test.ts b/tests/integration/find-near.test.ts index b2bf01cd..3fb235c8 100644 --- a/tests/integration/find-near.test.ts +++ b/tests/integration/find-near.test.ts @@ -9,7 +9,7 @@ * it). Now the anchor is fetched with its vector, and an anchor without one * refuses by name instead of failing inside the index. */ -import { describe, it, expect, beforeAll, afterAll } from 'vitest' +import { describe, it, expect, beforeAll } from 'vitest' import { Brainy } from '../../src/brainy' import { NounType } from '../../src/types/graphTypes' import { v5 } from '../../src/universal/uuid' @@ -28,10 +28,6 @@ describe('find({ near }) uses the anchor vector', () => { await brain.add({ id: 'far', data: 'far row', type: NounType.Thing, vector: generateTestVector() }) }) - afterAll(async () => { - await brain.close() - }) - it('returns the anchor\'s neighbours by its own vector', async () => { const results = await brain.find({ near: { id: 'anchor' }, limit: 3 }) expect(results.length).toBeGreaterThan(0) diff --git a/tests/integration/find-orderby-every-path.test.ts b/tests/integration/find-orderby-every-path.test.ts index e62ec670..7637a79b 100644 --- a/tests/integration/find-orderby-every-path.test.ts +++ b/tests/integration/find-orderby-every-path.test.ts @@ -40,7 +40,7 @@ * the covering is ASSERTED from the leg's own output rather than assumed. This * pin is about ordering, and it says nothing about recall. */ -import { describe, it, expect, beforeAll, afterAll } from 'vitest' +import { describe, it, expect, beforeAll } from 'vitest' import { Brainy } from '../../src/brainy' import { NounType, VerbType } from '../../src/types/graphTypes' import { resolveEntityId } from '../../src/utils/idNormalization' @@ -107,10 +107,6 @@ describe('find(): orderBy is the order on every path', () => { } }) - afterAll(async () => { - await brain.close() - }) - it('the fixture: the hybrid candidate set covers the whole filter universe', async () => { const universe: string[] = await (brain as any).filterIdsBelted({ lane: 'alpha' }) expect(universe).toHaveLength(ROWS) diff --git a/tests/integration/find-planner-door.test.ts b/tests/integration/find-planner-door.test.ts index e5224f6d..964b13f9 100644 --- a/tests/integration/find-planner-door.test.ts +++ b/tests/integration/find-planner-door.test.ts @@ -23,7 +23,7 @@ * against the adjacency before it is believed, so a not-serving graph refuses * loudly instead of answering `[]` as truth. */ -import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest' +import { describe, it, expect, beforeAll, vi } from 'vitest' import { Brainy } from '../../src/brainy' import { NounType, VerbType } from '../../src/types/graphTypes' import { generateTestVector } from '../helpers/test-factory' @@ -56,10 +56,6 @@ describe('find(): the optional planner door', () => { } }) - afterAll(async () => { - await brain.close() - }) - /** Install a planner door for one call, then remove it. */ const withDoor = async ( door: (...a: any[]) => Promise, diff --git a/tests/integration/find-unified-integration.test.ts b/tests/integration/find-unified-integration.test.ts index 3c4741c2..94053d55 100644 --- a/tests/integration/find-unified-integration.test.ts +++ b/tests/integration/find-unified-integration.test.ts @@ -48,7 +48,6 @@ describe('Unified Find() Integration Tests', () => { afterAll(async () => { await cleanup.cleanup() - await brain.close() brain = null as any }) diff --git a/tests/integration/graphIndex-pagination.test.ts b/tests/integration/graphIndex-pagination.test.ts index 8ad4d6d8..32a7673c 100644 --- a/tests/integration/graphIndex-pagination.test.ts +++ b/tests/integration/graphIndex-pagination.test.ts @@ -9,34 +9,9 @@ * 8.0 BigInt boundary: entity ints in (resolved via the metadata index's * idMapper), entity/verb ints out (`bigint[]`). Entity ints map back to UUIDs * via `idMapper.getUuid(Number(int))`; verb ints via `verbIntsToIds()`. - * - * COST NOTE (2026-09): this file's `beforeEach` used to recreate a fresh - * FileSystemStorage-backed Brainy plus 51 real-embedded entities before - * EVERY one of the 18 tests below (~950 add()/relate() calls total, each - * paying the real ONNX embedder — the whole file walled ~328s). Fixed - * without touching a single assertion: - * - * (1) `vector: []` on every add() below — these tests exercise graph - * pagination, never similarity, so a pre-supplied vector is honest, not - * a shortcut: `add()`'s `params.vector || (await this.embed(...))` never - * calls the embedder once `vector` is present, even the sanctioned - * unvectored `[]` shape (see brainy.ts's add(), the zero-norm-law - * comment) — and the `vector.length > 0` gate on dimension-pinning means - * `[]` never poisons `this.dimensions` for later real embeds. - * (2) `storage: { type: 'memory' }` instead of the 'auto' default - * (FileSystemStorage at ./brainy-data) — real disk I/O the pagination - * assertions never needed, and it sidesteps tests/setup.ts's global - * per-test `rm -rf brainy-data`, which would otherwise corrupt a brain - * shared across a describe's beforeAll out from under it. - * (3) the base fixture (one central hub + 50 outgoing-edge neighbors) now - * builds ONCE per describe (`beforeAll`) instead of once per test — safe - * because no test in a given describe block mutates the shared fixture - * in a way an earlier sibling test's assertion depends on (the one - * mutating case, the incoming-direction test, is the LAST test in its - * describe). */ -import { describe, it, expect, beforeAll, afterAll } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy } from '../../src/brainy.js' import { NounType, VerbType } from '../../src/types/graphTypes.js' @@ -64,21 +39,14 @@ describe('GraphAdjacencyIndex Pagination', () => { .map((i) => idMapper().getUuid(Number(i))) .filter((u: string | undefined): u is string => u !== undefined) - /** - * Builds one central hub + 50 neighbor entities (all outgoing edges from - * the hub), unvectored and on in-memory storage (see the file header). - * Assigns the describe-scoped `brain`/`centralId`/`neighborIds` above; - * called once per describe via `beforeAll`, not once per test. - */ - async function buildFixture(): Promise { + beforeEach(async () => { brain = new Brainy({ requireSubtype: false }) - await brain.init({ storage: { type: 'memory' } }) + await brain.init() // Create central entity centralId = await brain.add({ data: { name: 'Central Hub' }, - type: NounType.Thing, - vector: [] + type: NounType.Thing }) // Create 50 neighbor entities with relationships @@ -86,8 +54,7 @@ describe('GraphAdjacencyIndex Pagination', () => { for (let i = 0; i < 50; i++) { const neighborId = await brain.add({ data: { name: `Neighbor ${i}`, index: i }, - type: NounType.Thing, - vector: [] + type: NounType.Thing }) neighborIds.push(neighborId) @@ -98,14 +65,9 @@ describe('GraphAdjacencyIndex Pagination', () => { type: VerbType.RelatesTo }) } - } + }) describe('getNeighbors() Pagination', () => { - beforeAll(buildFixture) - afterAll(async () => { - await brain?.close() - }) - it('should return all neighbors without pagination', async () => { const neighborInts = await graphIndex().getNeighbors(entityInt(centralId)) const neighbors = intsToUuids(neighborInts) @@ -187,8 +149,7 @@ describe('GraphAdjacencyIndex Pagination', () => { // Create some incoming relationships const sourceId = await brain.add({ data: { name: 'Source' }, - type: NounType.Thing, - vector: [] + type: NounType.Thing }) await brain.relate({ @@ -208,11 +169,6 @@ describe('GraphAdjacencyIndex Pagination', () => { }) describe('getVerbIdsBySource() Pagination', () => { - beforeAll(buildFixture) - afterAll(async () => { - await brain?.close() - }) - it('should return all verb ints without pagination and resolve them back to ids', async () => { const verbInts: bigint[] = await graphIndex().getVerbIdsBySource(entityInt(centralId)) @@ -267,11 +223,6 @@ describe('GraphAdjacencyIndex Pagination', () => { }) describe('getVerbIdsByTarget() Pagination', () => { - beforeAll(buildFixture) - afterAll(async () => { - await brain?.close() - }) - it('should return all verb ints targeting an entity', async () => { // Pick a neighbor that's a target of relationships const targetId = neighborIds[0] @@ -285,16 +236,14 @@ describe('GraphAdjacencyIndex Pagination', () => { // Create entity with many incoming relationships const popularTarget = await brain.add({ data: { name: 'Popular Target' }, - type: NounType.Thing, - vector: [] + type: NounType.Thing }) // Create 30 relationships pointing to it for (let i = 0; i < 30; i++) { const sourceId = await brain.add({ data: { name: `Source ${i}` }, - type: NounType.Thing, - vector: [] + type: NounType.Thing }) await brain.relate({ from: sourceId, @@ -318,11 +267,6 @@ describe('GraphAdjacencyIndex Pagination', () => { }) describe('Performance with Pagination', () => { - beforeAll(buildFixture) - afterAll(async () => { - await brain?.close() - }) - it('should maintain sub-5ms performance with pagination', async () => { const central = entityInt(centralId) @@ -341,17 +285,11 @@ describe('GraphAdjacencyIndex Pagination', () => { }) describe('Real-World Use Cases', () => { - beforeAll(buildFixture) - afterAll(async () => { - await brain?.close() - }) - it('should efficiently paginate through high-degree node', async () => { // Simulate popular entity with 100+ relationships const hub = await brain.add({ data: { name: 'Popular Hub' }, - type: NounType.Thing, - vector: [] + type: NounType.Thing }) // Create 100 relationships @@ -359,8 +297,7 @@ describe('GraphAdjacencyIndex Pagination', () => { for (let i = 0; i < 100; i++) { const targetId = await brain.add({ data: { name: `Target ${i}` }, - type: NounType.Thing, - vector: [] + type: NounType.Thing }) targetIds.push(targetId) await brain.relate({ diff --git a/tests/integration/id-normalization.test.ts b/tests/integration/id-normalization.test.ts index 1eb14ab1..1ea1a221 100644 --- a/tests/integration/id-normalization.test.ts +++ b/tests/integration/id-normalization.test.ts @@ -18,7 +18,7 @@ * All entities carry explicit 384-dim vectors so no test invokes the embedder. */ -import { describe, it, expect, afterEach } from 'vitest' +import { describe, it, expect } from 'vitest' import { Brainy } from '../../src/brainy.js' import { NounType, VerbType } from '../../src/types/graphTypes.js' import { v5, v7, isUUID } from '../../src/universal/uuid.js' @@ -37,15 +37,8 @@ async function makeBrain(): Promise { } describe('id normalization — transparent string-key round-trips', () => { - const opened: Brainy[] = [] - - afterEach(async () => { - for (const b of opened.splice(0)) await b.close().catch(() => {}) - }) - it('1. add() returns v5(key); get(key) and get(returnedId) both resolve; _originalId preserved', async () => { const brain = await makeBrain() - opened.push(brain) const returnedId = await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person }) @@ -67,7 +60,6 @@ describe('id normalization — transparent string-key round-trips', () => { it('2. relate() by string keys; related(key) and related({from:key}) return the edge to v5(toKey)', async () => { const brain = await makeBrain() - opened.push(brain) await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person }) await brain.add({ id: 'doc-1', vector: vec(2), type: NounType.Document }) @@ -93,7 +85,6 @@ describe('id normalization — transparent string-key round-trips', () => { it('3. update() by string key reflects on get(key)', async () => { const brain = await makeBrain() - opened.push(brain) await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person, metadata: { role: 'admin' } }) await brain.update({ id: 'user-1', metadata: { role: 'owner' } }) @@ -107,7 +98,6 @@ describe('id normalization — transparent string-key round-trips', () => { it('4. remove() by string key deletes; get(key) is null', async () => { const brain = await makeBrain() - opened.push(brain) await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person }) expect(await brain.get('user-1')).not.toBeNull() @@ -120,7 +110,6 @@ describe('id normalization — transparent string-key round-trips', () => { it('5. find({ connected: { from: key } }) resolves the anchor key', async () => { const brain = await makeBrain() - opened.push(brain) await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person }) await brain.add({ id: 'doc-1', vector: vec(2), type: NounType.Document }) @@ -133,7 +122,6 @@ describe('id normalization — transparent string-key round-trips', () => { it('6. transact() add+relate by string keys round-trips with consistent canonical ids', async () => { const brain = await makeBrain() - opened.push(brain) // Seed user-1 so the relate op has a target to point at. await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person }) @@ -161,7 +149,6 @@ describe('id normalization — transparent string-key round-trips', () => { it('7. addMany() + relateMany() with string ids round-trip', async () => { const brain = await makeBrain() - opened.push(brain) const added = await brain.addMany({ items: [ @@ -188,7 +175,6 @@ describe('id normalization — transparent string-key round-trips', () => { it('8. determinism: same key maps to same UUID — two adds upsert ONE entity, not two', async () => { const brain = await makeBrain() - opened.push(brain) const id1 = await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person, metadata: { n: 1 } }) const id2 = await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person, metadata: { n: 2 } }) @@ -207,7 +193,6 @@ describe('id normalization — transparent string-key round-trips', () => { it('9. valid-UUID passthrough: a real UUID is kept verbatim with NO _originalId', async () => { const brain = await makeBrain() - opened.push(brain) const realUuid = v7() const returnedId = await brain.add({ id: realUuid, vector: vec(5), type: NounType.Thing }) @@ -222,7 +207,6 @@ describe('id normalization — transparent string-key round-trips', () => { it('10. no-id add() mints a v7; newId() mints a v7', async () => { const brain = await makeBrain() - opened.push(brain) const autoId = await brain.add({ vector: vec(6), type: NounType.Thing }) expect(isUUID(autoId)).toBe(true) diff --git a/tests/integration/idle-costs-nothing.test.ts b/tests/integration/idle-costs-nothing.test.ts index 7e664a28..b5c386cf 100644 --- a/tests/integration/idle-costs-nothing.test.ts +++ b/tests/integration/idle-costs-nothing.test.ts @@ -70,22 +70,8 @@ describe('an idle brain costs nothing', () => { await brain.flush() const logged: string[] = [] - // The STACK behind each narration, kept beside the line it belongs to. - // vitest tags a stdout block with the test that is RUNNING, not the brain - // that wrote it, so teeing these lines through would only ever name this - // test. The call stack does name the driver: `kickBackgroundFlush('idle')` - // under `armIdleFlushTimer` is a cadence flush on some brain, the deferred- - // embed worker's commit path is a brain still landing vectors, and a bare - // `flush()` is an explicit caller. That distinction is the whole question. - const stacks: string[] = [] const origLog = console.log - console.log = ((...a: unknown[]) => { - const line = a.map(String).join(' ') - logged.push(line) - if (/All indexes flushed to disk|Flushing Brainy indexes/.test(line)) { - stacks.push(new Error('flush narration').stack ?? '(no stack)') - } - }) as typeof console.log + console.log = ((...a: unknown[]) => { logged.push(a.map(String).join(' ')) }) as typeof console.log // Watch the providers directly: a flush that runs calls all of them. const storage = (brain as unknown as { storage: { flushCounts: () => Promise } }).storage @@ -102,38 +88,11 @@ describe('an idle brain costs nothing', () => { } // (a) + (b): nothing ran, nothing was said. - // - // THE SPIES COME FIRST, AND THEY ARE THE ATTRIBUTABLE HALF. They are bound - // to THIS brain's providers, so they answer "did this brain flush?" and - // nothing else. The console filters below cannot: the gate config runs the - // whole suite in ONE process (`pool: 'forks'`, `singleFork: true`), so - // `console.log` carries the narration of every brain alive in that - // process — including one a previous file opened and never closed, whose - // unref'd cadence timer is still doing honest work. A neighbour narrating - // is a REAL finding about suite hygiene, but it is not this brain failing - // its own law, and the two must not be reported as the same thing. - // - // So: spies first (whose failure means the engine broke the law), console - // second (whose failure means SOMETHING in the process narrated), and the - // console assertion carries the captured lines in its message. vitest's - // stdout blocks are prefixed `stdout | > `, so those lines - // plus the surrounding gate log name the brain that printed them. + expect(logged.filter((l) => /All indexes flushed to disk/.test(l))).toEqual([]) + expect(logged.filter((l) => /Flushing Brainy indexes/.test(l))).toEqual([]) expect(countsSpy).not.toHaveBeenCalled() expect(metadataSpy).not.toHaveBeenCalled() expect(graphSpy).not.toHaveBeenCalled() - - const flushChatter = logged.filter( - (l) => /All indexes flushed to disk/.test(l) || /Flushing Brainy indexes/.test(l) - ) - expect( - flushChatter, - `${flushChatter.length} flush line(s) narrated during the ${IDLE_WATCH_MS}ms idle ` + - `window. This brain's own providers were NOT called (asserted above), so another ` + - `brain alive in this process printed them — the suite runs every file in ONE ` + - `process and 67 test files create more brains than they close.\n` + - `${flushChatter.join('\n')}\n\n` + - `The stack behind the first one names the driver:\n${stacks[0] ?? '(none captured)'}` - ).toEqual([]) }, 180_000) it('an explicit flush over a clean brain calls no provider and prints nothing', async () => { diff --git a/tests/integration/metadata-vector-exclusion.test.ts b/tests/integration/metadata-vector-exclusion.test.ts index 1943b215..0ca25388 100644 --- a/tests/integration/metadata-vector-exclusion.test.ts +++ b/tests/integration/metadata-vector-exclusion.test.ts @@ -161,8 +161,7 @@ describe('Metadata Vector Exclusion Fix', () => { // silence at a bound of 10 — the field simply vanished from the index and // the row dropped out of every `where` on it, indistinguishably from "no // row matches". The bound is now MAX_INDEXED_ARRAY_LENGTH and it REFUSES. - const overTheBound = MAX_INDEXED_ARRAY_LENGTH + 1 - const largeArray = Array.from({ length: overTheBound }, (_, i) => `item${i}`) + const largeArray = Array.from({ length: 100 }, (_, i) => `item${i}`) const err = await brainy .add({ @@ -177,7 +176,7 @@ describe('Metadata Vector Exclusion Fix', () => { expect(err).toBeInstanceOf(MetadataArrayTooLargeError) expect(err.field).toBe('items') - expect(err.length).toBe(overTheBound) + expect(err.length).toBe(100) expect(err.limit).toBe(MAX_INDEXED_ARRAY_LENGTH) // Nothing was indexed from the refused write — no 'items' field, and above diff --git a/tests/integration/multi-process-safety.test.ts b/tests/integration/multi-process-safety.test.ts index dd1b8901..592d7969 100644 --- a/tests/integration/multi-process-safety.test.ts +++ b/tests/integration/multi-process-safety.test.ts @@ -107,11 +107,7 @@ describe('Multi-process safety + read-only mode', () => { const blocked = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } }) await expect(blocked.init()).rejects.toThrow(/another writer holds/i) - // A rejected init() still registered `blocked` in Brainy's global - // instance registry (the constructor does that unconditionally) — close() - // is safe to call even though init() never completed, and is what - // deregisters it (and, once idle, the process-level shutdown hooks). - await blocked.close().catch(() => {}) + // Don't track `blocked` for afterEach cleanup since init failed. }) it('takes over a STALE foreign lock (dead PID + old heartbeat) and claims atomically', async () => { @@ -155,7 +151,6 @@ describe('Multi-process safety + read-only mode', () => { const err: any = await blocked.init().catch((e) => e) expect(err.code).toBe('BRAINY_WRITER_LOCKED') expect(err.lockInfo?.pid).toBe(otherPid) - await blocked.close().catch(() => {}) }) it('release drains an in-flight heartbeat — no phantom lock re-created after unlink', async () => { diff --git a/tests/integration/related-verb-array.test.ts b/tests/integration/related-verb-array.test.ts index 7ed1bd3f..36a49850 100644 --- a/tests/integration/related-verb-array.test.ts +++ b/tests/integration/related-verb-array.test.ts @@ -30,7 +30,6 @@ describe('related() with a verb-type array returns every requested type', () => }) afterAll(async () => { - await brain.close() brain = null as any }) diff --git a/tests/integration/relationship-intelligence.test.ts b/tests/integration/relationship-intelligence.test.ts index c18057fb..b6e11cb5 100644 --- a/tests/integration/relationship-intelligence.test.ts +++ b/tests/integration/relationship-intelligence.test.ts @@ -59,8 +59,7 @@ describe('Relationship Intelligence', () => { await brain.init() }) - afterEach(async () => { - await brain.close() + afterEach(() => { if (fs.existsSync(testDir)) { fs.rmSync(testDir, { recursive: true }) } diff --git a/tests/integration/rev-and-ifabsent.test.ts b/tests/integration/rev-and-ifabsent.test.ts index 3bff59f1..64b184a3 100644 --- a/tests/integration/rev-and-ifabsent.test.ts +++ b/tests/integration/rev-and-ifabsent.test.ts @@ -9,7 +9,7 @@ * - addMany({ ifAbsent: true }) applies the flag to every item */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy } from '../../src/brainy.js' import { RevisionConflictError } from '../../src/transaction/RevisionConflictError.js' import { NounType } from '../../src/types/graphTypes.js' @@ -22,10 +22,6 @@ describe('7.31.0 — _rev CAS + ifAbsent', () => { await brain.init() }) - afterEach(async () => { - await brain.close() - }) - describe('_rev initialization + surface', () => { it('initializes _rev to 1 on add()', async () => { const id = await brain.add({ data: 'hello', type: NounType.Document }) diff --git a/tests/integration/triple-intelligence-correctness.test.ts b/tests/integration/triple-intelligence-correctness.test.ts deleted file mode 100644 index 53848d1a..00000000 --- a/tests/integration/triple-intelligence-correctness.test.ts +++ /dev/null @@ -1,172 +0,0 @@ -/** - * Triple Intelligence Correctness Tests - * - * Moved out of tests/performance/triple-intelligence-scale.test.ts (the - * perf-lane split excludes the whole `tests/performance/**` directory from - * the correctness gate — see vitest.config.ts's exclude list — which left - * this describe's 4 tests running nowhere by default). Every `expect(...)` - * below is byte-for-byte what the original file asserted — nothing here - * changes an assertion. - * - * Fixture-only fixes were required to make this run at all against the - * current engine — exactly the kind of drift that running nowhere hides - * (tsconfig.json excludes `**\/*.test.ts`, so tsc never typechecked this file - * either, and nothing else exercised it since the perf-lane split): - * `addMany()` now takes `{ items }`, not a bare array; `relate()`'s `type` is - * a `VerbType` enum value, not the string `'related'`; `add()`'s `type` is - * required at runtime (`type: NounType.Document` added — no test asserts on - * it); the `where` filter spells its operators bare (`gte`, not `$gte`); - * `storage: { type: 'memory' }` avoids tests/setup.ts's global per-test - * `rm -rf brainy-data` tearing the writer lock out from under this describe's - * shared (beforeAll) brain between tests. - * - * Two of the four tests are `it.skip` with a defect filed in a comment above - * each, not patched: `graphTraversal()` bypasses the 8.0 id-normalization law - * (a natural-key `connected.from` never resolves), and `vectorSearch()` - * throws a hardcoded O(log n) wall-time guard that a 6-row fixture's cold - * WASM/JIT cost blows through by 6-15x — both genuine TripleIntelligenceSystem - * defects the original file never surfaced because it ran (when it ran at - * all, in-process) after a 1M-item warm-up suite. See each skip's comment. - */ - -import { describe, it, expect, beforeAll, afterAll } from 'vitest' -import { Brainy } from '../../src/brainy.js' -import { TripleIntelligenceSystem } from '../../src/triple/TripleIntelligenceSystem.js' -import { NounType, VerbType } from '../../src/types/graphTypes.js' - -describe('Triple Intelligence Correctness', () => { - let brain: Brainy - let triple: TripleIntelligenceSystem - - beforeAll(async () => { - brain = new Brainy({ requireSubtype: false }) - await brain.init({ - enableMetadataIndex: true, - enableGraphIndex: true, - // Memory, not the 'auto' default's FileSystemStorage at ./brainy-data: - // tests/setup.ts's global per-test `rm -rf brainy-data` was ripping the - // writer lock out from under this describe's shared (beforeAll) brain - // between tests ("Writer fence lost" on close) — a store this test - // never needed to touch disk for. - storage: { type: 'memory' } - }) - - // Add test data with known patterns - const testData = [ - { id: 'doc1', data: 'Machine learning algorithms', type: NounType.Document, metadata: { topic: 'AI', year: 2023 } }, - { id: 'doc2', data: 'Deep learning neural networks', type: NounType.Document, metadata: { topic: 'AI', year: 2024 } }, - { id: 'doc3', data: 'Natural language processing', type: NounType.Document, metadata: { topic: 'AI', year: 2023 } }, - { id: 'doc4', data: 'Computer vision applications', type: NounType.Document, metadata: { topic: 'AI', year: 2024 } }, - { id: 'doc5', data: 'Quantum computing basics', type: NounType.Document, metadata: { topic: 'Physics', year: 2023 } }, - { id: 'doc6', data: 'Blockchain technology', type: NounType.Document, metadata: { topic: 'Crypto', year: 2024 } } - ] - - await brain.addMany({ items: testData }) - - // Add relationships - await brain.relate({ from: 'doc1', to: 'doc2', type: VerbType.RelatedTo }) - await brain.relate({ from: 'doc2', to: 'doc3', type: VerbType.RelatedTo }) - await brain.relate({ from: 'doc3', to: 'doc4', type: VerbType.RelatedTo }) - - triple = brain.getTripleIntelligence() - }) - - afterAll(async () => { - await brain?.close() - }) - - it('should return exact matches for field queries', async () => { - const results = await triple.find({ - where: { topic: 'AI' }, - limit: 10 - }) - - expect(results).toHaveLength(4) - for (const result of results) { - expect(result.metadata.topic).toBe('AI') - } - }) - - it('should handle range queries correctly', async () => { - const results = await triple.find({ - where: { year: { gte: 2024 } }, - limit: 10 - }) - - expect(results).toHaveLength(3) - for (const result of results) { - expect(result.metadata.year).toBeGreaterThanOrEqual(2024) - } - }) - - // SKIPPED — genuine TripleIntelligenceSystem defect, out of test-hygiene - // scope, filed rather than patched: graphTraversal() (TripleIntelligenceSystem.ts) - // calls storage.getNoun(id) / graphIndex.getNeighbors(id) directly with the - // caller's raw `connected.from` string, bypassing the 8.0 id-normalization - // law (Brainy.add() coerces a natural-key id like 'doc1' to a stable v5 - // UUID and stores the original only for translation at the public API - // surface — see coerceNewEntityId in brainy.ts). A caller passing a - // natural-key id here gets storage.getNoun('doc1') → undefined; every - // result's `id` is whatever raw string seeded the BFS queue, so results - // can never match by natural key either. Reproduces identically against - // the pre-move fixture and code — not introduced by this file's move, just - // never exercised (this describe ran nowhere since the perf-lane split). - it.skip('should traverse graph relationships', async () => { - const results = await triple.find({ - connected: { from: 'doc1', depth: 2 }, - limit: 10 - }) - - // Should find doc1, doc2 (depth 1), and doc3 (depth 2) - const ids = results.map(r => r.id) - expect(ids).toContain('doc1') - expect(ids).toContain('doc2') - expect(ids).toContain('doc3') - - // Check depth values - const doc1Result = results.find(r => r.id === 'doc1') - const doc2Result = results.find(r => r.id === 'doc2') - const doc3Result = results.find(r => r.id === 'doc3') - - expect(doc1Result?.depth).toBe(0) - expect(doc2Result?.depth).toBe(1) - expect(doc3Result?.depth).toBe(2) - }) - - // SKIPPED — genuine TripleIntelligenceSystem defect, out of test-hygiene - // scope, filed rather than patched: vectorSearch() (TripleIntelligenceSystem.ts) - // throws `Vector search O(log n) violation` when elapsed wall time exceeds - // `log2(hnswIndex.size()) * 5 * 2` — on a 6-row fixture that bound is - // ~25.8ms, which the real cost of a WASM/Candle embed call plus first-call - // JIT/cache warmup blows through by 6-15x (measured 166-375ms across - // repeated runs) — a hardcoded constant that assumes an already-warm, - // presumably-native runtime, not this environment. The ORIGINAL file never - // hit this: it ran after 'Triple Intelligence Performance at Scale', whose - // 1M-item setup + many queries left the embedder/HNSW thoroughly warm by - // the time this describe's tests ran in the same process — an accidental - // dependency on a sibling suite, not a property of this test. Standalone, - // cold, it is inherently flaky by the SUT's own design, not fixable by - // fixture changes (enlarging the fixture only pushes elapsed time up - // alongside the threshold's log-scaled — not linear — growth). - it.skip('should combine signals with proper fusion', async () => { - const results = await triple.find({ - similar: 'deep learning', - where: { topic: 'AI' }, - limit: 3 - }, { - fusion: { - strategy: 'rrf', - weights: { vector: 0.7, field: 0.3 } - } - }) - - // doc2 should rank highest (matches both signals) - expect(results[0].id).toBe('doc2') - expect(results[0].fusionScore).toBeGreaterThan(0) - - // All results should have AI topic - for (const result of results) { - expect(result.metadata.topic).toBe('AI') - } - }) -}) diff --git a/tests/integration/vfs-containment-batched.test.ts b/tests/integration/vfs-containment-batched.test.ts index 7bbad478..0a7919bf 100644 --- a/tests/integration/vfs-containment-batched.test.ts +++ b/tests/integration/vfs-containment-batched.test.ts @@ -81,7 +81,6 @@ describe('repairContainment: batched pass 2', () => { }) afterAll(async () => { - await brain.close() brain = null as any }) diff --git a/tests/integration/vfs-debug.test.ts b/tests/integration/vfs-debug.test.ts index 5eeb0ef5..7e781139 100644 --- a/tests/integration/vfs-debug.test.ts +++ b/tests/integration/vfs-debug.test.ts @@ -9,10 +9,9 @@ import * as XLSX from 'xlsx' describe('VFS Debug', () => { it('minimal VFS writeFile test', async () => { const brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' } }) - try { - await brain.init() + await brain.init() - console.log('✅ Brain initialized') + console.log('✅ Brain initialized') // Get VFS and initialize const vfs = brain.vfs @@ -78,8 +77,5 @@ describe('VFS Debug', () => { // THE REAL TEST: Can we query VFS? expect(children.length).toBeGreaterThan(0) expect(rootContents.length).toBeGreaterThan(0) - } finally { - await brain.close() - } }) }) diff --git a/tests/integration/writer-lock-fencing.test.ts b/tests/integration/writer-lock-fencing.test.ts index d5b82c30..e9f98dac 100644 --- a/tests/integration/writer-lock-fencing.test.ts +++ b/tests/integration/writer-lock-fencing.test.ts @@ -61,7 +61,6 @@ describe('writer-lock fencing', () => { // Old rule: heartbeat-age eviction → silent takeover → split brain. // New rule: live PID = live writer; the second opener throws typed. const second = new Brainy({ storage: { type: 'filesystem', path: dir }, requireSubtype: false }) - brains.push(second) await expect(second.init()).rejects.toMatchObject({ code: 'BRAINY_WRITER_LOCKED' }) }, 120000) diff --git a/tests/performance/triple-intelligence-scale.test.ts b/tests/performance/triple-intelligence-scale.test.ts index 1db7fc80..6687decd 100644 --- a/tests/performance/triple-intelligence-scale.test.ts +++ b/tests/performance/triple-intelligence-scale.test.ts @@ -352,8 +352,106 @@ describe('Triple Intelligence Performance at Scale', () => { }) }) -// The former 'Triple Intelligence Correctness' describe (4 tests, no timing -// assertions) moved to tests/integration/triple-intelligence-correctness.test.ts -// so it runs in the default correctness gate — this whole directory -// (tests/performance/**) is excluded from that gate (see vitest.config.ts), -// which had silently stopped running those 4 tests after the perf-lane split. \ No newline at end of file +describe('Triple Intelligence Correctness', () => { + let brain: Brainy + let triple: TripleIntelligenceSystem + + beforeAll(async () => { + brain = new Brainy({ requireSubtype: false }) + await brain.init({ + enableMetadataIndex: true, + enableGraphIndex: true + }) + + // Add test data with known patterns + const testData = [ + { id: 'doc1', data: 'Machine learning algorithms', metadata: { topic: 'AI', year: 2023 } }, + { id: 'doc2', data: 'Deep learning neural networks', metadata: { topic: 'AI', year: 2024 } }, + { id: 'doc3', data: 'Natural language processing', metadata: { topic: 'AI', year: 2023 } }, + { id: 'doc4', data: 'Computer vision applications', metadata: { topic: 'AI', year: 2024 } }, + { id: 'doc5', data: 'Quantum computing basics', metadata: { topic: 'Physics', year: 2023 } }, + { id: 'doc6', data: 'Blockchain technology', metadata: { topic: 'Crypto', year: 2024 } } + ] + + await brain.addMany(testData) + + // Add relationships + await brain.relate({ from: 'doc1', to: 'doc2', type: 'related' }) + await brain.relate({ from: 'doc2', to: 'doc3', type: 'related' }) + await brain.relate({ from: 'doc3', to: 'doc4', type: 'related' }) + + triple = brain.getTripleIntelligence() + }) + + afterAll(async () => { + await brain?.close() + }) + + it('should return exact matches for field queries', async () => { + const results = await triple.find({ + where: { topic: 'AI' }, + limit: 10 + }) + + expect(results).toHaveLength(4) + for (const result of results) { + expect(result.metadata.topic).toBe('AI') + } + }) + + it('should handle range queries correctly', async () => { + const results = await triple.find({ + where: { year: { $gte: 2024 } }, + limit: 10 + }) + + expect(results).toHaveLength(3) + for (const result of results) { + expect(result.metadata.year).toBeGreaterThanOrEqual(2024) + } + }) + + it('should traverse graph relationships', async () => { + const results = await triple.find({ + connected: { from: 'doc1', depth: 2 }, + limit: 10 + }) + + // Should find doc1, doc2 (depth 1), and doc3 (depth 2) + const ids = results.map(r => r.id) + expect(ids).toContain('doc1') + expect(ids).toContain('doc2') + expect(ids).toContain('doc3') + + // Check depth values + const doc1Result = results.find(r => r.id === 'doc1') + const doc2Result = results.find(r => r.id === 'doc2') + const doc3Result = results.find(r => r.id === 'doc3') + + expect(doc1Result?.depth).toBe(0) + expect(doc2Result?.depth).toBe(1) + expect(doc3Result?.depth).toBe(2) + }) + + it('should combine signals with proper fusion', async () => { + const results = await triple.find({ + similar: 'deep learning', + where: { topic: 'AI' }, + limit: 3 + }, { + fusion: { + strategy: 'rrf', + weights: { vector: 0.7, field: 0.3 } + } + }) + + // doc2 should rank highest (matches both signals) + expect(results[0].id).toBe('doc2') + expect(results[0].fusionScore).toBeGreaterThan(0) + + // All results should have AI topic + for (const result of results) { + expect(result.metadata.topic).toBe('AI') + } + }) +}) \ No newline at end of file diff --git a/tests/performance/typeAware.bench.test.ts b/tests/performance/typeAware.bench.test.ts index b1153662..72d96fe5 100644 --- a/tests/performance/typeAware.bench.test.ts +++ b/tests/performance/typeAware.bench.test.ts @@ -17,7 +17,7 @@ * - Note limitations and edge cases */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy } from '../../src/brainy.js' import { TypeAwareStorageAdapter } from '../../src/storage/adapters/typeAwareStorageAdapter.js' import { FileSystemStorage } from '../../src/storage/adapters/fileSystemStorage.js' @@ -67,10 +67,6 @@ describe('TypeAware Performance Benchmarks', () => { } }) - afterEach(async () => { - await brainMemory.close() - }) - it('should measure type-based query performance', async () => { // MEASURED: Query for one type (200 entities) const start = performance.now() diff --git a/tests/regression/metadata-field-typing.unit.test.ts b/tests/regression/metadata-field-typing.unit.test.ts deleted file mode 100644 index 910d4f2a..00000000 --- a/tests/regression/metadata-field-typing.unit.test.ts +++ /dev/null @@ -1,122 +0,0 @@ -/** - * @module metadata-field-typing.unit.test - * @description Regression: a metadata field that holds more than one value - * KIND stays fully filterable on every kind it holds. - * - * The defect this pins, reproduced on the released engine: the metadata index - * fixed a field's value type from the FIRST value it saw, and every later value - * of a different type was coerced to that type or, when coercion failed, - * dropped from the index in silence. Writing `category: 'electronics'` rows and - * then `category: 5` rows left `find({ where: { category: 5 } })` returning - * nothing — while the same rows in a numbers-only corpus answered correctly. - * The rows themselves were never lost: they stayed readable by id and by vector - * search, and only ever went missing from equality filters on that one field, - * which is what made it so quiet. - * - * Order is the whole point of these cases. Neither writer owns the field, so - * strings-then-numbers and numbers-then-strings must give the same answers. - */ - -import { describe, it, expect } from 'vitest' -import { Brainy } from '../../src/brainy.js' -import { NounType } from '../../src/types/graphTypes.js' - -/** A brain over memory storage, with a corpus written in the given order. */ -async function brainWith( - rows: Array<{ label: string; category: unknown }> -): Promise { - const brainy = new Brainy({ requireSubtype: false, storage: { type: 'memory' } }) - await brainy.init() - for (const row of rows) { - await brainy.add({ - data: `item ${row.label}`, - type: NounType.Thing, - metadata: { label: row.label, category: row.category } - }) - } - return brainy -} - -const labelsOf = (results: Array<{ metadata?: Record }>): string[] => - results.map((r) => String(r.metadata?.label)).sort() - -describe('regression: a mixed-kind metadata field filters on every kind', { timeout: 180_000 }, () => { - it('finds number rows written after string rows', async () => { - const brainy = await brainWith([ - { label: 'e1', category: 'electronics' }, - { label: 'f1', category: 'furniture' }, - { label: 'n1', category: 5 }, - { label: 'n2', category: 5 }, - { label: 'n3', category: 7 } - ]) - try { - expect(labelsOf(await brainy.find({ where: { category: 5 }, limit: 100 }))).toEqual(['n1', 'n2']) - expect(labelsOf(await brainy.find({ where: { category: 7 }, limit: 100 }))).toEqual(['n3']) - expect(labelsOf(await brainy.find({ where: { category: 'electronics' }, limit: 100 }))).toEqual(['e1']) - expect(labelsOf(await brainy.find({ where: { category: 'furniture' }, limit: 100 }))).toEqual(['f1']) - } finally { - await brainy.close() - } - }) - - it('finds string rows written after number rows', async () => { - const brainy = await brainWith([ - { label: 'n1', category: 5 }, - { label: 'n2', category: 5 }, - { label: 'e1', category: 'electronics' }, - { label: 'e2', category: 'electronics' } - ]) - try { - expect(labelsOf(await brainy.find({ where: { category: 'electronics' }, limit: 100 }))).toEqual(['e1', 'e2']) - expect(labelsOf(await brainy.find({ where: { category: 5 }, limit: 100 }))).toEqual(['n1', 'n2']) - } finally { - await brainy.close() - } - }) - - it('keeps `5` and `\'5\'` apart — a kind is part of the value, not a formatting detail', async () => { - const brainy = await brainWith([ - { label: 'num', category: 5 }, - { label: 'str', category: '5' } - ]) - try { - expect(labelsOf(await brainy.find({ where: { category: 5 }, limit: 100 }))).toEqual(['num']) - expect(labelsOf(await brainy.find({ where: { category: '5' }, limit: 100 }))).toEqual(['str']) - } finally { - await brainy.close() - } - }) - - it('serves booleans mixed into a field that already holds strings', async () => { - const brainy = await brainWith([ - { label: 's1', category: 'yes' }, - { label: 'b1', category: true }, - { label: 'b2', category: false } - ]) - try { - expect(labelsOf(await brainy.find({ where: { category: true }, limit: 100 }))).toEqual(['b1']) - expect(labelsOf(await brainy.find({ where: { category: false }, limit: 100 }))).toEqual(['b2']) - expect(labelsOf(await brainy.find({ where: { category: 'yes' }, limit: 100 }))).toEqual(['s1']) - } finally { - await brainy.close() - } - }) - - it('ranges over the numeric part of a mixed field', async () => { - const brainy = await brainWith([ - { label: 'unpriced', category: 'on request' }, - { label: 'cheap', category: 100 }, - { label: 'mid', category: 500 }, - { label: 'dear', category: 900 } - ]) - try { - const found = await brainy.find({ - where: { category: { greaterThan: 200 } }, - limit: 100 - }) - expect(labelsOf(found)).toEqual(['dear', 'mid']) - } finally { - await brainy.close() - } - }) -}) diff --git a/tests/unit/brainy-core.unit.test.ts b/tests/unit/brainy-core.unit.test.ts index 0488057d..eb6614e4 100644 --- a/tests/unit/brainy-core.unit.test.ts +++ b/tests/unit/brainy-core.unit.test.ts @@ -5,7 +5,7 @@ * No mocks, no fakes, real implementation */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy } from '../../src/brainy.js' import { NounType } from '../../src/types/graphTypes.js' @@ -21,10 +21,6 @@ describe('Brainy 3.0 Core (Unit Tests)', () => { await brain.init() }) - afterEach(async () => { - await brain.close() - }) - describe('CRUD Operations', () => { it('should create items with add', async () => { const id = await brain.add({ diff --git a/tests/unit/brainy/degraded-reads-surfaced.test.ts b/tests/unit/brainy/degraded-reads-surfaced.test.ts index 004adeaa..29a8a77c 100644 --- a/tests/unit/brainy/degraded-reads-surfaced.test.ts +++ b/tests/unit/brainy/degraded-reads-surfaced.test.ts @@ -19,19 +19,13 @@ import { prodLog } from '../../../src/utils/logger.js' const UUID = (suffix: string): string => `00000000-0000-4000-8000-0000000000${suffix}` describe('Finding 10 — degraded derived-index state is surfaced on reads', () => { - const opened: Brainy[] = [] - beforeEach(() => { process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true' }) - afterEach(async () => { - vi.restoreAllMocks() - for (const b of opened.splice(0)) await b.close().catch(() => {}) - }) + afterEach(() => vi.restoreAllMocks()) it('checkHealth() reports adopt-forward degraded ids as unhealthy', async () => { const brain = new Brainy({ storage: { type: 'memory' }, dimensions: 384, requireSubtype: false }) - opened.push(brain) await brain.init() ;(brain as any)._indexDegradedIds.add(UUID('de')) @@ -43,7 +37,6 @@ describe('Finding 10 — degraded derived-index state is surfaced on reads', () it('find()/get() warn loudly while degraded, ONCE, then repairIndex() clears it', async () => { const warn = vi.spyOn(prodLog, 'warn').mockImplementation(() => {}) const brain = new Brainy({ storage: { type: 'memory' }, dimensions: 384, requireSubtype: false }) - opened.push(brain) await brain.init() await brain.add({ id: UUID('a1'), data: 'x', type: NounType.Document }) ;(brain as any)._indexRebuildFailed = new Error('rebuild boom') @@ -66,7 +59,6 @@ describe('Finding 10 — degraded derived-index state is surfaced on reads', () it('persistSingleOp records receipt.degraded (widened return type, not dropped)', async () => { const brain = new Brainy({ storage: { type: 'memory' }, dimensions: 384, requireSubtype: false }) - opened.push(brain) await brain.init() // Simulate a degraded receipt by wrapping the generation store's commitSingleOp. const gs: any = (brain as any).generationStore diff --git a/tests/unit/brainy/find-complement-operators.test.ts b/tests/unit/brainy/find-complement-operators.test.ts index 710fbbbf..76fbb017 100644 --- a/tests/unit/brainy/find-complement-operators.test.ts +++ b/tests/unit/brainy/find-complement-operators.test.ts @@ -7,7 +7,7 @@ * soft-delete semantic: `field !== value` MUST include entities that have no * such field at all. */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy } from '../../../src/brainy' import { NounType } from '../../../src/types/graphTypes' @@ -26,10 +26,6 @@ describe('find() complement operators (ne / exists:false / missing:true)', () => ids.noField2 = await brain.add({ data: 'n2', type: NounType.Thing, metadata: { other: 2 } }) }) - afterEach(async () => { - await brain.close() - }) - it('ne returns everything except the matching value — INCLUDING entities without the field', async () => { const rows = await brain.find({ where: { status: { ne: 'active' } }, limit: 100 }) const got = new Set(rows.map((r) => r.id)) diff --git a/tests/unit/brainy/find-index-integrity-guard.test.ts b/tests/unit/brainy/find-index-integrity-guard.test.ts index 3e63d790..30cfdf1b 100644 --- a/tests/unit/brainy/find-index-integrity-guard.test.ts +++ b/tests/unit/brainy/find-index-integrity-guard.test.ts @@ -12,7 +12,7 @@ * returns an id whose record matches NEITHER the type nor the where filter) and * assert the phantom is dropped while the genuine matches survive. */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy } from '../../../src/brainy' import { NounType } from '../../../src/types/graphTypes' @@ -48,10 +48,6 @@ describe('find() index-integrity guard (phantom row class)', () => { }) }) - afterEach(async () => { - await brain.close() - }) - it('healthy index: the discriminant query returns only the staff Person', async () => { const rows = await brain.find({ type: NounType.Person, where: { entityType: 'staff' }, limit: 100 }) expect(rows.map((r) => r.id)).toEqual([staffId]) diff --git a/tests/unit/brainy/find.test.ts b/tests/unit/brainy/find.test.ts index 59601456..5bead272 100644 --- a/tests/unit/brainy/find.test.ts +++ b/tests/unit/brainy/find.test.ts @@ -1,4 +1,4 @@ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy } from '../../../src/brainy' import { createAddParams } from '../../helpers/test-factory' import { NounType } from '../../../src/types/graphTypes' @@ -12,11 +12,7 @@ describe('Brainy.find()', () => { }) await brain.init() }) - - afterEach(async () => { - await brain.close() - }) - + describe('success paths', () => { it('should find entities by text query', async () => { // Arrange diff --git a/tests/unit/brainy/get.test.ts b/tests/unit/brainy/get.test.ts index 97a19125..b39bf2e1 100644 --- a/tests/unit/brainy/get.test.ts +++ b/tests/unit/brainy/get.test.ts @@ -5,8 +5,7 @@ import { describe, it, expect, beforeEach, afterEach } from 'vitest' import { Brainy } from '../../../src/brainy' -import { MetadataArrayTooLargeError, MAX_INDEXED_ARRAY_LENGTH } from '../../../src/errors/brainyError' -import { +import { createAddParams, generateTestVector, createTestConfig, @@ -269,75 +268,32 @@ describe('Brainy.get()', () => { expect(entity!.id).toBe(id) }) - // THE INDEXABLE-ARRAY BOUND, from get()'s side. This case used to park a - // 1000-element array in the metadata bag and assert it came back. That - // shape is refused at the write door now — an array field mints one - // posting per element, so an unbounded array is an unbounded write — so - // the case pins BOTH halves of the law that replaced it: a large SCALAR - // payload still round-trips whole, and an array over the bound refuses by - // name. Every length derives from MAX_INDEXED_ARRAY_LENGTH so the pin - // follows the constant wherever it moves. - it('should get an entity with a large scalar metadata payload', async () => { - // Arrange — large in every dimension EXCEPT array length: a long string, - // many fields, deep nesting, and an array sitting exactly ON the bound. + it('should get entity with very large metadata', async () => { + // Arrange const largeMetadata = { - atTheBound: Array.from({ length: MAX_INDEXED_ARRAY_LENGTH }, (_, i) => `item${i}`), + bigArray: new Array(1000).fill('item'), bigObject: Object.fromEntries( Array.from({ length: 100 }, (_, i) => [`key${i}`, `value${i}`]) ), - longString: 'x'.repeat(10_000), deepNesting: Array(10).fill(null).reduce( (acc) => ({ nested: acc }), { value: 'deep' } ) } - + const id = await brain.add(createAddParams({ data: 'Large metadata', type: 'thing', metadata: largeMetadata })) - + // Act const entity = await brain.get(id) - - // Assert — the payload comes back whole, first element to last + + // Assert expect(entity).not.toBeNull() - expect(entity!.metadata.atTheBound).toHaveLength(MAX_INDEXED_ARRAY_LENGTH) - expect(entity!.metadata.atTheBound[0]).toBe('item0') - expect(entity!.metadata.atTheBound[MAX_INDEXED_ARRAY_LENGTH - 1]) - .toBe(`item${MAX_INDEXED_ARRAY_LENGTH - 1}`) + expect(entity!.metadata.bigArray).toHaveLength(1000) expect(Object.keys(entity!.metadata.bigObject)).toHaveLength(100) - expect(entity!.metadata.longString).toHaveLength(10_000) - - // ...including the deep nest, walked to the bottom. - let cursor: any = entity!.metadata.deepNesting - for (let depth = 0; depth < 10; depth++) cursor = cursor.nested - expect(cursor.value).toBe('deep') - }) - - it('should refuse a metadata array over the indexing bound, by name', async () => { - // Arrange - const overTheBound = MAX_INDEXED_ARRAY_LENGTH + 1 - - // Act - const err = await brain - .add(createAddParams({ - data: 'Large metadata', - type: 'thing', - metadata: { bigArray: new Array(overTheBound).fill('item') } - })) - .catch((e: any) => e) - - // Assert — the field, the length and the bound, on the error and in the - // message, so a handler can report or repair without parsing prose. - expect(err).toBeInstanceOf(MetadataArrayTooLargeError) - expect(err.field).toBe('bigArray') - expect(err.length).toBe(overTheBound) - expect(err.limit).toBe(MAX_INDEXED_ARRAY_LENGTH) - expect(err.message).toContain('bigArray') - expect(err.message).toContain(String(overTheBound)) - expect(err.message).toContain(String(MAX_INDEXED_ARRAY_LENGTH)) }) }) diff --git a/tests/unit/brainy/metadata-provider-contract.test.ts b/tests/unit/brainy/metadata-provider-contract.test.ts index 466fc654..945c0670 100644 --- a/tests/unit/brainy/metadata-provider-contract.test.ts +++ b/tests/unit/brainy/metadata-provider-contract.test.ts @@ -18,7 +18,7 @@ * exercised by cor's combined matrix); they inject probe/spy hooks onto the live JS * metadata index, which has neither method by default. */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy } from '../../../src/brainy' import { NounType } from '../../../src/types/graphTypes' @@ -34,10 +34,6 @@ describe('metadata-provider contract wiring (getIdsForFilter opts)', () => { mi = (brain as any).metadataIndex }) - afterEach(async () => { - await brain.close() - }) - it('RETIRED: a read never calls probeConsistency() / self-heals via detectAndRepairCorruption — that is the read-triggered dark rebuild the health-gate law forbids', async () => { let probes = 0 let repairs = 0 diff --git a/tests/unit/brainy/migration-gate-family-scoped.test.ts b/tests/unit/brainy/migration-gate-family-scoped.test.ts index ce510a4e..b71c3899 100644 --- a/tests/unit/brainy/migration-gate-family-scoped.test.ts +++ b/tests/unit/brainy/migration-gate-family-scoped.test.ts @@ -8,7 +8,7 @@ * gate that hung getStats / readdir / readFile behind an unrelated family's * migration until the wait timed out. */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy } from '../../../src/brainy.js' import { MigrationInProgressError } from '../../../src/errors/brainyError.js' @@ -38,19 +38,12 @@ const jam = (provider: unknown) => { } describe('migration LOCK is family-scoped', () => { - const opened: Brainy[] = [] - beforeEach(() => { process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true' }) - afterEach(async () => { - for (const b of opened.splice(0)) await b.close().catch(() => {}) - }) - it('a stuck VECTOR migration does not block canonical or graph/metadata reads', async () => { const brain = await seed() - opened.push(brain) const childId = ( (await brain.vfs.readdir('/notes', { withFileTypes: true })) as Array<{ entityId: string }> )[0].entityId @@ -67,7 +60,6 @@ describe('migration LOCK is family-scoped', () => { it('a stuck VECTOR migration STILL blocks a read that needs the vector family', async () => { const brain = await seed() - opened.push(brain) jam((brain as any).index) // A semantic query consults the vector index — it must wait, and (bounded by @@ -78,7 +70,6 @@ describe('migration LOCK is family-scoped', () => { it('a stuck GRAPH migration blocks traversal but not vector/canonical reads', async () => { const brain = await seed() - opened.push(brain) const childId = ( (await brain.vfs.readdir('/notes', { withFileTypes: true })) as Array<{ entityId: string }> )[0].entityId @@ -96,7 +87,6 @@ describe('migration LOCK is family-scoped', () => { it('with no migration in flight, every read serves (the fast path is a no-op)', async () => { const brain = await seed() - opened.push(brain) await expect(brain.getStats()).resolves.toBeDefined() await expect(brain.find({ query: 'doc' })).resolves.toBeDefined() await expect(brain.vfs.readdir('/notes')).resolves.toHaveLength(1) diff --git a/tests/unit/brainy/relate-duplicate-optimization.test.ts b/tests/unit/brainy/relate-duplicate-optimization.test.ts index 910d057d..8bcb7c7a 100644 --- a/tests/unit/brainy/relate-duplicate-optimization.test.ts +++ b/tests/unit/brainy/relate-duplicate-optimization.test.ts @@ -18,7 +18,7 @@ describe('Duplicate Check Optimization', () => { }) afterEach(async () => { - await brain.close() + // Cleanup is automatic with memory storage }) it('should detect duplicate relationships using GraphAdjacencyIndex', async () => { diff --git a/tests/unit/brainy/relate.test.ts b/tests/unit/brainy/relate.test.ts index bea35ba3..eb1a036e 100644 --- a/tests/unit/brainy/relate.test.ts +++ b/tests/unit/brainy/relate.test.ts @@ -5,8 +5,7 @@ import { describe, it, expect, beforeEach, afterEach } from 'vitest' import { Brainy } from '../../../src/brainy' -import { MetadataArrayTooLargeError, MAX_INDEXED_ARRAY_LENGTH } from '../../../src/errors/brainyError' -import { +import { createAddParams, createTestConfig, } from '../../helpers/test-factory' @@ -249,23 +248,16 @@ describe('Brainy.relate()', () => { expect(matches.length).toBe(1) // Only one relationship should exist }) - // THE INDEXABLE-ARRAY BOUND, from relate()'s side. This case used to pass a - // 100-element array through relate() and assert it came back — a length - // hardcoded either side of a bound it never named, so it read green or red - // purely by where the constant happened to sit. Both halves of the law are - // pinned here instead, and every length derives from - // MAX_INDEXED_ARRAY_LENGTH so the pin follows the constant. - it('should handle a large scalar metadata payload on a relation', async () => { - // Arrange — large in every dimension EXCEPT array length: a long string, - // many fields, and an array sitting exactly ON the bound. + it('should handle very long metadata', async () => { + // Arrange const largeMetadata = { - atTheBound: Array.from({ length: MAX_INDEXED_ARRAY_LENGTH }, (_, i) => `item${i}`), + bigArray: new Array(100).fill('item'), bigObject: Object.fromEntries( Array.from({ length: 50 }, (_, i) => [`key${i}`, `value${i}`]) ), - longString: 'x'.repeat(10_000) + longString: 'x'.repeat(1000) } - + // Act await brain.relate({ from: entity1Id, @@ -273,46 +265,12 @@ describe('Brainy.relate()', () => { type: 'relatedTo', metadata: largeMetadata }) - - // Assert — the payload comes back whole, first element to last + + // Assert const relations = await brain.related({ from: entity1Id }) const relation = relations.find(r => r.to === entity2Id) expect(relation).toBeDefined() - expect(relation!.metadata?.atTheBound).toHaveLength(MAX_INDEXED_ARRAY_LENGTH) - expect(relation!.metadata?.atTheBound[0]).toBe('item0') - expect(relation!.metadata?.atTheBound[MAX_INDEXED_ARRAY_LENGTH - 1]) - .toBe(`item${MAX_INDEXED_ARRAY_LENGTH - 1}`) - expect(Object.keys(relation!.metadata?.bigObject)).toHaveLength(50) - expect(relation!.metadata?.longString).toHaveLength(10_000) - }) - - it('should refuse a relation metadata array over the indexing bound, by name', async () => { - // Arrange - const overTheBound = MAX_INDEXED_ARRAY_LENGTH + 1 - - // Act - const err = await brain - .relate({ - from: entity1Id, - to: entity3Id, - type: 'relatedTo', - metadata: { bigArray: new Array(overTheBound).fill('item') } - }) - .catch((e: any) => e) - - // Assert — the field, the length and the bound, on the error and in the - // message, so a handler can report or repair without parsing prose. - expect(err).toBeInstanceOf(MetadataArrayTooLargeError) - expect(err.field).toBe('bigArray') - expect(err.length).toBe(overTheBound) - expect(err.limit).toBe(MAX_INDEXED_ARRAY_LENGTH) - expect(err.message).toContain('bigArray') - expect(err.message).toContain(String(overTheBound)) - expect(err.message).toContain(String(MAX_INDEXED_ARRAY_LENGTH)) - - // Refused means not written: no relation of this shape exists. - const relations = await brain.related({ from: entity1Id }) - expect(relations.some(r => r.to === entity3Id && r.metadata?.bigArray)).toBe(false) + expect(relation!.metadata?.bigArray).toHaveLength(100) }) it('should handle special characters in metadata', async () => { diff --git a/tests/unit/brainy/update.test.ts b/tests/unit/brainy/update.test.ts index ec5f3fff..19fdad19 100644 --- a/tests/unit/brainy/update.test.ts +++ b/tests/unit/brainy/update.test.ts @@ -5,8 +5,7 @@ import { describe, it, expect, beforeEach, afterEach } from 'vitest' import { Brainy } from '../../../src/brainy' -import { MetadataArrayTooLargeError, MAX_INDEXED_ARRAY_LENGTH } from '../../../src/errors/brainyError' -import { +import { createAddParams, createTestConfig, } from '../../helpers/test-factory' @@ -356,88 +355,36 @@ describe('Brainy.update()', () => { expect(final!.metadata.counter).toBeLessThanOrEqual(10) }) - // THE INDEXABLE-ARRAY BOUND, from update()'s side. This case used to write - // a 1000-element array through update() and assert it came back. That - // shape is refused at the write door now — an array field mints one - // posting per element, so an unbounded array is an unbounded write — so - // the case pins BOTH halves of the law that replaced it. Every length - // derives from MAX_INDEXED_ARRAY_LENGTH so the pin follows the constant. - it('should handle a large scalar metadata update', async () => { + it('should handle very large metadata updates', async () => { // Arrange const id = await brain.add(createAddParams({ data: 'Large metadata test', type: 'thing' })) - - // Large in every dimension EXCEPT array length: a long string, many - // fields, deep nesting, and an array sitting exactly ON the bound. + const largeMetadata = { - atTheBound: Array.from({ length: MAX_INDEXED_ARRAY_LENGTH }, (_, i) => `item${i}`), + bigArray: new Array(1000).fill('item'), bigObject: Object.fromEntries( Array.from({ length: 100 }, (_, i) => [`key${i}`, `value${i}`]) ), - longString: 'x'.repeat(10_000), deepNesting: Array(10).fill(null).reduce( (acc) => ({ nested: acc }), { value: 'deep' } ) } - + // Act await brain.update({ id, metadata: largeMetadata, merge: false }) - - // Assert — the payload comes back whole, first element to last + + // Assert const updated = await brain.get(id) expect(updated).not.toBeNull() - expect(updated!.metadata.atTheBound).toHaveLength(MAX_INDEXED_ARRAY_LENGTH) - expect(updated!.metadata.atTheBound[0]).toBe('item0') - expect(updated!.metadata.atTheBound[MAX_INDEXED_ARRAY_LENGTH - 1]) - .toBe(`item${MAX_INDEXED_ARRAY_LENGTH - 1}`) + expect(updated!.metadata.bigArray).toHaveLength(1000) expect(Object.keys(updated!.metadata.bigObject)).toHaveLength(100) - expect(updated!.metadata.longString).toHaveLength(10_000) - - // ...including the deep nest, walked to the bottom. - let cursor: any = updated!.metadata.deepNesting - for (let depth = 0; depth < 10; depth++) cursor = cursor.nested - expect(cursor.value).toBe('deep') - }) - - it('should refuse an update whose metadata array is over the indexing bound, by name', async () => { - // Arrange - const id = await brain.add(createAddParams({ - data: 'Large metadata test', - type: 'thing', - metadata: { keep: 'me' } - })) - const overTheBound = MAX_INDEXED_ARRAY_LENGTH + 1 - - // Act - const err = await brain - .update({ - id, - metadata: { bigArray: new Array(overTheBound).fill('item') }, - merge: false - }) - .catch((e: any) => e) - - // Assert — the field, the length and the bound, on the error and in the - // message, so a handler can report or repair without parsing prose. - expect(err).toBeInstanceOf(MetadataArrayTooLargeError) - expect(err.field).toBe('bigArray') - expect(err.length).toBe(overTheBound) - expect(err.limit).toBe(MAX_INDEXED_ARRAY_LENGTH) - expect(err.message).toContain('bigArray') - expect(err.message).toContain(String(overTheBound)) - expect(err.message).toContain(String(MAX_INDEXED_ARRAY_LENGTH)) - - // Refused means unchanged: the row still carries what it had before. - const unchanged = await brain.get(id) - expect(unchanged!.metadata.keep).toBe('me') - expect(unchanged!.metadata.bigArray).toBeUndefined() }) it('should preserve entity ID during update', async () => { diff --git a/tests/unit/get-index-status-readiness.test.ts b/tests/unit/get-index-status-readiness.test.ts index 5e283bc8..7f82ec5d 100644 --- a/tests/unit/get-index-status-readiness.test.ts +++ b/tests/unit/get-index-status-readiness.test.ts @@ -7,7 +7,7 @@ * _indexRebuildFailed / _indexDegradedIds degraded states (mirroring * validateIndexConsistency / checkHealth). */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy, NounType } from '../../src/index.js' describe('getIndexStatus honest readiness (Finding 9)', () => { @@ -20,10 +20,6 @@ describe('getIndexStatus honest readiness (Finding 9)', () => { await brain.flush() }) - afterEach(async () => { - await brain.close() - }) - it('a not-ready provider makes populated honest (false) and exposes ready:false', async () => { brain.index.isReady = () => false // count present, serving structure NOT loaded const status = await brain.getIndexStatus() diff --git a/tests/unit/graph/graph-fastpath-honest-readiness.test.ts b/tests/unit/graph/graph-fastpath-honest-readiness.test.ts index 95a6c0c4..46d318b4 100644 --- a/tests/unit/graph/graph-fastpath-honest-readiness.test.ts +++ b/tests/unit/graph/graph-fastpath-honest-readiness.test.ts @@ -8,7 +8,7 @@ * scan; and a one-shot probe self-heals a no-isReady provider whose adjacency * did not cold-load. */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy, NounType, VerbType } from '../../../src/index.js' describe('graph fast-path honest readiness (Finding 2)', () => { @@ -33,10 +33,6 @@ describe('graph fast-path honest readiness (Finding 2)', () => { await storage.getVerbsBySource(a) }) - afterEach(async () => { - await brain.close() - }) - it('not-ready provider → shard scan returns the REAL edges, not a silent []', async () => { const gi = storage.graphIndex // Simulate a cold native provider: count/manifest loaded (isInitialized) but diff --git a/tests/unit/indexes/columnStore/column-store-mixed-kind.test.ts b/tests/unit/indexes/columnStore/column-store-mixed-kind.test.ts deleted file mode 100644 index 1ce21d1f..00000000 --- a/tests/unit/indexes/columnStore/column-store-mixed-kind.test.ts +++ /dev/null @@ -1,241 +0,0 @@ -/** - * @module column-store-mixed-kind.test - * @description Typed posting lists: one field, several value KINDS, each - * answerable on its own. - * - * The behaviour these pin replaced a first-writer type freeze. The first value - * a field ever saw fixed that field's type; every later value of another kind - * was coerced to it, and when coercion failed — `Number('electronics')` — the - * value was dropped from the index with no error at all. The row stayed - * readable by id and by vector and vanished from every equality filter on the - * field. These tests therefore care about ORDER: strings-then-numbers and - * numbers-then-strings have to behave identically, because neither writer owns - * the field. - * - * Kinds never coerce into one another at query time either. `5` and `'5'` are - * different values and match different rows. - */ - -import { describe, it, expect, beforeEach, afterEach } from 'vitest' -import { ColumnStore } from '../../../../src/indexes/columnStore/ColumnStore.js' -import { MemoryStorage } from '../../../../src/storage/adapters/memoryStorage.js' -import { EntityIdMapper } from '../../../../src/utils/entityIdMapper.js' - -describe('ColumnStore — typed posting lists per (field, kind)', () => { - let storage: MemoryStorage - let idMapper: EntityIdMapper - let store: ColumnStore - - beforeEach(async () => { - storage = new MemoryStorage() - await storage.init() - idMapper = new EntityIdMapper({ storage, storageKey: 'test:idMapper' }) - await idMapper.init() - - store = new ColumnStore({ flushThreshold: 10 }) - await store.init(storage, idMapper) - }) - - afterEach(async () => { - await store.close() - }) - - /** Resolve a filter to the sorted UUIDs it matched. */ - const uuidsOf = async (field: string, value: unknown): Promise => { - const bitmap = await store.filter(field, value) - return Array.from(bitmap) - .map((id) => idMapper.getUuid(Number(id))) - .filter((u): u is string => u !== undefined) - .sort() - } - - describe('equality answers on the query value’s own kind', () => { - it('serves numbers written AFTER strings on the same field', async () => { - store.addEntity(BigInt(idMapper.getOrAssign('s1')), { category: 'electronics' }) - store.addEntity(BigInt(idMapper.getOrAssign('s2')), { category: 'furniture' }) - store.addEntity(BigInt(idMapper.getOrAssign('n1')), { category: 5 }) - store.addEntity(BigInt(idMapper.getOrAssign('n2')), { category: 5 }) - store.addEntity(BigInt(idMapper.getOrAssign('n3')), { category: 7 }) - - // The numbers are in the index, though a string got there first. - expect(await uuidsOf('category', 5)).toEqual(['n1', 'n2']) - expect(await uuidsOf('category', 7)).toEqual(['n3']) - // And the strings did not move. - expect(await uuidsOf('category', 'electronics')).toEqual(['s1']) - expect(await uuidsOf('category', 'furniture')).toEqual(['s2']) - }) - - it('serves strings written AFTER numbers on the same field', async () => { - store.addEntity(BigInt(idMapper.getOrAssign('n1')), { category: 5 }) - store.addEntity(BigInt(idMapper.getOrAssign('n2')), { category: 5 }) - store.addEntity(BigInt(idMapper.getOrAssign('s1')), { category: 'electronics' }) - store.addEntity(BigInt(idMapper.getOrAssign('s2')), { category: 'electronics' }) - - // 'electronics' would have become NaN and been dropped under the freeze. - expect(await uuidsOf('category', 'electronics')).toEqual(['s1', 's2']) - expect(await uuidsOf('category', 5)).toEqual(['n1', 'n2']) - }) - - it('does not coerce a number query into the string postings, or back', async () => { - store.addEntity(BigInt(idMapper.getOrAssign('num')), { code: 5 }) - store.addEntity(BigInt(idMapper.getOrAssign('str')), { code: '5' }) - - expect(await uuidsOf('code', 5)).toEqual(['num']) - expect(await uuidsOf('code', '5')).toEqual(['str']) - }) - - it('serves booleans mixed into a field that already holds strings and numbers', async () => { - store.addEntity(BigInt(idMapper.getOrAssign('s1')), { flag: 'yes' }) - store.addEntity(BigInt(idMapper.getOrAssign('n1')), { flag: 1 }) - store.addEntity(BigInt(idMapper.getOrAssign('b1')), { flag: true }) - store.addEntity(BigInt(idMapper.getOrAssign('b2')), { flag: false }) - - expect(await uuidsOf('flag', true)).toEqual(['b1']) - expect(await uuidsOf('flag', false)).toEqual(['b2']) - // `true` stores as 1 internally; that is an encoding, not a value. - expect(await uuidsOf('flag', 1)).toEqual(['n1']) - expect(await uuidsOf('flag', 'yes')).toEqual(['s1']) - }) - - it('answers nothing — not something coerced — for a kind the field never held', async () => { - store.addEntity(BigInt(idMapper.getOrAssign('s1')), { category: 'electronics' }) - - expect(await uuidsOf('category', 5)).toEqual([]) - expect(await uuidsOf('category', true)).toEqual([]) - }) - - it('holds every kind across a flush, not just the one in the tail buffer', async () => { - store.addEntity(BigInt(idMapper.getOrAssign('s1')), { category: 'electronics' }) - store.addEntity(BigInt(idMapper.getOrAssign('n1')), { category: 5 }) - await store.flush() - store.addEntity(BigInt(idMapper.getOrAssign('s2')), { category: 'electronics' }) - store.addEntity(BigInt(idMapper.getOrAssign('n2')), { category: 5 }) - - expect(await uuidsOf('category', 'electronics')).toEqual(['s1', 's2']) - expect(await uuidsOf('category', 5)).toEqual(['n1', 'n2']) - }) - }) - - describe('range filters read the numeric postings', () => { - it('ranges over the numeric subset of a mixed field, ignoring its strings', async () => { - store.addEntity(BigInt(idMapper.getOrAssign('cheap')), { price: 100 }) - store.addEntity(BigInt(idMapper.getOrAssign('mid')), { price: 500 }) - store.addEntity(BigInt(idMapper.getOrAssign('dear')), { price: 900 }) - store.addEntity(BigInt(idMapper.getOrAssign('unpriced')), { price: 'on request' }) - await store.flush() - - const inRange = await store.rangeQuery('price', 200, 1000) - const uuids = Array.from(inRange) - .map((id) => idMapper.getUuid(Number(id))) - .sort() - expect(uuids).toEqual(['dear', 'mid']) - }) - - it('an unbounded range still reports every kind — it is the “has a value” probe', async () => { - store.addEntity(BigInt(idMapper.getOrAssign('n1')), { mixed: 42 }) - store.addEntity(BigInt(idMapper.getOrAssign('s1')), { mixed: 'text' }) - store.addEntity(BigInt(idMapper.getOrAssign('b1')), { mixed: true }) - await store.flush() - - const anyValue = await store.rangeQuery('mixed') - const uuids = Array.from(anyValue) - .map((id) => idMapper.getUuid(Number(id))) - .sort() - expect(uuids).toEqual(['b1', 'n1', 's1']) - }) - }) - - describe('the index reports what a field actually holds', () => { - it('names every kind present, not the one that got there first', async () => { - store.addEntity(BigInt(idMapper.getOrAssign('s1')), { category: 'electronics' }) - expect(store.getFieldKinds('category')).toEqual(['string']) - - store.addEntity(BigInt(idMapper.getOrAssign('n1')), { category: 5 }) - store.addEntity(BigInt(idMapper.getOrAssign('b1')), { category: true }) - expect(store.getFieldKinds('category')).toEqual(['number', 'string', 'boolean']) - - // And the field is still ONE field by name. - expect(store.getIndexedFields()).toEqual(['category']) - expect(store.hasField('category')).toBe(true) - }) - - it('reports an unknown field as holding nothing', () => { - expect(store.getFieldKinds('never-written')).toEqual([]) - }) - }) - - describe('an integer column widens rather than rounding', () => { - it('keeps a non-integer written after integers as itself', async () => { - store.addEntity(BigInt(idMapper.getOrAssign('a')), { score: 4 }) - store.addEntity(BigInt(idMapper.getOrAssign('b')), { score: 4.5 }) - store.addEntity(BigInt(idMapper.getOrAssign('c')), { score: 5 }) - await store.flush() - - // 4.5 used to round to 5 and answer `score === 5` alongside c. - expect(await uuidsOf('score', 4.5)).toEqual(['b']) - expect(await uuidsOf('score', 5)).toEqual(['c']) - expect(await uuidsOf('score', 4)).toEqual(['a']) - }) - }) - - describe('close then reopen', () => { - it('keeps every typed posting, on the same storage', async () => { - store.addEntity(BigInt(idMapper.getOrAssign('s1')), { category: 'electronics' }) - store.addEntity(BigInt(idMapper.getOrAssign('n1')), { category: 5 }) - store.addEntity(BigInt(idMapper.getOrAssign('b1')), { category: true }) - store.addEntity(BigInt(idMapper.getOrAssign('f1')), { score: 1.5 }) - await store.flush() - await store.close() - - store = new ColumnStore({ flushThreshold: 10 }) - await store.init(storage, idMapper) - - expect(store.getFieldKinds('category')).toEqual(['number', 'string', 'boolean']) - expect(await uuidsOf('category', 'electronics')).toEqual(['s1']) - expect(await uuidsOf('category', 5)).toEqual(['n1']) - expect(await uuidsOf('category', true)).toEqual(['b1']) - expect(await uuidsOf('score', 1.5)).toEqual(['f1']) - }) - - it('accepts new values of every kind after the reopen', async () => { - store.addEntity(BigInt(idMapper.getOrAssign('s1')), { category: 'electronics' }) - store.addEntity(BigInt(idMapper.getOrAssign('n1')), { category: 5 }) - await store.flush() - await store.close() - - store = new ColumnStore({ flushThreshold: 10 }) - await store.init(storage, idMapper) - - store.addEntity(BigInt(idMapper.getOrAssign('s2')), { category: 'electronics' }) - store.addEntity(BigInt(idMapper.getOrAssign('n2')), { category: 5 }) - store.addEntity(BigInt(idMapper.getOrAssign('b1')), { category: false }) - await store.flush() - - expect(await uuidsOf('category', 'electronics')).toEqual(['s1', 's2']) - expect(await uuidsOf('category', 5)).toEqual(['n1', 'n2']) - expect(await uuidsOf('category', false)).toEqual(['b1']) - }) - - it('opens an index written by the pre-typed-postings shape and reads it unchanged', async () => { - // A single-kind field is byte-identical to what the old writer produced: - // one manifest at `_column_index//MANIFEST.json`, no kind - // subdirectory anywhere. That IS the old on-disk shape, so proving the - // new reader serves it proves an old index still opens. - store.addEntity(BigInt(idMapper.getOrAssign('a')), { status: 'active' }) - store.addEntity(BigInt(idMapper.getOrAssign('b')), { status: 'archived' }) - await store.flush() - - const keys = await (storage as unknown as { - listObjectsUnderPath: (prefix: string) => Promise - }).listObjectsUnderPath('_column_index/') - expect(keys.some((k) => k.includes('/k/'))).toBe(false) - - await store.close() - store = new ColumnStore({ flushThreshold: 10 }) - await store.init(storage, idMapper) - - expect(store.getFieldKinds('status')).toEqual(['string']) - expect(await uuidsOf('status', 'active')).toEqual(['a']) - }) - }) -}) diff --git a/tests/unit/metadata-cold-read-guard.test.ts b/tests/unit/metadata-cold-read-guard.test.ts index d079982e..b4f82f15 100644 --- a/tests/unit/metadata-cold-read-guard.test.ts +++ b/tests/unit/metadata-cold-read-guard.test.ts @@ -15,7 +15,7 @@ * The 8.0 JS index cold-loads correctly, so we simulate the cold native failure * mode by intercepting the provider's getIdsForFilter/rebuild. */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy, NounType, MetadataIndexNotReadyError } from '../../src/index.js' const V = () => Array.from({ length: 384 }, (_, i) => Math.sin(i * 0.1) + 0.001) @@ -31,10 +31,6 @@ describe('Metadata cold-read guard (#venue silent-[])', () => { await brain.flush() }) - afterEach(async () => { - await brain.close() - }) - it('warm brain: filtered find is correct and the guard does not rebuild', async () => { const mi = brain.metadataIndex let rebuilds = 0 diff --git a/tests/unit/migration-lock.test.ts b/tests/unit/migration-lock.test.ts index 63f6953e..f0fbbe4c 100644 --- a/tests/unit/migration-lock.test.ts +++ b/tests/unit/migration-lock.test.ts @@ -18,7 +18,7 @@ * the production feature-detection reads it. */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy, NounType, MigrationInProgressError } from '../../src/index.js' import { GraphAdjacencyIndex } from '../../src/graph/graphAdjacencyIndex.js' @@ -39,12 +39,6 @@ describe('Migration LOCK (#18) — coordinated 7.x→8.0 auto-upgrade', () => { await brain.init() }) - afterEach(async () => { - // The "close() is not gated" test already closes `brain` itself as its - // own assertion — closing an already-closed brain is a safe no-op here. - await brain.close().catch(() => {}) - }) - it('does not gate operations when no provider is migrating (fast path)', async () => { const id = await brain.add({ data: 'hello', type: NounType.Concept }) expect(id).toBeTruthy() @@ -136,9 +130,6 @@ describe('Migration LOCK (#18) — coordinated 7.x→8.0 auto-upgrade', () => { expect(e).toBeInstanceOf(MigrationInProgressError) expect(e.retryable).toBe(true) expect(typeof e.elapsedMs).toBe('number') - } finally { - // close() is proven not-gated by the test below — safe even mid-migration. - await shortBrain.close() } }) diff --git a/tests/unit/neural/signals/EmbeddingSignal.test.ts b/tests/unit/neural/signals/EmbeddingSignal.test.ts index ad08e045..54d34b64 100644 --- a/tests/unit/neural/signals/EmbeddingSignal.test.ts +++ b/tests/unit/neural/signals/EmbeddingSignal.test.ts @@ -13,11 +13,10 @@ describe('EmbeddingSignal', () => { signal = new EmbeddingSignal(brain) }) - afterEach(async () => { + afterEach(() => { signal.clearCache() signal.clearHistory() signal.resetStats() - await brain.close() }) describe('initialization', () => { diff --git a/tests/unit/plugin-autodetect.test.ts b/tests/unit/plugin-autodetect.test.ts index ee830c17..37c181ba 100644 --- a/tests/unit/plugin-autodetect.test.ts +++ b/tests/unit/plugin-autodetect.test.ts @@ -89,14 +89,12 @@ describe('Guarded plugin auto-detection (plugins: undefined)', () => { }) const brain: any = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, silent: true }) await expect(brain.init()).rejects.toThrow(/installed but failed to load/) - await brain.close().catch(() => {}) }) it('installed but not a valid plugin (missing activate) → init() throws', async () => { stubImport(async () => ({ default: { name: '@soulcraft/cor' } })) // no activate() const brain: any = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, silent: true }) await expect(brain.init()).rejects.toThrow(/not a valid Brainy plugin/) - await brain.close().catch(() => {}) }) it('installed but activation fails → init() throws (activateAll posture applies)', async () => { @@ -110,7 +108,6 @@ describe('Guarded plugin auto-detection (plugins: undefined)', () => { })) const brain: any = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, silent: true }) await expect(brain.init()).rejects.toThrow(/failed to activate/) - await brain.close().catch(() => {}) }) it('plugins: [] and plugins: false → no probe at all (explicit opt-out)', async () => { @@ -135,6 +132,5 @@ describe('Guarded plugin auto-detection (plugins: undefined)', () => { silent: true }) await expect(brain.init()).rejects.toThrow(/listed in config\.plugins but could not be loaded/) - await brain.close().catch(() => {}) }) }) diff --git a/tests/unit/plugin-version-coupling.test.ts b/tests/unit/plugin-version-coupling.test.ts index d4685ae2..ffcc2a88 100644 --- a/tests/unit/plugin-version-coupling.test.ts +++ b/tests/unit/plugin-version-coupling.test.ts @@ -143,6 +143,5 @@ describe('version coupling at init() — no silent fallback', () => { plugins: ['@soulcraft/this-package-does-not-exist-xyz'] }) await expect(brain.init()).rejects.toThrow(/could not be loaded|config\.plugins/) - await brain.close().catch(() => {}) }) }) diff --git a/tests/unit/plugin.test.ts b/tests/unit/plugin.test.ts index 82543120..f4064188 100644 --- a/tests/unit/plugin.test.ts +++ b/tests/unit/plugin.test.ts @@ -298,10 +298,9 @@ describe('Brainy plugin integration', () => { // must surface as a failed init(), NOT a silent degrade to the default // engine (the version-coupling guard; see plugin-version-coupling.test.ts). await expect(brain.init()).rejects.toThrow(/failed to activate|native module not found/) - await brain.close().catch(() => {}) }) - it('should use() return this for chaining', async () => { + it('should use() return this for chaining', () => { const plugin: BrainyPlugin = { name: 'chain-test', activate: async () => true @@ -310,8 +309,5 @@ describe('Brainy plugin integration', () => { const brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' } }) const result = brain.use(plugin) expect(result).toBe(brain) - // Never init()'d — the constructor still registered it in Brainy's global - // instance registry, so it still needs a close() to deregister. - await brain.close().catch(() => {}) }) }) diff --git a/tests/unit/release/wall-entry.test.ts b/tests/unit/release/wall-entry.test.ts deleted file mode 100644 index b29ae326..00000000 --- a/tests/unit/release/wall-entry.test.ts +++ /dev/null @@ -1,403 +0,0 @@ -/** - * scripts/wall-entry.mjs — the mechanical releases-wall entry. - * - * The script's only real interface is its CLI (it has no importable - * exports by design — one door, no parallel API to drift from it), so - * these tests spawn it exactly as scripts/release.sh does: as a child - * process, against a fixture CHANGELOG and a throwaway local bare repo - * standing in for git@source.soulcraft.com:soulcraftlabs/releases.git - * (--remote) plus a throwaway cache directory (--cache-dir) standing in - * for ~/.cache/soulcraft-releases — never the real remote, never the - * real developer cache. - */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' -import { execFileSync } from 'node:child_process' -import { mkdtempSync, rmSync, writeFileSync, readFileSync, chmodSync } from 'node:fs' -import { tmpdir } from 'node:os' -import { join } from 'node:path' - -const SCRIPT = join(process.cwd(), 'scripts/wall-entry.mjs') - -/** Run the script and capture the outcome without throwing on a non-zero exit. */ -function run(args: string[], cwd: string): { status: number; stdout: string; stderr: string } { - try { - const stdout = execFileSync('node', [SCRIPT, ...args], { cwd, encoding: 'utf8' }) - return { status: 0, stdout, stderr: '' } - } catch (err: any) { - return { status: err.status ?? 1, stdout: err.stdout ?? '', stderr: err.stderr ?? '' } - } -} - -function git(args: string[], cwd: string): string { - return execFileSync('git', ['-C', cwd, ...args], { encoding: 'utf8' }).trim() -} - -const CHANGELOG_HEADER = '# Changelog\n\nAll notable changes, in this fixture.\n' - -/** Build a CHANGELOG.md with one entry per [version, bullets[]] pair, newest first. */ -function buildChangelog(entries: Array<{ version: string; date: string; bullets: string[] }>): string { - const body = entries - .map( - (e) => - `### [${e.version}](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/vX...v${e.version}) (${e.date})\n\n` + - e.bullets.map((b) => `- ${b} (abc1234)`).join('\n') + - '\n', - ) - .join('\n') - return CHANGELOG_HEADER + '\n' + body -} - -function wallFile(product: string, entries: unknown[]): string { - return JSON.stringify({ product, entries }, null, 2) + '\n' -} - -const BASE_ENTRY = { - version: '10.4.11', - date: '2026-09-02', - headline: 'A faster open', - items: ['A faster open.'], - url: 'https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.11', - thumb: null, -} - -/** A throwaway bare repo standing in for the real soulcraftlabs/releases remote. */ -function initBareRemote(): string { - const remoteDir = mkdtempSync(join(tmpdir(), 'wall-remote-')) - execFileSync('git', ['init', '--bare', '-b', 'main', remoteDir]) - return remoteDir -} - -/** Seed the bare remote with an initial .json, via a throwaway clone. */ -function seedRemote(remoteDir: string, product: string, entries: unknown[]): void { - const seedDir = mkdtempSync(join(tmpdir(), 'wall-seed-')) - execFileSync('git', ['clone', remoteDir, seedDir], { stdio: 'ignore' }) - git(['config', 'user.email', 'seed@example.com'], seedDir) - git(['config', 'user.name', 'Seed'], seedDir) - writeFileSync(join(seedDir, `${product}.json`), wallFile(product, entries)) - git(['add', `${product}.json`], seedDir) - git(['commit', '-m', 'seed'], seedDir) - git(['push', 'origin', 'main'], seedDir) - rmSync(seedDir, { recursive: true, force: true }) -} - -/** Read .json back out of the bare remote's main tip, via a throwaway clone. */ -function readRemote(remoteDir: string, product: string): any { - const readDir = mkdtempSync(join(tmpdir(), 'wall-read-')) - execFileSync('git', ['clone', remoteDir, readDir], { stdio: 'ignore' }) - const data = JSON.parse(readFileSync(join(readDir, `${product}.json`), 'utf8')) - rmSync(readDir, { recursive: true, force: true }) - return data -} - -/** Reject every push — stands in for any push failure (including a genuine - * non-fast-forward raced by a concurrent release rail), which this script - * treats identically: refuse loudly, name the cure, touch nothing further. */ -function makeRemoteRejectPushes(remoteDir: string): void { - const hookPath = join(remoteDir, 'hooks', 'pre-receive') - writeFileSync(hookPath, '#!/bin/sh\necho "remote: simulated push rejection" >&2\nexit 1\n') - chmodSync(hookPath, 0o755) -} - -let dir: string -let remoteDir: string -let cacheDir: string - -beforeEach(() => { - dir = mkdtempSync(join(tmpdir(), 'wall-entry-test-')) - // wall-entry.mjs is run with this dir as its cwd, standing in for the real - // developer checkout it reads its commit identity from (process.cwd()) — - // give it a repo-local identity the same way seedRemote gives one to the - // seed clone, so the suite is deterministic on a host with no global git - // config (a bare CI box) as much as one with a developer's own. - execFileSync('git', ['init', '-q', dir]) - git(['config', 'user.name', 'Wall Entry Test'], dir) - git(['config', 'user.email', 'wall-entry-test@example.com'], dir) - remoteDir = initBareRemote() - cacheDir = join(mkdtempSync(join(tmpdir(), 'wall-cache-')), 'soulcraft-releases') -}) - -afterEach(() => { - rmSync(dir, { recursive: true, force: true }) - rmSync(remoteDir, { recursive: true, force: true }) - rmSync(cacheDir, { recursive: true, force: true }) -}) - -describe('wall-entry.mjs — generate + publish', () => { - it('derives headline from the first bullet and items from every bullet, hashes stripped, and pushes it to the remote', () => { - seedRemote(remoteDir, 'open-brainy', [BASE_ENTRY]) - writeFileSync( - join(dir, 'CHANGELOG.md'), - buildChangelog([{ version: '10.4.12', date: '2026-09-03', bullets: ['fix(wall): mechanize the entry', 'test(wall): pin the shape'] }]), - ) - - const result = run( - ['--product', 'open-brainy', '--version', '10.4.12', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir], - dir, - ) - expect(result.status).toBe(0) - expect(result.stdout).toMatch(/wrote v10\.4\.12.*pushed/i) - - const wall = readRemote(remoteDir, 'open-brainy') - expect(wall.entries).toHaveLength(2) - expect(wall.entries[0]).toEqual({ - version: '10.4.12', - date: '2026-09-03', - headline: 'fix(wall): mechanize the entry', - items: ['fix(wall): mechanize the entry', 'test(wall): pin the shape'], - url: 'https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.12', - thumb: null, - }) - // the older entry stays put, still second - expect(wall.entries[1].version).toBe('10.4.11') - }) - - it('prepends newest-first — the new entry lands at index 0 ahead of every existing one', () => { - seedRemote(remoteDir, 'open-brainy', [BASE_ENTRY, { ...BASE_ENTRY, version: '10.4.10' }]) - writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.5.0', date: '2026-09-03', bullets: ['feat: ten five'] }])) - - run(['--product', 'open-brainy', '--version', '10.5.0', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir], dir) - - const wall = readRemote(remoteDir, 'open-brainy') - expect(wall.entries.map((e: any) => e.version)).toEqual(['10.5.0', '10.4.11', '10.4.10']) - }) - - it('replaces an entry with the same version instead of duplicating it — idempotent re-runs', () => { - seedRemote(remoteDir, 'open-brainy', [ - { ...BASE_ENTRY, headline: 'stale headline, pre-fix' }, - { ...BASE_ENTRY, version: '10.4.10' }, - ]) - writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.4.11', date: '2026-09-02', bullets: ['fix: the corrected headline'] }])) - - const result = run( - ['--product', 'open-brainy', '--version', '10.4.11', '--date', '2026-09-02', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir], - dir, - ) - expect(result.status).toBe(0) - expect(result.stdout).toMatch(/replaced v10\.4\.11/i) - - const wall = readRemote(remoteDir, 'open-brainy') - expect(wall.entries).toHaveLength(2) // not 3 — replaced, not duplicated - expect(wall.entries[0].version).toBe('10.4.11') - expect(wall.entries[0].headline).toBe('fix: the corrected headline') - expect(wall.entries[1].version).toBe('10.4.10') - }) - - it('a re-run with byte-identical content commits nothing and still succeeds', () => { - // headline always equals items[0] for a derived entry, so this fixture - // (unlike BASE_ENTRY, whose headline/items intentionally diverge for the - // shape-only tests below) has to keep the two in lockstep to ever roundtrip. - const stableEntry = { ...BASE_ENTRY, headline: 'A faster open.', items: ['A faster open.'] } - seedRemote(remoteDir, 'open-brainy', [stableEntry]) - writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.4.11', date: '2026-09-02', bullets: ['A faster open.'] }])) - const before = readRemote(remoteDir, 'open-brainy') - - const result = run( - ['--product', 'open-brainy', '--version', '10.4.11', '--date', '2026-09-02', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir], - dir, - ) - expect(result.status).toBe(0) - expect(result.stdout).toMatch(/nothing to commit/i) - expect(readRemote(remoteDir, 'open-brainy')).toEqual(before) - }) - - it('derives the public package-page permalink for the product engine (private repo, never null)', () => { - seedRemote(remoteDir, 'brainy', [{ ...BASE_ENTRY, version: '11.0.5', url: 'https://source.soulcraft.com/soulcraft/-/packages/npm/@soulcraft%2Fbrainy/11.0.5' }]) - writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '11.0.6', date: '2026-09-03', bullets: ['fix: a native-only fix'] }])) - - const result = run( - ['--product', 'brainy', '--version', '11.0.6', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir], - dir, - ) - expect(result.status).toBe(0) - - const wall = readRemote(remoteDir, 'brainy') - expect(wall.entries[0].url).toBe('https://source.soulcraft.com/soulcraft/-/packages/npm/@soulcraft%2Fbrainy/11.0.6') - expect(wall.entries[0].thumb).toBeNull() - }) - - it('refuses a product with no permalink pattern, naming the cure', () => { - seedRemote(remoteDir, 'open-brainy', [BASE_ENTRY]) - writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '1.0.0', date: '2026-09-03', bullets: ['feat: first'] }])) - - const result = run(['--product', 'mystery', '--version', '1.0.0', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir], dir) - expect(result.status).not.toBe(0) - expect(result.stderr).toMatch(/no permalink pattern for product "mystery"/) - expect(result.stderr).toMatch(/never carry url: null/) - }) - - it('refuses when the CHANGELOG has no entry yet for the target version, and touches no remote', () => { - seedRemote(remoteDir, 'open-brainy', []) - writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.4.11', date: '2026-09-02', bullets: ['fix: whatever'] }])) - const beforeSha = git(['rev-parse', 'main'], remoteDir) - - const result = run( - ['--product', 'open-brainy', '--version', '99.0.0', '--date', '2026-09-02', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir], - dir, - ) - - expect(result.status).toBe(1) - expect(result.stderr).toMatch(/no CHANGELOG entry yet/i) - expect(git(['rev-parse', 'main'], remoteDir)).toBe(beforeSha) - }) - - it('refuses by naming the cure when the remote cannot be cloned', () => { - writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.4.12', date: '2026-09-03', bullets: ['fix: whatever'] }])) - const noSuchRemote = join(tmpdir(), 'wall-remote-does-not-exist-' + Date.now()) - - const result = run( - ['--product', 'open-brainy', '--version', '10.4.12', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', noSuchRemote, '--cache-dir', cacheDir], - dir, - ) - - expect(result.status).toBe(1) - expect(result.stderr).toMatch(/cannot clone/i) - expect(result.stderr).toMatch(/cure:/i) - }) - - it('refuses by naming the cure, and touches no remote, when the fetched wall fails shape validation', () => { - const seedDir = mkdtempSync(join(tmpdir(), 'wall-seed-broken-')) - execFileSync('git', ['clone', remoteDir, seedDir], { stdio: 'ignore' }) - git(['config', 'user.email', 'seed@example.com'], seedDir) - git(['config', 'user.name', 'Seed'], seedDir) - writeFileSync( - join(seedDir, 'open-brainy.json'), - JSON.stringify({ product: 'open-brainy', entries: [{ version: '10.4.11', date: '2026-09-02', items: ['x'], url: null }] }, null, 2), - ) - git(['add', 'open-brainy.json'], seedDir) - git(['commit', '-m', 'seed broken'], seedDir) - git(['push', 'origin', 'main'], seedDir) - rmSync(seedDir, { recursive: true, force: true }) - const beforeSha = git(['rev-parse', 'main'], remoteDir) - - writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.4.12', date: '2026-09-03', bullets: ['fix: whatever'] }])) - - const result = run( - ['--product', 'open-brainy', '--version', '10.4.12', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir], - dir, - ) - - expect(result.status).toBe(1) - expect(result.stderr).toMatch(/fails shape validation/i) - expect(result.stderr).toMatch(/missing key\(s\) headline/i) - expect(git(['rev-parse', 'main'], remoteDir)).toBe(beforeSha) - }) - - it('refuses by naming the cure when the remote rejects the push (stands in for a raced non-fast-forward)', () => { - seedRemote(remoteDir, 'open-brainy', [BASE_ENTRY]) - makeRemoteRejectPushes(remoteDir) - writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.4.12', date: '2026-09-03', bullets: ['fix: whatever'] }])) - - const result = run( - ['--product', 'open-brainy', '--version', '10.4.12', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir], - dir, - ) - - expect(result.status).toBe(1) - expect(result.stderr).toMatch(/push to .* failed/i) - expect(result.stderr).toMatch(/cure:/i) - }) - - it('refuses a cross-product write when the file\'s "product" field does not match --product', () => { - seedRemote(remoteDir, 'open-brainy', [BASE_ENTRY]) - const seedDir = mkdtempSync(join(tmpdir(), 'wall-seed-mismatch-')) - execFileSync('git', ['clone', remoteDir, seedDir], { stdio: 'ignore' }) - git(['config', 'user.email', 'seed@example.com'], seedDir) - git(['config', 'user.name', 'Seed'], seedDir) - const corrupted = JSON.parse(readFileSync(join(seedDir, 'open-brainy.json'), 'utf8')) - corrupted.product = 'brainy' - writeFileSync(join(seedDir, 'open-brainy.json'), JSON.stringify(corrupted, null, 2) + '\n') - git(['add', 'open-brainy.json'], seedDir) - git(['commit', '-m', 'corrupt product field'], seedDir) - git(['push', 'origin', 'main'], seedDir) - rmSync(seedDir, { recursive: true, force: true }) - - writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '1.0.0', date: '2026-09-03', bullets: ['fix: wrong repo'] }])) - - const result = run( - ['--product', 'open-brainy', '--version', '1.0.0', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir], - dir, - ) - - expect(result.status).toBe(1) - expect(result.stderr).toMatch(/product "brainy".*--product "open-brainy"/i) - }) -}) - -describe('wall-entry.mjs — --dry-run', () => { - it('prints the entry and the target path, and touches neither the cache dir nor the remote', () => { - seedRemote(remoteDir, 'open-brainy', [BASE_ENTRY]) - writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.4.12', date: '2026-09-03', bullets: ['fix: a dry run'] }])) - const beforeSha = git(['rev-parse', 'main'], remoteDir) - - const result = run( - ['--dry-run', '--product', 'open-brainy', '--version', '10.4.12', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir], - dir, - ) - - expect(result.status).toBe(0) - expect(result.stdout).toMatch(/would write to/i) - expect(result.stdout).toMatch(/"version": "10\.4\.12"/) - expect(git(['rev-parse', 'main'], remoteDir)).toBe(beforeSha) - }) -}) - -describe('wall-entry.mjs — --check', () => { - it('passes a well-formed, newest-first file with no duplicates', () => { - writeFileSync(join(dir, 'wall.json'), wallFile('open-brainy', [BASE_ENTRY, { ...BASE_ENTRY, version: '10.4.10' }])) - const result = run(['--check', '--file', 'wall.json'], dir) - expect(result.status).toBe(0) - expect(result.stdout).toMatch(/OK/) - }) - - it('passes a file where "thumb" is entirely absent (optional per the HQ contract)', () => { - const { thumb, ...noThumb } = BASE_ENTRY as any - writeFileSync(join(dir, 'wall.json'), wallFile('open-brainy', [noThumb])) - const result = run(['--check', '--file', 'wall.json'], dir) - expect(result.status).toBe(0) - }) - - it('catches a missing entry key', () => { - const broken = { version: '1.0.0', date: '2026-09-03', headline: 'h', items: ['i'] } // no "url" - writeFileSync(join(dir, 'wall.json'), wallFile('open-brainy', [broken])) - const result = run(['--check', '--file', 'wall.json'], dir) - expect(result.status).toBe(1) - expect(result.stderr).toMatch(/missing key\(s\) url/) - }) - - it('catches an unexpected top-level key (e.g. the retired "history" field)', () => { - const raw = JSON.parse(wallFile('open-brainy', [BASE_ENTRY])) - raw.history = 'retired field' - writeFileSync(join(dir, 'wall.json'), JSON.stringify(raw)) - const result = run(['--check', '--file', 'wall.json'], dir) - expect(result.status).toBe(1) - expect(result.stderr).toMatch(/unexpected key\(s\) history/) - }) - - it('catches entries that are not newest-first', () => { - writeFileSync(join(dir, 'wall.json'), wallFile('open-brainy', [{ ...BASE_ENTRY, version: '10.4.10' }, BASE_ENTRY])) - const result = run(['--check', '--file', 'wall.json'], dir) - expect(result.status).toBe(1) - expect(result.stderr).toMatch(/not newest-first/) - }) - - it('catches a duplicate version even with identical entries', () => { - writeFileSync(join(dir, 'wall.json'), wallFile('open-brainy', [BASE_ENTRY, { ...BASE_ENTRY }])) - const result = run(['--check', '--file', 'wall.json'], dir) - expect(result.status).toBe(1) - expect(result.stderr).toMatch(/duplicate version 10\.4\.11/) - }) - - it('catches an empty items array', () => { - writeFileSync(join(dir, 'wall.json'), wallFile('open-brainy', [{ ...BASE_ENTRY, items: [] }])) - const result = run(['--check', '--file', 'wall.json'], dir) - expect(result.status).toBe(1) - expect(result.stderr).toMatch(/"items" must be a non-empty array/) - }) - - it('catches a malformed date', () => { - writeFileSync(join(dir, 'wall.json'), wallFile('open-brainy', [{ ...BASE_ENTRY, date: '09/03/2026' }])) - const result = run(['--check', '--file', 'wall.json'], dir) - expect(result.status).toBe(1) - expect(result.stderr).toMatch(/"date" must be a YYYY-MM-DD string/) - }) -}) diff --git a/tests/unit/storage/pagination-parallel-hydration.test.ts b/tests/unit/storage/pagination-parallel-hydration.test.ts index a98fe8c9..ada324bb 100644 --- a/tests/unit/storage/pagination-parallel-hydration.test.ts +++ b/tests/unit/storage/pagination-parallel-hydration.test.ts @@ -7,7 +7,7 @@ * hydration (zero per-entity reads when unfiltered). Both must preserve the exact * pagination contract: same order, cursor continuation, filters, totalCount. */ -import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' +import { describe, it, expect, beforeEach, vi } from 'vitest' import { Brainy, NounType } from '../../../src/index.js' describe('paginated enumeration — parallel hydration + id-only (cortex heal-cost)', () => { @@ -30,10 +30,6 @@ describe('paginated enumeration — parallel hydration + id-only (cortex heal-co storage = brain.storage }) - afterEach(async () => { - await brain.close() - }) - /** Page the whole dataset through a small limit via cursor and collect ordered ids. */ const pageAll = async (fn: (opts: any) => Promise, key: 'items' | 'ids') => { const out: string[] = [] diff --git a/tests/unit/type-filtering.unit.test.ts b/tests/unit/type-filtering.unit.test.ts index a1943da9..9e4700b2 100644 --- a/tests/unit/type-filtering.unit.test.ts +++ b/tests/unit/type-filtering.unit.test.ts @@ -4,7 +4,7 @@ * Tests to verify that brain.find({ type: NounType.X }) correctly filters entities */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy, NounType } from '../../src/index.js' describe('Type Filtering (A Consumer Team Issue)', () => { @@ -17,10 +17,6 @@ describe('Type Filtering (A Consumer Team Issue)', () => { await brain.init() }) - afterEach(async () => { - await brain.close() - }) - it('should filter entities by NounType.Person', async () => { // Add 3 people await brain.add({ data: 'John Smith', type: NounType.Person, metadata: { name: 'John' } }) diff --git a/tests/unit/utils/metadataIndex-array-bound.test.ts b/tests/unit/utils/metadataIndex-array-bound.test.ts index 32bf5d8c..cbbf6b63 100644 --- a/tests/unit/utils/metadataIndex-array-bound.test.ts +++ b/tests/unit/utils/metadataIndex-array-bound.test.ts @@ -15,10 +15,9 @@ * and the caller had no way to tell that from "no row matches". Eleven tags is * not an exotic shape; the eleventh tag made the row invisible. * - * THE LAW. Arrays of scalars index up to {@link MAX_INDEXED_ARRAY_LENGTH}, + * THE LAW. Arrays of scalars index up to {@link MAX_INDEXED_ARRAY_LENGTH} = 64, * hardcoded (the zero-config law: no knob), which clears every legitimate - * multi-value field — tags, authors, keyword lists — and stays below the - * narrowest embedding this engine meets (384 dimensions). Above it the WRITE + * multi-value field and stays far below any embedding width. Above it the WRITE * IS REFUSED by name — `MetadataArrayTooLargeError`, carrying the field, the * length and the bound — at `add`, `update`, `relate` and `updateRelation` * alike. Nothing is skipped in silence. @@ -48,10 +47,6 @@ describe('the indexable-array bound', () => { await brain.init() }) - afterEach(async () => { - await brain.close() - }) - describe('BELOW the bound: the array indexes, every element of it', () => { it('the eleven-element array that used to vanish is searchable', async () => { // ELEVEN — one over the old silent limit, the whole shape of the defect. @@ -70,7 +65,7 @@ describe('the indexable-array bound', () => { } }) - it('indexes right up to the bound — every element of it', async () => { + it('indexes right up to the bound — all 64 elements', async () => { await brain.add({ id: 'at-bound', data: 'a row at the bound', @@ -79,9 +74,8 @@ describe('the indexable-array bound', () => { vector: [] }) - // The first, the last, and one in the middle — all derived from the - // bound, so the case follows the constant wherever it moves. - for (const tag of ['t0', `t${MAX_INDEXED_ARRAY_LENGTH - 1}`, `t${Math.floor(MAX_INDEXED_ARRAY_LENGTH / 2)}`]) { + // The first, the last, and one in the middle. + for (const tag of ['t0', `t${MAX_INDEXED_ARRAY_LENGTH - 1}`, 't31']) { const hits = await brain.find({ where: { tags: tag }, limit: 10 } as any) expect(hits.map((r: any) => r.id)).toContain(resolveEntityId('at-bound')) } diff --git a/tests/unit/utils/metadataIndex-sparse-range-collation.test.ts b/tests/unit/utils/metadataIndex-sparse-range-collation.test.ts index 7a2bf0a7..d6d00568 100644 --- a/tests/unit/utils/metadataIndex-sparse-range-collation.test.ts +++ b/tests/unit/utils/metadataIndex-sparse-range-collation.test.ts @@ -42,7 +42,7 @@ * column store adopts the field. It is named in `getIdsFromChunksForRange`'s * doc comment rather than papered over. */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy } from '../../../src/brainy' import { NounType } from '../../../src/types/graphTypes' import { SparseIndex, ChunkManager } from '../../../src/utils/metadataIndexChunking' @@ -122,10 +122,6 @@ describe('legacy sparse index: range queries order values, or refuse', () => { expect(index.columnStore.hasField(FIELD)).toBe(false) }) - afterEach(async () => { - await brain.close() - }) - describe('(a) a long BOUND against ordinary short values', () => { // 'apple' < 'mango' < 'zebra', and every bound below is compared against // these three raw keys. diff --git a/tests/unit/validate-invariants-delegation.test.ts b/tests/unit/validate-invariants-delegation.test.ts index 69133733..a5def81f 100644 --- a/tests/unit/validate-invariants-delegation.test.ts +++ b/tests/unit/validate-invariants-delegation.test.ts @@ -6,7 +6,7 @@ * validateInvariants(), and repairIndex() maps a failing invariant with heal:'rebuild' * to that provider's rebuild(). "healthy-while-broken must be impossible." */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy, NounType } from '../../src/index.js' import type { ProviderInvariantReport } from '../../src/index.js' @@ -48,10 +48,6 @@ describe('validateIndexConsistency delegates to provider validateInvariants() (P await brain.flush() }) - afterEach(async () => { - await brain.close() - }) - it('a broken provider report makes the store unhealthy and names the failing invariant', async () => { brain.index.validateInvariants = async () => brokenReport('vector') const v = await brain.validateIndexConsistency() diff --git a/tests/unit/vector-cold-read-guard.test.ts b/tests/unit/vector-cold-read-guard.test.ts index 963009b7..0905f298 100644 --- a/tests/unit/vector-cold-read-guard.test.ts +++ b/tests/unit/vector-cold-read-guard.test.ts @@ -12,7 +12,7 @@ * signal (from either strategy) THROWS VectorIndexNotReadyError immediately, * with no rebuild attempt in between — never a silent empty result. */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy, NounType, VectorIndexNotReadyError } from '../../src/index.js' const V = (): number[] => Array.from({ length: 384 }, (_, i) => Math.sin(i * 0.1) + 0.001) @@ -28,10 +28,6 @@ describe('Vector cold-read guard (verifyVectorLive) — silent-[] on cold semant await brain.flush() }) - afterEach(async () => { - await brain.close() - }) - it('warm brain: semantic find is correct and the guard does not rebuild', async () => { const vi = brain.index let rebuilds = 0 diff --git a/tests/unit/vfs-multi-instance-diagnostic.test.ts b/tests/unit/vfs-multi-instance-diagnostic.test.ts index 85ff1002..deaa4615 100644 --- a/tests/unit/vfs-multi-instance-diagnostic.test.ts +++ b/tests/unit/vfs-multi-instance-diagnostic.test.ts @@ -4,7 +4,7 @@ * Tests to verify VFS import behavior and identify if VFS creates only wrappers or also graph entities */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy, NounType } from '../../src/index.js' describe('VFS Multi-instance Diagnostic', () => { @@ -17,10 +17,6 @@ describe('VFS Multi-instance Diagnostic', () => { await brain.init() }) - afterEach(async () => { - await brain.close() - }) - it('should verify VFS creates document wrappers AND allows entity filtering', async () => { console.log('\n🔬 VFS Multi-instance Diagnostic Test\n') console.log('='.repeat(70)) diff --git a/tests/vfs/tree-operations.unit.test.ts b/tests/vfs/tree-operations.unit.test.ts index 91743227..8c717115 100644 --- a/tests/vfs/tree-operations.unit.test.ts +++ b/tests/vfs/tree-operations.unit.test.ts @@ -3,7 +3,7 @@ * Ensures tree methods prevent recursion and work correctly */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy } from '../../src/brainy.js' import { VirtualFileSystem } from '../../src/vfs/VirtualFileSystem.js' import { VFSTreeUtils } from '../../src/vfs/TreeUtils.js' @@ -24,10 +24,6 @@ describe('VFS Tree Operations', () => { await vfs.init() }) - afterEach(async () => { - await brain.close() - }) - describe('Critical: No Self-Inclusion Bug', () => { it('should NEVER return a directory as its own child', async () => { // Create test structure diff --git a/tests/vfs/vfs-bug-fixes.unit.test.ts b/tests/vfs/vfs-bug-fixes.unit.test.ts index 12199c8b..f98d6a76 100644 --- a/tests/vfs/vfs-bug-fixes.unit.test.ts +++ b/tests/vfs/vfs-bug-fixes.unit.test.ts @@ -6,7 +6,7 @@ * - Issue #2: File read decompression error */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy } from '../../src/brainy.js' import { VirtualFileSystem } from '../../src/vfs/VirtualFileSystem.js' @@ -25,10 +25,6 @@ describe('VFS Bug Fixes', () => { await vfs.init() }) - afterEach(async () => { - await brain.close() - }) - describe('Issue #1: Duplicate Directory Nodes', () => { it('should not create duplicate directory entries when writing multiple files to same directory', async () => { // Write multiple files to the same directory (reproduce the bug scenario) diff --git a/tests/vfs/vfs-bulkwrite-race.unit.test.ts b/tests/vfs/vfs-bulkwrite-race.unit.test.ts index 09d68568..238ac6b9 100644 --- a/tests/vfs/vfs-bulkwrite-race.unit.test.ts +++ b/tests/vfs/vfs-bulkwrite-race.unit.test.ts @@ -12,7 +12,7 @@ * other operations in parallel batches. */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach } from 'vitest' import { Brainy } from '../../src/brainy.js' import { VirtualFileSystem } from '../../src/vfs/VirtualFileSystem.js' @@ -30,10 +30,6 @@ describe('VFS bulkWrite Race Condition Fix', () => { await vfs.init() }) - afterEach(async () => { - await brain.close() - }) - describe('operation ordering', () => { it('should create directories before files when mixed in same batch', async () => { // This is the exact scenario that triggered the race condition: diff --git a/tests/vfs/vfs.unit.test.ts b/tests/vfs/vfs.unit.test.ts index b4024155..4b4ba8d2 100644 --- a/tests/vfs/vfs.unit.test.ts +++ b/tests/vfs/vfs.unit.test.ts @@ -389,14 +389,7 @@ describe('VirtualFileSystem - Production Tests', () => { }) describe('Performance', () => { - it('should handle many files efficiently', async (ctx) => { - // Wall-clock budget assertion — belongs to the perf lane (npm run - // test:perf), not the correctness gate: 121ms alone but 16.5s under - // the gate's sibling-file contention, a flake the code never caused - // (same pattern as storage-batch-operations.test.ts's batch-vs- - // individual timing case). - ctx.skip(!process.env.BRAINY_PERF_LANE, 'wall-clock budget assertion — runs only under the perf lane (npm run test:perf)') - + it('should handle many files efficiently', async () => { const dir = '/performance-test' await vfs.mkdir(dir)