Compare commits
6 commits
main
...
fix/10412-
| Author | SHA1 | Date | |
|---|---|---|---|
| b8a80b9900 | |||
| 3a339ce4af | |||
| 47cfaa7669 | |||
| 59003fd8bc | |||
| 913b4ffc6b | |||
| 819a5c5a8a |
79 changed files with 379 additions and 4042 deletions
|
|
@ -12,11 +12,6 @@ on:
|
|||
push:
|
||||
tags:
|
||||
- 'v*'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
ref_reason:
|
||||
description: 'why this manual run (e.g. tag event dropped)'
|
||||
required: false
|
||||
|
||||
jobs:
|
||||
publish:
|
||||
|
|
|
|||
22
CHANGELOG.md
22
CHANGELOG.md
|
|
@ -2,28 +2,6 @@
|
|||
|
||||
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
|
||||
|
||||
|
||||
### [10.4.13](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.12...v10.4.13) (2026-09-03)
|
||||
|
||||
- A shutdown that holds its listener until the exit decision, and a test suite that closes every brain it opens
|
||||
- fix(shutdown): the engine's signal handler keeps its listener registered until the exit decision is made — closing the last live instance no longer deregisters the handler mid-run, so a second signal delivery during a clean shutdown can never kill the process after the work is done (a2ea21b3)
|
||||
- fix(release): the release wall entry commits under an explicit git identity read from the developer's checkout; a host with no identity refuses by name instead of failing inside git (aac853d3)
|
||||
- test(hygiene): every brain a test file creates is closed by that file — 40 files fixed, the leaks that let a stray cadence narrate into later files are gone; brains whose init() was expected to fail are closed too (6eb5e448)
|
||||
|
||||
### [10.4.12](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.11...v10.4.12) (2026-09-03)
|
||||
|
||||
- Mixed-kind fields index exactly, arrays to 256, a drained loop is not a shutdown, and finds project from the column store
|
||||
- fix(index): a metadata field holds every value kind it was written with — one posting column per (field, kind); an equality filter reads the query value's own kind, a range routes by its bounds; nothing is refused and nothing is silently dropped; an index written by the old shape opens unchanged (a128f0ed)
|
||||
- fix(metadata): metadata arrays index up to 256 elements; a longer array refuses at write time by name (MetadataArrayTooLargeError) — a vector parked in metadata now throws; move it to `vector` (e435da78)
|
||||
- fix(shutdown): beforeExit runs a non-closing flush only — a script that never calls close() exits with the writer lock on disk and no clean-shutdown marker, and the next open evicts the stale lock and folds the log, bounded; SIGTERM and SIGINT are unchanged (6baa4d7f)
|
||||
- feat(find): field projection — find({fields}) and get({fields}) resolve scalars from the column store on every leg, including vector-leg finds; absent fields stay absent (ad0f493f)
|
||||
- fix(find): orderBy is the order on every find path, not only the metadata-only one (5e720d17)
|
||||
- fix(metadata): the legacy sparse range path orders values, or refuses by name — never ranks by hash (a7eb7f52)
|
||||
- fix(close): a read-only brain writes nothing under `_system/` (f27a7776)
|
||||
- fix(contract): the flush gate's internals are private, not doors (72c8ee6a)
|
||||
- test(hygiene): the triple-intelligence correctness cases sit in the gate; the idle and connected-find pins name the brain they measure (28083981)
|
||||
- ci(release): the rail writes its own wall entry into the shared releases repo — never hand-written again (adcb883e)
|
||||
|
||||
### [10.4.11](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.9...v10.4.11) (2026-09-02)
|
||||
|
||||
- ci: superseded pushes cancel their own runs (concurrency per ref) (6053f6d4)
|
||||
|
|
|
|||
|
|
@ -4,11 +4,6 @@
|
|||
|
||||
<h1 align="center">Brainy</h1>
|
||||
|
||||
> **Frozen at 10.4.13 (2026-09-03).** This repository is the reference implementation of the Brainy store format and API,
|
||||
> published under the MIT license. Version 10.4.13 is its last release; the repository is read-only from here. The engine
|
||||
> continues as `@soulcraft/brainy`, which bundles this layer as owned code; every published version of this package stays
|
||||
> available on The Source. Use this repository to read a Brainy store independently or to verify the conformance contract.
|
||||
|
||||
<p align="center">
|
||||
<b>Three database paradigms. One API. Zero configuration.</b><br>
|
||||
The in-process knowledge database for TypeScript — vector search, graph traversal,<br>
|
||||
|
|
|
|||
10
RELEASES.md
10
RELEASES.md
|
|
@ -1,15 +1,5 @@
|
|||
# @soulcraft/brainy — Release Notes for Consumers
|
||||
|
||||
> **Frozen at 10.4.13 (2026-09-03).** 10.4.13 is the last release of `@soulcraftlabs/brainy`; this repository is read-only from here.
|
||||
> Release notes for the product engine continue on its own wall.
|
||||
|
||||
Machine-readable release notes are published at
|
||||
https://source.soulcraft.com/soulcraftlabs/releases/raw/branch/main/open-brainy.json
|
||||
(this engine) and
|
||||
https://source.soulcraft.com/soulcraftlabs/releases/raw/branch/main/brainy.json
|
||||
(the product engine) — read by HQ's `/hq/releases` door, and the source of
|
||||
truth ahead of this file.
|
||||
|
||||
This file is the **quick reference for downstream sessions** tracking Brainy changes.
|
||||
Full auto-generated changelog: `CHANGELOG.md` · Releases: https://source.soulcraft.com/soulcraftlabs/open-brainy/releases
|
||||
|
||||
|
|
|
|||
|
|
@ -369,71 +369,6 @@ return results.slice(offset, offset + limit)
|
|||
// → Auto-correction: Use most likely alternative based on affinity data
|
||||
```
|
||||
|
||||
## Field Projection (`fields`)
|
||||
|
||||
`find()` and `get()` accept a `fields` list. Without it they return the whole
|
||||
record; with it they return only the fields you name — and, where the index can
|
||||
supply them, without opening the canonical record at all.
|
||||
|
||||
```ts
|
||||
// A list page: two user fields and one engine scalar. No document bodies.
|
||||
await brain.find({
|
||||
where: { kind: 'post' },
|
||||
fields: ['title', 'slug', 'system.createdAt'],
|
||||
limit: 50
|
||||
})
|
||||
|
||||
await brain.get(id, { fields: ['title'] })
|
||||
```
|
||||
|
||||
### Why it exists
|
||||
|
||||
A list view that renders a title and a date does not need the body, but without
|
||||
a projection every row hydrates its full record and throws almost all of it
|
||||
away. On a posts list that is the dominant cost of the query.
|
||||
|
||||
### The rules
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **`fields` absent** | The full record, byte-identical to before. Nothing changes. |
|
||||
| **Field names** | The one addressing law: a bare name is user metadata (`'title'`), `system.*` is an engine scalar (`'system.createdAt'`). |
|
||||
| **A field the row lacks** | Simply **absent** from the result. Never an error. |
|
||||
| **Identity** | Every row keeps its `id` (and `score` on `find`) regardless — a row you cannot identify is not a row. |
|
||||
| **Where values come from** | The **column store**, which holds raw values. Never the sparse index, which buckets timestamps for range queries. |
|
||||
| **A field the column cannot serve** | The canonical record is read for that field only. Correct, just not free. |
|
||||
|
||||
### Missing fields are absent, not errors
|
||||
|
||||
This is deliberate and differs from `orderBy`, which throws
|
||||
`UnresolvableFieldError` for an unknown field. A typo in `orderBy` silently
|
||||
changes the ordering, so it must be loud. A projection asks "give me these if
|
||||
you have them", and an optional field must not turn a list into a failure — so
|
||||
`fields` uses the permissive path.
|
||||
|
||||
### Cost
|
||||
|
||||
When every named field is column-served, a projected page performs **zero**
|
||||
canonical reads. When one is not, only that read happens and the rest still come
|
||||
from the index. Both are pinned by counting reads rather than timing them, in
|
||||
`tests/integration/find-fields-projection.test.ts`.
|
||||
|
||||
### `related()` takes no `fields`
|
||||
|
||||
A `Relation` carries `from` and `to` as **ids** and hydrates no entity record,
|
||||
so there is nothing for a projection to trim. Projecting the endpoints would be
|
||||
a new capability rather than a projection of an existing one.
|
||||
|
||||
### For engine implementers
|
||||
|
||||
Projection is served through an optional provider door,
|
||||
`getScalarsForIds(ids, fields)` on `MetadataIndexProvider`. The contract is in
|
||||
`src/plugin.ts`; the short version is **return only what you can serve exactly,
|
||||
and say what you served**. The caller diffs the answer against the request and
|
||||
reads records for the remainder, so omission costs a read while a wrong value is
|
||||
a wrong answer nobody can see. An engine without the door still works — every
|
||||
field falls back to the record.
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Query Performance by Type
|
||||
|
|
|
|||
|
|
@ -217,40 +217,6 @@ membership queries at scale:
|
|||
`__words__` for tokenized text…).
|
||||
- `_blobs/_column_index/{field}/L0-NNNNNN.bin` — the actual level-0 run
|
||||
segments, stored through the shared `_blobs/<key>.bin` binary convention.
|
||||
- `_column_index/{field}/k/{kind}/…` — the same two files again, for a
|
||||
**second value kind** on the same field (see below). Absent for a field that
|
||||
holds one kind, which is nearly all of them.
|
||||
|
||||
### One posting column per (field, kind)
|
||||
|
||||
A field is not obliged to hold one type of value. `category` may carry
|
||||
`'electronics'` on some rows and `5` on others, and both are real values of
|
||||
that field. A segment, though, has one encoding — i64, f64, UTF-8, or boolean
|
||||
— so a field that holds several kinds gets **one column per kind**:
|
||||
|
||||
- The first kind a field ever sees owns the plain `_column_index/{field}/`
|
||||
layout above. A single-kind field is therefore byte-identical to what earlier
|
||||
versions wrote, and an index written before typed postings opens unchanged.
|
||||
- Every later kind gets its own column beside it at
|
||||
`_column_index/{field}/k/{kind}/`, where `{kind}` is `number`, `string` or
|
||||
`boolean`.
|
||||
|
||||
What that buys at query time:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Equality** | Answered from the column matching the **query value's own kind**. `where {category: 5}` reads the number postings; `where {category: '5'}` reads the string postings. Neither borrows the other's rows — a row written with the number `5` is not a row whose category is the text `'5'`. |
|
||||
| **A kind the field never held** | Matches nothing. That is the true answer, not a coerced one. |
|
||||
| **Ranges** | Routed by the kind of the bounds: numeric bounds read the numeric postings and ignore the field's strings. An **unbounded** range is the "has any value here" probe behind `exists`, and reads every kind. |
|
||||
| **`orderBy`** | A number and a string have no order between them, so a mixed field orders by kind first (number, string, boolean) and by value within a kind. A single-kind field sorts exactly as it always did. |
|
||||
| **Numbers** | One kind, one column: an integer column is written as i64 and widens to f64 the first time a non-integer arrives, so `4.5` is stored as itself rather than rounded. |
|
||||
|
||||
`null` and `undefined` are not kinds and are never posted; their absence is
|
||||
what the `exists` / `missing` operators read.
|
||||
|
||||
Older readers are unaffected by the additional columns: they see the field's
|
||||
primary column exactly where it has always been, and a `k/{kind}` directory is
|
||||
simply a name they never query.
|
||||
|
||||
Sparse per-field indexes, roaring-bitmap chunks, and zone-map/bloom segments
|
||||
additionally live as bucketed keys under `_system/idx/` (see §3). Which path
|
||||
|
|
|
|||
|
|
@ -95,15 +95,8 @@ The heartbeat interval rewrites the lock file every 10 seconds. The timer
|
|||
is unref'd, so it does not keep the event loop alive on its own.
|
||||
|
||||
On normal shutdown the writer releases the lock in `close()`. The shutdown
|
||||
hooks Brainy registers for `SIGTERM` and `SIGINT` close every live brain by
|
||||
that same `close()`, so a container restart doesn't strand the directory.
|
||||
|
||||
`beforeExit` is not one of them. Node emits it whenever the event loop has
|
||||
no ref'd work left — a state a healthy script reaches routinely, because
|
||||
Brainy's own idle and cadence timers are unref'd — and a drained event loop
|
||||
is not a shutdown. That hook only persists derived state with a non-closing
|
||||
`flush()`: it closes nothing, releases no lock, and leaves every brain open
|
||||
and usable. If you want a shutdown, call `close()` or send `SIGTERM`.
|
||||
hooks Brainy registers for `SIGTERM`, `SIGINT`, and `beforeExit` also
|
||||
release the lock so a container restart doesn't strand the directory.
|
||||
|
||||
## How to inspect a live writer
|
||||
|
||||
|
|
|
|||
4
package-lock.json
generated
4
package-lock.json
generated
|
|
@ -1,12 +1,12 @@
|
|||
{
|
||||
"name": "@soulcraftlabs/brainy",
|
||||
"version": "10.4.13",
|
||||
"version": "10.4.11",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@soulcraftlabs/brainy",
|
||||
"version": "10.4.13",
|
||||
"version": "10.4.11",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@msgpack/msgpack": "^3.1.2",
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"name": "@soulcraftlabs/brainy",
|
||||
"version": "10.4.13",
|
||||
"version": "10.4.11",
|
||||
"brainyContract": 1,
|
||||
"description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.",
|
||||
"main": "dist/index.js",
|
||||
|
|
|
|||
|
|
@ -154,26 +154,13 @@ else
|
|||
fi
|
||||
|
||||
# Create new changelog entry
|
||||
RELEASE_DATE=$(date +%Y-%m-%d)
|
||||
CHANGELOG_ENTRY="### [${NEW_VERSION}](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) (${RELEASE_DATE})
|
||||
CHANGELOG_ENTRY="### [${NEW_VERSION}](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d))
|
||||
|
||||
${COMMITS}
|
||||
"
|
||||
# A CURATED entry wins over the generated one. When a release is cut from a
|
||||
# lineage that diverged from the previous tag (a candidate branch carrying
|
||||
# main's history), `git log <last-tag>..HEAD` lists every commit the tag never
|
||||
# saw — old notes, already-shipped fixes under new hashes, merge commits — and a
|
||||
# wall entry derived from it would misreport the release. If CHANGELOG.md
|
||||
# already carries a `### [NEW_VERSION]` heading, it was written on purpose:
|
||||
# keep it, and skip the generated prepend entirely.
|
||||
CURATED_ENTRY=false
|
||||
if grep -qE "^### \[${NEW_VERSION}\]" CHANGELOG.md 2>/dev/null; then
|
||||
CURATED_ENTRY=true
|
||||
echo -e "${YELLOW}CHANGELOG already carries a curated ### [${NEW_VERSION}] entry — keeping it, not generating one from commits${NC}"
|
||||
fi
|
||||
|
||||
# Prepend to CHANGELOG.md after header
|
||||
if [ "$CURATED_ENTRY" = false ] && [ -f "CHANGELOG.md" ]; then
|
||||
if [ -f "CHANGELOG.md" ]; then
|
||||
# Read header (first 4 lines)
|
||||
HEADER=$(head -n 4 CHANGELOG.md)
|
||||
# Read rest of file
|
||||
|
|
@ -187,19 +174,6 @@ if [ "$CURATED_ENTRY" = false ] && [ -f "CHANGELOG.md" ]; then
|
|||
fi
|
||||
echo -e "${GREEN}✅ CHANGELOG updated${NC}\n"
|
||||
|
||||
# Step 6b: Update the releases wall entry — mechanical, derived from the
|
||||
# CHANGELOG entry just composed. The fleet's HQ page reads open-brainy.json
|
||||
# from the one shared releases repo, soulcraftlabs/releases on The Source —
|
||||
# this used to be hand-written after every release (David: never again —
|
||||
# make it a step of the rail, landed in the one shared home; this repo no
|
||||
# longer hosts its own copy). This step clones/fetches that repo into a
|
||||
# local cache, prepends the entry, and pushes it directly — a real
|
||||
# cross-repo push, refusing loudly (never skipping) on any
|
||||
# clone/validation/commit/push failure.
|
||||
echo -e "${BLUE}5️⃣▸ Updating the releases wall...${NC}"
|
||||
node scripts/wall-entry.mjs --product open-brainy --version "${NEW_VERSION}" --date "${RELEASE_DATE}" --from-changelog CHANGELOG.md
|
||||
echo -e "${GREEN}✅ Releases wall updated${NC}\n"
|
||||
|
||||
# Step 7: Create release commit
|
||||
echo -e "${BLUE}6️⃣ Creating release commit...${NC}"
|
||||
git add package.json package-lock.json CHANGELOG.md
|
||||
|
|
@ -263,7 +237,7 @@ fi
|
|||
# and RELEASES.md are the record; this just gives The Source's UI a release page).
|
||||
echo -e "${BLUE}🔟 Creating release page on The Source...${NC}"
|
||||
if [ -n "${FORGEJO_RELEASE_TOKEN:-}" ]; then
|
||||
if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraftlabs/open-brainy/releases" \
|
||||
if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraft/brainy/releases" \
|
||||
-H "Authorization: token ${FORGEJO_RELEASE_TOKEN}" -H "Content-Type: application/json" \
|
||||
-d "{\"tag_name\":\"v${NEW_VERSION}\",\"name\":\"v${NEW_VERSION}\",\"prerelease\":${PRERELEASE}}" >/dev/null; then
|
||||
echo -e "${GREEN}✅ Release page created on The Source${NC}\n"
|
||||
|
|
|
|||
|
|
@ -1,539 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
/**
|
||||
* @module scripts/wall-entry
|
||||
* @description The releases-wall entry, made mechanical. The fleet's HQ page
|
||||
* reads one public JSON per product from the ONE releases repo on The Source
|
||||
* (soulcraftlabs/releases, files <product>.json at its root — shape
|
||||
* {product, entries:[{version, date, headline, items, url, thumb?}]}), at
|
||||
* https://source.soulcraft.com/soulcraftlabs/releases/raw/branch/main/<product>.json.
|
||||
* Those entries were hand-written after every release, then briefly written
|
||||
* into this repo's own releases/<product>.json; this script is the one door
|
||||
* that composes an entry and lands it in the shared repo, so it is never
|
||||
* hand-written and never forked across repos again.
|
||||
*
|
||||
* Two modes:
|
||||
*
|
||||
* 1. Generate + publish (default):
|
||||
* node wall-entry.mjs --product <p> --version <v> --date <YYYY-MM-DD> \
|
||||
* --from-changelog <CHANGELOG.md>
|
||||
* Derives an entry from the CHANGELOG.md entry for <v> (headline = the
|
||||
* entry's first bullet, items = every bullet, trimmed of its trailing
|
||||
* commit hash), then:
|
||||
* - clones (or, if a cached clone already exists, fetches and resets)
|
||||
* the releases repo into a local cache directory,
|
||||
* - prepends the entry to <cache>/<p>.json, newest first — replacing
|
||||
* any existing entry for the same version so a re-run is idempotent,
|
||||
* - validates the file's shape before and after,
|
||||
* - commits the change as "chore(wall): <p> <v>" and pushes main.
|
||||
* A failure at any step (clone, validation, commit, push, a
|
||||
* non-fast-forward remote) exits non-zero naming the cure. Nothing is
|
||||
* ever skipped — the wall either lands correctly or the release fails.
|
||||
*
|
||||
* 2. Dry run:
|
||||
* node wall-entry.mjs --dry-run --product <p> --version <v> \
|
||||
* --date <YYYY-MM-DD> --from-changelog <CHANGELOG.md>
|
||||
* Derives the entry exactly as above and prints it, along with the file
|
||||
* it would be written to, but touches no clone and no remote — usable
|
||||
* from a fresh checkout with no cache and no network.
|
||||
*
|
||||
* 3. Validate only (--check):
|
||||
* node wall-entry.mjs --check --file <path/to/product.json>
|
||||
* Validates an arbitrary wall file's exact key set (top-level and
|
||||
* per-entry), field types, and strict-descending semver ordering with
|
||||
* no duplicates. Read-only; never writes. Exit 0 = clean, exit 1 =
|
||||
* named violations printed to stderr.
|
||||
*
|
||||
* The remote and the local cache directory are each overridable
|
||||
* (--remote / --cache-dir, or WALL_ENTRY_RELEASES_REMOTE /
|
||||
* WALL_ENTRY_RELEASES_CACHE_DIR) so tests can point at a throwaway local
|
||||
* bare repo and a throwaway cache directory — never the real remote or the
|
||||
* real developer cache.
|
||||
*
|
||||
* No dependencies beyond the system `git` binary — CHANGELOG parsing,
|
||||
* semver comparison, and JSON shape checking are all hand-rolled below.
|
||||
*/
|
||||
|
||||
import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs'
|
||||
import { execFileSync } from 'node:child_process'
|
||||
import { homedir } from 'node:os'
|
||||
import { dirname, join } from 'node:path'
|
||||
|
||||
const DEFAULT_REMOTE = 'git@source.soulcraft.com:soulcraftlabs/releases.git'
|
||||
|
||||
/** @returns {string} */
|
||||
function defaultCacheDir() {
|
||||
const base = process.env.XDG_CACHE_HOME || join(homedir(), '.cache')
|
||||
return join(base, 'soulcraft-releases')
|
||||
}
|
||||
|
||||
// Required on every entry; "thumb" is optional (may be absent, or present as
|
||||
// string | null) — matching the HQ contract's {..., thumb?}.
|
||||
const ENTRY_REQUIRED_KEYS = ['version', 'date', 'headline', 'items', 'url']
|
||||
const ENTRY_OPTIONAL_KEYS = ['thumb']
|
||||
const ENTRY_ALLOWED_KEYS = [...ENTRY_REQUIRED_KEYS, ...ENTRY_OPTIONAL_KEYS]
|
||||
const FILE_KEYS = ['product', 'entries']
|
||||
|
||||
// The public permalink pattern, by product. Every entry MUST carry an https
|
||||
// permalink: HQ's parser rejects a wall whose entries carry url: null (the
|
||||
// whole feed became unreadable on 2026-09-02). A product whose forge repo is
|
||||
// private links its PUBLIC package page on The Source instead of a release
|
||||
// page that would 404 for HQ's readers.
|
||||
const RELEASE_URL_PATTERNS = {
|
||||
'open-brainy': (version) => `https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v${version}`,
|
||||
'brainy': (version) => `https://source.soulcraft.com/soulcraft/-/packages/npm/@soulcraft%2Fbrainy/${version}`,
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse argv into a flag map. `--flag value` sets a string; `--flag` alone
|
||||
* (end of argv, or followed by another `--flag`) sets boolean true.
|
||||
* @param {string[]} argv
|
||||
* @returns {Record<string, string | true>}
|
||||
*/
|
||||
function parseArgs(argv) {
|
||||
/** @type {Record<string, string | true>} */
|
||||
const args = {}
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i]
|
||||
if (!a.startsWith('--')) continue
|
||||
const key = a.slice(2)
|
||||
const next = argv[i + 1]
|
||||
if (next === undefined || next.startsWith('--')) {
|
||||
args[key] = true
|
||||
} else {
|
||||
args[key] = next
|
||||
i++
|
||||
}
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
/**
|
||||
* Print a loud, named error and exit 1. Every refusal in this script goes
|
||||
* through here so the failure mode is always the same shape: "wall-entry: <what>".
|
||||
* @param {string} message
|
||||
* @returns {never}
|
||||
*/
|
||||
function fail(message) {
|
||||
console.error(`wall-entry: ${message}`)
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} version
|
||||
* @returns {{major: number, minor: number, patch: number, pre: string | null} | null}
|
||||
*/
|
||||
function parseSemver(version) {
|
||||
const m = /^(\d+)\.(\d+)\.(\d+)(?:-([0-9A-Za-z.-]+))?$/.exec(version)
|
||||
if (!m) return null
|
||||
return { major: Number(m[1]), minor: Number(m[2]), patch: Number(m[3]), pre: m[4] ?? null }
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} a
|
||||
* @param {string} b
|
||||
* @returns {number} positive if a > b, negative if a < b, 0 if equal.
|
||||
*/
|
||||
function compareSemver(a, b) {
|
||||
const pa = parseSemver(a)
|
||||
const pb = parseSemver(b)
|
||||
if (!pa || !pb) throw new Error(`cannot compare non-semver versions "${a}" vs "${b}"`)
|
||||
if (pa.major !== pb.major) return pa.major - pb.major
|
||||
if (pa.minor !== pb.minor) return pa.minor - pb.minor
|
||||
if (pa.patch !== pb.patch) return pa.patch - pb.patch
|
||||
if (pa.pre === pb.pre) return 0
|
||||
if (pa.pre === null) return 1 // a release outranks any prerelease of the same core version
|
||||
if (pb.pre === null) return -1
|
||||
return pa.pre < pb.pre ? -1 : pa.pre > pb.pre ? 1 : 0
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate a wall file's full shape: top-level keys ("product", "entries" —
|
||||
* no more, no less), per-entry keys and field types ("thumb" optional), and
|
||||
* strict-descending semver ordering with no duplicates. Collects every
|
||||
* violation instead of failing on the first, so a caller reports the whole
|
||||
* picture in one pass.
|
||||
* @param {unknown} data
|
||||
* @returns {string[]} Violation messages; empty means the file is clean.
|
||||
*/
|
||||
function validateShape(data) {
|
||||
/** @type {string[]} */
|
||||
const errors = []
|
||||
|
||||
if (typeof data !== 'object' || data === null || Array.isArray(data)) {
|
||||
return ['top level: expected a JSON object']
|
||||
}
|
||||
const obj = /** @type {Record<string, unknown>} */ (data)
|
||||
|
||||
const topKeys = Object.keys(obj)
|
||||
const missingTop = FILE_KEYS.filter((k) => !(k in obj))
|
||||
const extraTop = topKeys.filter((k) => !FILE_KEYS.includes(k))
|
||||
if (missingTop.length) errors.push(`top level: missing key(s) ${missingTop.join(', ')}`)
|
||||
if (extraTop.length) errors.push(`top level: unexpected key(s) ${extraTop.join(', ')}`)
|
||||
|
||||
if (typeof obj.product !== 'string' || obj.product.trim() === '') {
|
||||
errors.push('top level: "product" must be a non-empty string')
|
||||
}
|
||||
if (!Array.isArray(obj.entries)) {
|
||||
errors.push('top level: "entries" must be an array')
|
||||
return errors // nothing further to check without an array
|
||||
}
|
||||
|
||||
const entries = /** @type {unknown[]} */ (obj.entries)
|
||||
entries.forEach((rawEntry, i) => {
|
||||
const label = `entries[${i}]`
|
||||
if (typeof rawEntry !== 'object' || rawEntry === null || Array.isArray(rawEntry)) {
|
||||
errors.push(`${label}: expected an object`)
|
||||
return
|
||||
}
|
||||
const entry = /** @type {Record<string, unknown>} */ (rawEntry)
|
||||
const keys = Object.keys(entry)
|
||||
const missing = ENTRY_REQUIRED_KEYS.filter((k) => !(k in entry))
|
||||
const extra = keys.filter((k) => !ENTRY_ALLOWED_KEYS.includes(k))
|
||||
if (missing.length) errors.push(`${label}: missing key(s) ${missing.join(', ')}`)
|
||||
if (extra.length) errors.push(`${label}: unexpected key(s) ${extra.join(', ')}`)
|
||||
|
||||
if (typeof entry.version !== 'string' || !parseSemver(entry.version)) {
|
||||
errors.push(`${label}: "version" must be a semver string (got ${JSON.stringify(entry.version)})`)
|
||||
}
|
||||
if (typeof entry.date !== 'string' || !/^\d{4}-\d{2}-\d{2}$/.test(entry.date) || Number.isNaN(Date.parse(entry.date))) {
|
||||
errors.push(`${label}: "date" must be a YYYY-MM-DD string (got ${JSON.stringify(entry.date)})`)
|
||||
}
|
||||
if (typeof entry.headline !== 'string' || entry.headline.trim() === '') {
|
||||
errors.push(`${label}: "headline" must be a non-empty string`)
|
||||
}
|
||||
if (!Array.isArray(entry.items) || entry.items.length === 0 || entry.items.some((it) => typeof it !== 'string' || it.trim() === '')) {
|
||||
errors.push(`${label}: "items" must be a non-empty array of non-empty strings`)
|
||||
}
|
||||
if (typeof entry.url !== 'string' || !/^https:\/\/\S+$/.test(entry.url)) {
|
||||
errors.push(`${label}: "url" must be an https permalink — never null; HQ's parser rejects the whole feed`)
|
||||
}
|
||||
if ('thumb' in entry && !(entry.thumb === null || typeof entry.thumb === 'string')) {
|
||||
errors.push(`${label}: "thumb" must be a string or null when present`)
|
||||
}
|
||||
})
|
||||
|
||||
// Ordering: newest first, strictly descending, no duplicate versions —
|
||||
// checked only over entries whose version parsed (a bad version is
|
||||
// already reported above; comparing it too would just be noise).
|
||||
const versioned = entries
|
||||
.map((e, i) => ({ i, version: /** @type {any} */ (e)?.version }))
|
||||
.filter((e) => typeof e.version === 'string' && parseSemver(e.version))
|
||||
for (let i = 0; i < versioned.length - 1; i++) {
|
||||
const a = versioned[i]
|
||||
const b = versioned[i + 1]
|
||||
const cmp = compareSemver(a.version, b.version)
|
||||
if (cmp === 0) {
|
||||
errors.push(`entries[${a.i}] and entries[${b.i}]: duplicate version ${a.version}`)
|
||||
} else if (cmp < 0) {
|
||||
errors.push(`entries[${a.i}] (${a.version}) sits above entries[${b.i}] (${b.version}) — not newest-first`)
|
||||
}
|
||||
}
|
||||
|
||||
return errors
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract one version's entry body from a standard-version-style CHANGELOG.md
|
||||
* (headings `### [version](url) (date)`, followed by `- bullet (hash)` lines
|
||||
* until the next heading or EOF).
|
||||
* @param {string} changelog
|
||||
* @param {string} version
|
||||
* @returns {string[]} Bullet lines, trimmed of their leading "- " and
|
||||
* trailing " (hash)".
|
||||
*/
|
||||
function extractChangelogBullets(changelog, version) {
|
||||
const lines = changelog.split('\n')
|
||||
const headingRe = /^### \[([^\]]+)\]\(.*\)\s*\(\d{4}-\d{2}-\d{2}\)\s*$/
|
||||
let start = -1
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const m = headingRe.exec(lines[i])
|
||||
if (m && m[1] === version) {
|
||||
start = i + 1
|
||||
break
|
||||
}
|
||||
}
|
||||
if (start === -1) {
|
||||
fail(
|
||||
`version ${version} has no CHANGELOG entry yet — run this after the CHANGELOG step composes "### [${version}]", not before`,
|
||||
)
|
||||
}
|
||||
/** @type {string[]} */
|
||||
const bullets = []
|
||||
for (let i = start; i < lines.length; i++) {
|
||||
if (headingRe.test(lines[i])) break // next entry starts
|
||||
const bulletMatch = /^- (.+?)(?:\s\(([0-9a-f]{6,40})\))?$/.exec(lines[i].trim())
|
||||
if (lines[i].trim().startsWith('- ') && bulletMatch) {
|
||||
const text = bulletMatch[1].trim()
|
||||
if (text) bullets.push(text)
|
||||
}
|
||||
}
|
||||
if (bullets.length === 0) {
|
||||
fail(`version ${version}'s CHANGELOG entry has no bullets to derive a headline/items from`)
|
||||
}
|
||||
return bullets
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive a wall entry from a CHANGELOG.md.
|
||||
* @param {{product: string, version: string, date: string, changelogPath: string, url?: string, thumb?: string | null}} opts
|
||||
* @returns {{version: string, date: string, headline: string, items: string[], url: string, thumb: string | null}}
|
||||
*/
|
||||
function deriveEntry({ product, version, date, changelogPath, url, thumb }) {
|
||||
if (!parseSemver(version)) fail(`--version "${version}" is not a semver string`)
|
||||
if (!/^\d{4}-\d{2}-\d{2}$/.test(date) || Number.isNaN(Date.parse(date))) {
|
||||
fail(`--date "${date}" is not a YYYY-MM-DD date`)
|
||||
}
|
||||
if (!existsSync(changelogPath)) fail(`--from-changelog "${changelogPath}" does not exist`)
|
||||
|
||||
const changelog = readFileSync(changelogPath, 'utf8')
|
||||
const items = extractChangelogBullets(changelog, version)
|
||||
const headline = items[0]
|
||||
|
||||
const pattern = RELEASE_URL_PATTERNS[product]
|
||||
if (url === undefined && pattern === undefined) {
|
||||
throw new Error(`wall-entry: no permalink pattern for product "${product}" — add one to RELEASE_URL_PATTERNS or pass --url; entries never carry url: null`)
|
||||
}
|
||||
const resolvedUrl = url !== undefined ? url : pattern(version)
|
||||
const resolvedThumb = thumb !== undefined ? thumb : null
|
||||
|
||||
return { version, date, headline, items, url: resolvedUrl, thumb: resolvedThumb }
|
||||
}
|
||||
|
||||
/**
|
||||
* Load and shape-validate a wall file.
|
||||
* @param {string} filePath
|
||||
* @returns {Record<string, any>}
|
||||
*/
|
||||
function loadWallFile(filePath) {
|
||||
if (!existsSync(filePath)) fail(`"${filePath}" does not exist`)
|
||||
/** @type {unknown} */
|
||||
let data
|
||||
try {
|
||||
data = JSON.parse(readFileSync(filePath, 'utf8'))
|
||||
} catch (err) {
|
||||
fail(`"${filePath}" is not valid JSON: ${/** @type {Error} */ (err).message}`)
|
||||
}
|
||||
const errors = validateShape(data)
|
||||
if (errors.length) {
|
||||
fail(`"${filePath}" fails shape validation —\n ${errors.join('\n ')}`)
|
||||
}
|
||||
return /** @type {Record<string, any>} */ (data)
|
||||
}
|
||||
|
||||
/**
|
||||
* Run a git command, throwing an Error whose message is git's own stderr
|
||||
* (trimmed) on failure — every caller wraps this to name the cure.
|
||||
* @param {string[]} args
|
||||
* @param {string} cwd
|
||||
* @returns {string} stdout, trimmed.
|
||||
*/
|
||||
function git(args, cwd) {
|
||||
try {
|
||||
return execFileSync('git', args, { cwd, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }).trim()
|
||||
} catch (err) {
|
||||
const stderr = /** @type {any} */ (err).stderr
|
||||
const message = (typeof stderr === 'string' && stderr.trim()) || /** @type {Error} */ (err).message
|
||||
throw new Error(message)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the git identity for the wall commit from the repository the rail
|
||||
* is actually running in — the developer's own checkout (`process.cwd()`;
|
||||
* `release.sh` invokes this script from the repo root with no `cd`), via
|
||||
* git's normal config precedence (repo-local, then global, then system).
|
||||
* Never guessed and never left to git's own "who are you?" prompt: a host
|
||||
* with no configured identity anywhere (a bare CI box, say) must refuse
|
||||
* loudly rather than have git manufacture a placeholder identity or hang.
|
||||
* @returns {{name: string, email: string}}
|
||||
*/
|
||||
function resolveWallCommitIdentity() {
|
||||
const repo = process.cwd()
|
||||
let name = ''
|
||||
let email = ''
|
||||
try {
|
||||
name = git(['config', 'user.name'], repo)
|
||||
} catch {
|
||||
name = ''
|
||||
}
|
||||
try {
|
||||
email = git(['config', 'user.email'], repo)
|
||||
} catch {
|
||||
email = ''
|
||||
}
|
||||
if (!name || !email) {
|
||||
fail('no git identity for the wall commit — set user.name/user.email')
|
||||
}
|
||||
return { name, email }
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensure a clean, up-to-date local clone of the releases repo at
|
||||
* `cacheDir`, checked out on `main` — cloning fresh if `cacheDir` has no
|
||||
* `.git`, otherwise fetching and hard-resetting onto `origin/main` (so a
|
||||
* stray local commit or edit left by a previous failed run can never leak
|
||||
* into the next one).
|
||||
* @param {string} remote
|
||||
* @param {string} cacheDir
|
||||
*/
|
||||
function ensureReleasesClone(remote, cacheDir) {
|
||||
if (existsSync(join(cacheDir, '.git'))) {
|
||||
try {
|
||||
git(['remote', 'set-url', 'origin', remote], cacheDir)
|
||||
git(['fetch', '--prune', 'origin'], cacheDir)
|
||||
git(['checkout', 'main'], cacheDir)
|
||||
git(['reset', '--hard', 'origin/main'], cacheDir)
|
||||
git(['clean', '-fd'], cacheDir)
|
||||
} catch (err) {
|
||||
fail(
|
||||
`cannot refresh the cached releases checkout at "${cacheDir}" from "${remote}" — ${/** @type {Error} */ (err).message}\n` +
|
||||
` cure: delete "${cacheDir}" and re-run so it re-clones from scratch, or confirm SSH access with "ssh -T git@source.soulcraft.com"`,
|
||||
)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
mkdirSync(dirname(cacheDir), { recursive: true })
|
||||
try {
|
||||
git(['clone', remote, cacheDir], dirname(cacheDir))
|
||||
} catch (err) {
|
||||
fail(
|
||||
`cannot clone "${remote}" — ${/** @type {Error} */ (err).message}\n` +
|
||||
` cure: confirm SSH access with "ssh -T git@source.soulcraft.com" and that the soulcraftlabs/releases repo exists yet`,
|
||||
)
|
||||
}
|
||||
try {
|
||||
git(['checkout', 'main'], cacheDir)
|
||||
} catch (err) {
|
||||
fail(
|
||||
`cloned "${remote}" into "${cacheDir}" but could not check out "main" — ${/** @type {Error} */ (err).message}\n` +
|
||||
` cure: confirm the releases repo's default branch is named "main"`,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Prepend `entry` to the wall at `<cacheDir>/<product>.json`, replacing any
|
||||
* existing entry for the same version (idempotent re-runs), validating
|
||||
* before and after, committing, and pushing — or refusing loudly, naming
|
||||
* the cure, at whichever step fails.
|
||||
* @param {{version: string, date: string, headline: string, items: string[], url: string, thumb: string | null}} entry
|
||||
* @param {string} product
|
||||
* @param {string} remote
|
||||
* @param {string} cacheDir
|
||||
*/
|
||||
function publishEntry(entry, product, remote, cacheDir) {
|
||||
ensureReleasesClone(remote, cacheDir)
|
||||
|
||||
const filePath = join(cacheDir, `${product}.json`)
|
||||
if (!existsSync(filePath)) {
|
||||
fail(
|
||||
`"${filePath}" does not exist in the releases repo — cure: seed "${product}.json" at the repo root first (it must exist before any release rail can prepend to it)`,
|
||||
)
|
||||
}
|
||||
const wall = loadWallFile(filePath)
|
||||
|
||||
if (wall.product !== product) {
|
||||
fail(`"${filePath}" has product "${wall.product}", but --product "${product}" was given — refusing a cross-product write`)
|
||||
}
|
||||
|
||||
const replacing = wall.entries.some((e) => e.version === entry.version)
|
||||
wall.entries = [entry, ...wall.entries.filter((e) => e.version !== entry.version)]
|
||||
|
||||
const postErrors = validateShape(wall)
|
||||
if (postErrors.length) {
|
||||
fail(`the entry for ${entry.version} would leave "${filePath}" invalid —\n ${postErrors.join('\n ')}`)
|
||||
}
|
||||
|
||||
writeFileSync(filePath, JSON.stringify(wall, null, 2) + '\n', 'utf8')
|
||||
|
||||
const status = git(['status', '--porcelain', '--', `${product}.json`], cacheDir)
|
||||
if (status === '') {
|
||||
console.log(`wall-entry: "${product}.json" already carries an identical entry for ${entry.version} — nothing to commit or push`)
|
||||
return
|
||||
}
|
||||
|
||||
const identity = resolveWallCommitIdentity()
|
||||
|
||||
try {
|
||||
git(['add', `${product}.json`], cacheDir)
|
||||
git(
|
||||
['-c', `user.name=${identity.name}`, '-c', `user.email=${identity.email}`, 'commit', '-m', `chore(wall): ${product} ${entry.version}`],
|
||||
cacheDir,
|
||||
)
|
||||
} catch (err) {
|
||||
fail(`cannot commit the wall entry in "${cacheDir}" — ${/** @type {Error} */ (err).message}\n cure: inspect "${cacheDir}" by hand and re-run once its git state is clean`)
|
||||
}
|
||||
|
||||
try {
|
||||
git(['push', 'origin', 'main'], cacheDir)
|
||||
} catch (err) {
|
||||
fail(
|
||||
`push to "${remote}" failed (likely a non-fast-forward — another release landed on main first) — ${/** @type {Error} */ (err).message}\n` +
|
||||
` cure: re-run this release step; it re-fetches and resets onto the latest origin/main before retrying`,
|
||||
)
|
||||
}
|
||||
|
||||
const sha = git(['rev-parse', 'HEAD'], cacheDir)
|
||||
console.log(
|
||||
`wall-entry: ${replacing ? 'replaced' : 'wrote'} v${entry.version} in "${product}.json" (${wall.entries.length} entries, newest first) — pushed ${sha} to ${remote} main`,
|
||||
)
|
||||
}
|
||||
|
||||
function main() {
|
||||
const args = parseArgs(process.argv.slice(2))
|
||||
|
||||
if (args.check) {
|
||||
const filePath = /** @type {string | undefined} */ (args.file)
|
||||
if (!filePath) fail('--check needs --file <path>')
|
||||
const wall = loadWallFile(/** @type {string} */ (filePath))
|
||||
console.log(`wall-entry --check: "${filePath}" OK — product "${wall.product}", ${wall.entries.length} entries, newest-first, no duplicates`)
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
// Generate mode (default, also covers --dry-run): --product, --version,
|
||||
// --date, --from-changelog required.
|
||||
const product = /** @type {string | undefined} */ (args.product)
|
||||
const version = /** @type {string | undefined} */ (args.version)
|
||||
const date = /** @type {string | undefined} */ (args.date)
|
||||
const fromChangelog = /** @type {string | undefined} */ (args['from-changelog'])
|
||||
|
||||
const missing = []
|
||||
if (!product) missing.push('--product')
|
||||
if (!version) missing.push('--version')
|
||||
if (!date) missing.push('--date')
|
||||
if (!fromChangelog) missing.push('--from-changelog')
|
||||
if (missing.length) {
|
||||
fail(
|
||||
`missing required flag(s): ${missing.join(', ')}\n` +
|
||||
'Usage:\n' +
|
||||
' wall-entry.mjs --product <p> --version <v> --date <YYYY-MM-DD> --from-changelog <CHANGELOG.md> [--dry-run]\n' +
|
||||
' wall-entry.mjs --check --file <path/to/product.json>',
|
||||
)
|
||||
}
|
||||
|
||||
const urlArg = args.url === true ? undefined : /** @type {string | undefined} */ (args.url)
|
||||
const thumbArg = args.thumb === true ? undefined : /** @type {string | undefined} */ (args.thumb)
|
||||
|
||||
const entry = deriveEntry({
|
||||
product: /** @type {string} */ (product),
|
||||
version: /** @type {string} */ (version),
|
||||
date: /** @type {string} */ (date),
|
||||
changelogPath: /** @type {string} */ (fromChangelog),
|
||||
url: urlArg,
|
||||
thumb: thumbArg,
|
||||
})
|
||||
|
||||
const remote = /** @type {string} */ (args.remote ?? process.env.WALL_ENTRY_RELEASES_REMOTE ?? DEFAULT_REMOTE)
|
||||
const cacheDir = /** @type {string} */ (args['cache-dir'] ?? process.env.WALL_ENTRY_RELEASES_CACHE_DIR ?? defaultCacheDir())
|
||||
|
||||
if (args['dry-run']) {
|
||||
console.log(`wall-entry --dry-run: would write to "${join(cacheDir, `${product}.json`)}" in ${remote} (main), pushed as "chore(wall): ${product} ${version}"`)
|
||||
console.log(JSON.stringify(entry, null, 2))
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
publishEntry(entry, /** @type {string} */ (product), remote, cacheDir)
|
||||
}
|
||||
|
||||
main()
|
||||
481
src/brainy.ts
481
src/brainy.ts
|
|
@ -531,36 +531,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
private static sigintListener?: () => void
|
||||
private static beforeExitListener?: () => void
|
||||
|
||||
/** True while the `beforeExit` pass is running its flushes. Node re-emits
|
||||
* 'beforeExit' after every loop drain and that pass schedules async work, so
|
||||
* a second emit can arrive on top of the first; it returns instead of
|
||||
* stacking a parallel pass. NOT a one-shot: every genuine drain still gets a
|
||||
* flush. See {@link registerShutdownHooks}. */
|
||||
private static beforeExitFlushInFlight = false
|
||||
|
||||
/** Whether the drained-event-loop notice has been printed for this
|
||||
* registration cycle. Printed ONCE — `console.log` to a pipe is itself
|
||||
* event-loop work, so narrating on every emit would keep the loop turning
|
||||
* and narrate forever. Reset by {@link deregisterShutdownHooksIfIdle}. */
|
||||
private static beforeExitNarrated = false
|
||||
|
||||
/** True for the entire duration of ONE `closeOnShutdown()` run (the
|
||||
* signal-path handler in {@link registerShutdownHooks}) — from before it
|
||||
* starts closing instances until after it has decided whether to exit.
|
||||
* THE RACE THIS CLOSES: closing the LAST live instance calls
|
||||
* `close()` → `deregisterShutdownHooksIfIdle()` synchronously, which
|
||||
* removes `Brainy.sigtermListener` from `process` — while `closeOnShutdown`
|
||||
* (that very listener's OWN still-running invocation) hasn't yet reached
|
||||
* `exitIfSoleShutdownOwner()`'s `process.exit(0)`. In that window Node has
|
||||
* NO registered SIGTERM listener, so a second/concurrent delivery of the
|
||||
* same signal (a raced re-send, common on a loaded host) falls through to
|
||||
* Node's default disposition and kills the process outright — the
|
||||
* clean-shutdown work already finished, but the process never reports the
|
||||
* 0 it earned. `deregisterShutdownHooksIfIdle()` checks this flag and
|
||||
* defers; `closeOnShutdown()`'s `finally` re-runs the deregistration check
|
||||
* once it is done, so the listener never actually leaks past its use. */
|
||||
private static shutdownSignalHandlerActive = false
|
||||
|
||||
/** Poll cadence (ms) for the migration LOCK when a provider exposes no
|
||||
* event-driven `whenMigrationComplete()` signal. See {@link awaitMigrationLock}. */
|
||||
private static readonly MIGRATION_POLL_INTERVAL_MS = 250
|
||||
|
|
@ -2160,11 +2130,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* Critical for Cloud Run, Fargate, Lambda, and other containerized deployments.
|
||||
*
|
||||
* Handles:
|
||||
* - SIGTERM: Graceful termination (Cloud Run, Fargate, Lambda) — CLOSES.
|
||||
* - SIGINT: Ctrl+C (development/local testing) — CLOSES.
|
||||
* - beforeExit: the event loop drained — FLUSHES, and closes NOTHING. A
|
||||
* drained loop is not a shutdown; see {@link flushOnDrainedEventLoop}'s
|
||||
* contract below.
|
||||
* - SIGTERM: Graceful termination (Cloud Run, Fargate, Lambda)
|
||||
* - SIGINT: Ctrl+C (development/local testing)
|
||||
* - beforeExit: Node.js cleanup hook (fallback)
|
||||
*
|
||||
* NOTE: Registers globally (once for all instances) to avoid MaxListenersExceededWarning
|
||||
*/
|
||||
|
|
@ -2213,174 +2181,51 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
*/
|
||||
const closeOnShutdown = async () => {
|
||||
console.log('Shutdown signal received - flushing pending data...')
|
||||
// HOLD THE LISTENER FOR THE WHOLE RUN. Closing the LAST live instance
|
||||
// below calls close() → deregisterShutdownHooksIfIdle(), which removes
|
||||
// Brainy's own SIGTERM/SIGINT listeners from `process` — synchronously,
|
||||
// before THIS invocation has reached exitIfSoleShutdownOwner()'s
|
||||
// process.exit(0). Left alone, that opens a window with no registered
|
||||
// listener for the signal at all, so a second/concurrent delivery of
|
||||
// the same signal (a raced re-send — not rare on a loaded host) falls
|
||||
// through to Node's default disposition and kills the process outright
|
||||
// AFTER the clean-shutdown work already finished, reporting a signal
|
||||
// kill instead of the 0 the shutdown earned. Setting this flag makes
|
||||
// deregisterShutdownHooksIfIdle() defer; the `finally` below re-checks
|
||||
// it once this run is fully done — closeOnShutdown, not a nested
|
||||
// close(), owns exactly when the listener actually comes off.
|
||||
Brainy.shutdownSignalHandlerActive = true
|
||||
try {
|
||||
// DEFER ONE MACROTASK. A host application registers its own listener
|
||||
// on the same signal, and Node runs listeners in registration order —
|
||||
// ours is usually first, because the brain was opened before the
|
||||
// host wired its shutdown. Yielding once lets every other listener
|
||||
// for this signal run its synchronous prologue, so a host that calls
|
||||
// close() gets to be the owner. It is only a courtesy, never the
|
||||
// safety: close()'s own single-flight gate is what makes a lost race
|
||||
// harmless.
|
||||
await new Promise<void>((resolve) => setImmediate(resolve))
|
||||
// DEFER ONE MACROTASK. A host application registers its own listener on
|
||||
// the same signal, and Node runs listeners in registration order — ours
|
||||
// is usually first, because the brain was opened before the host wired
|
||||
// its shutdown. Yielding once lets every other listener for this signal
|
||||
// run its synchronous prologue, so a host that calls close() gets to be
|
||||
// the owner. It is only a courtesy, never the safety: close()'s own
|
||||
// single-flight gate is what makes a lost race harmless.
|
||||
await new Promise<void>((resolve) => setImmediate(resolve))
|
||||
|
||||
let closedCount = 0
|
||||
let deferredCount = 0
|
||||
let failedCount = 0
|
||||
// Snapshot: close() splices Brainy.instances while we iterate.
|
||||
for (const instance of [...Brainy.instances]) {
|
||||
if (!instance.initialized) continue
|
||||
// SOMEONE ELSE OWNS THIS ONE. Not a flush, not a lock release, not a
|
||||
// component close — nothing. Touching a brain whose close is running
|
||||
// is the whole defect this handler was rewritten for.
|
||||
if (instance.closed || instance._closeInFlight !== null) {
|
||||
deferredCount++
|
||||
continue
|
||||
}
|
||||
try {
|
||||
// Law 1: this try/catch is the isolation — the loop continues.
|
||||
await instance.close()
|
||||
closedCount++
|
||||
} catch (error) {
|
||||
failedCount++
|
||||
console.error('Failed to close one Brainy instance on shutdown:', error)
|
||||
}
|
||||
let closedCount = 0
|
||||
let deferredCount = 0
|
||||
let failedCount = 0
|
||||
// Snapshot: close() splices Brainy.instances while we iterate.
|
||||
for (const instance of [...Brainy.instances]) {
|
||||
if (!instance.initialized) continue
|
||||
// SOMEONE ELSE OWNS THIS ONE. Not a flush, not a lock release, not a
|
||||
// component close — nothing. Touching a brain whose close is running
|
||||
// is the whole defect this handler was rewritten for.
|
||||
if (instance.closed || instance._closeInFlight !== null) {
|
||||
deferredCount++
|
||||
continue
|
||||
}
|
||||
if (closedCount > 0) {
|
||||
console.log(`Flushed successfully (${closedCount} instance${closedCount > 1 ? 's' : ''})`)
|
||||
try {
|
||||
// Law 1: this try/catch is the isolation — the loop continues.
|
||||
await instance.close()
|
||||
closedCount++
|
||||
} catch (error) {
|
||||
failedCount++
|
||||
console.error('Failed to close one Brainy instance on shutdown:', error)
|
||||
}
|
||||
if (deferredCount > 0) {
|
||||
console.log(
|
||||
`${deferredCount} Brainy instance${deferredCount > 1 ? 's are' : ' is'} already ` +
|
||||
`closing — left to the caller that owns that close.`
|
||||
)
|
||||
}
|
||||
if (failedCount > 0) {
|
||||
console.error(
|
||||
`${failedCount} Brainy instance${failedCount > 1 ? 's' : ''} did not complete shutdown — ` +
|
||||
`their writer locks were released, but their next open will run crash recovery.`
|
||||
)
|
||||
}
|
||||
} finally {
|
||||
// Release the hold and run the deferred check ourselves — the last
|
||||
// close() above may have found the flag set and skipped its own
|
||||
// deregistration, so nobody else will do this if we don't.
|
||||
Brainy.shutdownSignalHandlerActive = false
|
||||
Brainy.deregisterShutdownHooksIfIdle()
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* THE DRAINED-EVENT-LOOP PATH. A DRAINED LOOP IS NOT A SHUTDOWN.
|
||||
*
|
||||
* Node emits `'beforeExit'` whenever the event loop has no REF'd work
|
||||
* left — NOT when the process is ending, and with no signal involved. A
|
||||
* perfectly healthy script reaches that state routinely: this engine
|
||||
* unref's its idle and cadence timers ("an idle brain costs nothing"), so
|
||||
* a script awaiting anything those timers drive is, for that instant,
|
||||
* a process with no ref'd work and an open brain.
|
||||
*
|
||||
* MEASURED on the 11.1 rehearsal lane against a copy of a real store: the
|
||||
* `beforeExit` listener was wired to the SIGNAL path, so after the heal
|
||||
* phase the log printed `Shutdown signal received - flushing pending
|
||||
* data...` and `Flushed successfully (1 instance)` with NO signal ever
|
||||
* sent, and the script's very next `add()` threw `Brainy instance is not
|
||||
* initialized: it was closed via close(). Create a new instance.` The
|
||||
* engine had closed a live brain out from under a running script.
|
||||
*
|
||||
* SO, THE LAW: this path NEVER closes, deregisters, tears down or
|
||||
* force-exits anything, and never releases a writer lock. It runs
|
||||
* `flush()` — the engine's own non-closing durability door — on each live
|
||||
* brain, and leaves every one of them open and usable.
|
||||
*
|
||||
* WHY flush() AND NOT NOTHING. Each claim checked against the code it
|
||||
* names:
|
||||
* 1. IT CANNOT CLOSE ANYTHING. `flush()` → `_flushSteps()` persists
|
||||
* DERIVED state only: the count ledger, the metadata/graph/vector
|
||||
* projections, the generation counter, aggregation state, the
|
||||
* entity-tree stamp. It closes no component, deactivates no plugin,
|
||||
* touches neither `initialized` nor `closed`, and never calls
|
||||
* `releaseWriterLock()` — the clean-shutdown marker is written by
|
||||
* `generationStore.close()` alone, reached only from `close()`.
|
||||
* 2. IT CANNOT RACE A LATER WRITE INTO CORRUPTION. A background flush
|
||||
* concurrent with live writes is the engine's ORDINARY steady state:
|
||||
* `noteWriteForPersistence()` kicks exactly this call off an unref'd
|
||||
* timer on every busy brain. `flush()` is single-flight with one queued
|
||||
* follow-up, and a write landing mid-flush re-sets the dirty witness,
|
||||
* so its work is never lost — it belongs to the next flush.
|
||||
* 3. IT CANNOT SPIN. `flush()` on a clean brain returns without touching a
|
||||
* provider or scheduling I/O, so the second emit does no event-loop
|
||||
* work and the process exits. That is also why the listener is NOT
|
||||
* self-deregistered any more: a one-shot listener spent on a spurious
|
||||
* mid-script drain leaves the genuine end-of-script drain with nothing.
|
||||
* 4. A FAILED FLUSH IS SURVIVABLE AND LOUD. The write path is durable at
|
||||
* ack via the fact log; derived state is rebuildable. A throw is
|
||||
* reported per instance and the loop continues — exactly how
|
||||
* `kickBackgroundFlush()` already treats the same failure.
|
||||
*
|
||||
* The one thing lost against a closing handler is the clean-shutdown
|
||||
* marker for a script that opens a brain and never closes it: its next
|
||||
* open folds the log. That is the correct trade — a missing marker costs
|
||||
* a recovery fold, closing a live brain costs the caller its brain — and
|
||||
* the narration below names the cure.
|
||||
*/
|
||||
const flushOnDrainedEventLoop = async () => {
|
||||
// A second emit can land on top of the first (this pass schedules async
|
||||
// work, the loop turns, the loop drains again). One pass at a time.
|
||||
if (Brainy.beforeExitFlushInFlight) return
|
||||
|
||||
// Step aside for anyone whose close is running or done — the same
|
||||
// ownership rule the signal path follows.
|
||||
const live = [...Brainy.instances].filter(
|
||||
(instance) => instance.initialized && !instance.closed && instance._closeInFlight === null
|
||||
)
|
||||
if (live.length === 0) return
|
||||
|
||||
// ONCE per registration cycle: a `console.log` to a pipe is itself
|
||||
// event-loop work, so narrating on every emit would keep the loop
|
||||
// turning and narrate forever.
|
||||
if (!Brainy.beforeExitNarrated) {
|
||||
Brainy.beforeExitNarrated = true
|
||||
if (closedCount > 0) {
|
||||
console.log(`Flushed successfully (${closedCount} instance${closedCount > 1 ? 's' : ''})`)
|
||||
}
|
||||
if (deferredCount > 0) {
|
||||
console.log(
|
||||
`[Brainy] event loop drained with ${live.length} brain${live.length > 1 ? 's' : ''} ` +
|
||||
`open — persisting derived state; NOTHING was closed. A drained loop is not a ` +
|
||||
`shutdown: call close() (or send SIGTERM) when you mean one.`
|
||||
`${deferredCount} Brainy instance${deferredCount > 1 ? 's are' : ' is'} already ` +
|
||||
`closing — left to the caller that owns that close.`
|
||||
)
|
||||
}
|
||||
|
||||
Brainy.beforeExitFlushInFlight = true
|
||||
try {
|
||||
for (const instance of live) {
|
||||
try {
|
||||
await instance.flush()
|
||||
} catch (error) {
|
||||
// Per-instance isolation, and never fatal: canonical data is
|
||||
// durable at ack, so a failed derived-state flush costs the next
|
||||
// open a rebuild — it must not cost this one its brain.
|
||||
console.error(
|
||||
'[Brainy] flush on a drained event loop failed for one open brain ' +
|
||||
'(the brain stays open and usable; derived-state persistence retries at the ' +
|
||||
'next flush, and canonical data is unaffected):',
|
||||
error
|
||||
)
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
Brainy.beforeExitFlushInFlight = false
|
||||
if (failedCount > 0) {
|
||||
console.error(
|
||||
`${failedCount} Brainy instance${failedCount > 1 ? 's' : ''} did not complete shutdown — ` +
|
||||
`their writer locks were released, but their next open will run crash recovery.`
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -2409,14 +2254,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* last brain deregisters Brainy's own listeners — so a host application's
|
||||
* single remaining listener would look like `<= 1` and get force-exited
|
||||
* out of its own graceful shutdown, precisely the failure above.
|
||||
*
|
||||
* SIGNALS ONLY — NEVER `beforeExit`. The reasoning above is entirely about
|
||||
* a signal Brainy has suppressed Node's default terminate behaviour for.
|
||||
* `beforeExit` suppresses nothing: Node exits by itself once the loop is
|
||||
* genuinely done, and the script that is still running when it fires is
|
||||
* not shutting down at all. Calling this from that path would end a live
|
||||
* script at exit code 0 mid-work. It is called from the two signal
|
||||
* listeners below and from nowhere else.
|
||||
*/
|
||||
const exitIfSoleShutdownOwner = (ownersWhenSignalled: number): void => {
|
||||
if (ownersWhenSignalled <= 1) {
|
||||
|
|
@ -2433,7 +2270,18 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
await closeOnShutdown()
|
||||
exitIfSoleShutdownOwner(owners)
|
||||
}
|
||||
Brainy.beforeExitListener = flushOnDrainedEventLoop
|
||||
Brainy.beforeExitListener = async () => {
|
||||
// Self-deregister FIRST: Node re-emits 'beforeExit' after every event-
|
||||
// loop drain, and this flush schedules new async work — with the
|
||||
// listener still attached, a script that never calls close() would spin
|
||||
// flush → drain → flush forever and never exit. One flush, then the
|
||||
// next drain finds no listener and the process exits.
|
||||
if (Brainy.beforeExitListener) {
|
||||
process.off('beforeExit', Brainy.beforeExitListener)
|
||||
Brainy.beforeExitListener = undefined
|
||||
}
|
||||
await closeOnShutdown()
|
||||
}
|
||||
process.on('SIGTERM', Brainy.sigtermListener)
|
||||
process.on('SIGINT', Brainy.sigintListener)
|
||||
process.on('beforeExit', Brainy.beforeExitListener)
|
||||
|
|
@ -2444,17 +2292,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* script that closed every brain exits on its own — a library must never
|
||||
* keep its host process alive. Re-initializing later re-registers them
|
||||
* (the `shutdownHooksRegisteredGlobally` flag resets here).
|
||||
*
|
||||
* Deferred (not skipped — {@link closeOnShutdown}'s `finally` always
|
||||
* re-checks) while a signal-path shutdown is actively running: that
|
||||
* handler's OWN still-in-flight invocation is `Brainy.sigtermListener`, and
|
||||
* removing it out from under itself — which closing the LAST instance here
|
||||
* would otherwise do, synchronously, mid-run — would leave `process` with
|
||||
* no listener for the signal for the remainder of that run. See
|
||||
* {@link shutdownSignalHandlerActive}'s doc for the exact race this closes.
|
||||
*/
|
||||
private static deregisterShutdownHooksIfIdle(): void {
|
||||
if (Brainy.instances.length > 0 || !Brainy.shutdownHooksRegisteredGlobally || Brainy.shutdownSignalHandlerActive) {
|
||||
if (Brainy.instances.length > 0 || !Brainy.shutdownHooksRegisteredGlobally) {
|
||||
return
|
||||
}
|
||||
if (Brainy.sigtermListener) process.off('SIGTERM', Brainy.sigtermListener)
|
||||
|
|
@ -2463,11 +2303,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
Brainy.sigtermListener = undefined
|
||||
Brainy.sigintListener = undefined
|
||||
Brainy.beforeExitListener = undefined
|
||||
// A later re-init is a fresh cycle: it may narrate its own drained-loop
|
||||
// notice, and no pass of the previous cycle can still be running (the last
|
||||
// close() drained the flush chain).
|
||||
Brainy.beforeExitNarrated = false
|
||||
Brainy.beforeExitFlushInFlight = false
|
||||
Brainy.shutdownHooksRegisteredGlobally = false
|
||||
}
|
||||
|
||||
|
|
@ -4241,16 +4076,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
|
||||
// Route to metadata-only or full entity based on options
|
||||
// A PROJECTED get goes through the same seam every list page uses, so a
|
||||
// detail read of two scalars costs an index read rather than a record read.
|
||||
// It is checked before `includeVectors` because the two are incompatible by
|
||||
// construction: a projection returns the named fields, and a vector is not
|
||||
// one of them unless it was named.
|
||||
if (options?.fields !== undefined && options.fields.length > 0) {
|
||||
const page = await this.#hydratePage([id], options.fields)
|
||||
return page.get(id) ?? null
|
||||
}
|
||||
|
||||
const includeVectors = options?.includeVectors ?? false // Default: metadata-only (fast)
|
||||
|
||||
if (includeVectors) {
|
||||
|
|
@ -4297,170 +4122,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* const children = childIds.map(id => childrenMap.get(id)).filter(Boolean)
|
||||
* ```
|
||||
*/
|
||||
/**
|
||||
* **The projection seam** — hydrate a page of ids under an optional `fields`
|
||||
* projection, opening the canonical record only when the index cannot serve
|
||||
* what was asked for.
|
||||
*
|
||||
* Without a projection this is exactly `batchGet`, byte for byte: the whole
|
||||
* point is that `fields` absent changes nothing.
|
||||
*
|
||||
* With one, the order is: ask the index for the named scalars in a single
|
||||
* batched door; see which requested fields it actually served; and read
|
||||
* records ONLY if something is still missing — and only to fill those fields.
|
||||
* A page whose every requested field is index-served performs zero canonical
|
||||
* reads, which is the whole reason the door exists.
|
||||
*
|
||||
* `guardFields` are fetched ALONGSIDE the projection and trimmed off before
|
||||
* the caller sees them. find()'s index-integrity guard re-validates every row
|
||||
* against its own predicate, and it reads the entity to do so — so a row
|
||||
* projected down to `title` would fail a `where: { kind }` it genuinely
|
||||
* matches, and the whole page would vanish. The fields a filter names are
|
||||
* fields the index can serve by definition, so carrying them costs nothing
|
||||
* and keeps the guard honest.
|
||||
*
|
||||
* A field nothing can supply is simply absent from the row. That is the
|
||||
* permissive law: a projection asks "these, if you have them", and an
|
||||
* optional field must not turn a list into an exception. It deliberately does
|
||||
* NOT route through the strict address resolver, which throws
|
||||
* `UnresolvableFieldError` for an unknown key — that strictness is right for
|
||||
* `orderBy`, where a typo silently changes the order, and wrong here, where
|
||||
* the honest answer is "this row does not have that".
|
||||
*
|
||||
* @param ids - Canonical ids for the page.
|
||||
* @param fields - The projection, or undefined for the full record.
|
||||
* @returns `id → entity`, projected when `fields` was given.
|
||||
*/
|
||||
/**
|
||||
* The index keys find()'s integrity guard reads when it re-validates a row.
|
||||
*
|
||||
* The guard calls `entityMatchesFind(entity, params)`, so a projected entity
|
||||
* must still carry whatever the params constrain — otherwise a row that
|
||||
* genuinely matches is dropped for lacking the evidence. These are fetched
|
||||
* with the projection and trimmed off before the caller sees them.
|
||||
*
|
||||
* @param params - The find params.
|
||||
* @returns Index keys to carry through hydration.
|
||||
*/
|
||||
#guardFieldsFor(params: FindParams<T>): string[] {
|
||||
const keys: string[] = []
|
||||
if (params.where && typeof params.where === 'object') {
|
||||
// Top-level where keys only: nested `anyOf`/`allOf` branches are carried
|
||||
// by their own keys when the guard walks them, and a filter whose
|
||||
// evidence is missing keeps the row (the guard's own catch) rather than
|
||||
// dropping it.
|
||||
for (const key of Object.keys(params.where as Record<string, unknown>)) {
|
||||
if (key === 'anyOf' || key === 'allOf' || key === 'not') continue
|
||||
keys.push(key)
|
||||
}
|
||||
}
|
||||
if (params.type !== undefined) keys.push('system.type')
|
||||
if (params.subtype !== undefined) keys.push('system.subtype')
|
||||
if (params.service !== undefined) keys.push('system.service')
|
||||
if (params.excludeVFS === true) keys.push('vfsType', 'isVFSEntity')
|
||||
return keys
|
||||
}
|
||||
|
||||
async #hydratePage(
|
||||
ids: string[],
|
||||
fields?: readonly string[],
|
||||
guardFields: readonly string[] = []
|
||||
): Promise<Map<string, Entity<T>>> {
|
||||
if (fields === undefined || fields.length === 0) return this.batchGet(ids)
|
||||
|
||||
const wanted = [...new Set([...fields, ...guardFields])]
|
||||
const provider = this.metadataIndex as unknown as MetadataIndexProvider
|
||||
let served = new Map<string, Record<string, unknown>>()
|
||||
if (typeof provider.getScalarsForIds === 'function') {
|
||||
served = await provider.getScalarsForIds(ids, wanted)
|
||||
}
|
||||
|
||||
// Which ids still owe a field? Only those cost a record read, and a page
|
||||
// that owes nothing costs none at all.
|
||||
const owing: string[] = []
|
||||
for (const id of ids) {
|
||||
const row = served.get(id)
|
||||
if (row === undefined || wanted.some((f) => !(f in row))) owing.push(id)
|
||||
}
|
||||
|
||||
// The records are read for the OWED fields only; everything the index
|
||||
// already served is used as-is, so a body field pulls its own record and
|
||||
// no more than that.
|
||||
const records = owing.length > 0 ? await this.batchGet(owing) : new Map<string, Entity<T>>()
|
||||
|
||||
const out = new Map<string, Entity<T>>()
|
||||
for (const id of ids) {
|
||||
const fromIndex = served.get(id)
|
||||
const record = records.get(id)
|
||||
// An id neither the index nor storage knows is not a row.
|
||||
if (fromIndex === undefined && record === undefined) continue
|
||||
out.set(id, this.#projectEntity(id, wanted, fromIndex, record))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
/**
|
||||
* Build one projected entity: `id`, plus exactly the requested fields that
|
||||
* something could supply.
|
||||
*
|
||||
* Values come from the index first and the record second, and they must agree
|
||||
* — the index only reports what it can serve exactly, so a field it served is
|
||||
* the record's value. A field neither has is omitted rather than set to
|
||||
* `undefined`: absent and present-and-undefined are different answers, and a
|
||||
* caller checking `'slug' in row.metadata` deserves the true one.
|
||||
*
|
||||
* @param id - The entity id, always present on the result.
|
||||
* @param fields - The requested index keys.
|
||||
* @param fromIndex - What the index served for this id, if anything.
|
||||
* @param record - The canonical entity, if one had to be read.
|
||||
* @returns The projected entity.
|
||||
*/
|
||||
#projectEntity(
|
||||
id: string,
|
||||
fields: readonly string[],
|
||||
fromIndex: Record<string, unknown> | undefined,
|
||||
record: Entity<T> | undefined
|
||||
): Entity<T> {
|
||||
const projected: Record<string, unknown> = { id }
|
||||
const metadata: Record<string, unknown> = {}
|
||||
let sawMetadata = false
|
||||
|
||||
for (const field of fields) {
|
||||
let value: unknown
|
||||
let found = false
|
||||
if (fromIndex !== undefined && field in fromIndex) {
|
||||
value = fromIndex[field]
|
||||
found = true
|
||||
} else if (record !== undefined) {
|
||||
if (field.startsWith('system.')) {
|
||||
const inner = field.slice('system.'.length)
|
||||
const bag = record as unknown as Record<string, unknown>
|
||||
if (inner in bag && bag[inner] !== undefined) {
|
||||
value = bag[inner]
|
||||
found = true
|
||||
}
|
||||
} else {
|
||||
const bag = (record.metadata ?? {}) as Record<string, unknown>
|
||||
if (field in bag) {
|
||||
value = bag[field]
|
||||
found = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!found) continue
|
||||
|
||||
if (field.startsWith('system.')) {
|
||||
projected[field.slice('system.'.length)] = value
|
||||
} else {
|
||||
metadata[field] = value
|
||||
sawMetadata = true
|
||||
}
|
||||
}
|
||||
|
||||
if (sawMetadata) projected.metadata = metadata
|
||||
return projected as unknown as Entity<T>
|
||||
}
|
||||
|
||||
async batchGet(ids: string[], options?: GetOptions): Promise<Map<string, Entity<T>>> {
|
||||
// Canonical read (see get): resolves by id from storage, no derived index.
|
||||
await this.ensureInitialized({ needs: [] })
|
||||
|
|
@ -8259,7 +7920,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
|
||||
// Batch-load entities for 10x faster cloud storage performance
|
||||
// GCS: 10 entities = 1×50ms vs 10×50ms = 500ms (10x faster)
|
||||
const entitiesMap = await this.#hydratePage(pageIds, params.fields, this.#guardFieldsFor(params))
|
||||
const entitiesMap = await this.batchGet(pageIds)
|
||||
for (const id of pageIds) {
|
||||
const entity = entitiesMap.get(id)
|
||||
if (entity) {
|
||||
|
|
@ -8296,7 +7957,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
if (hiddenIds.size > 0) allUuids = allUuids.filter((id) => !hiddenIds.has(id))
|
||||
const pageIds = allUuids.slice(offset, offset + limit)
|
||||
|
||||
const entitiesMap = await this.#hydratePage(pageIds, params.fields, this.#guardFieldsFor(params))
|
||||
const entitiesMap = await this.batchGet(pageIds)
|
||||
for (const id of pageIds) {
|
||||
const entity = entitiesMap.get(id)
|
||||
if (entity) {
|
||||
|
|
@ -8324,7 +7985,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
const pageIds = filteredIds.slice(offset, offset + limit)
|
||||
|
||||
// Batch-load entities for 10x faster cloud storage performance
|
||||
const entitiesMap = await this.#hydratePage(pageIds, params.fields, this.#guardFieldsFor(params))
|
||||
const entitiesMap = await this.batchGet(pageIds)
|
||||
for (const id of pageIds) {
|
||||
const entity = entitiesMap.get(id)
|
||||
if (entity) {
|
||||
|
|
@ -8575,7 +8236,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
|
||||
// Batch-load entities for current page - O(page_size) instead of O(total_results)
|
||||
// GCS: 10 entities = 1×50ms vs 10×50ms = 500ms (10x faster)
|
||||
const entitiesMap = await this.#hydratePage(pageIds, params.fields, this.#guardFieldsFor(params))
|
||||
const entitiesMap = await this.batchGet(pageIds)
|
||||
for (const id of pageIds) {
|
||||
const entity = entitiesMap.get(id)
|
||||
if (entity) {
|
||||
|
|
@ -8603,7 +8264,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
|
||||
// Batch-load entities for paginated results (10x faster on GCS)
|
||||
const sortedResults: Result<T>[] = []
|
||||
const entitiesMap = await this.#hydratePage(pageIds, params.fields, this.#guardFieldsFor(params))
|
||||
const entitiesMap = await this.batchGet(pageIds)
|
||||
for (const id of pageIds) {
|
||||
const entity = entitiesMap.get(id)
|
||||
if (entity) {
|
||||
|
|
@ -8708,28 +8369,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
})
|
||||
}
|
||||
|
||||
// PROJECTION TRIM — applied once, here, AFTER the integrity guard, so every
|
||||
// find() path is trimmed uniformly and the guard still saw the evidence it
|
||||
// needs. Hydration carried the guard's fields alongside the projection;
|
||||
// this removes them, leaving exactly what the caller named.
|
||||
//
|
||||
// Rows that reached here from a path the seam does not hydrate (a vector or
|
||||
// text leg builds its own entities) are trimmed from what they already
|
||||
// hold, so the ANSWER is the same everywhere — only the cost differs, and
|
||||
// only on the paths that still read a record.
|
||||
if (params.fields !== undefined && params.fields.length > 0 && result.length > 0) {
|
||||
const named = [...new Set(params.fields)]
|
||||
result = result.map((r) => {
|
||||
const projected = this.#projectEntity(
|
||||
r.id,
|
||||
named,
|
||||
undefined,
|
||||
r.entity as unknown as Entity<T>
|
||||
)
|
||||
return { ...r, entity: projected } as typeof r
|
||||
})
|
||||
}
|
||||
|
||||
// includeVectors — opt-in vector hydration. Default (false) keeps the perf
|
||||
// contract: every result path above builds entities via the metadata-only
|
||||
// fast path, so `entity.vector` is the empty stub. When requested, fetch the
|
||||
|
|
@ -17112,7 +16751,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
ordered = valued.map((v) => v.id)
|
||||
}
|
||||
const pageIds = ordered.slice(offset, offset + limit)
|
||||
const entitiesMap = await this.#hydratePage(pageIds, params.fields, this.#guardFieldsFor(params))
|
||||
const entitiesMap = await this.batchGet(pageIds)
|
||||
const results: Result<T>[] = []
|
||||
for (const id of pageIds) {
|
||||
const entity = entitiesMap.get(id)
|
||||
|
|
|
|||
|
|
@ -412,23 +412,18 @@ export class MigrationInProgressError extends BrainyError {
|
|||
* embedding parked in the metadata bag would mint 384 postings for one row.
|
||||
* The bound exists to keep that out of the index.
|
||||
*
|
||||
* 256 is hardcoded on purpose (the zero-config law: no knob). It sits far above
|
||||
* 64 is hardcoded on purpose (the zero-config law: no knob). It sits far above
|
||||
* every legitimate multi-value field the engine has seen — tags, authors,
|
||||
* categories, labels, keyword lists, participant lists — and still below the
|
||||
* narrowest embedding this engine will ever meet (384 dimensions, the smallest
|
||||
* model it ships), so the two populations do not overlap and no caller has to
|
||||
* tune it. A vector parked in metadata is refused; a long keyword list is not.
|
||||
* categories, labels, participant lists — and far below any real embedding
|
||||
* width, so the two populations do not overlap and no caller has to tune it.
|
||||
*
|
||||
* It replaces a limit of 10 that was applied SILENTLY: a row whose `tags` array
|
||||
* held eleven entries had that field skipped entirely and dropped out of every
|
||||
* filtered search on it, with no error, no warning and no way to tell the
|
||||
* difference from "no row matches". A rule this consequential is a law with a
|
||||
* name and a refusal, not a `continue`.
|
||||
*
|
||||
* This is the ONE place the number lives. Every message, warning, doc line and
|
||||
* pin derives it from here — never a literal.
|
||||
*/
|
||||
export const MAX_INDEXED_ARRAY_LENGTH = 256
|
||||
export const MAX_INDEXED_ARRAY_LENGTH = 64
|
||||
|
||||
/**
|
||||
* A metadata field carries an array longer than {@link MAX_INDEXED_ARRAY_LENGTH}.
|
||||
|
|
|
|||
|
|
@ -23,10 +23,7 @@ import type { ColumnStoreProvider, SegmentMeta } from './types.js'
|
|||
import {
|
||||
ValueType,
|
||||
DEFAULT_FLUSH_THRESHOLD,
|
||||
FLAG_MULTI_VALUE,
|
||||
POSTING_KINDS,
|
||||
KIND_PATH_SEGMENT,
|
||||
type PostingKind
|
||||
FLAG_MULTI_VALUE
|
||||
} from './types.js'
|
||||
import { ColumnTailBuffer } from './ColumnTailBuffer.js'
|
||||
import { ColumnManifest } from './ColumnManifest.js'
|
||||
|
|
@ -55,89 +52,10 @@ interface HeapEntry {
|
|||
value: number | string
|
||||
entityIntId: number
|
||||
cursorIndex: number
|
||||
/**
|
||||
* Rank of the posting kind this entry came from, from {@link POSTING_KINDS}.
|
||||
* A mixed-kind field has no natural total order, so the merge orders by kind
|
||||
* first and by value within a kind.
|
||||
*/
|
||||
kindRank: number
|
||||
/** Iterator for the cursor — call next() to advance */
|
||||
iterator: Generator<CursorEntry>
|
||||
}
|
||||
|
||||
/**
|
||||
* One physical posting column: a (field, kind) pair and the key every internal
|
||||
* map and every storage path uses for it.
|
||||
*/
|
||||
interface KindColumn {
|
||||
/** The field as the query language names it. */
|
||||
field: string
|
||||
/** The kind of value this column holds. */
|
||||
kind: PostingKind
|
||||
/**
|
||||
* Internal map / storage key. The field's PRIMARY kind uses the bare field
|
||||
* name — the historical layout — and every other kind uses
|
||||
* `<field>/<KIND_PATH_SEGMENT>/<kind>`.
|
||||
*/
|
||||
key: string
|
||||
}
|
||||
|
||||
/**
|
||||
* The KIND a value indexes under — its JavaScript `typeof` class, not its
|
||||
* storage encoding.
|
||||
*
|
||||
* Anything that is not a number, string or boolean indexes as a string, which
|
||||
* is the `String(value)` treatment those values already received. `null` and
|
||||
* `undefined` never reach here: `addEntity` skips them, and their absence is
|
||||
* what the `exists` / `missing` operators read.
|
||||
*
|
||||
* @param value - The value about to be indexed or queried
|
||||
* @returns The posting kind that owns this value
|
||||
*/
|
||||
function kindOfValue(value: unknown): PostingKind {
|
||||
const t = typeof value
|
||||
if (t === 'number') return 'number'
|
||||
if (t === 'boolean') return 'boolean'
|
||||
return 'string'
|
||||
}
|
||||
|
||||
/**
|
||||
* The segment encoding a fresh column of this kind starts with.
|
||||
*
|
||||
* Only the number kind has a choice: an integer column starts as i64 and
|
||||
* widens to f64 the first time a non-integer arrives
|
||||
* ({@link ColumnTailBuffer.promoteToFloat}).
|
||||
*/
|
||||
function initialValueTypeFor(kind: PostingKind, firstValue: unknown): ValueType {
|
||||
switch (kind) {
|
||||
case 'boolean':
|
||||
return ValueType.Boolean
|
||||
case 'string':
|
||||
return ValueType.String
|
||||
case 'number':
|
||||
return Number.isInteger(firstValue) ? ValueType.Number : ValueType.Float
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The kind a column of this encoding holds — the inverse of
|
||||
* {@link initialValueTypeFor}, used to read a kind back off a manifest written
|
||||
* before typed postings existed.
|
||||
*/
|
||||
function kindOfValueType(valueType: ValueType): PostingKind {
|
||||
switch (valueType) {
|
||||
case ValueType.Boolean:
|
||||
return 'boolean'
|
||||
case ValueType.String:
|
||||
return 'string'
|
||||
case ValueType.Number:
|
||||
case ValueType.Float:
|
||||
return 'number'
|
||||
default:
|
||||
throw new Error(`Unknown ValueType: ${valueType}`)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Unified column store coordinator.
|
||||
*
|
||||
|
|
@ -203,19 +121,9 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
*/
|
||||
private deletedEntities: Map<string, RoaringBitmap32> = new Map()
|
||||
|
||||
/** Segment encoding per COLUMN key (not per field — a field has one per kind). */
|
||||
/** Known field value types (inferred from first write). */
|
||||
private fieldTypes: Map<string, ValueType> = new Map()
|
||||
|
||||
/**
|
||||
* Every posting column a field owns: field → kind → column key.
|
||||
*
|
||||
* This is the map that ends the first-writer type freeze. A field's first
|
||||
* kind takes the bare field name as its column key, keeping the historical
|
||||
* on-disk layout; each later kind takes its own column beside it. Nothing is
|
||||
* coerced across kinds and nothing is dropped for being the wrong type.
|
||||
*/
|
||||
private fieldColumns: Map<string, Map<PostingKind, string>> = new Map()
|
||||
|
||||
/** Whether init() has completed. */
|
||||
private initialized = false
|
||||
|
||||
|
|
@ -232,128 +140,6 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
this.l0CompactionTrigger = config?.l0CompactionTrigger ?? 4
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// Posting columns: (field, kind) → one physical column
|
||||
// =========================================================================
|
||||
|
||||
/**
|
||||
* Storage / map key for a (field, kind) column.
|
||||
*
|
||||
* `primary` is the kind that owns the bare field name. It is whichever kind
|
||||
* the field saw first, which for an index written before typed postings is
|
||||
* simply the kind of its single manifest — so the historical layout is
|
||||
* preserved rather than migrated.
|
||||
*/
|
||||
private static columnKeyFor(field: string, kind: PostingKind, primary: PostingKind | null): string {
|
||||
return primary === null || kind === primary
|
||||
? field
|
||||
: `${field}/${KIND_PATH_SEGMENT}/${kind}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Split a discovered manifest path back into its (field, kind) column, or
|
||||
* `null` when the path names a field's primary column rather than a kind
|
||||
* column. `<field>/k/<kind>` is the only shape that reads as a kind column,
|
||||
* and only for a `<kind>` this version knows.
|
||||
*/
|
||||
private static parseKindColumnKey(key: string): { field: string; kind: PostingKind } | null {
|
||||
const marker = `/${KIND_PATH_SEGMENT}/`
|
||||
const at = key.lastIndexOf(marker)
|
||||
if (at <= 0) return null
|
||||
const kind = key.slice(at + marker.length)
|
||||
if (!POSTING_KINDS.includes(kind as PostingKind)) return null
|
||||
return { field: key.slice(0, at), kind: kind as PostingKind }
|
||||
}
|
||||
|
||||
/** Record a discovered or freshly created column against its field. */
|
||||
private registerColumn(field: string, kind: PostingKind, key: string): void {
|
||||
let byKind = this.fieldColumns.get(field)
|
||||
if (!byKind) {
|
||||
byKind = new Map()
|
||||
this.fieldColumns.set(field, byKind)
|
||||
}
|
||||
const existing = byKind.get(kind)
|
||||
if (existing !== undefined && existing !== key) {
|
||||
// Two columns claiming one (field, kind) means the layout on disk is not
|
||||
// one this writer could have produced. Serving it would silently answer
|
||||
// from half the postings, so say which two and stop.
|
||||
throw new Error(
|
||||
`ColumnStore: field '${field}' has two '${kind}' posting columns on ` +
|
||||
`disk ('${existing}' and '${key}'). The column index layout is ` +
|
||||
`inconsistent — rebuild/repair the metadata index rather than ` +
|
||||
`serving from one half of it.`
|
||||
)
|
||||
}
|
||||
byKind.set(kind, key)
|
||||
}
|
||||
|
||||
/** The column key for this (field, kind), or `null` if the field has no such kind. */
|
||||
private columnKey(field: string, kind: PostingKind): string | null {
|
||||
return this.fieldColumns.get(field)?.get(kind) ?? null
|
||||
}
|
||||
|
||||
/**
|
||||
* The column key for this (field, kind), creating the registration if the
|
||||
* field has not seen this kind before. Write path only.
|
||||
*/
|
||||
private ensureColumnKey(field: string, kind: PostingKind): string {
|
||||
const byKind = this.fieldColumns.get(field)
|
||||
const existing = byKind?.get(kind)
|
||||
if (existing !== undefined) return existing
|
||||
|
||||
// The primary kind is the one already holding the bare field name, if any.
|
||||
let primary: PostingKind | null = null
|
||||
if (byKind) {
|
||||
for (const [k, key] of byKind) {
|
||||
if (key === field) { primary = k; break }
|
||||
}
|
||||
}
|
||||
const key = ColumnStore.columnKeyFor(field, kind, primary)
|
||||
this.registerColumn(field, kind, key)
|
||||
return key
|
||||
}
|
||||
|
||||
/**
|
||||
* Every posting column this field owns, in {@link POSTING_KINDS} order.
|
||||
*
|
||||
* Read doors that are not about one particular value — an unbounded range
|
||||
* used as an "any value present" probe, distinct values, sorting — fan out
|
||||
* over all of them.
|
||||
*/
|
||||
private columnsForField(field: string): KindColumn[] {
|
||||
const byKind = this.fieldColumns.get(field)
|
||||
if (!byKind) return []
|
||||
const out: KindColumn[] = []
|
||||
for (const kind of POSTING_KINDS) {
|
||||
const key = byKind.get(kind)
|
||||
if (key !== undefined) out.push({ field, kind, key })
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
/**
|
||||
* Which value kinds this field actually holds, in {@link POSTING_KINDS}
|
||||
* order — the honest answer to "what type is this field?".
|
||||
*
|
||||
* A field that carries both `'electronics'` and `5` reports
|
||||
* `['number', 'string']`, not whichever of them was written first.
|
||||
*
|
||||
* @param field - Field name
|
||||
* @returns Every kind with at least one posting, or `[]` for an unknown field
|
||||
*/
|
||||
getFieldKinds(field: string): PostingKind[] {
|
||||
return this.columnsForField(field)
|
||||
.filter((c) => this.columnHasData(c.key))
|
||||
.map((c) => c.kind)
|
||||
}
|
||||
|
||||
/** Does this physical column hold any postings (persisted or buffered)? */
|
||||
private columnHasData(key: string): boolean {
|
||||
const manifest = this.manifests.get(key)
|
||||
const buffer = this.tailBuffers.get(key)
|
||||
return (manifest !== undefined && !manifest.isEmpty()) || (buffer !== undefined && buffer.size > 0)
|
||||
}
|
||||
|
||||
/**
|
||||
* Initialize the column store: discover existing field manifests.
|
||||
*/
|
||||
|
|
@ -371,23 +157,11 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
}).listObjectsUnderPath(this.basePath + '/')
|
||||
for (const path of paths) {
|
||||
if (path.endsWith('/MANIFEST.json')) {
|
||||
// The discovered name is a COLUMN key: either a bare field (that
|
||||
// field's primary kind, which is every column an index written
|
||||
// before typed postings has) or `<field>/k/<kind>` for a second
|
||||
// kind that arrived on a field later.
|
||||
const columnKey = path.replace(this.basePath + '/', '').replace('/MANIFEST.json', '')
|
||||
const manifest = new ColumnManifest(columnKey, this.basePath)
|
||||
const fieldName = path.replace(this.basePath + '/', '').replace('/MANIFEST.json', '')
|
||||
const manifest = new ColumnManifest(fieldName, this.basePath)
|
||||
await manifest.load(storage)
|
||||
this.manifests.set(columnKey, manifest)
|
||||
this.fieldTypes.set(columnKey, manifest.valueType)
|
||||
|
||||
const parsed = ColumnStore.parseKindColumnKey(columnKey)
|
||||
if (parsed) {
|
||||
this.registerColumn(parsed.field, parsed.kind, columnKey)
|
||||
} else {
|
||||
this.registerColumn(columnKey, kindOfValueType(manifest.valueType), columnKey)
|
||||
}
|
||||
const fieldName = columnKey
|
||||
this.manifests.set(fieldName, manifest)
|
||||
this.fieldTypes.set(fieldName, manifest.valueType)
|
||||
|
||||
// Load global deleted bitmap if it exists. Raw blob preferred
|
||||
// (2.4.0 #4 cortex-shared format); legacy envelope fallback for
|
||||
|
|
@ -490,43 +264,26 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
/**
|
||||
* Point filter: find entities where field equals value.
|
||||
*
|
||||
* The QUERY VALUE'S OWN KIND picks the posting column, and only that column
|
||||
* is read. `where {category: 5}` answers from the number postings and
|
||||
* `where {category: '5'}` from the string postings — neither borrows the
|
||||
* other's rows, because a row written with the number `5` is not a row whose
|
||||
* category is the text `'5'`.
|
||||
*
|
||||
* A field that has never seen this kind matches nothing, which is the true
|
||||
* answer rather than a coerced one.
|
||||
*
|
||||
* Searches all segments + tail buffer of that column, returns the union as a
|
||||
* roaring bitmap. Excludes globally deleted entities.
|
||||
* Searches all segments + tail buffer, returns union as roaring bitmap.
|
||||
* Excludes globally deleted entities.
|
||||
*/
|
||||
async filter(field: string, value: unknown): Promise<RoaringBitmap32> {
|
||||
const result = new RoaringBitmap32()
|
||||
const columnKey = this.columnKey(field, kindOfValue(value))
|
||||
if (columnKey === null) return result
|
||||
|
||||
// The query value takes the column's encoding — a boolean queried against
|
||||
// a boolean column has to become the 1/0 the column stores.
|
||||
const encoded = this.normalizeValue(value, this.fieldTypes.get(columnKey) ?? ValueType.String)
|
||||
if (encoded === undefined) return result
|
||||
|
||||
const deleted = this.deletedEntities.get(columnKey)
|
||||
const deleted = this.deletedEntities.get(field)
|
||||
|
||||
// Search segments
|
||||
const cursors = await this.getSegmentCursors(columnKey)
|
||||
const cursors = await this.getSegmentCursors(field)
|
||||
for (const cursor of cursors) {
|
||||
const ids = cursor.getEntityIdsForValue(encoded)
|
||||
const ids = cursor.getEntityIdsForValue(value as number | string)
|
||||
for (const id of ids) {
|
||||
if (!deleted || !deleted.has(id)) result.add(id)
|
||||
}
|
||||
}
|
||||
|
||||
// Search tail buffer
|
||||
const tailCursor = this.getTailBufferCursor(columnKey)
|
||||
const tailCursor = this.getTailBufferCursor(field)
|
||||
if (tailCursor) {
|
||||
const ids = tailCursor.getEntityIdsForValue(encoded)
|
||||
const ids = tailCursor.getEntityIdsForValue(value as number | string)
|
||||
for (const id of ids) {
|
||||
if (!deleted || !deleted.has(id)) result.add(id)
|
||||
}
|
||||
|
|
@ -535,62 +292,6 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
return result
|
||||
}
|
||||
|
||||
/**
|
||||
* Read this column's value for each of `entityIntIds` — the per-id read
|
||||
* behind `find({ fields })`.
|
||||
*
|
||||
* Every other read door here answers "which entities have this value". A
|
||||
* projection asks the opposite — "what value does this entity have" — and
|
||||
* without it a projection has to go to the canonical record for a field the
|
||||
* column is already holding.
|
||||
*
|
||||
* The column is walked ONCE and the wanted ids are picked out as they pass,
|
||||
* so the cost is O(column) per field rather than O(ids x column). Later
|
||||
* sources win: the tail buffer holds writes newer than any segment, and
|
||||
* within the segments a later one supersedes an earlier, exactly as `filter`
|
||||
* treats them.
|
||||
*
|
||||
* Values are EXACT — this store keeps raw values, not the bucketed form the
|
||||
* sparse index uses for range queries — which is what makes it safe to
|
||||
* project from. Deleted entities are skipped; an id with no value in this
|
||||
* column is simply absent from the result.
|
||||
*
|
||||
* @param field - Field name to read.
|
||||
* @param entityIntIds - Entity integer ids to read values for.
|
||||
* @returns `entityIntId -> value` for the ids this column holds.
|
||||
*/
|
||||
async valuesForIds(
|
||||
field: string,
|
||||
entityIntIds: Iterable<number>
|
||||
): Promise<Map<number, number | string>> {
|
||||
const wanted = new Set<number>(entityIntIds)
|
||||
const out = new Map<number, number | string>()
|
||||
if (wanted.size === 0 || !this.hasField(field)) return out
|
||||
|
||||
// Every kind the field holds is read, in POSTING_KINDS order — a value an
|
||||
// entity wrote as a string is still that entity's value for this field.
|
||||
for (const column of this.columnsForField(field)) {
|
||||
const deleted = this.deletedEntities.get(column.key)
|
||||
const take = (entry: { value: number | string; entityIntId: number }): void => {
|
||||
if (!wanted.has(entry.entityIntId)) return
|
||||
if (deleted && deleted.has(entry.entityIntId)) return
|
||||
out.set(entry.entityIntId, entry.value)
|
||||
}
|
||||
|
||||
// Segments oldest -> newest, then the tail: a later write overwrites an
|
||||
// earlier one for the same id.
|
||||
const cursors = await this.getSegmentCursors(column.key)
|
||||
for (const cursor of cursors) {
|
||||
for (const entry of cursor.iterateForward()) take(entry)
|
||||
}
|
||||
const tailCursor = this.getTailBufferCursor(column.key)
|
||||
if (tailCursor) {
|
||||
for (const entry of tailCursor.iterateForward()) take(entry)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
/**
|
||||
* Range filter: find entities where field is within the bounds.
|
||||
*
|
||||
|
|
@ -610,59 +311,41 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
includeMax: boolean = true
|
||||
): Promise<RoaringBitmap32> {
|
||||
const result = new RoaringBitmap32()
|
||||
const cursors = await this.getSegmentCursors(field)
|
||||
|
||||
const hasMin = min !== undefined && min !== null
|
||||
const hasMax = max !== undefined && max !== null
|
||||
|
||||
// The BOUNDS pick the column: numeric bounds read the numeric postings,
|
||||
// string bounds the string postings. An unbounded call is not a range at
|
||||
// all — it is the "has any value here" probe behind `exists` — so it fans
|
||||
// out over every kind the field holds.
|
||||
const columns: KindColumn[] = hasMin
|
||||
? this.columnsForKind(field, kindOfValue(min))
|
||||
: hasMax
|
||||
? this.columnsForKind(field, kindOfValue(max))
|
||||
: this.columnsForField(field)
|
||||
for (const cursor of cursors) {
|
||||
const lo = hasMin ? min as number | string : cursor.minValue
|
||||
const hi = hasMax ? max as number | string : cursor.maxValue
|
||||
if (lo === undefined || hi === undefined) continue
|
||||
// Exclusivity applies only to an explicitly provided bound. A bound taken
|
||||
// from the segment's own min/max is a real stored value and must stay
|
||||
// inclusive, or the segment's boundary entities would be wrongly dropped.
|
||||
const ids = cursor.getEntityIdsInRange(
|
||||
lo,
|
||||
hi,
|
||||
hasMin ? includeMin : true,
|
||||
hasMax ? includeMax : true
|
||||
)
|
||||
for (const id of ids) result.add(id)
|
||||
}
|
||||
|
||||
for (const column of columns) {
|
||||
const cursors = await this.getSegmentCursors(column.key)
|
||||
for (const cursor of cursors) {
|
||||
const lo = hasMin ? min as number | string : cursor.minValue
|
||||
const hi = hasMax ? max as number | string : cursor.maxValue
|
||||
if (lo === undefined || hi === undefined) continue
|
||||
// Exclusivity applies only to an explicitly provided bound. A bound taken
|
||||
// from the segment's own min/max is a real stored value and must stay
|
||||
// inclusive, or the segment's boundary entities would be wrongly dropped.
|
||||
const ids = cursor.getEntityIdsInRange(
|
||||
lo,
|
||||
hi,
|
||||
hasMin ? includeMin : true,
|
||||
hasMax ? includeMax : true
|
||||
)
|
||||
for (const id of ids) result.add(id)
|
||||
}
|
||||
|
||||
// Tail buffer range: linear scan (tail is small)
|
||||
const tailCursor = this.getTailBufferCursor(column.key)
|
||||
if (tailCursor) {
|
||||
for (const entry of tailCursor.iterateForward()) {
|
||||
const v = entry.value as any
|
||||
const loOk = !hasMin || (includeMin ? v >= (min as any) : v > (min as any))
|
||||
const hiOk = !hasMax || (includeMax ? v <= (max as any) : v < (max as any))
|
||||
if (loOk && hiOk) result.add(entry.entityIntId)
|
||||
}
|
||||
// Tail buffer range: linear scan (tail is small)
|
||||
const tailCursor = this.getTailBufferCursor(field)
|
||||
if (tailCursor) {
|
||||
for (const entry of tailCursor.iterateForward()) {
|
||||
const v = entry.value as any
|
||||
const loOk = !hasMin || (includeMin ? v >= (min as any) : v > (min as any))
|
||||
const hiOk = !hasMax || (includeMax ? v <= (max as any) : v < (max as any))
|
||||
if (loOk && hiOk) result.add(entry.entityIntId)
|
||||
}
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
/** The single column for this (field, kind), as a list, or empty if absent. */
|
||||
private columnsForKind(field: string, kind: PostingKind): KindColumn[] {
|
||||
const key = this.columnKey(field, kind)
|
||||
return key === null ? [] : [{ field, kind, key }]
|
||||
}
|
||||
|
||||
/**
|
||||
* Sort top-K: return K entity int IDs in sorted order (u64-safe BigInt).
|
||||
*
|
||||
|
|
@ -693,21 +376,18 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
*/
|
||||
async getFilterValues(field: string): Promise<string[]> {
|
||||
const valueSet = new Set<string>()
|
||||
const cursors = await this.getSegmentCursors(field)
|
||||
|
||||
for (const column of this.columnsForField(field)) {
|
||||
const cursors = await this.getSegmentCursors(column.key)
|
||||
|
||||
for (const cursor of cursors) {
|
||||
for (const entry of cursor.iterateForward()) {
|
||||
valueSet.add(String(entry.value))
|
||||
}
|
||||
for (const cursor of cursors) {
|
||||
for (const entry of cursor.iterateForward()) {
|
||||
valueSet.add(String(entry.value))
|
||||
}
|
||||
}
|
||||
|
||||
const tailCursor = this.getTailBufferCursor(column.key)
|
||||
if (tailCursor) {
|
||||
for (const entry of tailCursor.iterateForward()) {
|
||||
valueSet.add(String(entry.value))
|
||||
}
|
||||
const tailCursor = this.getTailBufferCursor(field)
|
||||
if (tailCursor) {
|
||||
for (const entry of tailCursor.iterateForward()) {
|
||||
valueSet.add(String(entry.value))
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -718,7 +398,9 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
* Check if a field has any indexed data.
|
||||
*/
|
||||
hasField(field: string): boolean {
|
||||
return this.columnsForField(field).some((c) => this.columnHasData(c.key))
|
||||
const manifest = this.manifests.get(field)
|
||||
const buffer = this.tailBuffers.get(field)
|
||||
return (manifest !== undefined && !manifest.isEmpty()) || (buffer !== undefined && buffer.size > 0)
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -728,11 +410,12 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
* store will actually serve queries from.
|
||||
*/
|
||||
getIndexedFields(): string[] {
|
||||
// Names FIELDS, not columns: a field carrying two kinds is one name here,
|
||||
// the same name a caller queries with.
|
||||
const fields = new Set<string>()
|
||||
for (const [field] of this.fieldColumns) {
|
||||
if (this.hasField(field)) fields.add(field)
|
||||
for (const [field, manifest] of this.manifests) {
|
||||
if (!manifest.isEmpty()) fields.add(field)
|
||||
}
|
||||
for (const [field, buffer] of this.tailBuffers) {
|
||||
if (buffer.size > 0) fields.add(field)
|
||||
}
|
||||
return Array.from(fields).sort()
|
||||
}
|
||||
|
|
@ -747,16 +430,12 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
getFieldSizeSummary(): Array<{ field: string; segmentCount: number; tailSize: number }> {
|
||||
const summary: Array<{ field: string; segmentCount: number; tailSize: number }> = []
|
||||
for (const field of this.getIndexedFields()) {
|
||||
// Summed across the field's kind columns — the caller asked about a
|
||||
// field, and a field's size is all of the postings under its name.
|
||||
let segmentCount = 0
|
||||
let tailSize = 0
|
||||
for (const column of this.columnsForField(field)) {
|
||||
const manifest = this.manifests.get(column.key)
|
||||
const buffer = this.tailBuffers.get(column.key)
|
||||
if (manifest && !manifest.isEmpty()) segmentCount += manifest.getAllSegments().length
|
||||
if (buffer) tailSize += buffer.size
|
||||
}
|
||||
const manifest = this.manifests.get(field)
|
||||
const buffer = this.tailBuffers.get(field)
|
||||
const segmentCount = manifest && !manifest.isEmpty()
|
||||
? manifest.getAllSegments().length
|
||||
: 0
|
||||
const tailSize = buffer ? buffer.size : 0
|
||||
summary.push({ field, segmentCount, tailSize })
|
||||
}
|
||||
return summary
|
||||
|
|
@ -784,8 +463,6 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
this.segmentCache.clear()
|
||||
this.manifests.clear()
|
||||
this.deletedEntities.clear()
|
||||
this.fieldColumns.clear()
|
||||
this.fieldTypes.clear()
|
||||
this.initialized = false
|
||||
}
|
||||
|
||||
|
|
@ -794,64 +471,32 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
// =========================================================================
|
||||
|
||||
/**
|
||||
* Push a single value to the posting column for its (field, KIND).
|
||||
*
|
||||
* The value's own kind picks the column — a string goes to the field's
|
||||
* string postings, a number to its number postings — so a field carrying
|
||||
* `'electronics'` and `5` keeps both, each answerable by an equality filter
|
||||
* of its own kind. Under the first-writer type freeze this method replaced,
|
||||
* the first value's type became the field's type and every later value of
|
||||
* another kind was coerced to it or, when coercion failed, dropped with no
|
||||
* error at all.
|
||||
*
|
||||
* Creates the column's buffer and manifest on its first value.
|
||||
* Push a single value to a field's tail buffer.
|
||||
* Creates the buffer and manifest if first write to this field.
|
||||
* Infers ValueType from the first value seen.
|
||||
*/
|
||||
private pushToBuffer(field: string, value: unknown, entityIntId: number, isMultiValue: boolean): void {
|
||||
const kind = kindOfValue(value)
|
||||
const columnKey = this.ensureColumnKey(field, kind)
|
||||
|
||||
let buffer = this.tailBuffers.get(columnKey)
|
||||
let buffer = this.tailBuffers.get(field)
|
||||
if (!buffer) {
|
||||
// A reopened column takes its encoding from its manifest — an integer
|
||||
// column that widened to f64 in an earlier session stays widened.
|
||||
const valueType =
|
||||
this.manifests.get(columnKey)?.valueType ?? initialValueTypeFor(kind, value)
|
||||
buffer = new ColumnTailBuffer(columnKey, valueType, this.flushThreshold)
|
||||
this.tailBuffers.set(columnKey, buffer)
|
||||
this.fieldTypes.set(columnKey, valueType)
|
||||
const valueType = this.inferValueType(value)
|
||||
buffer = new ColumnTailBuffer(field, valueType, this.flushThreshold)
|
||||
this.tailBuffers.set(field, buffer)
|
||||
this.fieldTypes.set(field, valueType)
|
||||
|
||||
// Ensure manifest exists
|
||||
if (!this.manifests.has(columnKey)) {
|
||||
const manifest = new ColumnManifest(columnKey, this.basePath)
|
||||
if (!this.manifests.has(field)) {
|
||||
const manifest = new ColumnManifest(field, this.basePath)
|
||||
manifest.valueType = valueType
|
||||
manifest.multiValue = isMultiValue
|
||||
this.manifests.set(columnKey, manifest)
|
||||
this.manifests.set(field, manifest)
|
||||
}
|
||||
}
|
||||
|
||||
// An integer column widens the first time a non-integer number arrives, so
|
||||
// the value is stored as itself instead of rounded to the nearest integer.
|
||||
if (kind === 'number' && buffer.valueType === ValueType.Number && !Number.isInteger(value)) {
|
||||
buffer.promoteToFloat()
|
||||
this.fieldTypes.set(columnKey, ValueType.Float)
|
||||
const manifest = this.manifests.get(columnKey)
|
||||
if (manifest) manifest.valueType = ValueType.Float
|
||||
}
|
||||
|
||||
// Normalize value to the column type
|
||||
const normalizedValue = this.normalizeValue(value, buffer.valueType)
|
||||
if (normalizedValue === undefined) {
|
||||
// Unreachable by construction: the column was chosen BY this value's
|
||||
// kind, so the encoding always accepts it. Reaching here would mean a
|
||||
// value had been silently dropped from the index — the exact failure
|
||||
// typed postings exist to end — so it is an error, never a skip.
|
||||
throw new Error(
|
||||
`ColumnStore: field '${field}' rejected a ${kind} value for its own ` +
|
||||
`${ValueType[buffer.valueType]} posting column. The value would have ` +
|
||||
`been dropped from the index while the row stayed readable by id — ` +
|
||||
`this is a kind-routing bug, not a value the caller may ignore.`
|
||||
)
|
||||
if (normalizedValue !== undefined) {
|
||||
buffer.add(normalizedValue, entityIntId)
|
||||
}
|
||||
buffer.add(normalizedValue, entityIntId)
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -980,15 +625,8 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
/** Torn-segment quarantine entries for a field (observability + heal input). */
|
||||
quarantinedSegments(field: string): Array<{ segment: string; error: string; hits: number }> {
|
||||
const out: Array<{ segment: string; error: string; hits: number }> = []
|
||||
// Across every kind column of the field — a torn segment in the string
|
||||
// postings is this field's torn segment as much as one in the numbers.
|
||||
for (const column of this.columnsForField(field)) {
|
||||
const prefix = `${column.key}:`
|
||||
for (const [key, q] of this.segmentQuarantine) {
|
||||
if (key.startsWith(prefix)) {
|
||||
out.push({ segment: key.slice(prefix.length), error: q.error, hits: q.hits })
|
||||
}
|
||||
}
|
||||
for (const [key, q] of this.segmentQuarantine) {
|
||||
if (key.startsWith(`${field}:`)) out.push({ segment: key.slice(field.length + 1), error: q.error, hits: q.hits })
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
|
@ -1160,22 +798,17 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
k: number,
|
||||
filterBitmap: RoaringBitmap32 | null
|
||||
): Promise<number[]> {
|
||||
// Collect cursors across EVERY kind the field holds. A single-kind field —
|
||||
// nearly all of them — merges exactly the cursors it always did.
|
||||
// Collect all cursors (segments + tail buffer)
|
||||
const segCursors = await this.getSegmentCursors(field)
|
||||
const tailCursor = this.getTailBufferCursor(field)
|
||||
|
||||
// Create iterators for each cursor in the specified direction
|
||||
const iterators: Generator<CursorEntry>[] = []
|
||||
const iteratorKindRank: number[] = []
|
||||
for (const column of this.columnsForField(field)) {
|
||||
const kindRank = POSTING_KINDS.indexOf(column.kind)
|
||||
const segCursors = await this.getSegmentCursors(column.key)
|
||||
for (const cursor of segCursors) {
|
||||
iterators.push(order === 'asc' ? cursor.iterateForward() : cursor.iterateBackward())
|
||||
iteratorKindRank.push(kindRank)
|
||||
}
|
||||
const tailCursor = this.getTailBufferCursor(column.key)
|
||||
if (tailCursor) {
|
||||
iterators.push(order === 'asc' ? tailCursor.iterateForward() : tailCursor.iterateBackward())
|
||||
iteratorKindRank.push(kindRank)
|
||||
}
|
||||
for (const cursor of segCursors) {
|
||||
iterators.push(order === 'asc' ? cursor.iterateForward() : cursor.iterateBackward())
|
||||
}
|
||||
if (tailCursor) {
|
||||
iterators.push(order === 'asc' ? tailCursor.iterateForward() : tailCursor.iterateBackward())
|
||||
}
|
||||
|
||||
if (iterators.length === 0) return []
|
||||
|
|
@ -1189,21 +822,16 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
value: next.value.value,
|
||||
entityIntId: next.value.entityIntId,
|
||||
cursorIndex: i,
|
||||
kindRank: iteratorKindRank[i],
|
||||
iterator: iterators[i]
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Heapify. A number and a string have no ordering between them, so a
|
||||
// mixed-kind field orders by KIND first (POSTING_KINDS order) and by value
|
||||
// within a kind — one defined total order instead of a comparison whose
|
||||
// answer depends on which value happened to be on the left.
|
||||
// Heapify
|
||||
const isString = (this.fieldTypes.get(field) ?? ValueType.Number) === ValueType.String
|
||||
const compare = (a: HeapEntry, b: HeapEntry): number => {
|
||||
let cmp: number
|
||||
if (a.kindRank !== b.kindRank) {
|
||||
cmp = a.kindRank - b.kindRank
|
||||
} else if (POSTING_KINDS[a.kindRank] === 'string') {
|
||||
if (isString) {
|
||||
cmp = compareCodePoints(String(a.value), String(b.value))
|
||||
} else {
|
||||
cmp = (a.value as number) - (b.value as number)
|
||||
|
|
@ -1235,7 +863,6 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
value: next.value.value,
|
||||
entityIntId: next.value.entityIntId,
|
||||
cursorIndex: top.cursorIndex,
|
||||
kindRank: top.kindRank,
|
||||
iterator: top.iterator
|
||||
}
|
||||
}
|
||||
|
|
@ -1243,11 +870,8 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
this.heapDown(heap, 0, compare)
|
||||
}
|
||||
|
||||
// Apply global deleted check, filter, and dedup. The deleted bitmap is
|
||||
// per COLUMN, and the entry came from the column its kind names.
|
||||
const deleted = this.deletedEntities.get(
|
||||
this.columnKey(field, POSTING_KINDS[top.kindRank]) ?? field
|
||||
)
|
||||
// Apply global deleted check, filter, and dedup
|
||||
const deleted = this.deletedEntities.get(field)
|
||||
if (deleted && deleted.has(top.entityIntId)) continue
|
||||
if (seen.has(top.entityIntId)) continue
|
||||
if (filterBitmap && !filterBitmap.has(top.entityIntId)) continue
|
||||
|
|
@ -1289,31 +913,35 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
}
|
||||
|
||||
/**
|
||||
* Encode a value for the column its own kind selected.
|
||||
*
|
||||
* This does NOT convert between kinds. It used to: a string reaching a
|
||||
* numeric column was run through `Number(value)`, and a number reaching a
|
||||
* numeric column was run through `Math.round`, so `'electronics'` became
|
||||
* `NaN` and vanished while `4.5` became `5` and answered the wrong query.
|
||||
* Kind routing removes the need for either — the only work left is picking
|
||||
* the encoding the column already committed to.
|
||||
*
|
||||
* @returns The encoded value, or `undefined` if the value does not belong in
|
||||
* this column at all — which the caller treats as a routing bug and
|
||||
* raises, never as a value to skip.
|
||||
* Infer ValueType from a JavaScript value.
|
||||
*/
|
||||
private inferValueType(value: unknown): ValueType {
|
||||
if (typeof value === 'boolean') return ValueType.Boolean
|
||||
if (typeof value === 'number') {
|
||||
return Number.isInteger(value) ? ValueType.Number : ValueType.Float
|
||||
}
|
||||
return ValueType.String
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize a JavaScript value to the column's ValueType.
|
||||
*/
|
||||
private normalizeValue(value: unknown, type: ValueType): number | string | undefined {
|
||||
switch (type) {
|
||||
case ValueType.Number:
|
||||
// Integer column. Non-integers widen it to Float before reaching here.
|
||||
return typeof value === 'number' && Number.isInteger(value) ? value : undefined
|
||||
if (typeof value === 'number') return Math.round(value)
|
||||
if (typeof value === 'string') { const n = Number(value); return isNaN(n) ? undefined : Math.round(n) }
|
||||
if (typeof value === 'boolean') return value ? 1 : 0
|
||||
return undefined
|
||||
case ValueType.Float:
|
||||
return typeof value === 'number' ? value : undefined
|
||||
if (typeof value === 'number') return value
|
||||
if (typeof value === 'string') { const n = Number(value); return isNaN(n) ? undefined : n }
|
||||
return undefined
|
||||
case ValueType.Boolean:
|
||||
return typeof value === 'boolean' ? (value ? 1 : 0) : undefined
|
||||
if (typeof value === 'boolean') return value ? 1 : 0
|
||||
if (typeof value === 'number') return value ? 1 : 0
|
||||
return undefined
|
||||
case ValueType.String:
|
||||
// The string kind is also where objects and bigints land, exactly as
|
||||
// they always did.
|
||||
return String(value)
|
||||
default:
|
||||
return undefined
|
||||
|
|
|
|||
|
|
@ -55,12 +55,8 @@ export class ColumnTailBuffer {
|
|||
/** Field name this buffer is for. */
|
||||
readonly fieldName: string
|
||||
|
||||
/**
|
||||
* Value type determines sort comparator and segment encoding.
|
||||
*
|
||||
* Widened in place by {@link promoteToFloat} — never otherwise reassigned.
|
||||
*/
|
||||
valueType: ValueType
|
||||
/** Value type determines sort comparator. */
|
||||
readonly valueType: ValueType
|
||||
|
||||
/** Flush threshold. */
|
||||
readonly threshold: number
|
||||
|
|
@ -85,38 +81,6 @@ export class ColumnTailBuffer {
|
|||
this.threshold = threshold
|
||||
}
|
||||
|
||||
/**
|
||||
* Widen an integer column to floating point, losslessly and in place.
|
||||
*
|
||||
* The number posting kind holds every JavaScript number, but a segment picks
|
||||
* ONE encoding: i64 for integers, f64 for the rest. A column that has only
|
||||
* ever seen integers is written as i64; the first non-integer to arrive
|
||||
* widens it here, so that value is stored as itself instead of being rounded
|
||||
* to the nearest integer with no error — the rounding that made `4.5` and
|
||||
* `5.5` both answer `where {score: 5}` and neither answer its own value.
|
||||
*
|
||||
* Widening is lossless in both directions it has to be: every value already
|
||||
* buffered is an integer, and every integer is exactly representable as f64.
|
||||
* Segments already on disk keep their own i64 encoding in their own headers
|
||||
* and keep decoding by it — only segments written from here on are f64.
|
||||
*
|
||||
* @throws Error if called on a column that is not an integer column — the
|
||||
* only legal widening is Number → Float, and any other request is a bug in
|
||||
* the caller's kind routing rather than something to absorb quietly.
|
||||
*/
|
||||
promoteToFloat(): void {
|
||||
if (this.valueType === ValueType.Float) return
|
||||
if (this.valueType !== ValueType.Number) {
|
||||
throw new Error(
|
||||
`ColumnTailBuffer '${this.fieldName}': cannot widen a ` +
|
||||
`${ValueType[this.valueType]} column to Float — only an integer ` +
|
||||
`(Number) column widens, and this call means a value reached the ` +
|
||||
`wrong kind's column`
|
||||
)
|
||||
}
|
||||
this.valueType = ValueType.Float
|
||||
}
|
||||
|
||||
/**
|
||||
* Add a (value, entityIntId) entry to the buffer.
|
||||
*
|
||||
|
|
|
|||
|
|
@ -58,53 +58,6 @@ export enum ValueType {
|
|||
Boolean = 3
|
||||
}
|
||||
|
||||
/**
|
||||
* The KIND of a value, as the query language sees it.
|
||||
*
|
||||
* A kind is a JavaScript `typeof` class, not a storage encoding: `5` and `5.5`
|
||||
* are one kind (`'number'`) held in one posting column, even though they need
|
||||
* different segment encodings (i64 vs f64 — see {@link ValueType}).
|
||||
*
|
||||
* A field holds ONE POSTING COLUMN PER KIND, so `category` may carry string
|
||||
* values and number values at the same time and answer equality on each. This
|
||||
* replaces the first-writer type freeze, under which the first value's type
|
||||
* became the field's type and every later value of another kind was coerced —
|
||||
* or, when coercion failed (`Number('electronics')`), dropped from the index
|
||||
* with no error: the row stayed readable by id and by vector but vanished from
|
||||
* every equality filter on that field.
|
||||
*
|
||||
* Kinds do not coerce into one another at query time either: `where {c: 5}`
|
||||
* matches rows written with the NUMBER `5`, and `where {c: '5'}` matches rows
|
||||
* written with the STRING `'5'`. Neither ever matches the other.
|
||||
*
|
||||
* Values that are none of these three (objects, bigints) index as strings —
|
||||
* the same `String(value)` treatment they received before.
|
||||
*/
|
||||
export type PostingKind = 'number' | 'string' | 'boolean'
|
||||
|
||||
/**
|
||||
* Every posting kind, in the order that defines cross-kind sort position.
|
||||
*
|
||||
* A mixed-kind field has no natural total order — a number does not compare
|
||||
* with a string — so `sortTopK` orders by KIND first (numbers, then strings,
|
||||
* then booleans) and by value within a kind. A single-kind field, which is
|
||||
* nearly every field, sorts exactly as it always did.
|
||||
*/
|
||||
export const POSTING_KINDS: readonly PostingKind[] = ['number', 'string', 'boolean']
|
||||
|
||||
/**
|
||||
* Path segment marking a field's NON-PRIMARY kind columns on disk.
|
||||
*
|
||||
* The first kind a field ever sees keeps the historical layout —
|
||||
* `<base>/<field>/MANIFEST.json` and `<base>/<field>/L0-NNNNNN` — so every
|
||||
* index written before typed postings opens unchanged, and the byte-for-byte
|
||||
* interchange with the native column store is untouched for the single-kind
|
||||
* fields that are nearly all of them. A second kind arriving on the same field
|
||||
* gets its own column at `<base>/<field>/k/<kind>/…` rather than overwriting or
|
||||
* being coerced into the first.
|
||||
*/
|
||||
export const KIND_PATH_SEGMENT = 'k'
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Segment header and footer
|
||||
// ---------------------------------------------------------------------------
|
||||
|
|
@ -314,19 +267,6 @@ export interface ColumnStoreProvider {
|
|||
*/
|
||||
hasField(field: string): boolean
|
||||
|
||||
/**
|
||||
* Which value KINDS this field actually holds, in {@link POSTING_KINDS}
|
||||
* order — the honest answer to "what type is this field?" for a field that
|
||||
* carries more than one.
|
||||
*
|
||||
* OPTIONAL so an implementation written against the pre-typed-postings
|
||||
* contract still satisfies this interface; feature-detect before calling.
|
||||
*
|
||||
* @param field - Field name
|
||||
* @returns Every kind with at least one posting, or `[]` for an unknown field
|
||||
*/
|
||||
getFieldKinds?(field: string): PostingKind[]
|
||||
|
||||
/**
|
||||
* Flush all in-memory tail buffers to L0 segments on disk.
|
||||
* Saves all manifests.
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
* 🧠 BRAINY EMBEDDED PATTERNS
|
||||
*
|
||||
* AUTO-GENERATED - DO NOT EDIT
|
||||
* Generated: 2026-08-27T09:18:45-07:00
|
||||
* Generated: 2025-09-29T10:10:00-07:00
|
||||
* Patterns: 220
|
||||
* Coverage: 94-98% of all queries
|
||||
*
|
||||
|
|
|
|||
|
|
@ -495,45 +495,6 @@ export interface MetadataIndexProvider {
|
|||
query: string,
|
||||
ids: readonly string[]
|
||||
): Promise<Array<{ id: string; matchCount: number }>>
|
||||
/**
|
||||
* @description OPTIONAL: read named SCALAR fields for many ids at once, from
|
||||
* the index's own value storage, WITHOUT touching the canonical record.
|
||||
*
|
||||
* This is the door behind `find/get/related({ fields })`. A list view that
|
||||
* needs a title and a slug currently hydrates the whole record for every row
|
||||
* — document bodies included — and then discards almost all of it. Serving
|
||||
* the named scalars from the index turns that into an index read.
|
||||
*
|
||||
* ## The contract, and the one rule that makes it safe
|
||||
*
|
||||
* **Return only what you can serve EXACTLY, and say what you served.** The
|
||||
* answer is a per-id map of the fields this index actually resolved; the
|
||||
* caller diffs it against what was requested and reads the canonical record
|
||||
* for the remainder. An implementation must therefore OMIT a field rather
|
||||
* than approximate it — and omission costs only a record read, while a wrong
|
||||
* value is a wrong answer nobody can see.
|
||||
*
|
||||
* That rule is not hypothetical. This engine's own index buckets
|
||||
* `system.createdAt` and `system.updatedAt` to the minute for range queries,
|
||||
* so it cannot serve them exactly and omits them. An engine whose column
|
||||
* store holds raw values can serve the same fields — so the two answer
|
||||
* differently in COST and identically in CONTENT, which is the only
|
||||
* difference a projection door is allowed to have.
|
||||
*
|
||||
* A field absent from an entity is simply absent from that entity's map. It
|
||||
* is never an error, and never a `null` standing in for one: absent and
|
||||
* present-and-null are different answers.
|
||||
*
|
||||
* @param ids - Canonical entity ids to read.
|
||||
* @param fields - Index KEYS (bare = user metadata, `system.*` = engine
|
||||
* scalar), already address-resolved by the caller.
|
||||
* @returns `id → { field: value }` for the fields this index served exactly.
|
||||
* Ids with nothing to serve may be omitted entirely.
|
||||
*/
|
||||
getScalarsForIds?(
|
||||
ids: readonly string[],
|
||||
fields: readonly string[]
|
||||
): Promise<Map<string, Record<string, unknown>>>
|
||||
getSortedIdsForFilter(filter: any, orderBy: string, order?: 'asc' | 'desc', topK?: number): Promise<string[]>
|
||||
getFilterValues(field: string): Promise<string[]>
|
||||
getFilterFields(): Promise<string[]>
|
||||
|
|
|
|||
|
|
@ -561,33 +561,6 @@ export interface UpdateRelationParams<T = any> {
|
|||
* refusal with the fix in hand beats a silent behavior flip.
|
||||
*/
|
||||
export interface FindParams<T = any> {
|
||||
/**
|
||||
* **Field projection** — return only these fields on each row, instead of the
|
||||
* whole record.
|
||||
*
|
||||
* A list view that shows a title and a slug does not need the document body,
|
||||
* yet without a projection every row hydrates its full record and throws
|
||||
* almost all of it away. Naming the fields lets them be served from the index
|
||||
* itself: a scalar the index holds exactly is read from the index, and the
|
||||
* canonical record is opened ONLY when a requested field cannot be.
|
||||
*
|
||||
* Field names follow the one addressing law: a bare name is the user's
|
||||
* metadata (`'title'`), and `system.*` is an engine scalar
|
||||
* (`'system.createdAt'`).
|
||||
*
|
||||
* - **Absent** ⇒ the full record, exactly as before.
|
||||
* - A requested field the entity does not carry is simply **absent** from the
|
||||
* row. It is never an error — a projection asks "give me these if you have
|
||||
* them", so an optional field must not turn a list into a failure.
|
||||
* - Every returned row carries `id` (and, on `find`, `score`) regardless: a
|
||||
* row you cannot identify is not a row.
|
||||
*
|
||||
* @example
|
||||
* // A list page: two user fields and one engine scalar, no document bodies.
|
||||
* await brain.find({ where: { kind: 'post' }, fields: ['title', 'slug', 'system.createdAt'], limit: 50 })
|
||||
*/
|
||||
fields?: readonly string[]
|
||||
|
||||
// Vector Intelligence
|
||||
/** Natural language or semantic search query (embedded and matched via HNSW + text index) */
|
||||
query?: string
|
||||
|
|
@ -816,12 +789,6 @@ export interface SimilarParams<T = any> {
|
|||
* Added string ID shorthand syntax
|
||||
*/
|
||||
export interface RelatedParams {
|
||||
// NOTE: `fields` is deliberately NOT offered here. A Relation carries `from`
|
||||
// and `to` as IDS and hydrates no entity record, so there is nothing for a
|
||||
// projection to trim — the param would be decorative. Projecting the
|
||||
// ENDPOINTS would be a new capability (related() hydrating entities), not a
|
||||
// projection of an existing one, and it belongs in its own decision.
|
||||
|
||||
/**
|
||||
* Filter by source entity ID
|
||||
*
|
||||
|
|
@ -1447,33 +1414,6 @@ export interface ImportResult {
|
|||
*
|
||||
*/
|
||||
export interface GetOptions {
|
||||
/**
|
||||
* **Field projection** — return only these fields on each row, instead of the
|
||||
* whole record.
|
||||
*
|
||||
* A list view that shows a title and a slug does not need the document body,
|
||||
* yet without a projection every row hydrates its full record and throws
|
||||
* almost all of it away. Naming the fields lets them be served from the index
|
||||
* itself: a scalar the index holds exactly is read from the index, and the
|
||||
* canonical record is opened ONLY when a requested field cannot be.
|
||||
*
|
||||
* Field names follow the one addressing law: a bare name is the user's
|
||||
* metadata (`'title'`), and `system.*` is an engine scalar
|
||||
* (`'system.createdAt'`).
|
||||
*
|
||||
* - **Absent** ⇒ the full record, exactly as before.
|
||||
* - A requested field the entity does not carry is simply **absent** from the
|
||||
* row. It is never an error — a projection asks "give me these if you have
|
||||
* them", so an optional field must not turn a list into a failure.
|
||||
* - Every returned row carries `id` (and, on `find`, `score`) regardless: a
|
||||
* row you cannot identify is not a row.
|
||||
*
|
||||
* @example
|
||||
* // A list page: two user fields and one engine scalar, no document bodies.
|
||||
* await brain.find({ where: { kind: 'post' }, fields: ['title', 'slug', 'system.createdAt'], limit: 50 })
|
||||
*/
|
||||
fields?: readonly string[]
|
||||
|
||||
/**
|
||||
* Include 384-dimensional vector embeddings in the response
|
||||
*
|
||||
|
|
|
|||
|
|
@ -55,30 +55,8 @@ export enum FieldType {
|
|||
*/
|
||||
export interface FieldTypeInfo {
|
||||
field: string
|
||||
/**
|
||||
* The DOMINANT reading of the field — one type, the most specific one every
|
||||
* sampled value satisfies.
|
||||
*
|
||||
* A field is not obliged to hold one kind, so this is not the whole answer
|
||||
* for a field that holds several. Read {@link kinds} beside it: a field
|
||||
* carrying `'electronics'` and `5` infers as STRING here and reports
|
||||
* `['number', 'string']` there, and the metadata index keeps a separate
|
||||
* posting column for each of them.
|
||||
*/
|
||||
inferredType: FieldType
|
||||
confidence: number // 0-1 confidence score
|
||||
/**
|
||||
* Every value KIND observed in the sample, in the order
|
||||
* number → string → boolean. More than one entry means a genuinely
|
||||
* mixed field, and every one of those kinds is independently filterable.
|
||||
*
|
||||
* Kinds are JavaScript `typeof` classes, one level coarser than
|
||||
* {@link FieldType}: a UUID and a category name are both `'string'`, and an
|
||||
* integer and a timestamp are both `'number'`.
|
||||
*
|
||||
* Optional only for cached analyses written before this was reported.
|
||||
*/
|
||||
kinds?: Array<'number' | 'string' | 'boolean'>
|
||||
sampleSize: number // Number of values analyzed
|
||||
lastUpdated: number // Timestamp of last analysis
|
||||
detectionMethod: 'value' // Always 'value' (no fallbacks!)
|
||||
|
|
@ -155,71 +133,14 @@ export class FieldTypeInference {
|
|||
}
|
||||
|
||||
/**
|
||||
* Analyze values to determine field type, and report every KIND the field
|
||||
* actually holds alongside it.
|
||||
*
|
||||
* The classification below picks ONE type, because every one of its
|
||||
* heuristics asks `samples.every(...)`: a field carrying `'electronics'` and
|
||||
* `5` satisfies none of them and lands on STRING. That single answer is true
|
||||
* as far as it goes — string is the dominant reading — but on its own it
|
||||
* says nothing about the numbers also in the field, and a caller that treats
|
||||
* it as the field's only type reproduces the first-writer freeze the index
|
||||
* itself no longer has. {@link FieldTypeInfo.kinds} carries the rest.
|
||||
*/
|
||||
private async analyzeValues(field: string, values: any[]): Promise<FieldTypeInfo> {
|
||||
const info = await this.classifyValues(field, values)
|
||||
info.kinds = FieldTypeInference.observedKinds(values)
|
||||
if (info.kinds.length > 1 && info.metadata) {
|
||||
info.metadata.format = `${info.metadata.format} (field also holds: ${info.kinds
|
||||
.filter((k) => k !== FieldTypeInference.kindOfType(info.inferredType))
|
||||
.join(', ')})`
|
||||
}
|
||||
return info
|
||||
}
|
||||
|
||||
/**
|
||||
* The distinct value kinds present in a sample, in a stable order.
|
||||
*
|
||||
* Kinds are JavaScript `typeof` classes — the same classes the metadata
|
||||
* index keeps separate posting columns for — not the finer
|
||||
* {@link FieldType} readings, which are interpretations layered on top of
|
||||
* them (a UUID and a category name are both the `string` kind).
|
||||
*/
|
||||
private static observedKinds(values: any[]): Array<'number' | 'string' | 'boolean'> {
|
||||
const order: Array<'number' | 'string' | 'boolean'> = ['number', 'string', 'boolean']
|
||||
const seen = new Set<'number' | 'string' | 'boolean'>()
|
||||
for (const v of values) {
|
||||
if (v === null || v === undefined) continue
|
||||
const t = typeof v
|
||||
seen.add(t === 'number' ? 'number' : t === 'boolean' ? 'boolean' : 'string')
|
||||
}
|
||||
return order.filter((k) => seen.has(k))
|
||||
}
|
||||
|
||||
/** The value kind a {@link FieldType} reading is an interpretation of. */
|
||||
private static kindOfType(type: FieldType): 'number' | 'string' | 'boolean' {
|
||||
switch (type) {
|
||||
case FieldType.BOOLEAN:
|
||||
return 'boolean'
|
||||
case FieldType.INTEGER:
|
||||
case FieldType.FLOAT:
|
||||
case FieldType.TIMESTAMP_MS:
|
||||
case FieldType.TIMESTAMP_S:
|
||||
return 'number'
|
||||
default:
|
||||
return 'string'
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify values into a single field type.
|
||||
* Analyze values to determine field type
|
||||
*
|
||||
* Uses DuckDB-inspired type detection order:
|
||||
* BOOLEAN → INTEGER → FLOAT → DATE → TIMESTAMP → UUID → STRING
|
||||
*
|
||||
* No fallbacks - pure value-based detection
|
||||
*/
|
||||
private async classifyValues(field: string, values: any[]): Promise<FieldTypeInfo> {
|
||||
private async analyzeValues(field: string, values: any[]): Promise<FieldTypeInfo> {
|
||||
// Filter null/undefined values
|
||||
const validValues = values.filter(v => v !== null && v !== undefined)
|
||||
|
||||
|
|
|
|||
|
|
@ -2930,67 +2930,6 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
|||
return order === 'asc' ? comparison : -comparison
|
||||
}
|
||||
|
||||
/**
|
||||
* Read named scalar fields for many ids from the COLUMN STORE, without
|
||||
* touching the canonical record — the `find({ fields })` door.
|
||||
*
|
||||
* ## Why the column store and not the sparse index
|
||||
*
|
||||
* The column store keeps RAW values; the sparse index keeps a normalized,
|
||||
* bucketed form built for range queries — `system.createdAt` is indexed at
|
||||
* minute precision there. A projection served from the sparse index would
|
||||
* hand back a value that differs from the record's, which is a wrong answer
|
||||
* nobody can see. So this door reads the column store, and a field the
|
||||
* column store does not hold is OMITTED rather than approximated.
|
||||
*
|
||||
* ## Why batched
|
||||
*
|
||||
* `getFieldValueForEntity` answers one (id, field) pair by walking the
|
||||
* field's storage; called per row it re-walks the same column for every id.
|
||||
* This walks each column ONCE and picks out every requested id as it passes:
|
||||
* O(fields x column) instead of O(ids x fields x column).
|
||||
*
|
||||
* Omission is always safe — it costs the caller a record read. The caller
|
||||
* diffs what it asked for against what came back and reads records for the
|
||||
* remainder, so an index that can serve nothing is slow, never wrong.
|
||||
*
|
||||
* @param ids - Canonical entity ids.
|
||||
* @param fields - Index keys (bare = user metadata, `system.*` = engine scalar).
|
||||
* @returns `id -> { field: value }` for exactly the pairs this index served.
|
||||
*/
|
||||
async getScalarsForIds(
|
||||
ids: readonly string[],
|
||||
fields: readonly string[]
|
||||
): Promise<Map<string, Record<string, unknown>>> {
|
||||
const out = new Map<string, Record<string, unknown>>()
|
||||
if (ids.length === 0 || fields.length === 0) return out
|
||||
|
||||
// int -> id, so a column hit resolves back to the caller's id. An id the
|
||||
// mapper does not know cannot be in any column, so it is simply absent.
|
||||
const idByInt = new Map<number, string>()
|
||||
for (const id of ids) {
|
||||
const intId = this.idMapper.getInt(id)
|
||||
if (intId !== undefined) idByInt.set(intId, id)
|
||||
}
|
||||
if (idByInt.size === 0) return out
|
||||
|
||||
for (const field of fields) {
|
||||
if (!this.columnStore.hasField(field)) continue
|
||||
const values = await this.columnStore.valuesForIds(field, idByInt.keys())
|
||||
for (const [intId, value] of values) {
|
||||
const id = idByInt.get(intId)
|
||||
if (id === undefined) continue
|
||||
let row = out.get(id)
|
||||
if (row === undefined) {
|
||||
row = {}
|
||||
out.set(id, row)
|
||||
}
|
||||
row[field] = value
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
async getFieldValueForEntity(entityId: string, field: string): Promise<any> {
|
||||
// `field` arrives as a FROZEN INDEX KEY (bare = user metadata;
|
||||
// 'system.<field>' = engine scalar). Storage fallbacks read the matching
|
||||
|
|
|
|||
|
|
@ -57,14 +57,7 @@ export default defineConfig({
|
|||
// otherwise-correctness integration suite (self-skipped everywhere
|
||||
// else via BRAINY_PERF_LANE). Stays in the integration gate's
|
||||
// include too, so every OTHER test in the file keeps running there.
|
||||
'tests/integration/storage-batch-operations.test.ts',
|
||||
// Same pattern: one wall-clock budget case (100-file write + readdir,
|
||||
// 5.5s budget) inside an otherwise-correctness VFS unit suite
|
||||
// (self-skipped everywhere else via BRAINY_PERF_LANE — see
|
||||
// tests/vfs/vfs.unit.test.ts's 'Performance > should handle many
|
||||
// files efficiently'). Stays in the unit gate's *.unit.test.ts match
|
||||
// too, so every OTHER test in the file keeps running there.
|
||||
'tests/vfs/vfs.unit.test.ts'
|
||||
'tests/integration/storage-batch-operations.test.ts'
|
||||
],
|
||||
|
||||
reporters: process.env.CI ? ['dot'] : ['basic'],
|
||||
|
|
|
|||
|
|
@ -34,10 +34,6 @@ describe('API Parameter Validation', () => {
|
|||
})
|
||||
})
|
||||
|
||||
afterAll(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('should use "where" parameter for metadata filtering', async () => {
|
||||
const results = await brain.find({
|
||||
where: { category: 'test-category' },
|
||||
|
|
|
|||
|
|
@ -1,309 +0,0 @@
|
|||
/**
|
||||
* @module tests/integration/beforeexit-never-closes
|
||||
* @description A DRAINED EVENT LOOP IS NOT A SHUTDOWN.
|
||||
*
|
||||
* MEASURED on the 11.1 rehearsal lane, against a copy of a real store. The
|
||||
* `beforeExit` listener had been wired to the SIGNAL path — the path whose job
|
||||
* is to `close()` every live brain — so after the heal phase the log printed
|
||||
*
|
||||
* "Shutdown signal received - flushing pending data..."
|
||||
* "Flushed successfully (1 instance)"
|
||||
*
|
||||
* with no signal ever sent, and the script's very next `add()` threw
|
||||
*
|
||||
* "Brainy instance is not initialized: it was closed via close().
|
||||
* Create a new instance."
|
||||
*
|
||||
* Node emits `'beforeExit'` whenever the event loop has no REF'd work left.
|
||||
* That is not "the process is ending" — it is a state a perfectly healthy
|
||||
* script reaches, because this engine unref's its idle and cadence timers
|
||||
* ("an idle brain costs nothing"), so a script awaiting anything those timers
|
||||
* drive is, for that instant, a process with no ref'd work and an open brain.
|
||||
* The engine closed a live brain out from under a running script.
|
||||
*
|
||||
* The contract pinned here:
|
||||
* (1) `'beforeExit'` firing while a brain is open closes NOTHING: the brain
|
||||
* is still open, `add()` and `find()` still work, the writer lock is
|
||||
* still held, and the process still exits 0 on its own afterwards.
|
||||
* (2) The pass DOES persist derived state — a non-closing `flush()` ran —
|
||||
* and it wrote no clean-shutdown marker and no clean-close record: those
|
||||
* are `close()`'s word about itself, and no close happened.
|
||||
* (3) The signal path is untouched: SIGTERM still closes through `close()`
|
||||
* (pinned by tests/integration/shutdown-single-owner.test.ts, re-run
|
||||
* with this change).
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { mkdtempSync, rmSync, existsSync, readFileSync, writeFileSync } from 'node:fs'
|
||||
import { spawn } from 'node:child_process'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { NounType } from '../../src/types/graphTypes.js'
|
||||
|
||||
const REPO_ROOT = process.cwd()
|
||||
const TSX = join(REPO_ROOT, 'node_modules', '.bin', 'tsx')
|
||||
const BRAINY_SRC = join(REPO_ROOT, 'src', 'brainy.ts')
|
||||
|
||||
function makeTempDir(prefix: string): string {
|
||||
return mkdtempSync(join(tmpdir(), prefix))
|
||||
}
|
||||
|
||||
/** The writer lock itself — present for as long as this process owns the store. */
|
||||
const writerLockPath = (dir: string) => join(dir, 'locks', '_writer.lock')
|
||||
/** The clean-close record — written by `releaseWriterLock()`, i.e. by close(). */
|
||||
const closeRecordPath = (dir: string) => join(dir, 'locks', '_writer.close')
|
||||
/**
|
||||
* The generation store's clean-shutdown marker — written by
|
||||
* `generationStore.close()` alone, reached only from `close()`. (Raw objects
|
||||
* are gzipped on disk, so both spellings are accepted.)
|
||||
*/
|
||||
const cleanShutdownWritten = (dir: string) =>
|
||||
existsSync(join(dir, '_system', 'clean-shutdown.json.gz')) ||
|
||||
existsSync(join(dir, '_system', 'clean-shutdown.json'))
|
||||
|
||||
/**
|
||||
* Write a child script and run it under tsx to completion, collecting stdout
|
||||
* and stderr and the exit code. (A file, not `tsx -e`: the eval form compiles
|
||||
* to CommonJS, which has no top-level await.)
|
||||
*/
|
||||
function runChild(
|
||||
scriptDir: string,
|
||||
body: string
|
||||
): Promise<{ code: number | null; out: string }> {
|
||||
const scriptPath = join(scriptDir, 'child-process.mts')
|
||||
writeFileSync(scriptPath, body)
|
||||
// The child is an ORDINARY consumer process, so it runs the real embedding
|
||||
// pipeline: this suite's deterministic-embedder switch is inherited through
|
||||
// the environment, and under it `find()` self-retrieval returns nothing —
|
||||
// which would make the read half of this pin vacuous. (That property is the
|
||||
// deterministic embedder's, not this change's: it reproduces in a plain
|
||||
// script with no 'beforeExit' involved.)
|
||||
const env = { ...process.env }
|
||||
delete env.BRAINY_DETERMINISTIC_EMBEDDINGS
|
||||
const child = spawn(TSX, [scriptPath], {
|
||||
cwd: REPO_ROOT,
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
env
|
||||
})
|
||||
let out = ''
|
||||
child.stdout?.on('data', (d) => { out += String(d) })
|
||||
child.stderr?.on('data', (d) => { out += String(d) })
|
||||
return new Promise((resolvePromise) => {
|
||||
child.on('exit', (code) => resolvePromise({ code, out }))
|
||||
})
|
||||
}
|
||||
|
||||
describe('beforeExit never closes a live brain', () => {
|
||||
let dir: string
|
||||
let scriptDir: string
|
||||
let resultPath: string
|
||||
|
||||
beforeEach(() => {
|
||||
dir = makeTempDir('brainy-beforeexit-')
|
||||
scriptDir = makeTempDir('brainy-beforeexit-script-')
|
||||
resultPath = join(scriptDir, 'result.json')
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
for (const d of [dir, scriptDir]) {
|
||||
try { rmSync(d, { recursive: true, force: true }) } catch { /* ignore */ }
|
||||
}
|
||||
})
|
||||
|
||||
it('(1)+(2) a drained event loop flushes, closes nothing, and the script keeps working', async () => {
|
||||
/**
|
||||
* THE DRAIN, and why the script survives it. The script awaits a promise
|
||||
* that only an UNREF'd timer will resolve — the shape every engine cadence
|
||||
* timer has, and the reason a healthy script reaches a loop with no ref'd
|
||||
* work. Node emits `'beforeExit'` there, with the brain wide open.
|
||||
*
|
||||
* The engine's listener runs first (registered by `init()`, before the
|
||||
* script's). The script's own listener is both its witness — it records
|
||||
* that the emit happened, and the flush count AT that moment — and its
|
||||
* belt: it resolves the same promise, so the pin never depends on how many
|
||||
* milliseconds the engine's pass happens to keep the loop turning.
|
||||
*
|
||||
* The brain is DIRTY at the drain (one add, after a settling flush), so
|
||||
* the pass has real work to do and pin (2) is about a flush that ran, not
|
||||
* a flush that was skipped as a no-op.
|
||||
*/
|
||||
const script = `
|
||||
import { writeFileSync as __writeFileSync, existsSync as __existsSync } from 'node:fs'
|
||||
import { join as __join } from 'node:path'
|
||||
import { Brainy } from ${JSON.stringify(BRAINY_SRC)}
|
||||
|
||||
const DIR = ${JSON.stringify(dir)}
|
||||
const brain = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: DIR } })
|
||||
await brain.init()
|
||||
|
||||
// Count every flush that RUNS on this brain. An own property shadows the
|
||||
// prototype for every caller, including the engine's own listeners.
|
||||
let flushes = 0
|
||||
const flushImpl = brain.flush.bind(brain)
|
||||
brain.flush = () => { flushes++; return flushImpl() }
|
||||
// ...and every close ENTERED. This must still be 0 after the drain.
|
||||
let closes = 0
|
||||
const closeImpl = brain.close.bind(brain)
|
||||
brain.close = () => { closes++; return closeImpl() }
|
||||
|
||||
await brain.add({ data: 'written before the drain', type: 'concept' })
|
||||
await brain.flush() // settle: clean brain
|
||||
await new Promise((r) => setTimeout(r, 250)) // let the cadence quiet down
|
||||
await brain.add({ data: 'the write the drain must persist', type: 'concept' })
|
||||
|
||||
const flushesBeforeDrain = flushes
|
||||
let drains = 0
|
||||
let flushesAtDrain = -1
|
||||
const drained = new Promise((resolve) => {
|
||||
const t = setTimeout(resolve, 5)
|
||||
if (typeof t.unref === 'function') t.unref()
|
||||
process.on('beforeExit', () => {
|
||||
drains++
|
||||
if (flushesAtDrain === -1) flushesAtDrain = flushes
|
||||
resolve()
|
||||
})
|
||||
})
|
||||
await drained
|
||||
|
||||
// GIVE THE ENGINE'S PASS ITS FULL TURN before judging it. The signal
|
||||
// path this listener used to share defers one macrotask before it
|
||||
// touches an instance, so a script that resumes on the same tick as the
|
||||
// emit would race past the damage and see an open brain that is about to
|
||||
// be closed underneath it. Wait it out (a ref'd timer — the drain has
|
||||
// already happened), then look.
|
||||
await new Promise((r) => setTimeout(r, 1000))
|
||||
|
||||
// ---- The script is still running. The brain must still be its brain. ----
|
||||
const stateAtResume = {
|
||||
drains,
|
||||
flushesBeforeDrain,
|
||||
flushesAtDrain,
|
||||
closes,
|
||||
isClosed: brain.isClosed,
|
||||
isClosing: brain.isClosing,
|
||||
writerLockHeld: __existsSync(__join(DIR, 'locks', '_writer.lock')),
|
||||
cleanCloseRecord: __existsSync(__join(DIR, 'locks', '_writer.close')),
|
||||
cleanShutdownMarker:
|
||||
__existsSync(__join(DIR, '_system', 'clean-shutdown.json.gz')) ||
|
||||
__existsSync(__join(DIR, '_system', 'clean-shutdown.json'))
|
||||
}
|
||||
|
||||
let addAfterDrain = null
|
||||
let addError = null
|
||||
try {
|
||||
addAfterDrain = await brain.add({ data: 'written AFTER the drained event loop', type: 'concept' })
|
||||
} catch (error) {
|
||||
addError = error instanceof Error ? error.message : String(error)
|
||||
}
|
||||
|
||||
let findHits = -1
|
||||
let findError = null
|
||||
try {
|
||||
const results = await brain.find('written AFTER the drained event loop')
|
||||
findHits = results.length
|
||||
} catch (error) {
|
||||
findError = error instanceof Error ? error.message : String(error)
|
||||
}
|
||||
|
||||
__writeFileSync(
|
||||
${JSON.stringify(resultPath)},
|
||||
JSON.stringify({ ...stateAtResume, addAfterDrain, addError, findHits, findError, closesBeforeOurs: closes })
|
||||
)
|
||||
|
||||
// The script ends the way a script ends: it closes its own brain, and
|
||||
// the process exits on its own because nothing is left holding the loop.
|
||||
await brain.close()
|
||||
`
|
||||
|
||||
const { code, out } = await runChild(scriptDir, script)
|
||||
|
||||
expect(existsSync(resultPath), `child wrote no result file:\n${out}`).toBe(true)
|
||||
const r = JSON.parse(readFileSync(resultPath, 'utf-8'))
|
||||
|
||||
// The drain really happened — this test proves nothing otherwise.
|
||||
expect(r.drains, `'beforeExit' never fired:\n${out}`).toBeGreaterThanOrEqual(1)
|
||||
|
||||
// (1) NOTHING WAS CLOSED. This is the regression: under 10.4.11 the pass
|
||||
// ran close() here and `addError` carried "it was closed via close()".
|
||||
expect(r.addError, `add() after the drain failed:\n${out}`).toBeNull()
|
||||
expect(r.findError, `find() after the drain failed:\n${out}`).toBeNull()
|
||||
expect(r.closes, 'the engine closed the brain on a drained event loop').toBe(0)
|
||||
expect(r.isClosed).toBe(false)
|
||||
expect(r.isClosing).toBe(false)
|
||||
expect(typeof r.addAfterDrain).toBe('string')
|
||||
expect(r.findHits, `find() returned nothing:\n${out}`).toBeGreaterThanOrEqual(1)
|
||||
|
||||
// (1) The writer lock was never given up — a drained loop is not a handover.
|
||||
expect(r.writerLockHeld, 'the writer lock was released on a drained event loop').toBe(true)
|
||||
|
||||
// (2) A flush RAN, and it wrote neither of close()'s markers.
|
||||
expect(
|
||||
r.flushesAtDrain,
|
||||
`the drained-loop pass ran no flush (before=${r.flushesBeforeDrain}):\n${out}`
|
||||
).toBeGreaterThan(r.flushesBeforeDrain)
|
||||
expect(r.cleanShutdownMarker, 'the drained-loop flush stamped a clean-shutdown marker').toBe(false)
|
||||
expect(r.cleanCloseRecord, 'the drained-loop flush wrote a clean-close record').toBe(false)
|
||||
expect(out).toMatch(/All indexes flushed to disk/)
|
||||
|
||||
// The narration says what happened, and never claims a shutdown.
|
||||
expect(out).toMatch(/event loop drained with 1 brain open/)
|
||||
expect(out).toMatch(/NOTHING was closed\. A drained loop is not a shutdown/)
|
||||
expect(out).not.toMatch(/Shutdown signal received/)
|
||||
expect(out).not.toMatch(/Flushed successfully/)
|
||||
expect(out).not.toMatch(/is not initialized/)
|
||||
|
||||
// (1) And the process still exits 0 on its own once the script closes up.
|
||||
expect(code, `child output:\n${out}`).toBe(0)
|
||||
|
||||
// The store the script left behind is clean: it closed properly at the end.
|
||||
expect(cleanShutdownWritten(dir), 'the script\'s own close() wrote no marker').toBe(true)
|
||||
expect(existsSync(closeRecordPath(dir)), 'the script\'s own close() left no clean-close record').toBe(true)
|
||||
expect(existsSync(writerLockPath(dir)), 'the writer lock outlived close()').toBe(false)
|
||||
}, 300_000)
|
||||
|
||||
it('(2) the pass is repeatable and idempotent: a second drain closes nothing either', async () => {
|
||||
// In-process, so the assertions are on the object itself rather than on a
|
||||
// report: 'beforeExit' is an ordinary event, and emitting it twice must
|
||||
// leave the brain exactly as usable as it was.
|
||||
const brain = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } })
|
||||
await brain.init()
|
||||
|
||||
const flushed: Promise<void>[] = []
|
||||
const flushImpl = brain.flush.bind(brain)
|
||||
;(brain as unknown as { flush: () => Promise<void> }).flush = () => {
|
||||
const p = flushImpl()
|
||||
flushed.push(p)
|
||||
return p
|
||||
}
|
||||
|
||||
await brain.add({ data: 'a write the drained loop must persist', type: NounType.Concept })
|
||||
|
||||
for (const pass of [1, 2]) {
|
||||
const before = flushed.length
|
||||
process.emit('beforeExit', 0)
|
||||
await Promise.all(flushed.slice(before).map((p) => p.catch(() => {})))
|
||||
// Let the pass's own `finally` run (it settles a microtask after ours),
|
||||
// so the next emit is not turned away by the in-flight guard.
|
||||
await new Promise((r) => setTimeout(r, 50))
|
||||
|
||||
expect(brain.isClosed, `pass ${pass} closed the brain`).toBe(false)
|
||||
expect(brain.isClosing, `pass ${pass} started a close`).toBe(false)
|
||||
expect(existsSync(writerLockPath(dir)), `pass ${pass} released the writer lock`).toBe(true)
|
||||
expect(existsSync(closeRecordPath(dir)), `pass ${pass} wrote a clean-close record`).toBe(false)
|
||||
expect(cleanShutdownWritten(dir), `pass ${pass} stamped a clean-shutdown marker`).toBe(false)
|
||||
|
||||
// Still a working brain, after every pass.
|
||||
const id = await brain.add({ data: `still writable after drain ${pass}`, type: NounType.Concept })
|
||||
expect(id).toBeTruthy()
|
||||
}
|
||||
|
||||
// The first pass had a dirty brain and flushed it; the second found it
|
||||
// clean and cost nothing. Either way, neither closed anything.
|
||||
expect(flushed.length).toBeGreaterThanOrEqual(2)
|
||||
|
||||
await brain.close()
|
||||
expect(brain.isClosed).toBe(true)
|
||||
expect(cleanShutdownWritten(dir)).toBe(true)
|
||||
}, 300_000)
|
||||
})
|
||||
|
|
@ -7,7 +7,7 @@
|
|||
* - Backward compatibility preserved
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { NounType } from '../../src/types/graphTypes.js'
|
||||
|
||||
|
|
@ -19,10 +19,6 @@ describe('Entity Confidence & Weight Exposure', () => {
|
|||
await brain.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
describe('Entity interface', () => {
|
||||
it('should expose confidence when adding entity with confidence', async () => {
|
||||
const id = await brain.add({
|
||||
|
|
|
|||
|
|
@ -67,13 +67,6 @@ describe('find({ connected }) is graph-first: neighbours → filter → page', (
|
|||
})
|
||||
|
||||
afterAll(async () => {
|
||||
// CLOSE IT. Dropping the reference does not close a brain — it only makes
|
||||
// it unreachable from here. The instance stays open and registered, its
|
||||
// unref'd cadence timer keeps running, and because the gate config runs the
|
||||
// whole suite in ONE process (pool: 'forks', singleFork: true) it goes on
|
||||
// narrating its flushes into every test file that runs after this one.
|
||||
// A test that leaks a brain is a defect of the test.
|
||||
await brain?.close()
|
||||
brain = null as any
|
||||
})
|
||||
|
||||
|
|
@ -144,34 +137,13 @@ describe('find({ connected }) is graph-first: neighbours → filter → page', (
|
|||
})
|
||||
|
||||
it('walks the vector leg over the neighbours only', async () => {
|
||||
// The SAME query without the vector leg, first. Both legs draw from the
|
||||
// one neighbour set, so this is the control: it says whether a short answer
|
||||
// came from the adjacency/filter (both legs short) or from the vector walk
|
||||
// alone (only the vector leg short). Cheap, and it turns a bare count
|
||||
// mismatch into a named half — this case has gone red on the gate box
|
||||
// while passing in isolation and beside its own predecessor, so the next
|
||||
// red must arrive already carrying the half it belongs to.
|
||||
const control = await brain.find({
|
||||
connected: { from: anchor, direction: 'out' },
|
||||
where: { kind: 'note' },
|
||||
limit: 5
|
||||
})
|
||||
|
||||
const results = await brain.find({
|
||||
vector: sharedVector,
|
||||
connected: { from: anchor, direction: 'out' },
|
||||
where: { kind: 'note' },
|
||||
limit: 5
|
||||
})
|
||||
|
||||
expect(
|
||||
results.length,
|
||||
`the vector leg returned ${results.length} of a requested 5. The same query ` +
|
||||
`WITHOUT the vector returned ${control.length}: if that is also short the ` +
|
||||
`neighbour set or the filter is the cause, and if it is 5 the vector walk is — ` +
|
||||
`note every row in this corpus carries an identical vector, so the walk is ` +
|
||||
`ranking an exact tie.`
|
||||
).toBe(5)
|
||||
expect(results).toHaveLength(5)
|
||||
for (const r of results) expect(neighbourIds.has(r.entity.id)).toBe(true)
|
||||
})
|
||||
|
||||
|
|
|
|||
|
|
@ -1,265 +0,0 @@
|
|||
/**
|
||||
* @module tests/integration/find-fields-projection
|
||||
* @description **Field projection** — `find/get({ fields })` returns only the
|
||||
* named fields, and serves them from the index when it can.
|
||||
*
|
||||
* A list view that shows a title and a slug does not need the document body,
|
||||
* yet without a projection every row hydrates its whole record and discards
|
||||
* almost all of it. These pins hold the two halves of the fix:
|
||||
*
|
||||
* **The answer.** A projected row is a SUBSET of the full row — for every
|
||||
* requested field, the projected value equals the value the same query returns
|
||||
* unprojected. Absent `fields` is byte-identical to today. A requested field the
|
||||
* entity does not carry is simply absent, never an error. `system.*` resolves to
|
||||
* the engine scalar, a bare name to the user's metadata.
|
||||
*
|
||||
* **The cost.** When every requested field is index-served, the canonical
|
||||
* record is never opened — asserted by counting reads, not by timing them, so
|
||||
* it cannot flake into a false green. When one requested field is NOT
|
||||
* index-served (a body field, or a bucketed timestamp), exactly the owing rows
|
||||
* are read and the rest are still served from the index.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy'
|
||||
import { NounType } from '../../src/types/graphTypes'
|
||||
import { generateTestVector } from '../helpers/test-factory'
|
||||
|
||||
/** Rows carrying a title, a slug, and a large body nobody wants in a list. */
|
||||
const ROWS = 12
|
||||
const BODY = 'x'.repeat(4096)
|
||||
|
||||
describe('find/get({ fields }) — projection', () => {
|
||||
let brain: Brainy<any>
|
||||
const ids: string[] = []
|
||||
|
||||
beforeAll(async () => {
|
||||
brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' } })
|
||||
await brain.init()
|
||||
for (let i = 0; i < ROWS; i++) {
|
||||
ids.push(
|
||||
await brain.add({
|
||||
id: `post-${i}`,
|
||||
data: `post ${i}`,
|
||||
type: NounType.Thing,
|
||||
metadata: {
|
||||
kind: 'post',
|
||||
title: `Title ${i}`,
|
||||
slug: `slug-${i}`,
|
||||
rank: i,
|
||||
body: BODY,
|
||||
// Only some rows carry this, so "missing is absent" is exercised
|
||||
// by real data rather than by a name nothing ever had.
|
||||
...(i % 2 === 0 ? { featured: true } : {})
|
||||
},
|
||||
vector: generateTestVector()
|
||||
})
|
||||
)
|
||||
}
|
||||
// Persist so the column store holds the values a projection reads from.
|
||||
await brain.flush()
|
||||
})
|
||||
|
||||
afterAll(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
/** Count canonical record reads for one call. */
|
||||
const countingReads = async <R>(body: () => Promise<R>): Promise<{ out: R; reads: number }> => {
|
||||
const spy = vi.spyOn(brain as any, 'batchGet')
|
||||
try {
|
||||
const out = await body()
|
||||
const reads = spy.mock.calls.reduce(
|
||||
(n, call) => n + ((call[0] as string[] | undefined)?.length ?? 0),
|
||||
0
|
||||
)
|
||||
return { out, reads }
|
||||
} finally {
|
||||
spy.mockRestore()
|
||||
}
|
||||
}
|
||||
|
||||
it('absent fields is byte-identical to today', async () => {
|
||||
const params = { where: { kind: 'post' }, limit: 5 }
|
||||
const a = await brain.find({ ...params })
|
||||
const b = await brain.find({ ...params, fields: undefined })
|
||||
expect(JSON.stringify(b)).toBe(JSON.stringify(a))
|
||||
})
|
||||
|
||||
it('a projected row is a SUBSET of the full row, field for field', async () => {
|
||||
const shapes: Array<Record<string, unknown>> = [
|
||||
{ where: { kind: 'post' }, limit: 6 },
|
||||
{ where: { kind: 'post' }, limit: 6, offset: 3 },
|
||||
{ where: { kind: 'post' }, orderBy: 'rank', order: 'asc', limit: 6 },
|
||||
{ where: { kind: 'post' }, orderBy: 'rank', order: 'desc', limit: 4 }
|
||||
]
|
||||
for (const shape of shapes) {
|
||||
const full = await brain.find(shape as never)
|
||||
const projected = await brain.find({ ...shape, fields: ['title', 'slug'] } as never)
|
||||
expect(projected.map((r) => r.id), JSON.stringify(shape)).toEqual(full.map((r) => r.id))
|
||||
for (let i = 0; i < full.length; i++) {
|
||||
const fullMeta = (full[i].entity.metadata ?? {}) as Record<string, unknown>
|
||||
const projMeta = (projected[i].entity.metadata ?? {}) as Record<string, unknown>
|
||||
expect(projMeta.title, `${JSON.stringify(shape)} row ${i}`).toEqual(fullMeta.title)
|
||||
expect(projMeta.slug).toEqual(fullMeta.slug)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
it('returns ONLY the named fields — the body never rides along', async () => {
|
||||
const rows = await brain.find({ where: { kind: 'post' }, fields: ['title'], limit: 4 })
|
||||
expect(rows).toHaveLength(4)
|
||||
for (const r of rows) {
|
||||
const meta = (r.entity.metadata ?? {}) as Record<string, unknown>
|
||||
expect(Object.keys(meta)).toEqual(['title'])
|
||||
expect(meta.body).toBeUndefined()
|
||||
// Identity always survives a projection: a row you cannot identify is
|
||||
// not a row.
|
||||
expect(typeof r.id).toBe('string')
|
||||
expect(r.entity.id).toBe(r.id)
|
||||
}
|
||||
})
|
||||
|
||||
it('a missing field is simply ABSENT — never an error', async () => {
|
||||
// `featured` exists on half the rows; `no-such-field` on none. Neither
|
||||
// throws, and neither appears as an explicit undefined.
|
||||
const rows = await brain.find({
|
||||
where: { kind: 'post' },
|
||||
fields: ['title', 'featured', 'no-such-field'],
|
||||
limit: ROWS
|
||||
})
|
||||
expect(rows.length).toBeGreaterThan(0)
|
||||
let withFeatured = 0
|
||||
for (const r of rows) {
|
||||
const meta = (r.entity.metadata ?? {}) as Record<string, unknown>
|
||||
expect('no-such-field' in meta).toBe(false)
|
||||
if ('featured' in meta) withFeatured += 1
|
||||
}
|
||||
// Real data, not a name nothing ever had: some rows carry it, some do not.
|
||||
expect(withFeatured).toBeGreaterThan(0)
|
||||
expect(withFeatured).toBeLessThan(rows.length)
|
||||
})
|
||||
|
||||
it('a strict address resolver is NOT on this path', async () => {
|
||||
// orderBy throws UnresolvableFieldError for an unknown user key, because a
|
||||
// typo there silently changes the order. A projection must not inherit that
|
||||
// strictness: the honest answer to "give me this if you have it" is silence.
|
||||
await expect(
|
||||
brain.find({ where: { kind: 'post' }, fields: ['definitely-not-a-field'], limit: 2 })
|
||||
).resolves.toBeInstanceOf(Array)
|
||||
})
|
||||
|
||||
it('system.* resolves to the engine scalar, a bare name to user metadata', async () => {
|
||||
const full = await brain.find({ where: { kind: 'post' }, limit: 3 })
|
||||
const rows = await brain.find({
|
||||
where: { kind: 'post' },
|
||||
fields: ['system.createdAt', 'title'],
|
||||
limit: 3
|
||||
})
|
||||
for (let i = 0; i < rows.length; i++) {
|
||||
expect((rows[i].entity as any).createdAt).toEqual((full[i].entity as any).createdAt)
|
||||
const meta = (rows[i].entity.metadata ?? {}) as Record<string, unknown>
|
||||
expect(meta.title).toEqual((full[i].entity.metadata as any).title)
|
||||
// The engine scalar lands at the top level, not in the metadata bag —
|
||||
// the two address spaces never shadow each other.
|
||||
expect('system.createdAt' in meta).toBe(false)
|
||||
expect('createdAt' in meta).toBe(false)
|
||||
}
|
||||
})
|
||||
|
||||
it('reads NO canonical record when every requested field is index-served', async () => {
|
||||
// The cost pin, counted rather than timed. `title` and `slug` are ordinary
|
||||
// indexed user fields, so the index can serve them exactly.
|
||||
const { out, reads } = await countingReads(() =>
|
||||
brain.find({ where: { kind: 'post' }, fields: ['title', 'slug'], limit: ROWS })
|
||||
)
|
||||
expect(out.length).toBeGreaterThan(0)
|
||||
expect(reads).toBe(0)
|
||||
})
|
||||
|
||||
it('reads records only for the fields the column cannot serve', async () => {
|
||||
// `system.data` is NOT a column the store holds (verified against
|
||||
// getIndexedFields), so the record must be opened for it — while `title`,
|
||||
// which the column does hold, still comes from the index.
|
||||
const { out, reads } = await countingReads(() =>
|
||||
brain.find({ where: { kind: 'post' }, fields: ['title', 'system.data'], limit: 4 })
|
||||
)
|
||||
expect(out).toHaveLength(4)
|
||||
expect(reads).toBe(4)
|
||||
for (const r of out) {
|
||||
const meta = (r.entity.metadata ?? {}) as Record<string, unknown>
|
||||
expect(Object.keys(meta)).toEqual(['title'])
|
||||
expect(typeof (r.entity as any).data).toBe('string')
|
||||
}
|
||||
})
|
||||
|
||||
it('a large field the column DOES hold costs no record read', async () => {
|
||||
// Worth pinning because it is the venue case: the body is column-served on
|
||||
// this engine, so a list that projects around it pays nothing for it, and
|
||||
// a list that projects it still pays no record read.
|
||||
const { reads } = await countingReads(() =>
|
||||
brain.find({ where: { kind: 'post' }, fields: ['body'], limit: 4 })
|
||||
)
|
||||
expect(reads).toBe(0)
|
||||
})
|
||||
|
||||
it('projects a vector-leg find too — the ANSWER is uniform, only the cost is not', async () => {
|
||||
// The seam hydrates the metadata and graph page paths. A vector or text leg
|
||||
// builds its own entities, so those rows are trimmed after the integrity
|
||||
// guard instead. That difference is a COST difference, and this pin exists
|
||||
// so it can never quietly become an ANSWER difference.
|
||||
const rows = await brain.find({ query: 'post', fields: ['title'], limit: 3 })
|
||||
for (const r of rows) {
|
||||
const meta = (r.entity.metadata ?? {}) as Record<string, unknown>
|
||||
expect(Object.keys(meta)).toEqual(['title'])
|
||||
expect(meta.body).toBeUndefined()
|
||||
expect(r.entity.id).toBe(r.id)
|
||||
}
|
||||
})
|
||||
|
||||
it('get({ fields }) projects a single row through the same seam', async () => {
|
||||
const full = await brain.get(ids[0])
|
||||
const projected = await brain.get(ids[0], { fields: ['title', 'slug'] })
|
||||
expect(projected).not.toBeNull()
|
||||
expect(projected!.id).toBe(full!.id)
|
||||
const fullMeta = (full!.metadata ?? {}) as Record<string, unknown>
|
||||
const projMeta = (projected!.metadata ?? {}) as Record<string, unknown>
|
||||
expect(projMeta.title).toEqual(fullMeta.title)
|
||||
expect(projMeta.slug).toEqual(fullMeta.slug)
|
||||
expect(Object.keys(projMeta).sort()).toEqual(['slug', 'title'])
|
||||
expect((projected as any).body).toBeUndefined()
|
||||
})
|
||||
|
||||
it('get({ fields }) reads no record when the index serves the fields', async () => {
|
||||
const { reads } = await countingReads(() => brain.get(ids[1], { fields: ['title'] }))
|
||||
expect(reads).toBe(0)
|
||||
})
|
||||
|
||||
it('the door serves EXACT values — the column, never the bucketed index', async () => {
|
||||
// The sparse index buckets `system.createdAt` to the minute for range
|
||||
// queries; the column store keeps raw ms. Serving a projection from the
|
||||
// former would hand back a value that differs from the record's, so the
|
||||
// door reads the column — and this pin is what proves which one it read.
|
||||
const index = (brain as any).metadataIndex
|
||||
const sample = ids.slice(0, 3)
|
||||
const served = await index.getScalarsForIds(sample, ['title', 'system.createdAt'])
|
||||
expect(served.size).toBe(sample.length)
|
||||
for (const id of sample) {
|
||||
const row = served.get(id)!
|
||||
const record = await brain.get(id)
|
||||
expect(row.title).toEqual((record!.metadata as any).title)
|
||||
// Exact to the millisecond — a bucketed value would be rounded down to
|
||||
// the minute and this would fail.
|
||||
expect(row['system.createdAt']).toEqual((record as any).createdAt)
|
||||
}
|
||||
})
|
||||
|
||||
it('a field the column store does not hold is OMITTED, not approximated', async () => {
|
||||
const index = (brain as any).metadataIndex
|
||||
const served = await index.getScalarsForIds(ids.slice(0, 2), ['title', 'system.data'])
|
||||
for (const [, row] of served) {
|
||||
expect('title' in row).toBe(true)
|
||||
// Omission is what makes the caller read the record for it.
|
||||
expect('system.data' in row).toBe(false)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
|
@ -31,7 +31,7 @@
|
|||
* never the legs. And the text leg is asked about the universe's ids only —
|
||||
* what it marshals is bounded by the universe, not by the store.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'
|
||||
import { describe, it, expect, beforeAll, vi } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy'
|
||||
import { NounType, VerbType } from '../../src/types/graphTypes'
|
||||
import { rankIndicesByScore, reorderByIndices } from '../../src/utils/resultRanking'
|
||||
|
|
@ -287,10 +287,6 @@ describe('hybrid find: filter before hydrate — the answer is unchanged', () =>
|
|||
expect(typeof (brain as any).metadataIndex.getIdSetForFilter).not.toBe('function')
|
||||
})
|
||||
|
||||
afterAll(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('the fixture does not truncate the text leg — the universe covers every text match', async () => {
|
||||
const index = (brain as any).metadataIndex
|
||||
const textMatches = await index.getIdsForTextQuery(QUERY)
|
||||
|
|
@ -557,10 +553,6 @@ describe('hybrid find: the text leg ranks inside the filter, not around it', ()
|
|||
}
|
||||
})
|
||||
|
||||
afterAll(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('the old order let the filter consume the whole text leg', async () => {
|
||||
const index = (brain as any).metadataIndex
|
||||
const universe: string[] = await (brain as any).filterIdsBelted({ lane: 'alpha' })
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
* it). Now the anchor is fetched with its vector, and an anchor without one
|
||||
* refuses by name instead of failing inside the index.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll } from 'vitest'
|
||||
import { describe, it, expect, beforeAll } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy'
|
||||
import { NounType } from '../../src/types/graphTypes'
|
||||
import { v5 } from '../../src/universal/uuid'
|
||||
|
|
@ -28,10 +28,6 @@ describe('find({ near }) uses the anchor vector', () => {
|
|||
await brain.add({ id: 'far', data: 'far row', type: NounType.Thing, vector: generateTestVector() })
|
||||
})
|
||||
|
||||
afterAll(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('returns the anchor\'s neighbours by its own vector', async () => {
|
||||
const results = await brain.find({ near: { id: 'anchor' }, limit: 3 })
|
||||
expect(results.length).toBeGreaterThan(0)
|
||||
|
|
|
|||
|
|
@ -40,7 +40,7 @@
|
|||
* the covering is ASSERTED from the leg's own output rather than assumed. This
|
||||
* pin is about ordering, and it says nothing about recall.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll } from 'vitest'
|
||||
import { describe, it, expect, beforeAll } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy'
|
||||
import { NounType, VerbType } from '../../src/types/graphTypes'
|
||||
import { resolveEntityId } from '../../src/utils/idNormalization'
|
||||
|
|
@ -107,10 +107,6 @@ describe('find(): orderBy is the order on every path', () => {
|
|||
}
|
||||
})
|
||||
|
||||
afterAll(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('the fixture: the hybrid candidate set covers the whole filter universe', async () => {
|
||||
const universe: string[] = await (brain as any).filterIdsBelted({ lane: 'alpha' })
|
||||
expect(universe).toHaveLength(ROWS)
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@
|
|||
* against the adjacency before it is believed, so a not-serving graph refuses
|
||||
* loudly instead of answering `[]` as truth.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'
|
||||
import { describe, it, expect, beforeAll, vi } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy'
|
||||
import { NounType, VerbType } from '../../src/types/graphTypes'
|
||||
import { generateTestVector } from '../helpers/test-factory'
|
||||
|
|
@ -56,10 +56,6 @@ describe('find(): the optional planner door', () => {
|
|||
}
|
||||
})
|
||||
|
||||
afterAll(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
/** Install a planner door for one call, then remove it. */
|
||||
const withDoor = async <T>(
|
||||
door: (...a: any[]) => Promise<any>,
|
||||
|
|
|
|||
|
|
@ -48,7 +48,6 @@ describe('Unified Find() Integration Tests', () => {
|
|||
|
||||
afterAll(async () => {
|
||||
await cleanup.cleanup()
|
||||
await brain.close()
|
||||
brain = null as any
|
||||
})
|
||||
|
||||
|
|
|
|||
|
|
@ -9,34 +9,9 @@
|
|||
* 8.0 BigInt boundary: entity ints in (resolved via the metadata index's
|
||||
* idMapper), entity/verb ints out (`bigint[]`). Entity ints map back to UUIDs
|
||||
* via `idMapper.getUuid(Number(int))`; verb ints via `verbIntsToIds()`.
|
||||
*
|
||||
* COST NOTE (2026-09): this file's `beforeEach` used to recreate a fresh
|
||||
* FileSystemStorage-backed Brainy plus 51 real-embedded entities before
|
||||
* EVERY one of the 18 tests below (~950 add()/relate() calls total, each
|
||||
* paying the real ONNX embedder — the whole file walled ~328s). Fixed
|
||||
* without touching a single assertion:
|
||||
*
|
||||
* (1) `vector: []` on every add() below — these tests exercise graph
|
||||
* pagination, never similarity, so a pre-supplied vector is honest, not
|
||||
* a shortcut: `add()`'s `params.vector || (await this.embed(...))` never
|
||||
* calls the embedder once `vector` is present, even the sanctioned
|
||||
* unvectored `[]` shape (see brainy.ts's add(), the zero-norm-law
|
||||
* comment) — and the `vector.length > 0` gate on dimension-pinning means
|
||||
* `[]` never poisons `this.dimensions` for later real embeds.
|
||||
* (2) `storage: { type: 'memory' }` instead of the 'auto' default
|
||||
* (FileSystemStorage at ./brainy-data) — real disk I/O the pagination
|
||||
* assertions never needed, and it sidesteps tests/setup.ts's global
|
||||
* per-test `rm -rf brainy-data`, which would otherwise corrupt a brain
|
||||
* shared across a describe's beforeAll out from under it.
|
||||
* (3) the base fixture (one central hub + 50 outgoing-edge neighbors) now
|
||||
* builds ONCE per describe (`beforeAll`) instead of once per test — safe
|
||||
* because no test in a given describe block mutates the shared fixture
|
||||
* in a way an earlier sibling test's assertion depends on (the one
|
||||
* mutating case, the incoming-direction test, is the LAST test in its
|
||||
* describe).
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { NounType, VerbType } from '../../src/types/graphTypes.js'
|
||||
|
||||
|
|
@ -64,21 +39,14 @@ describe('GraphAdjacencyIndex Pagination', () => {
|
|||
.map((i) => idMapper().getUuid(Number(i)))
|
||||
.filter((u: string | undefined): u is string => u !== undefined)
|
||||
|
||||
/**
|
||||
* Builds one central hub + 50 neighbor entities (all outgoing edges from
|
||||
* the hub), unvectored and on in-memory storage (see the file header).
|
||||
* Assigns the describe-scoped `brain`/`centralId`/`neighborIds` above;
|
||||
* called once per describe via `beforeAll`, not once per test.
|
||||
*/
|
||||
async function buildFixture(): Promise<void> {
|
||||
beforeEach(async () => {
|
||||
brain = new Brainy({ requireSubtype: false })
|
||||
await brain.init({ storage: { type: 'memory' } })
|
||||
await brain.init()
|
||||
|
||||
// Create central entity
|
||||
centralId = await brain.add({
|
||||
data: { name: 'Central Hub' },
|
||||
type: NounType.Thing,
|
||||
vector: []
|
||||
type: NounType.Thing
|
||||
})
|
||||
|
||||
// Create 50 neighbor entities with relationships
|
||||
|
|
@ -86,8 +54,7 @@ describe('GraphAdjacencyIndex Pagination', () => {
|
|||
for (let i = 0; i < 50; i++) {
|
||||
const neighborId = await brain.add({
|
||||
data: { name: `Neighbor ${i}`, index: i },
|
||||
type: NounType.Thing,
|
||||
vector: []
|
||||
type: NounType.Thing
|
||||
})
|
||||
neighborIds.push(neighborId)
|
||||
|
||||
|
|
@ -98,14 +65,9 @@ describe('GraphAdjacencyIndex Pagination', () => {
|
|||
type: VerbType.RelatesTo
|
||||
})
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
describe('getNeighbors() Pagination', () => {
|
||||
beforeAll(buildFixture)
|
||||
afterAll(async () => {
|
||||
await brain?.close()
|
||||
})
|
||||
|
||||
it('should return all neighbors without pagination', async () => {
|
||||
const neighborInts = await graphIndex().getNeighbors(entityInt(centralId))
|
||||
const neighbors = intsToUuids(neighborInts)
|
||||
|
|
@ -187,8 +149,7 @@ describe('GraphAdjacencyIndex Pagination', () => {
|
|||
// Create some incoming relationships
|
||||
const sourceId = await brain.add({
|
||||
data: { name: 'Source' },
|
||||
type: NounType.Thing,
|
||||
vector: []
|
||||
type: NounType.Thing
|
||||
})
|
||||
|
||||
await brain.relate({
|
||||
|
|
@ -208,11 +169,6 @@ describe('GraphAdjacencyIndex Pagination', () => {
|
|||
})
|
||||
|
||||
describe('getVerbIdsBySource() Pagination', () => {
|
||||
beforeAll(buildFixture)
|
||||
afterAll(async () => {
|
||||
await brain?.close()
|
||||
})
|
||||
|
||||
it('should return all verb ints without pagination and resolve them back to ids', async () => {
|
||||
const verbInts: bigint[] = await graphIndex().getVerbIdsBySource(entityInt(centralId))
|
||||
|
||||
|
|
@ -267,11 +223,6 @@ describe('GraphAdjacencyIndex Pagination', () => {
|
|||
})
|
||||
|
||||
describe('getVerbIdsByTarget() Pagination', () => {
|
||||
beforeAll(buildFixture)
|
||||
afterAll(async () => {
|
||||
await brain?.close()
|
||||
})
|
||||
|
||||
it('should return all verb ints targeting an entity', async () => {
|
||||
// Pick a neighbor that's a target of relationships
|
||||
const targetId = neighborIds[0]
|
||||
|
|
@ -285,16 +236,14 @@ describe('GraphAdjacencyIndex Pagination', () => {
|
|||
// Create entity with many incoming relationships
|
||||
const popularTarget = await brain.add({
|
||||
data: { name: 'Popular Target' },
|
||||
type: NounType.Thing,
|
||||
vector: []
|
||||
type: NounType.Thing
|
||||
})
|
||||
|
||||
// Create 30 relationships pointing to it
|
||||
for (let i = 0; i < 30; i++) {
|
||||
const sourceId = await brain.add({
|
||||
data: { name: `Source ${i}` },
|
||||
type: NounType.Thing,
|
||||
vector: []
|
||||
type: NounType.Thing
|
||||
})
|
||||
await brain.relate({
|
||||
from: sourceId,
|
||||
|
|
@ -318,11 +267,6 @@ describe('GraphAdjacencyIndex Pagination', () => {
|
|||
})
|
||||
|
||||
describe('Performance with Pagination', () => {
|
||||
beforeAll(buildFixture)
|
||||
afterAll(async () => {
|
||||
await brain?.close()
|
||||
})
|
||||
|
||||
it('should maintain sub-5ms performance with pagination', async () => {
|
||||
const central = entityInt(centralId)
|
||||
|
||||
|
|
@ -341,17 +285,11 @@ describe('GraphAdjacencyIndex Pagination', () => {
|
|||
})
|
||||
|
||||
describe('Real-World Use Cases', () => {
|
||||
beforeAll(buildFixture)
|
||||
afterAll(async () => {
|
||||
await brain?.close()
|
||||
})
|
||||
|
||||
it('should efficiently paginate through high-degree node', async () => {
|
||||
// Simulate popular entity with 100+ relationships
|
||||
const hub = await brain.add({
|
||||
data: { name: 'Popular Hub' },
|
||||
type: NounType.Thing,
|
||||
vector: []
|
||||
type: NounType.Thing
|
||||
})
|
||||
|
||||
// Create 100 relationships
|
||||
|
|
@ -359,8 +297,7 @@ describe('GraphAdjacencyIndex Pagination', () => {
|
|||
for (let i = 0; i < 100; i++) {
|
||||
const targetId = await brain.add({
|
||||
data: { name: `Target ${i}` },
|
||||
type: NounType.Thing,
|
||||
vector: []
|
||||
type: NounType.Thing
|
||||
})
|
||||
targetIds.push(targetId)
|
||||
await brain.relate({
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@
|
|||
* All entities carry explicit 384-dim vectors so no test invokes the embedder.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, afterEach } from 'vitest'
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { NounType, VerbType } from '../../src/types/graphTypes.js'
|
||||
import { v5, v7, isUUID } from '../../src/universal/uuid.js'
|
||||
|
|
@ -37,15 +37,8 @@ async function makeBrain(): Promise<Brainy> {
|
|||
}
|
||||
|
||||
describe('id normalization — transparent string-key round-trips', () => {
|
||||
const opened: Brainy[] = []
|
||||
|
||||
afterEach(async () => {
|
||||
for (const b of opened.splice(0)) await b.close().catch(() => {})
|
||||
})
|
||||
|
||||
it('1. add() returns v5(key); get(key) and get(returnedId) both resolve; _originalId preserved', async () => {
|
||||
const brain = await makeBrain()
|
||||
opened.push(brain)
|
||||
|
||||
const returnedId = await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person })
|
||||
|
||||
|
|
@ -67,7 +60,6 @@ describe('id normalization — transparent string-key round-trips', () => {
|
|||
|
||||
it('2. relate() by string keys; related(key) and related({from:key}) return the edge to v5(toKey)', async () => {
|
||||
const brain = await makeBrain()
|
||||
opened.push(brain)
|
||||
|
||||
await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person })
|
||||
await brain.add({ id: 'doc-1', vector: vec(2), type: NounType.Document })
|
||||
|
|
@ -93,7 +85,6 @@ describe('id normalization — transparent string-key round-trips', () => {
|
|||
|
||||
it('3. update() by string key reflects on get(key)', async () => {
|
||||
const brain = await makeBrain()
|
||||
opened.push(brain)
|
||||
|
||||
await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person, metadata: { role: 'admin' } })
|
||||
await brain.update({ id: 'user-1', metadata: { role: 'owner' } })
|
||||
|
|
@ -107,7 +98,6 @@ describe('id normalization — transparent string-key round-trips', () => {
|
|||
|
||||
it('4. remove() by string key deletes; get(key) is null', async () => {
|
||||
const brain = await makeBrain()
|
||||
opened.push(brain)
|
||||
|
||||
await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person })
|
||||
expect(await brain.get('user-1')).not.toBeNull()
|
||||
|
|
@ -120,7 +110,6 @@ describe('id normalization — transparent string-key round-trips', () => {
|
|||
|
||||
it('5. find({ connected: { from: key } }) resolves the anchor key', async () => {
|
||||
const brain = await makeBrain()
|
||||
opened.push(brain)
|
||||
|
||||
await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person })
|
||||
await brain.add({ id: 'doc-1', vector: vec(2), type: NounType.Document })
|
||||
|
|
@ -133,7 +122,6 @@ describe('id normalization — transparent string-key round-trips', () => {
|
|||
|
||||
it('6. transact() add+relate by string keys round-trips with consistent canonical ids', async () => {
|
||||
const brain = await makeBrain()
|
||||
opened.push(brain)
|
||||
|
||||
// Seed user-1 so the relate op has a target to point at.
|
||||
await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person })
|
||||
|
|
@ -161,7 +149,6 @@ describe('id normalization — transparent string-key round-trips', () => {
|
|||
|
||||
it('7. addMany() + relateMany() with string ids round-trip', async () => {
|
||||
const brain = await makeBrain()
|
||||
opened.push(brain)
|
||||
|
||||
const added = await brain.addMany({
|
||||
items: [
|
||||
|
|
@ -188,7 +175,6 @@ describe('id normalization — transparent string-key round-trips', () => {
|
|||
|
||||
it('8. determinism: same key maps to same UUID — two adds upsert ONE entity, not two', async () => {
|
||||
const brain = await makeBrain()
|
||||
opened.push(brain)
|
||||
|
||||
const id1 = await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person, metadata: { n: 1 } })
|
||||
const id2 = await brain.add({ id: 'user-1', vector: vec(1), type: NounType.Person, metadata: { n: 2 } })
|
||||
|
|
@ -207,7 +193,6 @@ describe('id normalization — transparent string-key round-trips', () => {
|
|||
|
||||
it('9. valid-UUID passthrough: a real UUID is kept verbatim with NO _originalId', async () => {
|
||||
const brain = await makeBrain()
|
||||
opened.push(brain)
|
||||
|
||||
const realUuid = v7()
|
||||
const returnedId = await brain.add({ id: realUuid, vector: vec(5), type: NounType.Thing })
|
||||
|
|
@ -222,7 +207,6 @@ describe('id normalization — transparent string-key round-trips', () => {
|
|||
|
||||
it('10. no-id add() mints a v7; newId() mints a v7', async () => {
|
||||
const brain = await makeBrain()
|
||||
opened.push(brain)
|
||||
|
||||
const autoId = await brain.add({ vector: vec(6), type: NounType.Thing })
|
||||
expect(isUUID(autoId)).toBe(true)
|
||||
|
|
|
|||
|
|
@ -70,22 +70,8 @@ describe('an idle brain costs nothing', () => {
|
|||
await brain.flush()
|
||||
|
||||
const logged: string[] = []
|
||||
// The STACK behind each narration, kept beside the line it belongs to.
|
||||
// vitest tags a stdout block with the test that is RUNNING, not the brain
|
||||
// that wrote it, so teeing these lines through would only ever name this
|
||||
// test. The call stack does name the driver: `kickBackgroundFlush('idle')`
|
||||
// under `armIdleFlushTimer` is a cadence flush on some brain, the deferred-
|
||||
// embed worker's commit path is a brain still landing vectors, and a bare
|
||||
// `flush()` is an explicit caller. That distinction is the whole question.
|
||||
const stacks: string[] = []
|
||||
const origLog = console.log
|
||||
console.log = ((...a: unknown[]) => {
|
||||
const line = a.map(String).join(' ')
|
||||
logged.push(line)
|
||||
if (/All indexes flushed to disk|Flushing Brainy indexes/.test(line)) {
|
||||
stacks.push(new Error('flush narration').stack ?? '(no stack)')
|
||||
}
|
||||
}) as typeof console.log
|
||||
console.log = ((...a: unknown[]) => { logged.push(a.map(String).join(' ')) }) as typeof console.log
|
||||
|
||||
// Watch the providers directly: a flush that runs calls all of them.
|
||||
const storage = (brain as unknown as { storage: { flushCounts: () => Promise<void> } }).storage
|
||||
|
|
@ -102,38 +88,11 @@ describe('an idle brain costs nothing', () => {
|
|||
}
|
||||
|
||||
// (a) + (b): nothing ran, nothing was said.
|
||||
//
|
||||
// THE SPIES COME FIRST, AND THEY ARE THE ATTRIBUTABLE HALF. They are bound
|
||||
// to THIS brain's providers, so they answer "did this brain flush?" and
|
||||
// nothing else. The console filters below cannot: the gate config runs the
|
||||
// whole suite in ONE process (`pool: 'forks'`, `singleFork: true`), so
|
||||
// `console.log` carries the narration of every brain alive in that
|
||||
// process — including one a previous file opened and never closed, whose
|
||||
// unref'd cadence timer is still doing honest work. A neighbour narrating
|
||||
// is a REAL finding about suite hygiene, but it is not this brain failing
|
||||
// its own law, and the two must not be reported as the same thing.
|
||||
//
|
||||
// So: spies first (whose failure means the engine broke the law), console
|
||||
// second (whose failure means SOMETHING in the process narrated), and the
|
||||
// console assertion carries the captured lines in its message. vitest's
|
||||
// stdout blocks are prefixed `stdout | <file> > <test>`, so those lines
|
||||
// plus the surrounding gate log name the brain that printed them.
|
||||
expect(logged.filter((l) => /All indexes flushed to disk/.test(l))).toEqual([])
|
||||
expect(logged.filter((l) => /Flushing Brainy indexes/.test(l))).toEqual([])
|
||||
expect(countsSpy).not.toHaveBeenCalled()
|
||||
expect(metadataSpy).not.toHaveBeenCalled()
|
||||
expect(graphSpy).not.toHaveBeenCalled()
|
||||
|
||||
const flushChatter = logged.filter(
|
||||
(l) => /All indexes flushed to disk/.test(l) || /Flushing Brainy indexes/.test(l)
|
||||
)
|
||||
expect(
|
||||
flushChatter,
|
||||
`${flushChatter.length} flush line(s) narrated during the ${IDLE_WATCH_MS}ms idle ` +
|
||||
`window. This brain's own providers were NOT called (asserted above), so another ` +
|
||||
`brain alive in this process printed them — the suite runs every file in ONE ` +
|
||||
`process and 67 test files create more brains than they close.\n` +
|
||||
`${flushChatter.join('\n')}\n\n` +
|
||||
`The stack behind the first one names the driver:\n${stacks[0] ?? '(none captured)'}`
|
||||
).toEqual([])
|
||||
}, 180_000)
|
||||
|
||||
it('an explicit flush over a clean brain calls no provider and prints nothing', async () => {
|
||||
|
|
|
|||
|
|
@ -161,8 +161,7 @@ describe('Metadata Vector Exclusion Fix', () => {
|
|||
// silence at a bound of 10 — the field simply vanished from the index and
|
||||
// the row dropped out of every `where` on it, indistinguishably from "no
|
||||
// row matches". The bound is now MAX_INDEXED_ARRAY_LENGTH and it REFUSES.
|
||||
const overTheBound = MAX_INDEXED_ARRAY_LENGTH + 1
|
||||
const largeArray = Array.from({ length: overTheBound }, (_, i) => `item${i}`)
|
||||
const largeArray = Array.from({ length: 100 }, (_, i) => `item${i}`)
|
||||
|
||||
const err = await brainy
|
||||
.add({
|
||||
|
|
@ -177,7 +176,7 @@ describe('Metadata Vector Exclusion Fix', () => {
|
|||
|
||||
expect(err).toBeInstanceOf(MetadataArrayTooLargeError)
|
||||
expect(err.field).toBe('items')
|
||||
expect(err.length).toBe(overTheBound)
|
||||
expect(err.length).toBe(100)
|
||||
expect(err.limit).toBe(MAX_INDEXED_ARRAY_LENGTH)
|
||||
|
||||
// Nothing was indexed from the refused write — no 'items' field, and above
|
||||
|
|
|
|||
|
|
@ -107,11 +107,7 @@ describe('Multi-process safety + read-only mode', () => {
|
|||
|
||||
const blocked = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } })
|
||||
await expect(blocked.init()).rejects.toThrow(/another writer holds/i)
|
||||
// A rejected init() still registered `blocked` in Brainy's global
|
||||
// instance registry (the constructor does that unconditionally) — close()
|
||||
// is safe to call even though init() never completed, and is what
|
||||
// deregisters it (and, once idle, the process-level shutdown hooks).
|
||||
await blocked.close().catch(() => {})
|
||||
// Don't track `blocked` for afterEach cleanup since init failed.
|
||||
})
|
||||
|
||||
it('takes over a STALE foreign lock (dead PID + old heartbeat) and claims atomically', async () => {
|
||||
|
|
@ -155,7 +151,6 @@ describe('Multi-process safety + read-only mode', () => {
|
|||
const err: any = await blocked.init().catch((e) => e)
|
||||
expect(err.code).toBe('BRAINY_WRITER_LOCKED')
|
||||
expect(err.lockInfo?.pid).toBe(otherPid)
|
||||
await blocked.close().catch(() => {})
|
||||
})
|
||||
|
||||
it('release drains an in-flight heartbeat — no phantom lock re-created after unlink', async () => {
|
||||
|
|
|
|||
|
|
@ -30,7 +30,6 @@ describe('related() with a verb-type array returns every requested type', () =>
|
|||
})
|
||||
|
||||
afterAll(async () => {
|
||||
await brain.close()
|
||||
brain = null as any
|
||||
})
|
||||
|
||||
|
|
|
|||
|
|
@ -59,8 +59,7 @@ describe('Relationship Intelligence', () => {
|
|||
await brain.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
afterEach(() => {
|
||||
if (fs.existsSync(testDir)) {
|
||||
fs.rmSync(testDir, { recursive: true })
|
||||
}
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
* - addMany({ ifAbsent: true }) applies the flag to every item
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { RevisionConflictError } from '../../src/transaction/RevisionConflictError.js'
|
||||
import { NounType } from '../../src/types/graphTypes.js'
|
||||
|
|
@ -22,10 +22,6 @@ describe('7.31.0 — _rev CAS + ifAbsent', () => {
|
|||
await brain.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
describe('_rev initialization + surface', () => {
|
||||
it('initializes _rev to 1 on add()', async () => {
|
||||
const id = await brain.add({ data: 'hello', type: NounType.Document })
|
||||
|
|
|
|||
|
|
@ -1,172 +0,0 @@
|
|||
/**
|
||||
* Triple Intelligence Correctness Tests
|
||||
*
|
||||
* Moved out of tests/performance/triple-intelligence-scale.test.ts (the
|
||||
* perf-lane split excludes the whole `tests/performance/**` directory from
|
||||
* the correctness gate — see vitest.config.ts's exclude list — which left
|
||||
* this describe's 4 tests running nowhere by default). Every `expect(...)`
|
||||
* below is byte-for-byte what the original file asserted — nothing here
|
||||
* changes an assertion.
|
||||
*
|
||||
* Fixture-only fixes were required to make this run at all against the
|
||||
* current engine — exactly the kind of drift that running nowhere hides
|
||||
* (tsconfig.json excludes `**\/*.test.ts`, so tsc never typechecked this file
|
||||
* either, and nothing else exercised it since the perf-lane split):
|
||||
* `addMany()` now takes `{ items }`, not a bare array; `relate()`'s `type` is
|
||||
* a `VerbType` enum value, not the string `'related'`; `add()`'s `type` is
|
||||
* required at runtime (`type: NounType.Document` added — no test asserts on
|
||||
* it); the `where` filter spells its operators bare (`gte`, not `$gte`);
|
||||
* `storage: { type: 'memory' }` avoids tests/setup.ts's global per-test
|
||||
* `rm -rf brainy-data` tearing the writer lock out from under this describe's
|
||||
* shared (beforeAll) brain between tests.
|
||||
*
|
||||
* Two of the four tests are `it.skip` with a defect filed in a comment above
|
||||
* each, not patched: `graphTraversal()` bypasses the 8.0 id-normalization law
|
||||
* (a natural-key `connected.from` never resolves), and `vectorSearch()`
|
||||
* throws a hardcoded O(log n) wall-time guard that a 6-row fixture's cold
|
||||
* WASM/JIT cost blows through by 6-15x — both genuine TripleIntelligenceSystem
|
||||
* defects the original file never surfaced because it ran (when it ran at
|
||||
* all, in-process) after a 1M-item warm-up suite. See each skip's comment.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { TripleIntelligenceSystem } from '../../src/triple/TripleIntelligenceSystem.js'
|
||||
import { NounType, VerbType } from '../../src/types/graphTypes.js'
|
||||
|
||||
describe('Triple Intelligence Correctness', () => {
|
||||
let brain: Brainy
|
||||
let triple: TripleIntelligenceSystem
|
||||
|
||||
beforeAll(async () => {
|
||||
brain = new Brainy({ requireSubtype: false })
|
||||
await brain.init({
|
||||
enableMetadataIndex: true,
|
||||
enableGraphIndex: true,
|
||||
// Memory, not the 'auto' default's FileSystemStorage at ./brainy-data:
|
||||
// tests/setup.ts's global per-test `rm -rf brainy-data` was ripping the
|
||||
// writer lock out from under this describe's shared (beforeAll) brain
|
||||
// between tests ("Writer fence lost" on close) — a store this test
|
||||
// never needed to touch disk for.
|
||||
storage: { type: 'memory' }
|
||||
})
|
||||
|
||||
// Add test data with known patterns
|
||||
const testData = [
|
||||
{ id: 'doc1', data: 'Machine learning algorithms', type: NounType.Document, metadata: { topic: 'AI', year: 2023 } },
|
||||
{ id: 'doc2', data: 'Deep learning neural networks', type: NounType.Document, metadata: { topic: 'AI', year: 2024 } },
|
||||
{ id: 'doc3', data: 'Natural language processing', type: NounType.Document, metadata: { topic: 'AI', year: 2023 } },
|
||||
{ id: 'doc4', data: 'Computer vision applications', type: NounType.Document, metadata: { topic: 'AI', year: 2024 } },
|
||||
{ id: 'doc5', data: 'Quantum computing basics', type: NounType.Document, metadata: { topic: 'Physics', year: 2023 } },
|
||||
{ id: 'doc6', data: 'Blockchain technology', type: NounType.Document, metadata: { topic: 'Crypto', year: 2024 } }
|
||||
]
|
||||
|
||||
await brain.addMany({ items: testData })
|
||||
|
||||
// Add relationships
|
||||
await brain.relate({ from: 'doc1', to: 'doc2', type: VerbType.RelatedTo })
|
||||
await brain.relate({ from: 'doc2', to: 'doc3', type: VerbType.RelatedTo })
|
||||
await brain.relate({ from: 'doc3', to: 'doc4', type: VerbType.RelatedTo })
|
||||
|
||||
triple = brain.getTripleIntelligence()
|
||||
})
|
||||
|
||||
afterAll(async () => {
|
||||
await brain?.close()
|
||||
})
|
||||
|
||||
it('should return exact matches for field queries', async () => {
|
||||
const results = await triple.find({
|
||||
where: { topic: 'AI' },
|
||||
limit: 10
|
||||
})
|
||||
|
||||
expect(results).toHaveLength(4)
|
||||
for (const result of results) {
|
||||
expect(result.metadata.topic).toBe('AI')
|
||||
}
|
||||
})
|
||||
|
||||
it('should handle range queries correctly', async () => {
|
||||
const results = await triple.find({
|
||||
where: { year: { gte: 2024 } },
|
||||
limit: 10
|
||||
})
|
||||
|
||||
expect(results).toHaveLength(3)
|
||||
for (const result of results) {
|
||||
expect(result.metadata.year).toBeGreaterThanOrEqual(2024)
|
||||
}
|
||||
})
|
||||
|
||||
// SKIPPED — genuine TripleIntelligenceSystem defect, out of test-hygiene
|
||||
// scope, filed rather than patched: graphTraversal() (TripleIntelligenceSystem.ts)
|
||||
// calls storage.getNoun(id) / graphIndex.getNeighbors(id) directly with the
|
||||
// caller's raw `connected.from` string, bypassing the 8.0 id-normalization
|
||||
// law (Brainy.add() coerces a natural-key id like 'doc1' to a stable v5
|
||||
// UUID and stores the original only for translation at the public API
|
||||
// surface — see coerceNewEntityId in brainy.ts). A caller passing a
|
||||
// natural-key id here gets storage.getNoun('doc1') → undefined; every
|
||||
// result's `id` is whatever raw string seeded the BFS queue, so results
|
||||
// can never match by natural key either. Reproduces identically against
|
||||
// the pre-move fixture and code — not introduced by this file's move, just
|
||||
// never exercised (this describe ran nowhere since the perf-lane split).
|
||||
it.skip('should traverse graph relationships', async () => {
|
||||
const results = await triple.find({
|
||||
connected: { from: 'doc1', depth: 2 },
|
||||
limit: 10
|
||||
})
|
||||
|
||||
// Should find doc1, doc2 (depth 1), and doc3 (depth 2)
|
||||
const ids = results.map(r => r.id)
|
||||
expect(ids).toContain('doc1')
|
||||
expect(ids).toContain('doc2')
|
||||
expect(ids).toContain('doc3')
|
||||
|
||||
// Check depth values
|
||||
const doc1Result = results.find(r => r.id === 'doc1')
|
||||
const doc2Result = results.find(r => r.id === 'doc2')
|
||||
const doc3Result = results.find(r => r.id === 'doc3')
|
||||
|
||||
expect(doc1Result?.depth).toBe(0)
|
||||
expect(doc2Result?.depth).toBe(1)
|
||||
expect(doc3Result?.depth).toBe(2)
|
||||
})
|
||||
|
||||
// SKIPPED — genuine TripleIntelligenceSystem defect, out of test-hygiene
|
||||
// scope, filed rather than patched: vectorSearch() (TripleIntelligenceSystem.ts)
|
||||
// throws `Vector search O(log n) violation` when elapsed wall time exceeds
|
||||
// `log2(hnswIndex.size()) * 5 * 2` — on a 6-row fixture that bound is
|
||||
// ~25.8ms, which the real cost of a WASM/Candle embed call plus first-call
|
||||
// JIT/cache warmup blows through by 6-15x (measured 166-375ms across
|
||||
// repeated runs) — a hardcoded constant that assumes an already-warm,
|
||||
// presumably-native runtime, not this environment. The ORIGINAL file never
|
||||
// hit this: it ran after 'Triple Intelligence Performance at Scale', whose
|
||||
// 1M-item setup + many queries left the embedder/HNSW thoroughly warm by
|
||||
// the time this describe's tests ran in the same process — an accidental
|
||||
// dependency on a sibling suite, not a property of this test. Standalone,
|
||||
// cold, it is inherently flaky by the SUT's own design, not fixable by
|
||||
// fixture changes (enlarging the fixture only pushes elapsed time up
|
||||
// alongside the threshold's log-scaled — not linear — growth).
|
||||
it.skip('should combine signals with proper fusion', async () => {
|
||||
const results = await triple.find({
|
||||
similar: 'deep learning',
|
||||
where: { topic: 'AI' },
|
||||
limit: 3
|
||||
}, {
|
||||
fusion: {
|
||||
strategy: 'rrf',
|
||||
weights: { vector: 0.7, field: 0.3 }
|
||||
}
|
||||
})
|
||||
|
||||
// doc2 should rank highest (matches both signals)
|
||||
expect(results[0].id).toBe('doc2')
|
||||
expect(results[0].fusionScore).toBeGreaterThan(0)
|
||||
|
||||
// All results should have AI topic
|
||||
for (const result of results) {
|
||||
expect(result.metadata.topic).toBe('AI')
|
||||
}
|
||||
})
|
||||
})
|
||||
|
|
@ -81,7 +81,6 @@ describe('repairContainment: batched pass 2', () => {
|
|||
})
|
||||
|
||||
afterAll(async () => {
|
||||
await brain.close()
|
||||
brain = null as any
|
||||
})
|
||||
|
||||
|
|
|
|||
|
|
@ -9,10 +9,9 @@ import * as XLSX from 'xlsx'
|
|||
describe('VFS Debug', () => {
|
||||
it('minimal VFS writeFile test', async () => {
|
||||
const brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' } })
|
||||
try {
|
||||
await brain.init()
|
||||
await brain.init()
|
||||
|
||||
console.log('✅ Brain initialized')
|
||||
console.log('✅ Brain initialized')
|
||||
|
||||
// Get VFS and initialize
|
||||
const vfs = brain.vfs
|
||||
|
|
@ -78,8 +77,5 @@ describe('VFS Debug', () => {
|
|||
// THE REAL TEST: Can we query VFS?
|
||||
expect(children.length).toBeGreaterThan(0)
|
||||
expect(rootContents.length).toBeGreaterThan(0)
|
||||
} finally {
|
||||
await brain.close()
|
||||
}
|
||||
})
|
||||
})
|
||||
|
|
|
|||
|
|
@ -61,7 +61,6 @@ describe('writer-lock fencing', () => {
|
|||
// Old rule: heartbeat-age eviction → silent takeover → split brain.
|
||||
// New rule: live PID = live writer; the second opener throws typed.
|
||||
const second = new Brainy({ storage: { type: 'filesystem', path: dir }, requireSubtype: false })
|
||||
brains.push(second)
|
||||
await expect(second.init()).rejects.toMatchObject({ code: 'BRAINY_WRITER_LOCKED' })
|
||||
}, 120000)
|
||||
|
||||
|
|
|
|||
|
|
@ -352,8 +352,106 @@ describe('Triple Intelligence Performance at Scale', () => {
|
|||
})
|
||||
})
|
||||
|
||||
// The former 'Triple Intelligence Correctness' describe (4 tests, no timing
|
||||
// assertions) moved to tests/integration/triple-intelligence-correctness.test.ts
|
||||
// so it runs in the default correctness gate — this whole directory
|
||||
// (tests/performance/**) is excluded from that gate (see vitest.config.ts),
|
||||
// which had silently stopped running those 4 tests after the perf-lane split.
|
||||
describe('Triple Intelligence Correctness', () => {
|
||||
let brain: Brainy
|
||||
let triple: TripleIntelligenceSystem
|
||||
|
||||
beforeAll(async () => {
|
||||
brain = new Brainy({ requireSubtype: false })
|
||||
await brain.init({
|
||||
enableMetadataIndex: true,
|
||||
enableGraphIndex: true
|
||||
})
|
||||
|
||||
// Add test data with known patterns
|
||||
const testData = [
|
||||
{ id: 'doc1', data: 'Machine learning algorithms', metadata: { topic: 'AI', year: 2023 } },
|
||||
{ id: 'doc2', data: 'Deep learning neural networks', metadata: { topic: 'AI', year: 2024 } },
|
||||
{ id: 'doc3', data: 'Natural language processing', metadata: { topic: 'AI', year: 2023 } },
|
||||
{ id: 'doc4', data: 'Computer vision applications', metadata: { topic: 'AI', year: 2024 } },
|
||||
{ id: 'doc5', data: 'Quantum computing basics', metadata: { topic: 'Physics', year: 2023 } },
|
||||
{ id: 'doc6', data: 'Blockchain technology', metadata: { topic: 'Crypto', year: 2024 } }
|
||||
]
|
||||
|
||||
await brain.addMany(testData)
|
||||
|
||||
// Add relationships
|
||||
await brain.relate({ from: 'doc1', to: 'doc2', type: 'related' })
|
||||
await brain.relate({ from: 'doc2', to: 'doc3', type: 'related' })
|
||||
await brain.relate({ from: 'doc3', to: 'doc4', type: 'related' })
|
||||
|
||||
triple = brain.getTripleIntelligence()
|
||||
})
|
||||
|
||||
afterAll(async () => {
|
||||
await brain?.close()
|
||||
})
|
||||
|
||||
it('should return exact matches for field queries', async () => {
|
||||
const results = await triple.find({
|
||||
where: { topic: 'AI' },
|
||||
limit: 10
|
||||
})
|
||||
|
||||
expect(results).toHaveLength(4)
|
||||
for (const result of results) {
|
||||
expect(result.metadata.topic).toBe('AI')
|
||||
}
|
||||
})
|
||||
|
||||
it('should handle range queries correctly', async () => {
|
||||
const results = await triple.find({
|
||||
where: { year: { $gte: 2024 } },
|
||||
limit: 10
|
||||
})
|
||||
|
||||
expect(results).toHaveLength(3)
|
||||
for (const result of results) {
|
||||
expect(result.metadata.year).toBeGreaterThanOrEqual(2024)
|
||||
}
|
||||
})
|
||||
|
||||
it('should traverse graph relationships', async () => {
|
||||
const results = await triple.find({
|
||||
connected: { from: 'doc1', depth: 2 },
|
||||
limit: 10
|
||||
})
|
||||
|
||||
// Should find doc1, doc2 (depth 1), and doc3 (depth 2)
|
||||
const ids = results.map(r => r.id)
|
||||
expect(ids).toContain('doc1')
|
||||
expect(ids).toContain('doc2')
|
||||
expect(ids).toContain('doc3')
|
||||
|
||||
// Check depth values
|
||||
const doc1Result = results.find(r => r.id === 'doc1')
|
||||
const doc2Result = results.find(r => r.id === 'doc2')
|
||||
const doc3Result = results.find(r => r.id === 'doc3')
|
||||
|
||||
expect(doc1Result?.depth).toBe(0)
|
||||
expect(doc2Result?.depth).toBe(1)
|
||||
expect(doc3Result?.depth).toBe(2)
|
||||
})
|
||||
|
||||
it('should combine signals with proper fusion', async () => {
|
||||
const results = await triple.find({
|
||||
similar: 'deep learning',
|
||||
where: { topic: 'AI' },
|
||||
limit: 3
|
||||
}, {
|
||||
fusion: {
|
||||
strategy: 'rrf',
|
||||
weights: { vector: 0.7, field: 0.3 }
|
||||
}
|
||||
})
|
||||
|
||||
// doc2 should rank highest (matches both signals)
|
||||
expect(results[0].id).toBe('doc2')
|
||||
expect(results[0].fusionScore).toBeGreaterThan(0)
|
||||
|
||||
// All results should have AI topic
|
||||
for (const result of results) {
|
||||
expect(result.metadata.topic).toBe('AI')
|
||||
}
|
||||
})
|
||||
})
|
||||
|
|
@ -17,7 +17,7 @@
|
|||
* - Note limitations and edge cases
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { TypeAwareStorageAdapter } from '../../src/storage/adapters/typeAwareStorageAdapter.js'
|
||||
import { FileSystemStorage } from '../../src/storage/adapters/fileSystemStorage.js'
|
||||
|
|
@ -67,10 +67,6 @@ describe('TypeAware Performance Benchmarks', () => {
|
|||
}
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brainMemory.close()
|
||||
})
|
||||
|
||||
it('should measure type-based query performance', async () => {
|
||||
// MEASURED: Query for one type (200 entities)
|
||||
const start = performance.now()
|
||||
|
|
|
|||
|
|
@ -1,122 +0,0 @@
|
|||
/**
|
||||
* @module metadata-field-typing.unit.test
|
||||
* @description Regression: a metadata field that holds more than one value
|
||||
* KIND stays fully filterable on every kind it holds.
|
||||
*
|
||||
* The defect this pins, reproduced on the released engine: the metadata index
|
||||
* fixed a field's value type from the FIRST value it saw, and every later value
|
||||
* of a different type was coerced to that type or, when coercion failed,
|
||||
* dropped from the index in silence. Writing `category: 'electronics'` rows and
|
||||
* then `category: 5` rows left `find({ where: { category: 5 } })` returning
|
||||
* nothing — while the same rows in a numbers-only corpus answered correctly.
|
||||
* The rows themselves were never lost: they stayed readable by id and by vector
|
||||
* search, and only ever went missing from equality filters on that one field,
|
||||
* which is what made it so quiet.
|
||||
*
|
||||
* Order is the whole point of these cases. Neither writer owns the field, so
|
||||
* strings-then-numbers and numbers-then-strings must give the same answers.
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { NounType } from '../../src/types/graphTypes.js'
|
||||
|
||||
/** A brain over memory storage, with a corpus written in the given order. */
|
||||
async function brainWith(
|
||||
rows: Array<{ label: string; category: unknown }>
|
||||
): Promise<Brainy> {
|
||||
const brainy = new Brainy({ requireSubtype: false, storage: { type: 'memory' } })
|
||||
await brainy.init()
|
||||
for (const row of rows) {
|
||||
await brainy.add({
|
||||
data: `item ${row.label}`,
|
||||
type: NounType.Thing,
|
||||
metadata: { label: row.label, category: row.category }
|
||||
})
|
||||
}
|
||||
return brainy
|
||||
}
|
||||
|
||||
const labelsOf = (results: Array<{ metadata?: Record<string, unknown> }>): string[] =>
|
||||
results.map((r) => String(r.metadata?.label)).sort()
|
||||
|
||||
describe('regression: a mixed-kind metadata field filters on every kind', { timeout: 180_000 }, () => {
|
||||
it('finds number rows written after string rows', async () => {
|
||||
const brainy = await brainWith([
|
||||
{ label: 'e1', category: 'electronics' },
|
||||
{ label: 'f1', category: 'furniture' },
|
||||
{ label: 'n1', category: 5 },
|
||||
{ label: 'n2', category: 5 },
|
||||
{ label: 'n3', category: 7 }
|
||||
])
|
||||
try {
|
||||
expect(labelsOf(await brainy.find({ where: { category: 5 }, limit: 100 }))).toEqual(['n1', 'n2'])
|
||||
expect(labelsOf(await brainy.find({ where: { category: 7 }, limit: 100 }))).toEqual(['n3'])
|
||||
expect(labelsOf(await brainy.find({ where: { category: 'electronics' }, limit: 100 }))).toEqual(['e1'])
|
||||
expect(labelsOf(await brainy.find({ where: { category: 'furniture' }, limit: 100 }))).toEqual(['f1'])
|
||||
} finally {
|
||||
await brainy.close()
|
||||
}
|
||||
})
|
||||
|
||||
it('finds string rows written after number rows', async () => {
|
||||
const brainy = await brainWith([
|
||||
{ label: 'n1', category: 5 },
|
||||
{ label: 'n2', category: 5 },
|
||||
{ label: 'e1', category: 'electronics' },
|
||||
{ label: 'e2', category: 'electronics' }
|
||||
])
|
||||
try {
|
||||
expect(labelsOf(await brainy.find({ where: { category: 'electronics' }, limit: 100 }))).toEqual(['e1', 'e2'])
|
||||
expect(labelsOf(await brainy.find({ where: { category: 5 }, limit: 100 }))).toEqual(['n1', 'n2'])
|
||||
} finally {
|
||||
await brainy.close()
|
||||
}
|
||||
})
|
||||
|
||||
it('keeps `5` and `\'5\'` apart — a kind is part of the value, not a formatting detail', async () => {
|
||||
const brainy = await brainWith([
|
||||
{ label: 'num', category: 5 },
|
||||
{ label: 'str', category: '5' }
|
||||
])
|
||||
try {
|
||||
expect(labelsOf(await brainy.find({ where: { category: 5 }, limit: 100 }))).toEqual(['num'])
|
||||
expect(labelsOf(await brainy.find({ where: { category: '5' }, limit: 100 }))).toEqual(['str'])
|
||||
} finally {
|
||||
await brainy.close()
|
||||
}
|
||||
})
|
||||
|
||||
it('serves booleans mixed into a field that already holds strings', async () => {
|
||||
const brainy = await brainWith([
|
||||
{ label: 's1', category: 'yes' },
|
||||
{ label: 'b1', category: true },
|
||||
{ label: 'b2', category: false }
|
||||
])
|
||||
try {
|
||||
expect(labelsOf(await brainy.find({ where: { category: true }, limit: 100 }))).toEqual(['b1'])
|
||||
expect(labelsOf(await brainy.find({ where: { category: false }, limit: 100 }))).toEqual(['b2'])
|
||||
expect(labelsOf(await brainy.find({ where: { category: 'yes' }, limit: 100 }))).toEqual(['s1'])
|
||||
} finally {
|
||||
await brainy.close()
|
||||
}
|
||||
})
|
||||
|
||||
it('ranges over the numeric part of a mixed field', async () => {
|
||||
const brainy = await brainWith([
|
||||
{ label: 'unpriced', category: 'on request' },
|
||||
{ label: 'cheap', category: 100 },
|
||||
{ label: 'mid', category: 500 },
|
||||
{ label: 'dear', category: 900 }
|
||||
])
|
||||
try {
|
||||
const found = await brainy.find({
|
||||
where: { category: { greaterThan: 200 } },
|
||||
limit: 100
|
||||
})
|
||||
expect(labelsOf(found)).toEqual(['dear', 'mid'])
|
||||
} finally {
|
||||
await brainy.close()
|
||||
}
|
||||
})
|
||||
})
|
||||
|
|
@ -5,7 +5,7 @@
|
|||
* No mocks, no fakes, real implementation
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { NounType } from '../../src/types/graphTypes.js'
|
||||
|
||||
|
|
@ -21,10 +21,6 @@ describe('Brainy 3.0 Core (Unit Tests)', () => {
|
|||
await brain.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
describe('CRUD Operations', () => {
|
||||
it('should create items with add', async () => {
|
||||
const id = await brain.add({
|
||||
|
|
|
|||
|
|
@ -19,19 +19,13 @@ import { prodLog } from '../../../src/utils/logger.js'
|
|||
const UUID = (suffix: string): string => `00000000-0000-4000-8000-0000000000${suffix}`
|
||||
|
||||
describe('Finding 10 — degraded derived-index state is surfaced on reads', () => {
|
||||
const opened: Brainy[] = []
|
||||
|
||||
beforeEach(() => {
|
||||
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
|
||||
})
|
||||
afterEach(async () => {
|
||||
vi.restoreAllMocks()
|
||||
for (const b of opened.splice(0)) await b.close().catch(() => {})
|
||||
})
|
||||
afterEach(() => vi.restoreAllMocks())
|
||||
|
||||
it('checkHealth() reports adopt-forward degraded ids as unhealthy', async () => {
|
||||
const brain = new Brainy({ storage: { type: 'memory' }, dimensions: 384, requireSubtype: false })
|
||||
opened.push(brain)
|
||||
await brain.init()
|
||||
;(brain as any)._indexDegradedIds.add(UUID('de'))
|
||||
|
||||
|
|
@ -43,7 +37,6 @@ describe('Finding 10 — degraded derived-index state is surfaced on reads', ()
|
|||
it('find()/get() warn loudly while degraded, ONCE, then repairIndex() clears it', async () => {
|
||||
const warn = vi.spyOn(prodLog, 'warn').mockImplementation(() => {})
|
||||
const brain = new Brainy({ storage: { type: 'memory' }, dimensions: 384, requireSubtype: false })
|
||||
opened.push(brain)
|
||||
await brain.init()
|
||||
await brain.add({ id: UUID('a1'), data: 'x', type: NounType.Document })
|
||||
;(brain as any)._indexRebuildFailed = new Error('rebuild boom')
|
||||
|
|
@ -66,7 +59,6 @@ describe('Finding 10 — degraded derived-index state is surfaced on reads', ()
|
|||
|
||||
it('persistSingleOp records receipt.degraded (widened return type, not dropped)', async () => {
|
||||
const brain = new Brainy({ storage: { type: 'memory' }, dimensions: 384, requireSubtype: false })
|
||||
opened.push(brain)
|
||||
await brain.init()
|
||||
// Simulate a degraded receipt by wrapping the generation store's commitSingleOp.
|
||||
const gs: any = (brain as any).generationStore
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@
|
|||
* soft-delete semantic: `field !== value` MUST include entities that have no
|
||||
* such field at all.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy } from '../../../src/brainy'
|
||||
import { NounType } from '../../../src/types/graphTypes'
|
||||
|
||||
|
|
@ -26,10 +26,6 @@ describe('find() complement operators (ne / exists:false / missing:true)', () =>
|
|||
ids.noField2 = await brain.add({ data: 'n2', type: NounType.Thing, metadata: { other: 2 } })
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('ne returns everything except the matching value — INCLUDING entities without the field', async () => {
|
||||
const rows = await brain.find({ where: { status: { ne: 'active' } }, limit: 100 })
|
||||
const got = new Set(rows.map((r) => r.id))
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@
|
|||
* returns an id whose record matches NEITHER the type nor the where filter) and
|
||||
* assert the phantom is dropped while the genuine matches survive.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy } from '../../../src/brainy'
|
||||
import { NounType } from '../../../src/types/graphTypes'
|
||||
|
||||
|
|
@ -48,10 +48,6 @@ describe('find() index-integrity guard (phantom row class)', () => {
|
|||
})
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('healthy index: the discriminant query returns only the staff Person', async () => {
|
||||
const rows = await brain.find({ type: NounType.Person, where: { entityType: 'staff' }, limit: 100 })
|
||||
expect(rows.map((r) => r.id)).toEqual([staffId])
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy } from '../../../src/brainy'
|
||||
import { createAddParams } from '../../helpers/test-factory'
|
||||
import { NounType } from '../../../src/types/graphTypes'
|
||||
|
|
@ -13,10 +13,6 @@ describe('Brainy.find()', () => {
|
|||
await brain.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
describe('success paths', () => {
|
||||
it('should find entities by text query', async () => {
|
||||
// Arrange
|
||||
|
|
|
|||
|
|
@ -5,7 +5,6 @@
|
|||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { Brainy } from '../../../src/brainy'
|
||||
import { MetadataArrayTooLargeError, MAX_INDEXED_ARRAY_LENGTH } from '../../../src/errors/brainyError'
|
||||
import {
|
||||
createAddParams,
|
||||
generateTestVector,
|
||||
|
|
@ -269,23 +268,13 @@ describe('Brainy.get()', () => {
|
|||
expect(entity!.id).toBe(id)
|
||||
})
|
||||
|
||||
// THE INDEXABLE-ARRAY BOUND, from get()'s side. This case used to park a
|
||||
// 1000-element array in the metadata bag and assert it came back. That
|
||||
// shape is refused at the write door now — an array field mints one
|
||||
// posting per element, so an unbounded array is an unbounded write — so
|
||||
// the case pins BOTH halves of the law that replaced it: a large SCALAR
|
||||
// payload still round-trips whole, and an array over the bound refuses by
|
||||
// name. Every length derives from MAX_INDEXED_ARRAY_LENGTH so the pin
|
||||
// follows the constant wherever it moves.
|
||||
it('should get an entity with a large scalar metadata payload', async () => {
|
||||
// Arrange — large in every dimension EXCEPT array length: a long string,
|
||||
// many fields, deep nesting, and an array sitting exactly ON the bound.
|
||||
it('should get entity with very large metadata', async () => {
|
||||
// Arrange
|
||||
const largeMetadata = {
|
||||
atTheBound: Array.from({ length: MAX_INDEXED_ARRAY_LENGTH }, (_, i) => `item${i}`),
|
||||
bigArray: new Array(1000).fill('item'),
|
||||
bigObject: Object.fromEntries(
|
||||
Array.from({ length: 100 }, (_, i) => [`key${i}`, `value${i}`])
|
||||
),
|
||||
longString: 'x'.repeat(10_000),
|
||||
deepNesting: Array(10).fill(null).reduce(
|
||||
(acc) => ({ nested: acc }),
|
||||
{ value: 'deep' }
|
||||
|
|
@ -301,43 +290,10 @@ describe('Brainy.get()', () => {
|
|||
// Act
|
||||
const entity = await brain.get(id)
|
||||
|
||||
// Assert — the payload comes back whole, first element to last
|
||||
// Assert
|
||||
expect(entity).not.toBeNull()
|
||||
expect(entity!.metadata.atTheBound).toHaveLength(MAX_INDEXED_ARRAY_LENGTH)
|
||||
expect(entity!.metadata.atTheBound[0]).toBe('item0')
|
||||
expect(entity!.metadata.atTheBound[MAX_INDEXED_ARRAY_LENGTH - 1])
|
||||
.toBe(`item${MAX_INDEXED_ARRAY_LENGTH - 1}`)
|
||||
expect(entity!.metadata.bigArray).toHaveLength(1000)
|
||||
expect(Object.keys(entity!.metadata.bigObject)).toHaveLength(100)
|
||||
expect(entity!.metadata.longString).toHaveLength(10_000)
|
||||
|
||||
// ...including the deep nest, walked to the bottom.
|
||||
let cursor: any = entity!.metadata.deepNesting
|
||||
for (let depth = 0; depth < 10; depth++) cursor = cursor.nested
|
||||
expect(cursor.value).toBe('deep')
|
||||
})
|
||||
|
||||
it('should refuse a metadata array over the indexing bound, by name', async () => {
|
||||
// Arrange
|
||||
const overTheBound = MAX_INDEXED_ARRAY_LENGTH + 1
|
||||
|
||||
// Act
|
||||
const err = await brain
|
||||
.add(createAddParams({
|
||||
data: 'Large metadata',
|
||||
type: 'thing',
|
||||
metadata: { bigArray: new Array(overTheBound).fill('item') }
|
||||
}))
|
||||
.catch((e: any) => e)
|
||||
|
||||
// Assert — the field, the length and the bound, on the error and in the
|
||||
// message, so a handler can report or repair without parsing prose.
|
||||
expect(err).toBeInstanceOf(MetadataArrayTooLargeError)
|
||||
expect(err.field).toBe('bigArray')
|
||||
expect(err.length).toBe(overTheBound)
|
||||
expect(err.limit).toBe(MAX_INDEXED_ARRAY_LENGTH)
|
||||
expect(err.message).toContain('bigArray')
|
||||
expect(err.message).toContain(String(overTheBound))
|
||||
expect(err.message).toContain(String(MAX_INDEXED_ARRAY_LENGTH))
|
||||
})
|
||||
})
|
||||
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@
|
|||
* exercised by cor's combined matrix); they inject probe/spy hooks onto the live JS
|
||||
* metadata index, which has neither method by default.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy } from '../../../src/brainy'
|
||||
import { NounType } from '../../../src/types/graphTypes'
|
||||
|
||||
|
|
@ -34,10 +34,6 @@ describe('metadata-provider contract wiring (getIdsForFilter opts)', () => {
|
|||
mi = (brain as any).metadataIndex
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('RETIRED: a read never calls probeConsistency() / self-heals via detectAndRepairCorruption — that is the read-triggered dark rebuild the health-gate law forbids', async () => {
|
||||
let probes = 0
|
||||
let repairs = 0
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
* gate that hung getStats / readdir / readFile behind an unrelated family's
|
||||
* migration until the wait timed out.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy } from '../../../src/brainy.js'
|
||||
import { MigrationInProgressError } from '../../../src/errors/brainyError.js'
|
||||
|
||||
|
|
@ -38,19 +38,12 @@ const jam = (provider: unknown) => {
|
|||
}
|
||||
|
||||
describe('migration LOCK is family-scoped', () => {
|
||||
const opened: Brainy[] = []
|
||||
|
||||
beforeEach(() => {
|
||||
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
for (const b of opened.splice(0)) await b.close().catch(() => {})
|
||||
})
|
||||
|
||||
it('a stuck VECTOR migration does not block canonical or graph/metadata reads', async () => {
|
||||
const brain = await seed()
|
||||
opened.push(brain)
|
||||
const childId = (
|
||||
(await brain.vfs.readdir('/notes', { withFileTypes: true })) as Array<{ entityId: string }>
|
||||
)[0].entityId
|
||||
|
|
@ -67,7 +60,6 @@ describe('migration LOCK is family-scoped', () => {
|
|||
|
||||
it('a stuck VECTOR migration STILL blocks a read that needs the vector family', async () => {
|
||||
const brain = await seed()
|
||||
opened.push(brain)
|
||||
jam((brain as any).index)
|
||||
|
||||
// A semantic query consults the vector index — it must wait, and (bounded by
|
||||
|
|
@ -78,7 +70,6 @@ describe('migration LOCK is family-scoped', () => {
|
|||
|
||||
it('a stuck GRAPH migration blocks traversal but not vector/canonical reads', async () => {
|
||||
const brain = await seed()
|
||||
opened.push(brain)
|
||||
const childId = (
|
||||
(await brain.vfs.readdir('/notes', { withFileTypes: true })) as Array<{ entityId: string }>
|
||||
)[0].entityId
|
||||
|
|
@ -96,7 +87,6 @@ describe('migration LOCK is family-scoped', () => {
|
|||
|
||||
it('with no migration in flight, every read serves (the fast path is a no-op)', async () => {
|
||||
const brain = await seed()
|
||||
opened.push(brain)
|
||||
await expect(brain.getStats()).resolves.toBeDefined()
|
||||
await expect(brain.find({ query: 'doc' })).resolves.toBeDefined()
|
||||
await expect(brain.vfs.readdir('/notes')).resolves.toHaveLength(1)
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ describe('Duplicate Check Optimization', () => {
|
|||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
// Cleanup is automatic with memory storage
|
||||
})
|
||||
|
||||
it('should detect duplicate relationships using GraphAdjacencyIndex', async () => {
|
||||
|
|
|
|||
|
|
@ -5,7 +5,6 @@
|
|||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { Brainy } from '../../../src/brainy'
|
||||
import { MetadataArrayTooLargeError, MAX_INDEXED_ARRAY_LENGTH } from '../../../src/errors/brainyError'
|
||||
import {
|
||||
createAddParams,
|
||||
createTestConfig,
|
||||
|
|
@ -249,21 +248,14 @@ describe('Brainy.relate()', () => {
|
|||
expect(matches.length).toBe(1) // Only one relationship should exist
|
||||
})
|
||||
|
||||
// THE INDEXABLE-ARRAY BOUND, from relate()'s side. This case used to pass a
|
||||
// 100-element array through relate() and assert it came back — a length
|
||||
// hardcoded either side of a bound it never named, so it read green or red
|
||||
// purely by where the constant happened to sit. Both halves of the law are
|
||||
// pinned here instead, and every length derives from
|
||||
// MAX_INDEXED_ARRAY_LENGTH so the pin follows the constant.
|
||||
it('should handle a large scalar metadata payload on a relation', async () => {
|
||||
// Arrange — large in every dimension EXCEPT array length: a long string,
|
||||
// many fields, and an array sitting exactly ON the bound.
|
||||
it('should handle very long metadata', async () => {
|
||||
// Arrange
|
||||
const largeMetadata = {
|
||||
atTheBound: Array.from({ length: MAX_INDEXED_ARRAY_LENGTH }, (_, i) => `item${i}`),
|
||||
bigArray: new Array(100).fill('item'),
|
||||
bigObject: Object.fromEntries(
|
||||
Array.from({ length: 50 }, (_, i) => [`key${i}`, `value${i}`])
|
||||
),
|
||||
longString: 'x'.repeat(10_000)
|
||||
longString: 'x'.repeat(1000)
|
||||
}
|
||||
|
||||
// Act
|
||||
|
|
@ -274,45 +266,11 @@ describe('Brainy.relate()', () => {
|
|||
metadata: largeMetadata
|
||||
})
|
||||
|
||||
// Assert — the payload comes back whole, first element to last
|
||||
// Assert
|
||||
const relations = await brain.related({ from: entity1Id })
|
||||
const relation = relations.find(r => r.to === entity2Id)
|
||||
expect(relation).toBeDefined()
|
||||
expect(relation!.metadata?.atTheBound).toHaveLength(MAX_INDEXED_ARRAY_LENGTH)
|
||||
expect(relation!.metadata?.atTheBound[0]).toBe('item0')
|
||||
expect(relation!.metadata?.atTheBound[MAX_INDEXED_ARRAY_LENGTH - 1])
|
||||
.toBe(`item${MAX_INDEXED_ARRAY_LENGTH - 1}`)
|
||||
expect(Object.keys(relation!.metadata?.bigObject)).toHaveLength(50)
|
||||
expect(relation!.metadata?.longString).toHaveLength(10_000)
|
||||
})
|
||||
|
||||
it('should refuse a relation metadata array over the indexing bound, by name', async () => {
|
||||
// Arrange
|
||||
const overTheBound = MAX_INDEXED_ARRAY_LENGTH + 1
|
||||
|
||||
// Act
|
||||
const err = await brain
|
||||
.relate({
|
||||
from: entity1Id,
|
||||
to: entity3Id,
|
||||
type: 'relatedTo',
|
||||
metadata: { bigArray: new Array(overTheBound).fill('item') }
|
||||
})
|
||||
.catch((e: any) => e)
|
||||
|
||||
// Assert — the field, the length and the bound, on the error and in the
|
||||
// message, so a handler can report or repair without parsing prose.
|
||||
expect(err).toBeInstanceOf(MetadataArrayTooLargeError)
|
||||
expect(err.field).toBe('bigArray')
|
||||
expect(err.length).toBe(overTheBound)
|
||||
expect(err.limit).toBe(MAX_INDEXED_ARRAY_LENGTH)
|
||||
expect(err.message).toContain('bigArray')
|
||||
expect(err.message).toContain(String(overTheBound))
|
||||
expect(err.message).toContain(String(MAX_INDEXED_ARRAY_LENGTH))
|
||||
|
||||
// Refused means not written: no relation of this shape exists.
|
||||
const relations = await brain.related({ from: entity1Id })
|
||||
expect(relations.some(r => r.to === entity3Id && r.metadata?.bigArray)).toBe(false)
|
||||
expect(relation!.metadata?.bigArray).toHaveLength(100)
|
||||
})
|
||||
|
||||
it('should handle special characters in metadata', async () => {
|
||||
|
|
|
|||
|
|
@ -5,7 +5,6 @@
|
|||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { Brainy } from '../../../src/brainy'
|
||||
import { MetadataArrayTooLargeError, MAX_INDEXED_ARRAY_LENGTH } from '../../../src/errors/brainyError'
|
||||
import {
|
||||
createAddParams,
|
||||
createTestConfig,
|
||||
|
|
@ -356,27 +355,18 @@ describe('Brainy.update()', () => {
|
|||
expect(final!.metadata.counter).toBeLessThanOrEqual(10)
|
||||
})
|
||||
|
||||
// THE INDEXABLE-ARRAY BOUND, from update()'s side. This case used to write
|
||||
// a 1000-element array through update() and assert it came back. That
|
||||
// shape is refused at the write door now — an array field mints one
|
||||
// posting per element, so an unbounded array is an unbounded write — so
|
||||
// the case pins BOTH halves of the law that replaced it. Every length
|
||||
// derives from MAX_INDEXED_ARRAY_LENGTH so the pin follows the constant.
|
||||
it('should handle a large scalar metadata update', async () => {
|
||||
it('should handle very large metadata updates', async () => {
|
||||
// Arrange
|
||||
const id = await brain.add(createAddParams({
|
||||
data: 'Large metadata test',
|
||||
type: 'thing'
|
||||
}))
|
||||
|
||||
// Large in every dimension EXCEPT array length: a long string, many
|
||||
// fields, deep nesting, and an array sitting exactly ON the bound.
|
||||
const largeMetadata = {
|
||||
atTheBound: Array.from({ length: MAX_INDEXED_ARRAY_LENGTH }, (_, i) => `item${i}`),
|
||||
bigArray: new Array(1000).fill('item'),
|
||||
bigObject: Object.fromEntries(
|
||||
Array.from({ length: 100 }, (_, i) => [`key${i}`, `value${i}`])
|
||||
),
|
||||
longString: 'x'.repeat(10_000),
|
||||
deepNesting: Array(10).fill(null).reduce(
|
||||
(acc) => ({ nested: acc }),
|
||||
{ value: 'deep' }
|
||||
|
|
@ -390,54 +380,11 @@ describe('Brainy.update()', () => {
|
|||
merge: false
|
||||
})
|
||||
|
||||
// Assert — the payload comes back whole, first element to last
|
||||
// Assert
|
||||
const updated = await brain.get(id)
|
||||
expect(updated).not.toBeNull()
|
||||
expect(updated!.metadata.atTheBound).toHaveLength(MAX_INDEXED_ARRAY_LENGTH)
|
||||
expect(updated!.metadata.atTheBound[0]).toBe('item0')
|
||||
expect(updated!.metadata.atTheBound[MAX_INDEXED_ARRAY_LENGTH - 1])
|
||||
.toBe(`item${MAX_INDEXED_ARRAY_LENGTH - 1}`)
|
||||
expect(updated!.metadata.bigArray).toHaveLength(1000)
|
||||
expect(Object.keys(updated!.metadata.bigObject)).toHaveLength(100)
|
||||
expect(updated!.metadata.longString).toHaveLength(10_000)
|
||||
|
||||
// ...including the deep nest, walked to the bottom.
|
||||
let cursor: any = updated!.metadata.deepNesting
|
||||
for (let depth = 0; depth < 10; depth++) cursor = cursor.nested
|
||||
expect(cursor.value).toBe('deep')
|
||||
})
|
||||
|
||||
it('should refuse an update whose metadata array is over the indexing bound, by name', async () => {
|
||||
// Arrange
|
||||
const id = await brain.add(createAddParams({
|
||||
data: 'Large metadata test',
|
||||
type: 'thing',
|
||||
metadata: { keep: 'me' }
|
||||
}))
|
||||
const overTheBound = MAX_INDEXED_ARRAY_LENGTH + 1
|
||||
|
||||
// Act
|
||||
const err = await brain
|
||||
.update({
|
||||
id,
|
||||
metadata: { bigArray: new Array(overTheBound).fill('item') },
|
||||
merge: false
|
||||
})
|
||||
.catch((e: any) => e)
|
||||
|
||||
// Assert — the field, the length and the bound, on the error and in the
|
||||
// message, so a handler can report or repair without parsing prose.
|
||||
expect(err).toBeInstanceOf(MetadataArrayTooLargeError)
|
||||
expect(err.field).toBe('bigArray')
|
||||
expect(err.length).toBe(overTheBound)
|
||||
expect(err.limit).toBe(MAX_INDEXED_ARRAY_LENGTH)
|
||||
expect(err.message).toContain('bigArray')
|
||||
expect(err.message).toContain(String(overTheBound))
|
||||
expect(err.message).toContain(String(MAX_INDEXED_ARRAY_LENGTH))
|
||||
|
||||
// Refused means unchanged: the row still carries what it had before.
|
||||
const unchanged = await brain.get(id)
|
||||
expect(unchanged!.metadata.keep).toBe('me')
|
||||
expect(unchanged!.metadata.bigArray).toBeUndefined()
|
||||
})
|
||||
|
||||
it('should preserve entity ID during update', async () => {
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@
|
|||
* _indexRebuildFailed / _indexDegradedIds degraded states (mirroring
|
||||
* validateIndexConsistency / checkHealth).
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy, NounType } from '../../src/index.js'
|
||||
|
||||
describe('getIndexStatus honest readiness (Finding 9)', () => {
|
||||
|
|
@ -20,10 +20,6 @@ describe('getIndexStatus honest readiness (Finding 9)', () => {
|
|||
await brain.flush()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('a not-ready provider makes populated honest (false) and exposes ready:false', async () => {
|
||||
brain.index.isReady = () => false // count present, serving structure NOT loaded
|
||||
const status = await brain.getIndexStatus()
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
* scan; and a one-shot probe self-heals a no-isReady provider whose adjacency
|
||||
* did not cold-load.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy, NounType, VerbType } from '../../../src/index.js'
|
||||
|
||||
describe('graph fast-path honest readiness (Finding 2)', () => {
|
||||
|
|
@ -33,10 +33,6 @@ describe('graph fast-path honest readiness (Finding 2)', () => {
|
|||
await storage.getVerbsBySource(a)
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('not-ready provider → shard scan returns the REAL edges, not a silent []', async () => {
|
||||
const gi = storage.graphIndex
|
||||
// Simulate a cold native provider: count/manifest loaded (isInitialized) but
|
||||
|
|
|
|||
|
|
@ -1,241 +0,0 @@
|
|||
/**
|
||||
* @module column-store-mixed-kind.test
|
||||
* @description Typed posting lists: one field, several value KINDS, each
|
||||
* answerable on its own.
|
||||
*
|
||||
* The behaviour these pin replaced a first-writer type freeze. The first value
|
||||
* a field ever saw fixed that field's type; every later value of another kind
|
||||
* was coerced to it, and when coercion failed — `Number('electronics')` — the
|
||||
* value was dropped from the index with no error at all. The row stayed
|
||||
* readable by id and by vector and vanished from every equality filter on the
|
||||
* field. These tests therefore care about ORDER: strings-then-numbers and
|
||||
* numbers-then-strings have to behave identically, because neither writer owns
|
||||
* the field.
|
||||
*
|
||||
* Kinds never coerce into one another at query time either. `5` and `'5'` are
|
||||
* different values and match different rows.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { ColumnStore } from '../../../../src/indexes/columnStore/ColumnStore.js'
|
||||
import { MemoryStorage } from '../../../../src/storage/adapters/memoryStorage.js'
|
||||
import { EntityIdMapper } from '../../../../src/utils/entityIdMapper.js'
|
||||
|
||||
describe('ColumnStore — typed posting lists per (field, kind)', () => {
|
||||
let storage: MemoryStorage
|
||||
let idMapper: EntityIdMapper
|
||||
let store: ColumnStore
|
||||
|
||||
beforeEach(async () => {
|
||||
storage = new MemoryStorage()
|
||||
await storage.init()
|
||||
idMapper = new EntityIdMapper({ storage, storageKey: 'test:idMapper' })
|
||||
await idMapper.init()
|
||||
|
||||
store = new ColumnStore({ flushThreshold: 10 })
|
||||
await store.init(storage, idMapper)
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await store.close()
|
||||
})
|
||||
|
||||
/** Resolve a filter to the sorted UUIDs it matched. */
|
||||
const uuidsOf = async (field: string, value: unknown): Promise<string[]> => {
|
||||
const bitmap = await store.filter(field, value)
|
||||
return Array.from(bitmap)
|
||||
.map((id) => idMapper.getUuid(Number(id)))
|
||||
.filter((u): u is string => u !== undefined)
|
||||
.sort()
|
||||
}
|
||||
|
||||
describe('equality answers on the query value’s own kind', () => {
|
||||
it('serves numbers written AFTER strings on the same field', async () => {
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('s1')), { category: 'electronics' })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('s2')), { category: 'furniture' })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('n1')), { category: 5 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('n2')), { category: 5 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('n3')), { category: 7 })
|
||||
|
||||
// The numbers are in the index, though a string got there first.
|
||||
expect(await uuidsOf('category', 5)).toEqual(['n1', 'n2'])
|
||||
expect(await uuidsOf('category', 7)).toEqual(['n3'])
|
||||
// And the strings did not move.
|
||||
expect(await uuidsOf('category', 'electronics')).toEqual(['s1'])
|
||||
expect(await uuidsOf('category', 'furniture')).toEqual(['s2'])
|
||||
})
|
||||
|
||||
it('serves strings written AFTER numbers on the same field', async () => {
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('n1')), { category: 5 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('n2')), { category: 5 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('s1')), { category: 'electronics' })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('s2')), { category: 'electronics' })
|
||||
|
||||
// 'electronics' would have become NaN and been dropped under the freeze.
|
||||
expect(await uuidsOf('category', 'electronics')).toEqual(['s1', 's2'])
|
||||
expect(await uuidsOf('category', 5)).toEqual(['n1', 'n2'])
|
||||
})
|
||||
|
||||
it('does not coerce a number query into the string postings, or back', async () => {
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('num')), { code: 5 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('str')), { code: '5' })
|
||||
|
||||
expect(await uuidsOf('code', 5)).toEqual(['num'])
|
||||
expect(await uuidsOf('code', '5')).toEqual(['str'])
|
||||
})
|
||||
|
||||
it('serves booleans mixed into a field that already holds strings and numbers', async () => {
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('s1')), { flag: 'yes' })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('n1')), { flag: 1 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('b1')), { flag: true })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('b2')), { flag: false })
|
||||
|
||||
expect(await uuidsOf('flag', true)).toEqual(['b1'])
|
||||
expect(await uuidsOf('flag', false)).toEqual(['b2'])
|
||||
// `true` stores as 1 internally; that is an encoding, not a value.
|
||||
expect(await uuidsOf('flag', 1)).toEqual(['n1'])
|
||||
expect(await uuidsOf('flag', 'yes')).toEqual(['s1'])
|
||||
})
|
||||
|
||||
it('answers nothing — not something coerced — for a kind the field never held', async () => {
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('s1')), { category: 'electronics' })
|
||||
|
||||
expect(await uuidsOf('category', 5)).toEqual([])
|
||||
expect(await uuidsOf('category', true)).toEqual([])
|
||||
})
|
||||
|
||||
it('holds every kind across a flush, not just the one in the tail buffer', async () => {
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('s1')), { category: 'electronics' })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('n1')), { category: 5 })
|
||||
await store.flush()
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('s2')), { category: 'electronics' })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('n2')), { category: 5 })
|
||||
|
||||
expect(await uuidsOf('category', 'electronics')).toEqual(['s1', 's2'])
|
||||
expect(await uuidsOf('category', 5)).toEqual(['n1', 'n2'])
|
||||
})
|
||||
})
|
||||
|
||||
describe('range filters read the numeric postings', () => {
|
||||
it('ranges over the numeric subset of a mixed field, ignoring its strings', async () => {
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('cheap')), { price: 100 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('mid')), { price: 500 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('dear')), { price: 900 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('unpriced')), { price: 'on request' })
|
||||
await store.flush()
|
||||
|
||||
const inRange = await store.rangeQuery('price', 200, 1000)
|
||||
const uuids = Array.from(inRange)
|
||||
.map((id) => idMapper.getUuid(Number(id)))
|
||||
.sort()
|
||||
expect(uuids).toEqual(['dear', 'mid'])
|
||||
})
|
||||
|
||||
it('an unbounded range still reports every kind — it is the “has a value” probe', async () => {
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('n1')), { mixed: 42 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('s1')), { mixed: 'text' })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('b1')), { mixed: true })
|
||||
await store.flush()
|
||||
|
||||
const anyValue = await store.rangeQuery('mixed')
|
||||
const uuids = Array.from(anyValue)
|
||||
.map((id) => idMapper.getUuid(Number(id)))
|
||||
.sort()
|
||||
expect(uuids).toEqual(['b1', 'n1', 's1'])
|
||||
})
|
||||
})
|
||||
|
||||
describe('the index reports what a field actually holds', () => {
|
||||
it('names every kind present, not the one that got there first', async () => {
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('s1')), { category: 'electronics' })
|
||||
expect(store.getFieldKinds('category')).toEqual(['string'])
|
||||
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('n1')), { category: 5 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('b1')), { category: true })
|
||||
expect(store.getFieldKinds('category')).toEqual(['number', 'string', 'boolean'])
|
||||
|
||||
// And the field is still ONE field by name.
|
||||
expect(store.getIndexedFields()).toEqual(['category'])
|
||||
expect(store.hasField('category')).toBe(true)
|
||||
})
|
||||
|
||||
it('reports an unknown field as holding nothing', () => {
|
||||
expect(store.getFieldKinds('never-written')).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('an integer column widens rather than rounding', () => {
|
||||
it('keeps a non-integer written after integers as itself', async () => {
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('a')), { score: 4 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('b')), { score: 4.5 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('c')), { score: 5 })
|
||||
await store.flush()
|
||||
|
||||
// 4.5 used to round to 5 and answer `score === 5` alongside c.
|
||||
expect(await uuidsOf('score', 4.5)).toEqual(['b'])
|
||||
expect(await uuidsOf('score', 5)).toEqual(['c'])
|
||||
expect(await uuidsOf('score', 4)).toEqual(['a'])
|
||||
})
|
||||
})
|
||||
|
||||
describe('close then reopen', () => {
|
||||
it('keeps every typed posting, on the same storage', async () => {
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('s1')), { category: 'electronics' })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('n1')), { category: 5 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('b1')), { category: true })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('f1')), { score: 1.5 })
|
||||
await store.flush()
|
||||
await store.close()
|
||||
|
||||
store = new ColumnStore({ flushThreshold: 10 })
|
||||
await store.init(storage, idMapper)
|
||||
|
||||
expect(store.getFieldKinds('category')).toEqual(['number', 'string', 'boolean'])
|
||||
expect(await uuidsOf('category', 'electronics')).toEqual(['s1'])
|
||||
expect(await uuidsOf('category', 5)).toEqual(['n1'])
|
||||
expect(await uuidsOf('category', true)).toEqual(['b1'])
|
||||
expect(await uuidsOf('score', 1.5)).toEqual(['f1'])
|
||||
})
|
||||
|
||||
it('accepts new values of every kind after the reopen', async () => {
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('s1')), { category: 'electronics' })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('n1')), { category: 5 })
|
||||
await store.flush()
|
||||
await store.close()
|
||||
|
||||
store = new ColumnStore({ flushThreshold: 10 })
|
||||
await store.init(storage, idMapper)
|
||||
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('s2')), { category: 'electronics' })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('n2')), { category: 5 })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('b1')), { category: false })
|
||||
await store.flush()
|
||||
|
||||
expect(await uuidsOf('category', 'electronics')).toEqual(['s1', 's2'])
|
||||
expect(await uuidsOf('category', 5)).toEqual(['n1', 'n2'])
|
||||
expect(await uuidsOf('category', false)).toEqual(['b1'])
|
||||
})
|
||||
|
||||
it('opens an index written by the pre-typed-postings shape and reads it unchanged', async () => {
|
||||
// A single-kind field is byte-identical to what the old writer produced:
|
||||
// one manifest at `_column_index/<field>/MANIFEST.json`, no kind
|
||||
// subdirectory anywhere. That IS the old on-disk shape, so proving the
|
||||
// new reader serves it proves an old index still opens.
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('a')), { status: 'active' })
|
||||
store.addEntity(BigInt(idMapper.getOrAssign('b')), { status: 'archived' })
|
||||
await store.flush()
|
||||
|
||||
const keys = await (storage as unknown as {
|
||||
listObjectsUnderPath: (prefix: string) => Promise<string[]>
|
||||
}).listObjectsUnderPath('_column_index/')
|
||||
expect(keys.some((k) => k.includes('/k/'))).toBe(false)
|
||||
|
||||
await store.close()
|
||||
store = new ColumnStore({ flushThreshold: 10 })
|
||||
await store.init(storage, idMapper)
|
||||
|
||||
expect(store.getFieldKinds('status')).toEqual(['string'])
|
||||
expect(await uuidsOf('status', 'active')).toEqual(['a'])
|
||||
})
|
||||
})
|
||||
})
|
||||
|
|
@ -15,7 +15,7 @@
|
|||
* The 8.0 JS index cold-loads correctly, so we simulate the cold native failure
|
||||
* mode by intercepting the provider's getIdsForFilter/rebuild.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy, NounType, MetadataIndexNotReadyError } from '../../src/index.js'
|
||||
|
||||
const V = () => Array.from({ length: 384 }, (_, i) => Math.sin(i * 0.1) + 0.001)
|
||||
|
|
@ -31,10 +31,6 @@ describe('Metadata cold-read guard (#venue silent-[])', () => {
|
|||
await brain.flush()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('warm brain: filtered find is correct and the guard does not rebuild', async () => {
|
||||
const mi = brain.metadataIndex
|
||||
let rebuilds = 0
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@
|
|||
* the production feature-detection reads it.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy, NounType, MigrationInProgressError } from '../../src/index.js'
|
||||
import { GraphAdjacencyIndex } from '../../src/graph/graphAdjacencyIndex.js'
|
||||
|
||||
|
|
@ -39,12 +39,6 @@ describe('Migration LOCK (#18) — coordinated 7.x→8.0 auto-upgrade', () => {
|
|||
await brain.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
// The "close() is not gated" test already closes `brain` itself as its
|
||||
// own assertion — closing an already-closed brain is a safe no-op here.
|
||||
await brain.close().catch(() => {})
|
||||
})
|
||||
|
||||
it('does not gate operations when no provider is migrating (fast path)', async () => {
|
||||
const id = await brain.add({ data: 'hello', type: NounType.Concept })
|
||||
expect(id).toBeTruthy()
|
||||
|
|
@ -136,9 +130,6 @@ describe('Migration LOCK (#18) — coordinated 7.x→8.0 auto-upgrade', () => {
|
|||
expect(e).toBeInstanceOf(MigrationInProgressError)
|
||||
expect(e.retryable).toBe(true)
|
||||
expect(typeof e.elapsedMs).toBe('number')
|
||||
} finally {
|
||||
// close() is proven not-gated by the test below — safe even mid-migration.
|
||||
await shortBrain.close()
|
||||
}
|
||||
})
|
||||
|
||||
|
|
|
|||
|
|
@ -13,11 +13,10 @@ describe('EmbeddingSignal', () => {
|
|||
signal = new EmbeddingSignal(brain)
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
afterEach(() => {
|
||||
signal.clearCache()
|
||||
signal.clearHistory()
|
||||
signal.resetStats()
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
describe('initialization', () => {
|
||||
|
|
|
|||
|
|
@ -89,14 +89,12 @@ describe('Guarded plugin auto-detection (plugins: undefined)', () => {
|
|||
})
|
||||
const brain: any = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, silent: true })
|
||||
await expect(brain.init()).rejects.toThrow(/installed but failed to load/)
|
||||
await brain.close().catch(() => {})
|
||||
})
|
||||
|
||||
it('installed but not a valid plugin (missing activate) → init() throws', async () => {
|
||||
stubImport(async () => ({ default: { name: '@soulcraft/cor' } })) // no activate()
|
||||
const brain: any = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, silent: true })
|
||||
await expect(brain.init()).rejects.toThrow(/not a valid Brainy plugin/)
|
||||
await brain.close().catch(() => {})
|
||||
})
|
||||
|
||||
it('installed but activation fails → init() throws (activateAll posture applies)', async () => {
|
||||
|
|
@ -110,7 +108,6 @@ describe('Guarded plugin auto-detection (plugins: undefined)', () => {
|
|||
}))
|
||||
const brain: any = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, silent: true })
|
||||
await expect(brain.init()).rejects.toThrow(/failed to activate/)
|
||||
await brain.close().catch(() => {})
|
||||
})
|
||||
|
||||
it('plugins: [] and plugins: false → no probe at all (explicit opt-out)', async () => {
|
||||
|
|
@ -135,6 +132,5 @@ describe('Guarded plugin auto-detection (plugins: undefined)', () => {
|
|||
silent: true
|
||||
})
|
||||
await expect(brain.init()).rejects.toThrow(/listed in config\.plugins but could not be loaded/)
|
||||
await brain.close().catch(() => {})
|
||||
})
|
||||
})
|
||||
|
|
|
|||
|
|
@ -143,6 +143,5 @@ describe('version coupling at init() — no silent fallback', () => {
|
|||
plugins: ['@soulcraft/this-package-does-not-exist-xyz']
|
||||
})
|
||||
await expect(brain.init()).rejects.toThrow(/could not be loaded|config\.plugins/)
|
||||
await brain.close().catch(() => {})
|
||||
})
|
||||
})
|
||||
|
|
|
|||
|
|
@ -298,10 +298,9 @@ describe('Brainy plugin integration', () => {
|
|||
// must surface as a failed init(), NOT a silent degrade to the default
|
||||
// engine (the version-coupling guard; see plugin-version-coupling.test.ts).
|
||||
await expect(brain.init()).rejects.toThrow(/failed to activate|native module not found/)
|
||||
await brain.close().catch(() => {})
|
||||
})
|
||||
|
||||
it('should use() return this for chaining', async () => {
|
||||
it('should use() return this for chaining', () => {
|
||||
const plugin: BrainyPlugin = {
|
||||
name: 'chain-test',
|
||||
activate: async () => true
|
||||
|
|
@ -310,8 +309,5 @@ describe('Brainy plugin integration', () => {
|
|||
const brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' } })
|
||||
const result = brain.use(plugin)
|
||||
expect(result).toBe(brain)
|
||||
// Never init()'d — the constructor still registered it in Brainy's global
|
||||
// instance registry, so it still needs a close() to deregister.
|
||||
await brain.close().catch(() => {})
|
||||
})
|
||||
})
|
||||
|
|
|
|||
|
|
@ -1,403 +0,0 @@
|
|||
/**
|
||||
* scripts/wall-entry.mjs — the mechanical releases-wall entry.
|
||||
*
|
||||
* The script's only real interface is its CLI (it has no importable
|
||||
* exports by design — one door, no parallel API to drift from it), so
|
||||
* these tests spawn it exactly as scripts/release.sh does: as a child
|
||||
* process, against a fixture CHANGELOG and a throwaway local bare repo
|
||||
* standing in for git@source.soulcraft.com:soulcraftlabs/releases.git
|
||||
* (--remote) plus a throwaway cache directory (--cache-dir) standing in
|
||||
* for ~/.cache/soulcraft-releases — never the real remote, never the
|
||||
* real developer cache.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { execFileSync } from 'node:child_process'
|
||||
import { mkdtempSync, rmSync, writeFileSync, readFileSync, chmodSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
|
||||
const SCRIPT = join(process.cwd(), 'scripts/wall-entry.mjs')
|
||||
|
||||
/** Run the script and capture the outcome without throwing on a non-zero exit. */
|
||||
function run(args: string[], cwd: string): { status: number; stdout: string; stderr: string } {
|
||||
try {
|
||||
const stdout = execFileSync('node', [SCRIPT, ...args], { cwd, encoding: 'utf8' })
|
||||
return { status: 0, stdout, stderr: '' }
|
||||
} catch (err: any) {
|
||||
return { status: err.status ?? 1, stdout: err.stdout ?? '', stderr: err.stderr ?? '' }
|
||||
}
|
||||
}
|
||||
|
||||
function git(args: string[], cwd: string): string {
|
||||
return execFileSync('git', ['-C', cwd, ...args], { encoding: 'utf8' }).trim()
|
||||
}
|
||||
|
||||
const CHANGELOG_HEADER = '# Changelog\n\nAll notable changes, in this fixture.\n'
|
||||
|
||||
/** Build a CHANGELOG.md with one entry per [version, bullets[]] pair, newest first. */
|
||||
function buildChangelog(entries: Array<{ version: string; date: string; bullets: string[] }>): string {
|
||||
const body = entries
|
||||
.map(
|
||||
(e) =>
|
||||
`### [${e.version}](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/vX...v${e.version}) (${e.date})\n\n` +
|
||||
e.bullets.map((b) => `- ${b} (abc1234)`).join('\n') +
|
||||
'\n',
|
||||
)
|
||||
.join('\n')
|
||||
return CHANGELOG_HEADER + '\n' + body
|
||||
}
|
||||
|
||||
function wallFile(product: string, entries: unknown[]): string {
|
||||
return JSON.stringify({ product, entries }, null, 2) + '\n'
|
||||
}
|
||||
|
||||
const BASE_ENTRY = {
|
||||
version: '10.4.11',
|
||||
date: '2026-09-02',
|
||||
headline: 'A faster open',
|
||||
items: ['A faster open.'],
|
||||
url: 'https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.11',
|
||||
thumb: null,
|
||||
}
|
||||
|
||||
/** A throwaway bare repo standing in for the real soulcraftlabs/releases remote. */
|
||||
function initBareRemote(): string {
|
||||
const remoteDir = mkdtempSync(join(tmpdir(), 'wall-remote-'))
|
||||
execFileSync('git', ['init', '--bare', '-b', 'main', remoteDir])
|
||||
return remoteDir
|
||||
}
|
||||
|
||||
/** Seed the bare remote with an initial <product>.json, via a throwaway clone. */
|
||||
function seedRemote(remoteDir: string, product: string, entries: unknown[]): void {
|
||||
const seedDir = mkdtempSync(join(tmpdir(), 'wall-seed-'))
|
||||
execFileSync('git', ['clone', remoteDir, seedDir], { stdio: 'ignore' })
|
||||
git(['config', 'user.email', 'seed@example.com'], seedDir)
|
||||
git(['config', 'user.name', 'Seed'], seedDir)
|
||||
writeFileSync(join(seedDir, `${product}.json`), wallFile(product, entries))
|
||||
git(['add', `${product}.json`], seedDir)
|
||||
git(['commit', '-m', 'seed'], seedDir)
|
||||
git(['push', 'origin', 'main'], seedDir)
|
||||
rmSync(seedDir, { recursive: true, force: true })
|
||||
}
|
||||
|
||||
/** Read <product>.json back out of the bare remote's main tip, via a throwaway clone. */
|
||||
function readRemote(remoteDir: string, product: string): any {
|
||||
const readDir = mkdtempSync(join(tmpdir(), 'wall-read-'))
|
||||
execFileSync('git', ['clone', remoteDir, readDir], { stdio: 'ignore' })
|
||||
const data = JSON.parse(readFileSync(join(readDir, `${product}.json`), 'utf8'))
|
||||
rmSync(readDir, { recursive: true, force: true })
|
||||
return data
|
||||
}
|
||||
|
||||
/** Reject every push — stands in for any push failure (including a genuine
|
||||
* non-fast-forward raced by a concurrent release rail), which this script
|
||||
* treats identically: refuse loudly, name the cure, touch nothing further. */
|
||||
function makeRemoteRejectPushes(remoteDir: string): void {
|
||||
const hookPath = join(remoteDir, 'hooks', 'pre-receive')
|
||||
writeFileSync(hookPath, '#!/bin/sh\necho "remote: simulated push rejection" >&2\nexit 1\n')
|
||||
chmodSync(hookPath, 0o755)
|
||||
}
|
||||
|
||||
let dir: string
|
||||
let remoteDir: string
|
||||
let cacheDir: string
|
||||
|
||||
beforeEach(() => {
|
||||
dir = mkdtempSync(join(tmpdir(), 'wall-entry-test-'))
|
||||
// wall-entry.mjs is run with this dir as its cwd, standing in for the real
|
||||
// developer checkout it reads its commit identity from (process.cwd()) —
|
||||
// give it a repo-local identity the same way seedRemote gives one to the
|
||||
// seed clone, so the suite is deterministic on a host with no global git
|
||||
// config (a bare CI box) as much as one with a developer's own.
|
||||
execFileSync('git', ['init', '-q', dir])
|
||||
git(['config', 'user.name', 'Wall Entry Test'], dir)
|
||||
git(['config', 'user.email', 'wall-entry-test@example.com'], dir)
|
||||
remoteDir = initBareRemote()
|
||||
cacheDir = join(mkdtempSync(join(tmpdir(), 'wall-cache-')), 'soulcraft-releases')
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(dir, { recursive: true, force: true })
|
||||
rmSync(remoteDir, { recursive: true, force: true })
|
||||
rmSync(cacheDir, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
describe('wall-entry.mjs — generate + publish', () => {
|
||||
it('derives headline from the first bullet and items from every bullet, hashes stripped, and pushes it to the remote', () => {
|
||||
seedRemote(remoteDir, 'open-brainy', [BASE_ENTRY])
|
||||
writeFileSync(
|
||||
join(dir, 'CHANGELOG.md'),
|
||||
buildChangelog([{ version: '10.4.12', date: '2026-09-03', bullets: ['fix(wall): mechanize the entry', 'test(wall): pin the shape'] }]),
|
||||
)
|
||||
|
||||
const result = run(
|
||||
['--product', 'open-brainy', '--version', '10.4.12', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir],
|
||||
dir,
|
||||
)
|
||||
expect(result.status).toBe(0)
|
||||
expect(result.stdout).toMatch(/wrote v10\.4\.12.*pushed/i)
|
||||
|
||||
const wall = readRemote(remoteDir, 'open-brainy')
|
||||
expect(wall.entries).toHaveLength(2)
|
||||
expect(wall.entries[0]).toEqual({
|
||||
version: '10.4.12',
|
||||
date: '2026-09-03',
|
||||
headline: 'fix(wall): mechanize the entry',
|
||||
items: ['fix(wall): mechanize the entry', 'test(wall): pin the shape'],
|
||||
url: 'https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.12',
|
||||
thumb: null,
|
||||
})
|
||||
// the older entry stays put, still second
|
||||
expect(wall.entries[1].version).toBe('10.4.11')
|
||||
})
|
||||
|
||||
it('prepends newest-first — the new entry lands at index 0 ahead of every existing one', () => {
|
||||
seedRemote(remoteDir, 'open-brainy', [BASE_ENTRY, { ...BASE_ENTRY, version: '10.4.10' }])
|
||||
writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.5.0', date: '2026-09-03', bullets: ['feat: ten five'] }]))
|
||||
|
||||
run(['--product', 'open-brainy', '--version', '10.5.0', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir], dir)
|
||||
|
||||
const wall = readRemote(remoteDir, 'open-brainy')
|
||||
expect(wall.entries.map((e: any) => e.version)).toEqual(['10.5.0', '10.4.11', '10.4.10'])
|
||||
})
|
||||
|
||||
it('replaces an entry with the same version instead of duplicating it — idempotent re-runs', () => {
|
||||
seedRemote(remoteDir, 'open-brainy', [
|
||||
{ ...BASE_ENTRY, headline: 'stale headline, pre-fix' },
|
||||
{ ...BASE_ENTRY, version: '10.4.10' },
|
||||
])
|
||||
writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.4.11', date: '2026-09-02', bullets: ['fix: the corrected headline'] }]))
|
||||
|
||||
const result = run(
|
||||
['--product', 'open-brainy', '--version', '10.4.11', '--date', '2026-09-02', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir],
|
||||
dir,
|
||||
)
|
||||
expect(result.status).toBe(0)
|
||||
expect(result.stdout).toMatch(/replaced v10\.4\.11/i)
|
||||
|
||||
const wall = readRemote(remoteDir, 'open-brainy')
|
||||
expect(wall.entries).toHaveLength(2) // not 3 — replaced, not duplicated
|
||||
expect(wall.entries[0].version).toBe('10.4.11')
|
||||
expect(wall.entries[0].headline).toBe('fix: the corrected headline')
|
||||
expect(wall.entries[1].version).toBe('10.4.10')
|
||||
})
|
||||
|
||||
it('a re-run with byte-identical content commits nothing and still succeeds', () => {
|
||||
// headline always equals items[0] for a derived entry, so this fixture
|
||||
// (unlike BASE_ENTRY, whose headline/items intentionally diverge for the
|
||||
// shape-only tests below) has to keep the two in lockstep to ever roundtrip.
|
||||
const stableEntry = { ...BASE_ENTRY, headline: 'A faster open.', items: ['A faster open.'] }
|
||||
seedRemote(remoteDir, 'open-brainy', [stableEntry])
|
||||
writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.4.11', date: '2026-09-02', bullets: ['A faster open.'] }]))
|
||||
const before = readRemote(remoteDir, 'open-brainy')
|
||||
|
||||
const result = run(
|
||||
['--product', 'open-brainy', '--version', '10.4.11', '--date', '2026-09-02', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir],
|
||||
dir,
|
||||
)
|
||||
expect(result.status).toBe(0)
|
||||
expect(result.stdout).toMatch(/nothing to commit/i)
|
||||
expect(readRemote(remoteDir, 'open-brainy')).toEqual(before)
|
||||
})
|
||||
|
||||
it('derives the public package-page permalink for the product engine (private repo, never null)', () => {
|
||||
seedRemote(remoteDir, 'brainy', [{ ...BASE_ENTRY, version: '11.0.5', url: 'https://source.soulcraft.com/soulcraft/-/packages/npm/@soulcraft%2Fbrainy/11.0.5' }])
|
||||
writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '11.0.6', date: '2026-09-03', bullets: ['fix: a native-only fix'] }]))
|
||||
|
||||
const result = run(
|
||||
['--product', 'brainy', '--version', '11.0.6', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir],
|
||||
dir,
|
||||
)
|
||||
expect(result.status).toBe(0)
|
||||
|
||||
const wall = readRemote(remoteDir, 'brainy')
|
||||
expect(wall.entries[0].url).toBe('https://source.soulcraft.com/soulcraft/-/packages/npm/@soulcraft%2Fbrainy/11.0.6')
|
||||
expect(wall.entries[0].thumb).toBeNull()
|
||||
})
|
||||
|
||||
it('refuses a product with no permalink pattern, naming the cure', () => {
|
||||
seedRemote(remoteDir, 'open-brainy', [BASE_ENTRY])
|
||||
writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '1.0.0', date: '2026-09-03', bullets: ['feat: first'] }]))
|
||||
|
||||
const result = run(['--product', 'mystery', '--version', '1.0.0', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir], dir)
|
||||
expect(result.status).not.toBe(0)
|
||||
expect(result.stderr).toMatch(/no permalink pattern for product "mystery"/)
|
||||
expect(result.stderr).toMatch(/never carry url: null/)
|
||||
})
|
||||
|
||||
it('refuses when the CHANGELOG has no entry yet for the target version, and touches no remote', () => {
|
||||
seedRemote(remoteDir, 'open-brainy', [])
|
||||
writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.4.11', date: '2026-09-02', bullets: ['fix: whatever'] }]))
|
||||
const beforeSha = git(['rev-parse', 'main'], remoteDir)
|
||||
|
||||
const result = run(
|
||||
['--product', 'open-brainy', '--version', '99.0.0', '--date', '2026-09-02', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir],
|
||||
dir,
|
||||
)
|
||||
|
||||
expect(result.status).toBe(1)
|
||||
expect(result.stderr).toMatch(/no CHANGELOG entry yet/i)
|
||||
expect(git(['rev-parse', 'main'], remoteDir)).toBe(beforeSha)
|
||||
})
|
||||
|
||||
it('refuses by naming the cure when the remote cannot be cloned', () => {
|
||||
writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.4.12', date: '2026-09-03', bullets: ['fix: whatever'] }]))
|
||||
const noSuchRemote = join(tmpdir(), 'wall-remote-does-not-exist-' + Date.now())
|
||||
|
||||
const result = run(
|
||||
['--product', 'open-brainy', '--version', '10.4.12', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', noSuchRemote, '--cache-dir', cacheDir],
|
||||
dir,
|
||||
)
|
||||
|
||||
expect(result.status).toBe(1)
|
||||
expect(result.stderr).toMatch(/cannot clone/i)
|
||||
expect(result.stderr).toMatch(/cure:/i)
|
||||
})
|
||||
|
||||
it('refuses by naming the cure, and touches no remote, when the fetched wall fails shape validation', () => {
|
||||
const seedDir = mkdtempSync(join(tmpdir(), 'wall-seed-broken-'))
|
||||
execFileSync('git', ['clone', remoteDir, seedDir], { stdio: 'ignore' })
|
||||
git(['config', 'user.email', 'seed@example.com'], seedDir)
|
||||
git(['config', 'user.name', 'Seed'], seedDir)
|
||||
writeFileSync(
|
||||
join(seedDir, 'open-brainy.json'),
|
||||
JSON.stringify({ product: 'open-brainy', entries: [{ version: '10.4.11', date: '2026-09-02', items: ['x'], url: null }] }, null, 2),
|
||||
)
|
||||
git(['add', 'open-brainy.json'], seedDir)
|
||||
git(['commit', '-m', 'seed broken'], seedDir)
|
||||
git(['push', 'origin', 'main'], seedDir)
|
||||
rmSync(seedDir, { recursive: true, force: true })
|
||||
const beforeSha = git(['rev-parse', 'main'], remoteDir)
|
||||
|
||||
writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.4.12', date: '2026-09-03', bullets: ['fix: whatever'] }]))
|
||||
|
||||
const result = run(
|
||||
['--product', 'open-brainy', '--version', '10.4.12', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir],
|
||||
dir,
|
||||
)
|
||||
|
||||
expect(result.status).toBe(1)
|
||||
expect(result.stderr).toMatch(/fails shape validation/i)
|
||||
expect(result.stderr).toMatch(/missing key\(s\) headline/i)
|
||||
expect(git(['rev-parse', 'main'], remoteDir)).toBe(beforeSha)
|
||||
})
|
||||
|
||||
it('refuses by naming the cure when the remote rejects the push (stands in for a raced non-fast-forward)', () => {
|
||||
seedRemote(remoteDir, 'open-brainy', [BASE_ENTRY])
|
||||
makeRemoteRejectPushes(remoteDir)
|
||||
writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.4.12', date: '2026-09-03', bullets: ['fix: whatever'] }]))
|
||||
|
||||
const result = run(
|
||||
['--product', 'open-brainy', '--version', '10.4.12', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir],
|
||||
dir,
|
||||
)
|
||||
|
||||
expect(result.status).toBe(1)
|
||||
expect(result.stderr).toMatch(/push to .* failed/i)
|
||||
expect(result.stderr).toMatch(/cure:/i)
|
||||
})
|
||||
|
||||
it('refuses a cross-product write when the file\'s "product" field does not match --product', () => {
|
||||
seedRemote(remoteDir, 'open-brainy', [BASE_ENTRY])
|
||||
const seedDir = mkdtempSync(join(tmpdir(), 'wall-seed-mismatch-'))
|
||||
execFileSync('git', ['clone', remoteDir, seedDir], { stdio: 'ignore' })
|
||||
git(['config', 'user.email', 'seed@example.com'], seedDir)
|
||||
git(['config', 'user.name', 'Seed'], seedDir)
|
||||
const corrupted = JSON.parse(readFileSync(join(seedDir, 'open-brainy.json'), 'utf8'))
|
||||
corrupted.product = 'brainy'
|
||||
writeFileSync(join(seedDir, 'open-brainy.json'), JSON.stringify(corrupted, null, 2) + '\n')
|
||||
git(['add', 'open-brainy.json'], seedDir)
|
||||
git(['commit', '-m', 'corrupt product field'], seedDir)
|
||||
git(['push', 'origin', 'main'], seedDir)
|
||||
rmSync(seedDir, { recursive: true, force: true })
|
||||
|
||||
writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '1.0.0', date: '2026-09-03', bullets: ['fix: wrong repo'] }]))
|
||||
|
||||
const result = run(
|
||||
['--product', 'open-brainy', '--version', '1.0.0', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir],
|
||||
dir,
|
||||
)
|
||||
|
||||
expect(result.status).toBe(1)
|
||||
expect(result.stderr).toMatch(/product "brainy".*--product "open-brainy"/i)
|
||||
})
|
||||
})
|
||||
|
||||
describe('wall-entry.mjs — --dry-run', () => {
|
||||
it('prints the entry and the target path, and touches neither the cache dir nor the remote', () => {
|
||||
seedRemote(remoteDir, 'open-brainy', [BASE_ENTRY])
|
||||
writeFileSync(join(dir, 'CHANGELOG.md'), buildChangelog([{ version: '10.4.12', date: '2026-09-03', bullets: ['fix: a dry run'] }]))
|
||||
const beforeSha = git(['rev-parse', 'main'], remoteDir)
|
||||
|
||||
const result = run(
|
||||
['--dry-run', '--product', 'open-brainy', '--version', '10.4.12', '--date', '2026-09-03', '--from-changelog', 'CHANGELOG.md', '--remote', remoteDir, '--cache-dir', cacheDir],
|
||||
dir,
|
||||
)
|
||||
|
||||
expect(result.status).toBe(0)
|
||||
expect(result.stdout).toMatch(/would write to/i)
|
||||
expect(result.stdout).toMatch(/"version": "10\.4\.12"/)
|
||||
expect(git(['rev-parse', 'main'], remoteDir)).toBe(beforeSha)
|
||||
})
|
||||
})
|
||||
|
||||
describe('wall-entry.mjs — --check', () => {
|
||||
it('passes a well-formed, newest-first file with no duplicates', () => {
|
||||
writeFileSync(join(dir, 'wall.json'), wallFile('open-brainy', [BASE_ENTRY, { ...BASE_ENTRY, version: '10.4.10' }]))
|
||||
const result = run(['--check', '--file', 'wall.json'], dir)
|
||||
expect(result.status).toBe(0)
|
||||
expect(result.stdout).toMatch(/OK/)
|
||||
})
|
||||
|
||||
it('passes a file where "thumb" is entirely absent (optional per the HQ contract)', () => {
|
||||
const { thumb, ...noThumb } = BASE_ENTRY as any
|
||||
writeFileSync(join(dir, 'wall.json'), wallFile('open-brainy', [noThumb]))
|
||||
const result = run(['--check', '--file', 'wall.json'], dir)
|
||||
expect(result.status).toBe(0)
|
||||
})
|
||||
|
||||
it('catches a missing entry key', () => {
|
||||
const broken = { version: '1.0.0', date: '2026-09-03', headline: 'h', items: ['i'] } // no "url"
|
||||
writeFileSync(join(dir, 'wall.json'), wallFile('open-brainy', [broken]))
|
||||
const result = run(['--check', '--file', 'wall.json'], dir)
|
||||
expect(result.status).toBe(1)
|
||||
expect(result.stderr).toMatch(/missing key\(s\) url/)
|
||||
})
|
||||
|
||||
it('catches an unexpected top-level key (e.g. the retired "history" field)', () => {
|
||||
const raw = JSON.parse(wallFile('open-brainy', [BASE_ENTRY]))
|
||||
raw.history = 'retired field'
|
||||
writeFileSync(join(dir, 'wall.json'), JSON.stringify(raw))
|
||||
const result = run(['--check', '--file', 'wall.json'], dir)
|
||||
expect(result.status).toBe(1)
|
||||
expect(result.stderr).toMatch(/unexpected key\(s\) history/)
|
||||
})
|
||||
|
||||
it('catches entries that are not newest-first', () => {
|
||||
writeFileSync(join(dir, 'wall.json'), wallFile('open-brainy', [{ ...BASE_ENTRY, version: '10.4.10' }, BASE_ENTRY]))
|
||||
const result = run(['--check', '--file', 'wall.json'], dir)
|
||||
expect(result.status).toBe(1)
|
||||
expect(result.stderr).toMatch(/not newest-first/)
|
||||
})
|
||||
|
||||
it('catches a duplicate version even with identical entries', () => {
|
||||
writeFileSync(join(dir, 'wall.json'), wallFile('open-brainy', [BASE_ENTRY, { ...BASE_ENTRY }]))
|
||||
const result = run(['--check', '--file', 'wall.json'], dir)
|
||||
expect(result.status).toBe(1)
|
||||
expect(result.stderr).toMatch(/duplicate version 10\.4\.11/)
|
||||
})
|
||||
|
||||
it('catches an empty items array', () => {
|
||||
writeFileSync(join(dir, 'wall.json'), wallFile('open-brainy', [{ ...BASE_ENTRY, items: [] }]))
|
||||
const result = run(['--check', '--file', 'wall.json'], dir)
|
||||
expect(result.status).toBe(1)
|
||||
expect(result.stderr).toMatch(/"items" must be a non-empty array/)
|
||||
})
|
||||
|
||||
it('catches a malformed date', () => {
|
||||
writeFileSync(join(dir, 'wall.json'), wallFile('open-brainy', [{ ...BASE_ENTRY, date: '09/03/2026' }]))
|
||||
const result = run(['--check', '--file', 'wall.json'], dir)
|
||||
expect(result.status).toBe(1)
|
||||
expect(result.stderr).toMatch(/"date" must be a YYYY-MM-DD string/)
|
||||
})
|
||||
})
|
||||
|
|
@ -7,7 +7,7 @@
|
|||
* hydration (zero per-entity reads when unfiltered). Both must preserve the exact
|
||||
* pagination contract: same order, cursor continuation, filters, totalCount.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
|
||||
import { describe, it, expect, beforeEach, vi } from 'vitest'
|
||||
import { Brainy, NounType } from '../../../src/index.js'
|
||||
|
||||
describe('paginated enumeration — parallel hydration + id-only (cortex heal-cost)', () => {
|
||||
|
|
@ -30,10 +30,6 @@ describe('paginated enumeration — parallel hydration + id-only (cortex heal-co
|
|||
storage = brain.storage
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
/** Page the whole dataset through a small limit via cursor and collect ordered ids. */
|
||||
const pageAll = async (fn: (opts: any) => Promise<any>, key: 'items' | 'ids') => {
|
||||
const out: string[] = []
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
* Tests to verify that brain.find({ type: NounType.X }) correctly filters entities
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy, NounType } from '../../src/index.js'
|
||||
|
||||
describe('Type Filtering (A Consumer Team Issue)', () => {
|
||||
|
|
@ -17,10 +17,6 @@ describe('Type Filtering (A Consumer Team Issue)', () => {
|
|||
await brain.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('should filter entities by NounType.Person', async () => {
|
||||
// Add 3 people
|
||||
await brain.add({ data: 'John Smith', type: NounType.Person, metadata: { name: 'John' } })
|
||||
|
|
|
|||
|
|
@ -15,10 +15,9 @@
|
|||
* and the caller had no way to tell that from "no row matches". Eleven tags is
|
||||
* not an exotic shape; the eleventh tag made the row invisible.
|
||||
*
|
||||
* THE LAW. Arrays of scalars index up to {@link MAX_INDEXED_ARRAY_LENGTH},
|
||||
* THE LAW. Arrays of scalars index up to {@link MAX_INDEXED_ARRAY_LENGTH} = 64,
|
||||
* hardcoded (the zero-config law: no knob), which clears every legitimate
|
||||
* multi-value field — tags, authors, keyword lists — and stays below the
|
||||
* narrowest embedding this engine meets (384 dimensions). Above it the WRITE
|
||||
* multi-value field and stays far below any embedding width. Above it the WRITE
|
||||
* IS REFUSED by name — `MetadataArrayTooLargeError`, carrying the field, the
|
||||
* length and the bound — at `add`, `update`, `relate` and `updateRelation`
|
||||
* alike. Nothing is skipped in silence.
|
||||
|
|
@ -48,10 +47,6 @@ describe('the indexable-array bound', () => {
|
|||
await brain.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
describe('BELOW the bound: the array indexes, every element of it', () => {
|
||||
it('the eleven-element array that used to vanish is searchable', async () => {
|
||||
// ELEVEN — one over the old silent limit, the whole shape of the defect.
|
||||
|
|
@ -70,7 +65,7 @@ describe('the indexable-array bound', () => {
|
|||
}
|
||||
})
|
||||
|
||||
it('indexes right up to the bound — every element of it', async () => {
|
||||
it('indexes right up to the bound — all 64 elements', async () => {
|
||||
await brain.add({
|
||||
id: 'at-bound',
|
||||
data: 'a row at the bound',
|
||||
|
|
@ -79,9 +74,8 @@ describe('the indexable-array bound', () => {
|
|||
vector: []
|
||||
})
|
||||
|
||||
// The first, the last, and one in the middle — all derived from the
|
||||
// bound, so the case follows the constant wherever it moves.
|
||||
for (const tag of ['t0', `t${MAX_INDEXED_ARRAY_LENGTH - 1}`, `t${Math.floor(MAX_INDEXED_ARRAY_LENGTH / 2)}`]) {
|
||||
// The first, the last, and one in the middle.
|
||||
for (const tag of ['t0', `t${MAX_INDEXED_ARRAY_LENGTH - 1}`, 't31']) {
|
||||
const hits = await brain.find({ where: { tags: tag }, limit: 10 } as any)
|
||||
expect(hits.map((r: any) => r.id)).toContain(resolveEntityId('at-bound'))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -42,7 +42,7 @@
|
|||
* column store adopts the field. It is named in `getIdsFromChunksForRange`'s
|
||||
* doc comment rather than papered over.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy } from '../../../src/brainy'
|
||||
import { NounType } from '../../../src/types/graphTypes'
|
||||
import { SparseIndex, ChunkManager } from '../../../src/utils/metadataIndexChunking'
|
||||
|
|
@ -122,10 +122,6 @@ describe('legacy sparse index: range queries order values, or refuse', () => {
|
|||
expect(index.columnStore.hasField(FIELD)).toBe(false)
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
describe('(a) a long BOUND against ordinary short values', () => {
|
||||
// 'apple' < 'mango' < 'zebra', and every bound below is compared against
|
||||
// these three raw keys.
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@
|
|||
* validateInvariants(), and repairIndex() maps a failing invariant with heal:'rebuild'
|
||||
* to that provider's rebuild(). "healthy-while-broken must be impossible."
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy, NounType } from '../../src/index.js'
|
||||
import type { ProviderInvariantReport } from '../../src/index.js'
|
||||
|
||||
|
|
@ -48,10 +48,6 @@ describe('validateIndexConsistency delegates to provider validateInvariants() (P
|
|||
await brain.flush()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('a broken provider report makes the store unhealthy and names the failing invariant', async () => {
|
||||
brain.index.validateInvariants = async () => brokenReport('vector')
|
||||
const v = await brain.validateIndexConsistency()
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@
|
|||
* signal (from either strategy) THROWS VectorIndexNotReadyError immediately,
|
||||
* with no rebuild attempt in between — never a silent empty result.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy, NounType, VectorIndexNotReadyError } from '../../src/index.js'
|
||||
|
||||
const V = (): number[] => Array.from({ length: 384 }, (_, i) => Math.sin(i * 0.1) + 0.001)
|
||||
|
|
@ -28,10 +28,6 @@ describe('Vector cold-read guard (verifyVectorLive) — silent-[] on cold semant
|
|||
await brain.flush()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('warm brain: semantic find is correct and the guard does not rebuild', async () => {
|
||||
const vi = brain.index
|
||||
let rebuilds = 0
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
* Tests to verify VFS import behavior and identify if VFS creates only wrappers or also graph entities
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy, NounType } from '../../src/index.js'
|
||||
|
||||
describe('VFS Multi-instance Diagnostic', () => {
|
||||
|
|
@ -17,10 +17,6 @@ describe('VFS Multi-instance Diagnostic', () => {
|
|||
await brain.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
it('should verify VFS creates document wrappers AND allows entity filtering', async () => {
|
||||
console.log('\n🔬 VFS Multi-instance Diagnostic Test\n')
|
||||
console.log('='.repeat(70))
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
* Ensures tree methods prevent recursion and work correctly
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { VirtualFileSystem } from '../../src/vfs/VirtualFileSystem.js'
|
||||
import { VFSTreeUtils } from '../../src/vfs/TreeUtils.js'
|
||||
|
|
@ -24,10 +24,6 @@ describe('VFS Tree Operations', () => {
|
|||
await vfs.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
describe('Critical: No Self-Inclusion Bug', () => {
|
||||
it('should NEVER return a directory as its own child', async () => {
|
||||
// Create test structure
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@
|
|||
* - Issue #2: File read decompression error
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { VirtualFileSystem } from '../../src/vfs/VirtualFileSystem.js'
|
||||
|
||||
|
|
@ -25,10 +25,6 @@ describe('VFS Bug Fixes', () => {
|
|||
await vfs.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
describe('Issue #1: Duplicate Directory Nodes', () => {
|
||||
it('should not create duplicate directory entries when writing multiple files to same directory', async () => {
|
||||
// Write multiple files to the same directory (reproduce the bug scenario)
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@
|
|||
* other operations in parallel batches.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { describe, it, expect, beforeEach } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { VirtualFileSystem } from '../../src/vfs/VirtualFileSystem.js'
|
||||
|
||||
|
|
@ -30,10 +30,6 @@ describe('VFS bulkWrite Race Condition Fix', () => {
|
|||
await vfs.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
describe('operation ordering', () => {
|
||||
it('should create directories before files when mixed in same batch', async () => {
|
||||
// This is the exact scenario that triggered the race condition:
|
||||
|
|
|
|||
|
|
@ -389,14 +389,7 @@ describe('VirtualFileSystem - Production Tests', () => {
|
|||
})
|
||||
|
||||
describe('Performance', () => {
|
||||
it('should handle many files efficiently', async (ctx) => {
|
||||
// Wall-clock budget assertion — belongs to the perf lane (npm run
|
||||
// test:perf), not the correctness gate: 121ms alone but 16.5s under
|
||||
// the gate's sibling-file contention, a flake the code never caused
|
||||
// (same pattern as storage-batch-operations.test.ts's batch-vs-
|
||||
// individual timing case).
|
||||
ctx.skip(!process.env.BRAINY_PERF_LANE, 'wall-clock budget assertion — runs only under the perf lane (npm run test:perf)')
|
||||
|
||||
it('should handle many files efficiently', async () => {
|
||||
const dir = '/performance-test'
|
||||
await vfs.mkdir(dir)
|
||||
|
||||
|
|
|
|||
Reference in a new issue