soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat(8.0): brain.fillSubtypes migration helper + pre-RC1 gap closure

Browse source 
- brain.fillSubtypes(rules): idempotent subtype back-fill for pre-8.0 data.
  One rule per NounType/VerbType (literal default or per-entry function);
  fills only entries still missing a subtype through the public update()/
  updateRelation() paths; returns { scanned, filled, skipped, errors, byType }.
  Full unit suite in tests/unit/brainy/fill-subtypes.test.ts.
- Fix getNouns/getVerbs pagination hasMore (peek one past the window) —
  was permanently false, silently truncating every multi-page walk.
- find({ near }) without near.id now throws a teaching error instead of an
  opaque storage sharding failure; CLI --threshold without --near applies a
  plain score floor.
- CLI init/close audit: every one-shot command init()s, close()s, and exits
  explicitly; delete the unmaintained interactive REPL; replace the cloud-era
  storage subcommands with status/batch-delete; new types/validate commands.
- requireSubtype JSDoc now documents the 8.0 default-on contract; audit()
  recommendation points at fillSubtypes.
- Docs: data-storage-architecture rewritten to the real 8.0 on-disk layout;
  README storage section reflects filesystem+memory and snapshots; eli5 and
  SEMANTIC_VFS /as-of/ semantics corrected; internal tracker IDs and
  .strategy references scrubbed from published files.
This commit is contained in:
David Snelling 2026-06-11 10:42:34 -07:00
parent 9b0f4acd5b
commit c44678390e
30 changed files with 1517 additions and 3226 deletions
Download patch file Download diff file Expand all files Collapse all files

40
README.md
View file

@ -278,11 +278,16 @@ brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
**Enforce the pairing.** Register a vocabulary per type or turn on brain-wide strict mode to ensure every entity AND relationship has both `type` AND `subtype`:
```javascript
// Per-type rule with vocabulary
// Per-type rule with a closed vocabulary
brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true })
// Or brain-wide strict mode
const brain = new Brainy({ requireSubtype: true })
// 8.0 default: every write requires a subtype. Exempt genuine catch-all types…
const brain = new Brainy({ requireSubtype: { except: [NounType.Thing] } })
// …or opt out while migrating pre-8.0 data, then audit and back-fill:
const legacy = new Brainy({ requireSubtype: false })
await legacy.audit() // gaps, grouped by type
await legacy.fillSubtypes({ [NounType.Person]: 'unspecified' }) // close them
```
For other facets you want counted (`status`, `source`, `role`), register them with `brain.trackField(name)`. Renaming an existing convention to `subtype`? Use `brain.migrateField({from, to, entityKind: 'both'})` to walk nouns AND verbs in one pass. Full guide: **[Subtypes & Facets](docs/guides/subtypes-and-facets.md)**.
@ -291,7 +296,7 @@ For other facets you want counted (`status`, `source`, `role`), register them wi
---
## Storage: Memory to Cloud
## Storage: Memory and Filesystem
The same API at every scale. Change one config line to go from prototype to production.
@ -301,28 +306,31 @@ The same API at every scale. Change one config line to go from prototype to prod
const brain = new Brainy()
```
### Production — Filesystem with Compression
### Production — Filesystem (gzip compression on by default)
```javascript
const brain = new Brainy({
storage: { type: 'filesystem', path: './data', compression: true }
storage: { type: 'filesystem', rootDirectory: './data' }
})
```
### Cloud — S3, GCS, Azure, Cloudflare R2
### Backups and Portability — Snapshots
```javascript
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: { bucketName: 'my-knowledge-base', region: 'us-east-1' }
}
})
const db = brain.now()
await db.persist('/backups/2026-06-11') // instant hard-link snapshot
await db.release()
const snapshot = await Brainy.load('/backups/2026-06-11') // read-only Db
const hits = await snapshot.search('quarterly invoices')
await snapshot.release()
```
A snapshot directory is self-contained — copy it to another machine, open it with `Brainy.load()`, or restore it wholesale with `brain.restore(path, { confirm: true })`.
Performance benchmarks and capacity planning in **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)**.
**[Cloud Deployment Guide](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** | **[Capacity Planning](docs/operations/capacity-planning.md)**
**[Capacity Planning](docs/operations/capacity-planning.md)**
---
@ -371,10 +379,8 @@ Performance benchmarks and capacity planning in **[docs/PERFORMANCE.md](docs/PER
### Operations
- **[Cloud Deployment](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** — AWS, GCS, Azure
- **[Capacity Planning](docs/operations/capacity-planning.md)** — Memory, storage, and scaling
- **[Performance](docs/PERFORMANCE.md)** — Benchmarks and architecture details
- Cost Optimization: **[AWS S3](docs/operations/cost-optimization-aws-s3.md)** | **[GCS](docs/operations/cost-optimization-gcs.md)** | **[Azure](docs/operations/cost-optimization-azure.md)** | **[R2](docs/operations/cost-optimization-cloudflare-r2.md)**
---
@ -387,7 +393,7 @@ bun install @soulcraft/brainy # Bun — best performance
npm install @soulcraft/brainy # Node.js — fully supported
```
> **Deprecation Notice:** Browser support (OPFS, Web Workers, WASM embeddings) is deprecated in v7.10.0 and will be removed in v8.0.0. Brainy v8+ will be server-only.
> Brainy 8.0 is server-only. Browser support (OPFS storage, Web Workers, in-browser WASM embeddings) was removed in 8.0 — the 7.x line remains available on npm if you need it.
## Single-Writer Model

23
RELEASES.md
View file

@ -167,8 +167,12 @@ this rename; only the API surface moved.
Escape hatches: `requireSubtype: false` (last-resort opt-out for legacy
data) or `requireSubtype: { except: [NounType.Thing, ...] }` (per-type
allowlist). Per-type rules registered via `brain.requireSubtype(type, opts)`
still compose. `brain.audit()` reports entries missing a subtype and
`brain.migrateField()` backfills them.
still compose. `brain.audit()` reports entries missing a subtype and the new
`brain.fillSubtypes(rules)` migration helper backfills them — one rule per
NounType/VerbType (literal default or per-entry function), applied only to
entries still missing a subtype, returning
`{ scanned, filled, skipped, errors, byType }`. Idempotent: re-running fills
nothing, so a crashed run is resumed by running it again.
- **Verb ids are Brainy-generated UUIDs — by contract.** `relate()` now throws
a teaching error if a caller passes an `id` field
(`src/utils/paramValidation.ts:526`). In 7.x a supplied id was silently
@ -270,8 +274,10 @@ The full cost model is in
`getHistory` / `streamHistory` / `versions` / `data()` per the table above.
6. **Subtypes:** if your data predates subtype discipline, start with
`requireSubtype: false`, run `brain.audit()`, backfill with
`brain.migrateField()`, then remove the opt-out so the 8.0 default
enforcement protects you going forward.
`brain.fillSubtypes(rules)` (one rule per type — a literal default or a
function deriving the subtype from each entry), re-run `audit()` until
`total === 0`, then remove the opt-out so the 8.0 default enforcement
protects you going forward.
7. **CLI scripts:** `fork` / `branch` / `checkout` / `migrate` →
`snapshot <path>` / `restore <path>` / `history` / `generation`.
8. **Plugin authors:** apply the contract renames and the BigInt graph
@ -553,8 +559,7 @@ alongside every other metadata field on the existing write paths.
`_rev`, `ifRev`, `RevisionConflictError`, and `ifAbsent` all survive the 8.0 Db
redesign unchanged. 8.0 layers `brain.transact(tx, { ifAtGeneration })` for
whole-tx CAS on top of the same per-entity mechanism — per-entity for single-record
patterns, generation-based for "did the world move under me." See
`.strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md` § C-6 (internal).
patterns, generation-based for "did the world move under me."
---
@ -672,8 +677,7 @@ in `validateFindParams()`, both fire before any storage/index/Cortex call. Corte
The new `docs/guides/find-limits.md` calls out that 8.0's Datomic-style `Db.find()`
may tighten per-call limits; that's a Brainy 8.0 / Cortex 3.0 coordination point
documented in `.strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md` (open question #4 covers
the rollout staging, which now also applies to query limits).
whose rollout staging now also covers query limits.
### What consumers should do
@ -830,8 +834,7 @@ Brainy before any native call.
- **Native `audit()` proxy.** For billion-scale brains, `audit()` walks every entity (O(N)).
A native implementation reading from a "null-subtype" bitmap in the column store would be
O(buckets). Listed as the 6th open question in the
[Brainy 8.0 spec doc](`/media/dpsifr/storage/home/Projects/brainy/.strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md`).
O(buckets).
- **Strict-mode parity test.** Cortex should mirror Brainy's new
`tests/integration/strict-mode-self-test.test.ts` against their native paths to catch any
latent bug where native writes bypass JS validation.

1615
docs/architecture/data-storage-architecture.md
View file

File diff suppressed because it is too large Load diff

5
docs/architecture/multiprocess-storage-mixin.md
View file

@ -86,9 +86,8 @@ Benefits:
`FileSystemStorage` is the only one in-tree, but Cortex's
`MmapFileSystemStorage` inherits from it — interface relocation would
ripple through the plugin ecosystem.
- The current state works. Real failure modes
(BR-CX-INTERFACE-GAP) were build/install artifacts, not type-system
failures.
- The current state works. The real failure modes seen in the field
were build/install artifacts, not type-system failures.
- 7.22.0 just shipped a clean fix. Stacking another refactor before
consumers absorb it adds churn without urgency.
- The `hasStorageMethod()` guard accomplishes the same runtime safety the

2
docs/eli5.md
View file

@ -133,7 +133,7 @@ Brainy is the only row with every box checked. And it runs all of them in a sing
### One library, any scale
Brainy scales from a single laptop to billions of entities without changing a line of code. Small datasets live in memory. Larger ones spill to disk. At cloud scale, Brainy uses S3-compatible storage and automatically shards across nodes — the same API the whole way. Up to ten billion entities is fully implemented today.
Brainy scales from a quick experiment to serious production datasets without changing a line of code. Small datasets live entirely in memory. Larger ones spill to disk, where Brainy shards and compresses everything automatically. Need a backup or a copy? Snapshots are instant — the same API the whole way.
Add Cortex and you also unlock memory-mapped storage — aggregate state lives directly in the operating system's memory with zero serialization overhead, as fast as the hardware allows.

55
docs/vfs/SEMANTIC_VFS.md
View file

@ -47,16 +47,18 @@ const authFiles = await vfs.readdir('/by-concept/authentication')
const aliceFiles = await vfs.readdir('/by-author/alice')
```
### 2. **Time Travel**
See your codebase as it existed at any point:
### 2. **Change Tracking by Date**
List the files that were modified on any given day:
```typescript
// Code from March 15th
const snapshot = await vfs.readdir('/as-of/2024-03-15')
// Files that changed on March 15th
const changed = await vfs.readdir('/as-of/2024-03-15')
// Compare with today
// Everything under /src right now
const current = await vfs.readdir('/src')
```
`/as-of/<date>` selects by *modification date* — it reads the files' current content, not historical versions. For true point-in-time queries over entity state, use the Db API (`brain.asOf(generation)`).
### 3. **Knowledge Graph Navigation**
Navigate by semantic relationships:
```typescript
@ -118,13 +120,14 @@ await vfs.stat('/by-author/alice/config.ts')
### 4. By Time (Temporal) ✅ **Production**
```typescript
await vfs.readdir('/as-of/2024-03-15')
// Files modified on March 15, 2024
// Files modified on March 15, 2024 (24-hour window)
await vfs.readFile('/as-of/2024-03-15/src/auth.ts')
// Read auth.ts as it existed that day
await vfs.readFile('/as-of/2024-03-15/auth.ts')
// Current content of auth.ts, addressed by modification date —
// the path only resolves if auth.ts was modified that day
```
**How it works:** Tracks `modified` timestamp. Uses B-tree range queries (`greaterEqual`/`lessEqual`) for O(log n) performance.
**How it works:** Tracks the `modified` timestamp on every file and runs a range query (`greaterEqual`/`lessEqual`) over one 24-hour window for O(log n) performance. The VFS does not store historical file contents — `/as-of/` filters by *when a file last changed*; reads return the current bytes. For point-in-time state, use the Db API (`brain.asOf(generation)`).
**Status:** ✅ Fully implemented and tested at 10K file scale
@ -222,7 +225,7 @@ console.log(authFiles)
// ['login.ts', 'signup.ts', 'oauth.ts']
```
### Example 2: Time Travel
### Example 2: Changes by Day
```typescript
// See what changed today
const today = new Date().toISOString().split('T')[0]
@ -232,8 +235,8 @@ const todaysFiles = await vfs.readdir(`/as-of/${today}`)
const yesterday = new Date(Date.now() - 86400000).toISOString().split('T')[0]
const yesterdaysFiles = await vfs.readdir(`/as-of/${yesterday}`)
const newFiles = todaysFiles.filter(f => !yesterdaysFiles.includes(f))
console.log('New files today:', newFiles)
const onlyToday = todaysFiles.filter(f => !yesterdaysFiles.includes(f))
console.log('Changed today (untouched yesterday):', onlyToday)
```
### Example 3: Graph Navigation
@ -432,25 +435,17 @@ await vfs.writeFile(path, code, {
})
```
### 3. Optimize for Your Scale
### 3. Combine Dimensions
```typescript
// For < 100K files: Post-filtering is fine
// For > 100K files: Use flattened indexes
// Force index refresh after bulk operations
await brain.storage.rebuildIndexes()
```
### 4. Combine Dimensions
```typescript
// Find security files Alice worked on this week
// Find security files Alice changed on a given day
// (each /as-of/<date> path covers exactly that one day)
const aliceFiles = await vfs.readdir('/by-author/alice')
const securityFiles = await vfs.readdir('/by-tag/security')
const thisWeek = await vfs.readdir(`/as-of/${weekAgo}`)
const changedThatDay = await vfs.readdir('/as-of/2024-03-15')
const intersection = aliceFiles
.filter(f => securityFiles.includes(f))
.filter(f => thisWeek.includes(f))
.filter(f => changedThatDay.includes(f))
```
---
@ -469,14 +464,14 @@ console.log(entity.metadata.concepts)
### Slow Queries on Large Datasets
```typescript
// Check if indexes are built
const stats = await brain.storage.getIndexStats()
// Check if indexes are built and populated
const stats = await brain.getIndexStats()
console.log(stats)
// Rebuild if needed
await brain.storage.rebuildIndexes()
```
If an index looks empty or inconsistent, rebuild from raw storage with the CLI
(stop the live writer first): `brainy inspect repair <data-dir>`.
### Semantic Path Returns Empty
```typescript
// Check if metadata exists

305
src/brainy.ts
View file

@ -94,7 +94,10 @@ import {
BatchResult,
BrainyConfig,
BrainyStats,
ScoreExplanation
ScoreExplanation,
FillSubtypeRule,
FillSubtypeRules,
FillSubtypesResult
} from './types/brainy.types.js'
import { NounType, VerbType, TypeUtils } from './types/graphTypes.js'
import { BrainyInterface } from './types/brainyInterface.js'
@ -2885,7 +2888,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* })
*
* @example Lock down management relationships
* brain.requireSubtype(VerbType.Manages, {
* brain.requireSubtype(VerbType.ReportsTo, {
* values: ['direct', 'dotted-line'],
* required: true
* })
@ -7330,10 +7333,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* @param subtype - Optional specific subtype string for O(1) point count
*
* @example
* brain.counts.byRelationshipSubtype(VerbType.Manages)
* brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
* // → { direct: 12, 'dotted-line': 3 }
*
* brain.counts.byRelationshipSubtype(VerbType.Manages, 'direct')
* brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct')
* // → 12
*/
byRelationshipSubtype: (verb: VerbType, subtype?: string): number | Record<string, number> => {
@ -7361,7 +7364,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* `topSubtypes` for verbs.
*
* @example
* brain.counts.topRelationshipSubtypes(VerbType.Manages, 5)
* brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 5)
* // → [['direct', 12], ['dotted-line', 3]]
*/
topRelationshipSubtypes: (verb: VerbType, n: number = 10): Array<[string, number]> => {
@ -7533,7 +7536,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* @returns Sorted list of distinct subtype strings (empty if none)
*
* @example
* const variants = brain.relationshipSubtypesOf(VerbType.Manages)
* const variants = brain.relationshipSubtypesOf(VerbType.ReportsTo)
* // → ['direct', 'dotted-line']
*/
relationshipSubtypesOf(verb: VerbType): string[] {
@ -7550,17 +7553,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
/**
* Find entities and relationships missing a `subtype` value, grouped by type.
*
* The diagnostic pair to `migrateField()` / `fillSubtypes()` — answers the
* question "what would break if I enabled strict subtype enforcement?". Run
* this before adopting an SDK that registers `requireSubtype()` rules on
* common NounTypes, or before upgrading to Brainy 8.0 (which makes
* `requireSubtype: true` the default).
* The diagnostic pair to `fillSubtypes()` / `migrateField()` — answers the
* question "what would strict subtype enforcement reject?". 8.0 makes
* `requireSubtype: true` the default, so run this when opening a pre-8.0
* brain (with `requireSubtype: false` as the temporary escape hatch), then
* back-fill the reported gaps with `fillSubtypes(rules)`.
*
* Streams the brain via the same paginated `storage.getNouns()` /
* `storage.getVerbs()` pattern `migrateField()` uses — safe for large brains
* but linear in `O(N)`. Cortex 3.0+ may proxy this through the native
* column-store null-subtype bitmap for sub-linear performance on billion-scale
* brains (tracked in `CTX-SUBTYPE-8.0-CONTRACT`).
* `storage.getVerbs()` pattern `fillSubtypes()` uses — safe for large brains
* but linear in `O(N)`. A native index provider may serve this from a
* column-store null-subtype bitmap in the future for sub-linear performance
* on billion-scale brains.
*
* @param options.includeVFS - When `false` (default), entities marked with
* `metadata.isVFSEntity` or `metadata.isVFS` are excluded from the report —
@ -7570,7 +7573,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* @param options.onProgress - Optional progress callback invoked after each batch.
* @returns Report with per-type counts of entities/relationships without a
* subtype, plus the overall total and a one-line recommendation pointing at
* `migrateField()` (7.x) or `fillSubtypes()` (8.0).
* `fillSubtypes()`.
*
* @example Find pre-existing gaps before turning on strict mode
* const report = await brain.audit()
@ -7655,7 +7658,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
const recommendation = missingSubtype === 0
? 'No subtype gaps detected — this brain is strict-mode-ready.'
: 'Found ' + missingSubtype + ' entries without subtype. Migrate via `brain.migrateField()` (7.x) — or wait for `brain.fillSubtypes()` (8.0) which closes the same gap with caller-supplied rules.'
: 'Found ' + missingSubtype + ' entries without subtype. Back-fill with `brain.fillSubtypes(rules)` — one rule per NounType/VerbType, literal default or per-entry function. To lift an existing field into `subtype` instead, use `brain.migrateField({ from, to: \'subtype\' })`.'
return {
entitiesWithoutSubtype,
@ -7666,6 +7669,262 @@ export class Brainy<T = any> implements BrainyInterface<T> {
}
}
/**
* Back-fill missing `subtype` values across the whole brain — the 8.0
* migration helper for data written before subtype became required.
*
* 8.0 enforces a non-empty `subtype` on every write by default
* (`requireSubtype: true`). A brain created on 7.x typically carries entities
* and relationships without one; this method clears that debt in a single
* idempotent pass so the opt-out (`requireSubtype: false`) can be removed.
* The intended upgrade flow:
*
* 1. Open the brain with `requireSubtype: false` (temporary escape hatch).
* 2. `await brain.audit()` — see what's missing, grouped by type.
* 3. `await brain.fillSubtypes(rules)` — back-fill with one rule per type.
* 4. Re-run `audit()` until `total === 0`, then drop the opt-out.
*
* **Rules.** One rule per NounType (entities) and/or VerbType
* (relationships) — the two vocabularies don't overlap, so a single map
* covers both sides. A rule is either a literal subtype string (blanket
* default) or a function deriving the subtype from the entry; functions
* subsume `where`-style filtering by returning `undefined` for entries they
* decline (those stay untouched and count as `skipped`, so a later run with
* a stricter rule can pick them up).
*
* **What is never touched:** entries that already carry a non-empty
* `subtype` (re-running is a no-op on them), and — unless
* `includeVFS: true` — Brainy's own VFS infrastructure entries
* (`metadata.isVFSEntity` / `metadata.isVFS`), which bypass enforcement
* anyway and are not migration debt.
*
* **Write strategy (deliberate):** each fill goes through the public
* `update()` / `updateRelation()` paths, so every write is individually
* atomic (storage record + indexes + subtype rollups commit together) and
* bumps the entry's `_rev` like any other update. The pass is *not* one
* whole-brain transaction: a migration over millions of entries inside a
* single transaction would hold an unbounded working set and turn one bad
* entry into an all-or-nothing failure. Idempotence is the recovery model —
* a crashed or partially-failed run is resumed safely by re-running, because
* only entries still missing a subtype are written.
*
* Streams via the same paginated `storage.getNouns()` / `storage.getVerbs()`
* walk `audit()` uses — `O(N)` but constant memory. The noun pass is skipped
* entirely when the map has no NounType rules, and vice versa.
*
* @param rules - Map of NounType/VerbType → literal subtype or rule function.
* Must contain at least one valid type key; invalid keys, empty-string
* literals, and non-string/non-function values throw before any data is
* touched.
* @param options.includeVFS - Also fill VFS infrastructure entries (default
* `false` — they bypass enforcement and don't need a subtype).
* @param options.batchSize - Pagination batch size (default `200`).
* @param options.onProgress - Optional callback invoked after each batch.
* @returns `{ scanned, filled, skipped, errors, byType }` — see
* {@link FillSubtypesResult}. After a clean run, `skipped` equals the
* remaining `audit().total`.
* @throws If the brain is read-only, the rule map is empty/malformed, or a
* key is not a valid NounType/VerbType. Per-entry write failures do NOT
* throw — they are collected in `errors` and the pass continues.
*
* @example Back-fill entities and relationships in one pass
* const report = await brain.fillSubtypes({
* [NounType.Person]: (e) => e.metadata?.kind ?? 'unspecified',
* [NounType.Document]: 'general',
* [VerbType.RelatedTo]: 'unspecified'
* })
* // → { scanned: 5200, filled: 1429, skipped: 0, errors: [], byType: { person: 800, document: 600, relatedTo: 29 } }
*
* @example Selective fill — decline entries a rule can't classify
* await brain.fillSubtypes({
* [NounType.Person]: (e) => e.metadata?.department ? 'employee' : undefined
* })
* // Persons without a department stay untouched (counted as skipped).
*
* @since 8.0.0
*/
async fillSubtypes(
rules: FillSubtypeRules<T>,
options: {
includeVFS?: boolean
batchSize?: number
onProgress?: (progress: { scanned: number; filled: number; skipped: number }) => void
} = {}
): Promise<FillSubtypesResult> {
this.assertWritable('fillSubtypes')
await this.ensureInitialized()
// Validate the rule map up front — fail fast on shape errors before any
// data is touched.
if (!rules || typeof rules !== 'object' || Array.isArray(rules)) {
throw new Error(
'fillSubtypes: rules must be a map of NounType/VerbType → subtype string or rule function'
)
}
const nounTypeValues = new Set<string>(Object.values(NounType))
const verbTypeValues = new Set<string>(Object.values(VerbType))
const nounRules = new Map<string, FillSubtypeRule<Entity<T>>>()
const verbRules = new Map<string, FillSubtypeRule<Relation<T>>>()
for (const [key, rule] of Object.entries(rules)) {
if (rule === undefined) continue
if (typeof rule !== 'string' && typeof rule !== 'function') {
throw new Error(
`fillSubtypes: rule for '${key}' must be a subtype string or a function (got ${typeof rule})`
)
}
if (typeof rule === 'string' && rule.length === 0) {
throw new Error(
`fillSubtypes: rule for '${key}' is an empty string — a subtype must be non-empty`
)
}
if (nounTypeValues.has(key)) {
nounRules.set(key, rule as FillSubtypeRule<Entity<T>>)
} else if (verbTypeValues.has(key)) {
verbRules.set(key, rule as FillSubtypeRule<Relation<T>>)
} else {
throw new Error(`fillSubtypes: '${key}' is not a valid NounType or VerbType`)
}
}
if (nounRules.size === 0 && verbRules.size === 0) {
throw new Error(
'fillSubtypes: rules map is empty — provide at least one NounType or VerbType rule'
)
}
const includeVFS = options.includeVFS === true
const batchSize = Math.max(1, options.batchSize ?? 200)
let scanned = 0
let filled = 0
let skipped = 0
const errors: Array<{ id: string; error: string }> = []
const byType: Record<string, number> = {}
const reportProgress = (): void => {
if (options.onProgress) options.onProgress({ scanned, filled, skipped })
}
// Normalize a rule's output: only a non-empty string is a fill; anything
// else (undefined, empty string) is a decline. An empty subtype would fail
// the very strict-mode check this helper exists to satisfy.
const normalize = (value: string | undefined): string | undefined =>
typeof value === 'string' && value.length > 0 ? value : undefined
// Pass 1: entities (skipped entirely when the map has no NounType rules).
if (nounRules.size > 0) {
let offset = 0
while (true) {
const page = await this.storage.getNouns({ pagination: { offset, limit: batchSize } })
if (page.items.length === 0) break
for (const noun of page.items) {
scanned++
// VFS infrastructure entries bypass enforcement via their marker, so
// they're not migration debt — leave them alone unless asked.
if (
!includeVFS &&
(noun.metadata?.isVFSEntity === true || noun.metadata?.isVFS === true)
) {
continue
}
if (typeof noun.subtype === 'string' && noun.subtype.length > 0) continue // already filled — never overwrite
const rule = nounRules.get(noun.type)
if (rule === undefined) {
skipped++
continue
}
try {
let subtype: string | undefined
if (typeof rule === 'function') {
const entity = await this.convertNounToEntity(noun)
subtype = normalize(rule(entity))
} else {
subtype = rule
}
if (subtype === undefined) {
skipped++
continue
}
await this.update({ id: noun.id, subtype })
filled++
byType[noun.type] = (byType[noun.type] || 0) + 1
} catch (err) {
errors.push({
id: noun.id,
error: err instanceof Error ? err.message : String(err)
})
}
}
reportProgress()
if (!page.hasMore) break
offset += page.items.length
}
}
// Pass 2: relationships (skipped entirely when the map has no VerbType rules).
if (verbRules.size > 0) {
let offset = 0
while (true) {
const page = await this.storage.getVerbs({ pagination: { offset, limit: batchSize } })
if (page.items.length === 0) break
for (const verb of page.items) {
scanned++
if (
!includeVFS &&
(verb.metadata?.isVFSEntity === true || verb.metadata?.isVFS === true)
) {
continue
}
if (typeof verb.subtype === 'string' && verb.subtype.length > 0) continue // already filled — never overwrite
const rule = verbRules.get(verb.verb)
if (rule === undefined) {
skipped++
continue
}
try {
let subtype: string | undefined
if (typeof rule === 'function') {
// Project the stored verb onto the public Relation<T> shape the
// rule function is typed against.
const relation: Relation<T> = {
id: verb.id,
from: verb.sourceId,
to: verb.targetId,
type: verb.verb,
weight: verb.weight,
data: verb.data,
metadata: verb.metadata as T,
service: verb.service,
createdAt: verb.createdAt,
updatedAt: verb.updatedAt,
confidence: verb.confidence
}
subtype = normalize(rule(relation))
} else {
subtype = rule
}
if (subtype === undefined) {
skipped++
continue
}
await this.updateRelation({ id: verb.id, subtype })
filled++
byType[verb.verb] = (byType[verb.verb] || 0) + 1
} catch (err) {
errors.push({
id: verb.id,
error: err instanceof Error ? err.message : String(err)
})
}
}
reportProgress()
if (!page.hasMore) break
offset += page.items.length
}
}
return { scanned, filled, skipped, errors, byType }
}
/**
* Stream-and-rewrite a field across every entity in the brain.
*
@ -8845,6 +9104,16 @@ export class Brainy<T = any> implements BrainyInterface<T> {
private async executeProximitySearch(params: FindParams<T>): Promise<Result<T>[]> {
if (!params.near) return []
// Teaching error: without an anchor id the constraint is meaningless, and
// letting it fall through produces an opaque storage-layer sharding error.
if (!params.near.id) {
throw new Error(
"find({ near }): 'near.id' is required — pass the entity to search around, " +
'e.g. near: { id, threshold }. To impose a minimum score on plain semantic ' +
'results, filter on result.score instead.'
)
}
const nearEntity = await this.get(params.near.id)
if (!nearEntity) return []
@ -9680,7 +9949,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
*/
private normalizeConfig(config?: BrainyConfig): Required<BrainyConfig> {
// Validate storage configuration. Brainy 8.0 ships two adapters only —
// FileSystemStorage and MemoryStorage — per BR-BRAINY-80-STORAGE-SIMPLIFY.
// FileSystemStorage and MemoryStorage (cloud + OPFS adapters were removed).
// Cloud backup remains supported via operator tooling (db.persist() +
// gsutil / aws s3 cp / rclone / azcopy). Pre-constructed adapter
// instances bypass the type check (they ARE the storage).

63
src/cli/commands/core.ts
View file

@ -132,6 +132,7 @@ export const coreCommands = {
const spinner = ora('Adding to neural database...').start()
const brain = getBrainy()
await brain.init()
let metadata: any = {}
if (options.metadata) {
@ -204,6 +205,12 @@ export const coreCommands = {
} else {
formatOutput({ id: result, metadata, confidence: addParams.confidence, weight: addParams.weight }, options)
}
// close() releases the writer lock and indexes, but global timers
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
// alive. CLI commands are one-shot — exit explicitly.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('Failed to add data')
console.error(chalk.red('Failed to add data:', error.message))
@ -280,6 +287,7 @@ export const coreCommands = {
const spinner = ora('Searching with Triple Intelligence™...').start()
const brain = getBrainy()
await brain.init()
// Build comprehensive search params
const searchParams: any = {
@ -292,11 +300,6 @@ export const coreCommands = {
searchParams.offset = parseInt(options.offset)
}
// Vector Intelligence - similarity threshold
if (options.threshold) {
searchParams.near = { threshold: parseFloat(options.threshold) }
}
// Metadata Intelligence - type filtering
if (options.type) {
const types = options.type.split(',').map(t => t.trim())
@ -314,7 +317,9 @@ export const coreCommands = {
}
}
// Vector Intelligence - proximity search
// Vector Intelligence - proximity search around an anchor entity.
// `near` requires an id; a bare --threshold (no --near) is applied as a
// plain score floor on the fused results after find() returns.
if (options.near) {
searchParams.near = {
id: options.near,
@ -371,7 +376,14 @@ export const coreCommands = {
}
}
const results = await brain.find(searchParams)
let results = await brain.find(searchParams)
// Without --near there is no proximity anchor; apply --threshold as a
// minimum-score filter on the fused results instead.
if (!options.near && options.threshold) {
const minScore = parseFloat(options.threshold)
results = results.filter((r) => r.score === undefined || r.score >= minScore)
}
spinner.succeed(`Found ${results.length} results`)
@ -457,6 +469,10 @@ export const coreCommands = {
} else {
formatOutput(results, options)
}
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('Search failed')
console.error(chalk.red('Search failed:', error.message))
@ -496,6 +512,7 @@ export const coreCommands = {
const spinner = ora('Fetching item...').start()
const brain = getBrainy()
await brain.init()
// Try to get the item
const item = await brain.get(id)
@ -529,6 +546,10 @@ export const coreCommands = {
} else {
formatOutput(item, options)
}
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('Failed to get item')
console.error(chalk.red('Failed to get item:', error.message))
@ -589,6 +610,7 @@ export const coreCommands = {
const spinner = ora('Creating relationship...').start()
const brain = getBrainy()
await brain.init()
let metadata: any = {}
if (options.metadata) {
@ -627,6 +649,10 @@ export const coreCommands = {
} else {
formatOutput({ id: result, source, verb, target, metadata }, options)
}
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('Failed to create relationship')
console.error(chalk.red('Failed to create relationship:', error.message))
@ -683,6 +709,7 @@ export const coreCommands = {
spinner = ora('Updating entity...').start()
const brain = getBrainy()
await brain.init()
// Get existing entity first
const existing = await brain.get(id)
@ -731,6 +758,10 @@ export const coreCommands = {
} else {
formatOutput({ id, updated: true }, options)
}
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('Failed to update entity')
console.error(chalk.red('Update failed:', error.message))
@ -784,6 +815,7 @@ export const coreCommands = {
spinner = ora('Deleting entity...').start()
const brain = getBrainy()
await brain.init()
await brain.delete(id)
@ -794,6 +826,10 @@ export const coreCommands = {
} else {
formatOutput({ id, deleted: true }, options)
}
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('Failed to delete entity')
console.error(chalk.red('Delete failed:', error.message))
@ -846,6 +882,7 @@ export const coreCommands = {
spinner = ora('Removing relationship...').start()
const brain = getBrainy()
await brain.init()
await brain.unrelate(id)
@ -856,6 +893,10 @@ export const coreCommands = {
} else {
formatOutput({ id, removed: true }, options)
}
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('Failed to remove relationship')
console.error(chalk.red('Unrelate failed:', error.message))
@ -875,7 +916,9 @@ export const coreCommands = {
if (options.json) {
formatOutput(diag, options)
return
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
}
console.log(chalk.bold('\nBrainy Diagnostics'))
@ -907,6 +950,10 @@ export const coreCommands = {
console.log(` Metadata: ${diag.indexes.metadata.type} (initialized: ${diag.indexes.metadata.initialized})`)
console.log(` Graph: ${diag.indexes.graph.type} (initialized: ${diag.indexes.graph.initialized}, wired: ${diag.indexes.graph.wiredToStorage})`)
console.log()
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
console.error(chalk.red('Diagnostics failed: ' + error.message))
process.exit(1)

14
src/cli/commands/import.ts
View file

@ -152,6 +152,7 @@ export const importCommands = {
spinner = ora('Initializing import...').start()
const brain = getBrainy()
await brain.init()
// Handle different source types
let result: any
@ -401,6 +402,12 @@ export const importCommands = {
} else if (options.json) {
formatOutput(result, options)
}
// close() releases the writer lock and indexes, but global timers
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
// alive. CLI commands are one-shot — exit explicitly.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('Import failed')
console.error(chalk.red('Import failed:', error.message))
@ -472,9 +479,10 @@ export const importCommands = {
spinner = ora('Initializing VFS import...').start()
const brain = getBrainy()
await brain.init()
// Get VFS
const vfs = await brain.vfs
const vfs = brain.vfs
// Load DirectoryImporter
const { DirectoryImporter } = await import('../../vfs/importers/DirectoryImporter.js')
@ -536,6 +544,10 @@ export const importCommands = {
} else if (options.json) {
formatOutput(result, options)
}
// One-shot command — see import() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('VFS import failed')
console.error(chalk.red('VFS import failed:', error.message))

23
src/cli/commands/insights.ts
View file

@ -41,6 +41,7 @@ export const insightsCommands = {
try {
const brain = getBrainy()
await brain.init()
// Get insights from Brainy
const insights = await brain.insights()
@ -102,6 +103,12 @@ export const insightsCommands = {
} else {
formatOutput(insights, options)
}
// close() releases the writer lock and indexes, but global timers
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
// alive. CLI commands are one-shot — exit explicitly.
await brain.close()
process.exit(0)
} catch (error: any) {
spinner.fail('Failed to get insights')
console.error(chalk.red('Insights failed:', error.message))
@ -120,6 +127,7 @@ export const insightsCommands = {
try {
const brain = getBrainy()
await brain.init()
// Get available fields from metadata index
const fields = await brain.getAvailableFields()
@ -166,6 +174,10 @@ export const insightsCommands = {
}))
formatOutput(fieldsWithStats, options)
}
// One-shot command — see insights() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
spinner.fail('Failed to get fields')
console.error(chalk.red('Fields analysis failed:', error.message))
@ -186,6 +198,7 @@ export const insightsCommands = {
if (!field) {
spinner = ora('Getting available fields...').start()
const brain = getBrainy()
await brain.init()
const availableFields = await brain.getAvailableFields()
spinner.stop()
@ -202,6 +215,7 @@ export const insightsCommands = {
spinner = ora(`Getting values for field: ${field}...`).start()
const brain = getBrainy()
await brain.init()
const values = await brain.getFieldValues(field)
const limit = options.limit ? parseInt(options.limit) : 100
@ -242,6 +256,10 @@ export const insightsCommands = {
} else {
formatOutput({ field, values, count: values.length }, options)
}
// One-shot command — see insights() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('Failed to get field values')
console.error(chalk.red('Field values failed:', error.message))
@ -289,6 +307,7 @@ export const insightsCommands = {
spinner = ora('Analyzing optimal query plan...').start()
const brain = getBrainy()
await brain.init()
const plan = await brain.getOptimalQueryPlan(filters)
@ -328,6 +347,10 @@ export const insightsCommands = {
} else {
formatOutput({ filters, plan }, options)
}
// One-shot command — see insights() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('Failed to generate query plan')
console.error(chalk.red('Query plan failed:', error.message))

273
src/cli/commands/neural.ts
View file

@ -1,7 +1,11 @@
/**
* 🧠 Neural Similarity API Commands
*
* CLI interface for semantic similarity, clustering, and neural operations
* @module cli/commands/neural
* @description Neural CLI commands: semantic similarity, clustering,
* hierarchy, neighbors, outlier detection, and visualization data export.
* Registered in `src/cli/index.ts` as `similar` / `cluster` / `related` /
* `hierarchy` / `outliers` / `visualize`. Each command is one-shot: it
* initializes the shared Brainy instance, runs the neural operation, then
* closes the store and exits explicitly.
*/
import inquirer from 'inquirer'
@ -25,136 +29,13 @@ interface CommandArguments {
_: string[];
}
export const neuralCommand = {
command: 'neural [action]',
describe: '🧠 Neural similarity and clustering operations',
builder: (yargs: any) => {
return yargs
.positional('action', {
describe: 'Neural operation to perform',
type: 'string',
choices: ['similar', 'clusters', 'hierarchy', 'neighbors', 'path', 'outliers', 'visualize']
})
.option('id', {
describe: 'Item ID for similarity operations',
type: 'string',
alias: 'i'
})
.option('query', {
describe: 'Query text for similarity search',
type: 'string',
alias: 'q'
})
.option('threshold', {
describe: 'Similarity threshold (0-1)',
type: 'number',
default: 0.7,
alias: 't'
})
.option('format', {
describe: 'Output format',
type: 'string',
choices: ['json', 'table', 'tree', 'graph'],
default: 'table',
alias: 'f'
})
.option('output', {
describe: 'Output file path',
type: 'string',
alias: 'o'
})
.option('limit', {
describe: 'Maximum number of results',
type: 'number',
default: 10,
alias: 'l'
})
.option('algorithm', {
describe: 'Clustering algorithm',
type: 'string',
choices: ['hierarchical', 'kmeans', 'dbscan', 'auto'],
default: 'auto',
alias: 'a'
})
.option('dimensions', {
describe: 'Visualization dimensions (2 or 3)',
type: 'number',
choices: [2, 3],
default: 2,
alias: 'd'
})
.option('explain', {
describe: 'Include detailed explanations',
type: 'boolean',
default: false,
alias: 'e'
})
},
handler: async (argv: CommandArguments) => {
console.log(chalk.cyan('\n🧠 NEURAL SIMILARITY API'))
console.log(chalk.gray('━'.repeat(50)))
let brainyInstance: Brainy | null = null
// Initialize Brainy and Neural API
const brain = new Brainy()
const neural = brain.neural()
try {
const action = argv.action || await promptForAction()
switch (action) {
case 'similar':
await handleSimilarCommand(neural, argv)
break
case 'clusters':
await handleClustersCommand(neural, argv)
break
case 'hierarchy':
await handleHierarchyCommand(neural, argv)
break
case 'neighbors':
await handleNeighborsCommand(neural, argv)
break
case 'path':
console.log(chalk.yellow('\n⚠️ Semantic path finding coming soon'))
console.log(chalk.dim('This feature requires implementing graph traversal algorithms'))
console.log(chalk.dim('Use "neighbors" and "hierarchy" commands to explore connections'))
break
case 'outliers':
await handleOutliersCommand(neural, argv)
break
case 'visualize':
await handleVisualizeCommand(neural, argv)
break
default:
console.log(chalk.red(`❌ Unknown action: ${action}`))
showHelp()
}
} catch (error) {
console.error(chalk.red('💥 Error:'), error instanceof Error ? error.message : error)
process.exit(1)
}
const getBrainy = (): Brainy => {
if (!brainyInstance) {
brainyInstance = new Brainy()
}
}
async function promptForAction(): Promise<string> {
const answer = await inquirer.prompt([{
type: 'list',
name: 'action',
message: 'Choose a neural operation:',
choices: [
{ name: '🔗 Calculate similarity between items', value: 'similar' },
{ name: '🎯 Find semantic clusters', value: 'clusters' },
{ name: '🌳 Show item hierarchy', value: 'hierarchy' },
{ name: '🕸️ Find semantic neighbors', value: 'neighbors' },
{ name: '🛣️ Find semantic path between items (coming soon)', value: 'path', disabled: true },
{ name: '🚨 Detect outliers', value: 'outliers' },
{ name: '📊 Generate visualization data', value: 'visualize' }
]
}])
return answer.action
return brainyInstance
}
async function handleSimilarCommand(neural: any, argv: CommandArguments): Promise<void> {
@ -436,63 +317,6 @@ async function handleNeighborsCommand(neural: any, argv: CommandArguments): Prom
}
}
async function handlePathCommand(neural: any, argv: CommandArguments): Promise<void> {
const spinner = ora('🛣️ Finding semantic path...').start()
try {
let fromId: string, toId: string
if (argv._ && argv._.length >= 3) {
fromId = argv._[1]
toId = argv._[2]
} else {
spinner.stop()
const answers = await inquirer.prompt([
{
type: 'input',
name: 'from',
message: 'From item ID:',
validate: (input: string) => input.length > 0
},
{
type: 'input',
name: 'to',
message: 'To item ID:',
validate: (input: string) => input.length > 0
}
])
fromId = answers.from
toId = answers.to
spinner.start()
}
const path = await neural.semanticPath(fromId, toId)
if (path.length === 0) {
spinner.warn('🚫 No semantic path found')
console.log(`No path found between ${chalk.cyan(fromId)} and ${chalk.cyan(toId)}`)
} else {
spinner.succeed(`✅ Found path with ${path.length} hops`)
console.log(`\n🛣️ Semantic Path from ${chalk.cyan(fromId)} to ${chalk.cyan(toId)}:`)
console.log(`${chalk.cyan(fromId)} (start)`)
path.forEach((hop, index) => {
console.log(`${' '.repeat(index + 1)}↓ ${(hop.similarity * 100).toFixed(1)}%`)
console.log(`${' '.repeat(index + 1)}${hop.id} (hop ${hop.hop})`)
})
}
if (argv.output) {
await saveToFile(argv.output, path, argv.format!)
}
} catch (error) {
spinner.fail('💥 Failed to find path')
throw error
}
}
async function handleOutliersCommand(neural: any, argv: CommandArguments): Promise<void> {
const spinner = ora('🚨 Detecting semantic outliers...').start()
@ -592,32 +416,32 @@ function formatAsTable(data: any): string {
return JSON.stringify(data, null, 2)
}
function showHelp(): void {
console.log('\n🧠 Neural Similarity API Commands:')
console.log('')
console.log(' brainy neural similar <item1> <item2> Calculate similarity')
console.log(' brainy neural clusters Find semantic clusters')
console.log(' brainy neural hierarchy <id> Show item hierarchy')
console.log(' brainy neural neighbors <id> Find semantic neighbors')
console.log(' brainy neural path <from> <to> Find semantic path')
console.log(' brainy neural outliers Detect outliers')
console.log(' brainy neural visualize Generate visualization data')
console.log('')
console.log('Options:')
console.log(' --threshold, -t Similarity threshold (0-1)')
console.log(' --format, -f Output format (json|table|tree|graph)')
console.log(' --output, -o Save to file')
console.log(' --limit, -l Maximum results')
console.log(' --explain, -e Include explanations')
console.log('')
/**
* @description Run a neural CLI handler with the one-shot lifecycle every
* Brainy command follows: init → work → `close()` → explicit `process.exit`.
* close() releases the writer lock and indexes, but global timers
* (UnifiedCache bookkeeping, PathResolver stats) keep the event loop alive,
* so one-shot commands must exit explicitly.
* @param work - The handler body, given the initialized neural API.
*/
async function runNeuralCommand(work: (neural: ReturnType<Brainy['neural']>) => Promise<void>): Promise<void> {
try {
const brain = getBrainy()
await brain.init()
await work(brain.neural())
await brain.close()
process.exit(0)
} catch (error) {
console.error(chalk.red('💥 Error:'), error instanceof Error ? error.message : error)
process.exit(1)
}
}
// Commander-compatible wrappers
export const neuralCommands = {
async similar(a?: string, b?: string, options?: any) {
const brain = new Brainy()
const neural = brain.neural()
// Build argv-style object for handler
const argv: CommandArguments = {
_: ['neural', 'similar', a || '', b || ''].filter(x => x),
@ -627,13 +451,10 @@ export const neuralCommands = {
...options
}
await handleSimilarCommand(neural, argv)
await runNeuralCommand((neural) => handleSimilarCommand(neural, argv))
},
async cluster(options?: any) {
const brain = new Brainy()
const neural = brain.neural()
const argv: CommandArguments = {
_: ['neural', 'cluster'],
algorithm: options?.algorithm || 'hierarchical',
@ -643,26 +464,20 @@ export const neuralCommands = {
...options
}
await handleClustersCommand(neural, argv)
await runNeuralCommand((neural) => handleClustersCommand(neural, argv))
},
async hierarchy(id?: string, options?: any) {
const brain = new Brainy()
const neural = brain.neural()
const argv: CommandArguments = {
_: ['neural', 'hierarchy', id || ''].filter(x => x),
id,
...options
}
await handleHierarchyCommand(neural, argv)
await runNeuralCommand((neural) => handleHierarchyCommand(neural, argv))
},
async related(id?: string, options?: any) {
const brain = new Brainy()
const neural = brain.neural()
const argv: CommandArguments = {
_: ['neural', 'related', id || ''].filter(x => x),
id,
@ -671,13 +486,10 @@ export const neuralCommands = {
...options
}
await handleNeighborsCommand(neural, argv)
await runNeuralCommand((neural) => handleNeighborsCommand(neural, argv))
},
async outliers(options?: any) {
const brain = new Brainy()
const neural = brain.neural()
const argv: CommandArguments = {
_: ['neural', 'outliers'],
threshold: options?.threshold ? parseFloat(options.threshold) : 0.3,
@ -685,13 +497,10 @@ export const neuralCommands = {
...options
}
await handleOutliersCommand(neural, argv)
await runNeuralCommand((neural) => handleOutliersCommand(neural, argv))
},
async visualize(options?: any) {
const brain = new Brainy()
const neural = brain.neural()
const argv: CommandArguments = {
_: ['neural', 'visualize'],
format: options?.format || 'json',
@ -701,8 +510,6 @@ export const neuralCommands = {
...options
}
await handleVisualizeCommand(neural, argv)
await runNeuralCommand((neural) => handleVisualizeCommand(neural, argv))
}
}
export default neuralCommand
}

17
src/cli/commands/nlp.ts
View file

@ -62,6 +62,7 @@ export const nlpCommands = {
spinner = ora('Extracting entities with neural NLP...').start()
const brain = getBrainy()
await brain.init()
// Extract entities using Brainy's neural entity extractor
const entities = await brain.extract(text)
@ -105,6 +106,12 @@ export const nlpCommands = {
} else {
formatOutput(entities, options)
}
// close() releases the writer lock and indexes, but global timers
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
// alive. CLI commands are one-shot — exit explicitly.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('Entity extraction failed')
console.error(chalk.red('Extraction failed:', error.message))
@ -147,6 +154,7 @@ export const nlpCommands = {
spinner = ora('Extracting concepts with neural analysis...').start()
const brain = getBrainy()
await brain.init()
const confidence = options.threshold ? parseFloat(options.threshold) : 0.5
const concepts = await brain.extractConcepts(text, { confidence })
@ -171,6 +179,10 @@ export const nlpCommands = {
} else {
formatOutput(concepts, options)
}
// One-shot command — see extract() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('Concept extraction failed')
console.error(chalk.red('Extraction failed:', error.message))
@ -201,6 +213,7 @@ export const nlpCommands = {
spinner = ora('Analyzing text with neural NLP...').start()
const brain = getBrainy()
await brain.init()
// Run both entity extraction and concept extraction
const [entities, concepts] = await Promise.all([
@ -265,6 +278,10 @@ export const nlpCommands = {
concepts
}, options)
}
// One-shot command — see extract() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('Analysis failed')
console.error(chalk.red('Analysis failed:', error.message))

914
src/cli/commands/storage.ts
View file

File diff suppressed because it is too large Load diff

61
src/cli/commands/utility.ts
View file

@ -22,8 +22,7 @@ interface StatsOptions extends UtilityOptions {
}
interface CleanOptions extends UtilityOptions {
removeOrphans?: boolean
rebuildIndex?: boolean
force?: boolean
}
interface BenchmarkOptions extends UtilityOptions {
@ -63,6 +62,7 @@ export const utilityCommands = {
try {
const brain = getBrainy()
await brain.init()
const nounCount = await brain.getNounCount()
const verbCount = await brain.getVerbCount()
const memUsage = process.memoryUsage()
@ -77,7 +77,11 @@ export const utilityCommands = {
if (options.json) {
formatOutput(stats, options)
return
// close() releases the writer lock and indexes, but global timers
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
// alive. CLI commands are one-shot — exit explicitly.
await brain.close()
process.exit(0)
}
console.log(chalk.cyan('\n📊 Database Statistics\n'))
@ -112,7 +116,10 @@ export const utilityCommands = {
)
console.log(memTable.toString())
// One-shot command — see the --json branch for why the explicit exit.
await brain.close()
process.exit(0)
} catch (error: any) {
spinner.fail('Failed to gather statistics')
console.error(chalk.red(error.message))
@ -121,30 +128,33 @@ export const utilityCommands = {
},
/**
* Clean and optimize database
* Clear the database (all entities, relationships, and indexes).
* Destructive — asks for confirmation unless --force is passed.
*/
async clean(options: CleanOptions) {
const spinner = ora('Cleaning database...').start()
let spinner: ReturnType<typeof ora> | null = null
try {
const brain = getBrainy()
// Destructive operation — confirm first (skipped with --force).
if (!options.force) {
const inquirer = (await import('inquirer')).default
const { confirm } = await inquirer.prompt([{
type: 'confirm',
name: 'confirm',
message: chalk.yellow('⚠️ Permanently delete ALL data (entities, relationships, indexes)?'),
default: false
}])
// For now, only support full clear
// removeOrphans and rebuildIndex would require new Brainy APIs
if (options.removeOrphans || options.rebuildIndex) {
spinner.warn('Advanced cleanup options not yet implemented')
console.log(chalk.yellow('\n⚠️ Advanced cleanup features coming soon:'))
console.log(chalk.dim(' • --remove-orphans: Remove disconnected items'))
console.log(chalk.dim(' • --rebuild-index: Rebuild vector index'))
console.log(chalk.dim('\nUse "brainy clean" without options to clear the database'))
return
if (!confirm) {
console.log(chalk.yellow('Clean cancelled'))
process.exit(0)
}
}
// Show warning before clearing
console.log(chalk.yellow('\n⚠️ WARNING: This will permanently delete ALL data!'))
const brain = getBrainy()
// Clear all data (entities, relationships, and every index)
spinner.text = 'Clearing all data...'
spinner = ora('Clearing all data...').start()
await brain.init()
await brain.clear()
@ -163,7 +173,7 @@ export const utilityCommands = {
await brain.close()
process.exit(0)
} catch (error: any) {
spinner.fail('Cleanup failed')
if (spinner) spinner.fail('Cleanup failed')
console.error(chalk.red(error.message))
process.exit(1)
}
@ -185,7 +195,8 @@ export const utilityCommands = {
try {
const brain = getBrainy()
await brain.init()
// Benchmark different operations
const benchmarks = [
{ name: 'add', enabled: operations === 'all' || operations.includes('add') },
@ -205,7 +216,8 @@ export const utilityCommands = {
switch (bench.name) {
case 'add':
await brain.add({ data: `Test item ${i}`, type: NounType.Thing, metadata: { benchmark: true } })
// 8.0 requires a subtype on every write by default.
await brain.add({ data: `Test item ${i}`, type: NounType.Thing, subtype: 'benchmark', metadata: { benchmark: true } })
break
case 'search':
await brain.find({ query: 'test', limit: 10 })
@ -284,7 +296,10 @@ export const utilityCommands = {
} else {
formatOutput(results, options)
}
// One-shot command — see stats() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
console.error(chalk.red('Benchmark failed:'), error.message)
process.exit(1)

48
src/cli/commands/vfs.ts
View file

@ -70,6 +70,12 @@ export const vfsCommands = {
} else {
formatOutput({ path, content: buffer.toString(), size: buffer.length }, options)
}
// close() releases the writer lock and indexes, but global timers
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
// alive. CLI commands are one-shot — exit explicitly.
await brain.close()
process.exit(0)
} catch (error: any) {
spinner.fail('Failed to read file')
console.error(chalk.red(error.message))
@ -109,6 +115,10 @@ export const vfsCommands = {
} else {
formatOutput({ path, size: Buffer.byteLength(data) }, options)
}
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
spinner.fail('Failed to write file')
console.error(chalk.red(error.message))
@ -132,7 +142,9 @@ export const vfsCommands = {
if (!options.json) {
if (!Array.isArray(entries) || entries.length === 0) {
console.log(chalk.yellow('Directory is empty'))
return
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
}
if (options.long) {
@ -171,6 +183,10 @@ export const vfsCommands = {
} else {
formatOutput(entries, options)
}
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
spinner.fail('Failed to list directory')
console.error(chalk.red(error.message))
@ -202,6 +218,10 @@ export const vfsCommands = {
} else {
formatOutput(stats, options)
}
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
spinner.fail('Failed to get stats')
console.error(chalk.red(error.message))
@ -227,6 +247,10 @@ export const vfsCommands = {
} else {
formatOutput({ path, created: true }, options)
}
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
spinner.fail('Failed to create directory')
console.error(chalk.red(error.message))
@ -258,12 +282,16 @@ export const vfsCommands = {
} else {
formatOutput({ path, removed: true }, options)
}
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
spinner.fail('Failed to remove')
console.error(chalk.red(error.message))
if (!options.force) {
process.exit(1)
}
// --force tolerates the failure but the process is still one-shot:
// exit cleanly (0) instead of falling off the event loop.
process.exit(options.force ? 0 : 1)
}
},
@ -303,6 +331,10 @@ export const vfsCommands = {
} else {
formatOutput(results, options)
}
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
spinner.fail('Search failed')
console.error(chalk.red(error.message))
@ -343,6 +375,10 @@ export const vfsCommands = {
} else {
formatOutput(results, options)
}
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
spinner.fail('Failed to find similar files')
console.error(chalk.red(error.message))
@ -371,6 +407,10 @@ export const vfsCommands = {
} else {
formatOutput(tree, options)
}
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
spinner.fail('Failed to build tree')
console.error(chalk.red(error.message))

131
src/cli/index.ts
View file

@ -19,6 +19,7 @@ import { insightsCommands } from './commands/insights.js'
import { importCommands } from './commands/import.js'
import { snapshotCommands } from './commands/snapshot.js'
import { inspectCommands } from './commands/inspect.js'
import { types as typesCommand, validate as validateCommand } from './commands/types.js'
import { readFileSync } from 'fs'
import { fileURLToPath } from 'url'
import { dirname, join } from 'path'
@ -70,14 +71,9 @@ ${chalk.cyan('Examples:')}
$ brainy vfs similar /code/Button.tsx
${chalk.dim('# Storage management')}
$ brainy storage status --quota
$ brainy storage lifecycle set ${chalk.dim('# Interactive mode')}
$ brainy storage cost-estimate
$ brainy storage status
$ brainy storage batch-delete old-ids.txt
${chalk.dim('# Interactive mode')}
$ brainy interactive
${chalk.cyan('Documentation:')}
${chalk.dim('Full docs:')} https://github.com/soulcraftlabs/brainy
${chalk.dim('Report issues:')} https://github.com/soulcraftlabs/brainy/issues
@ -109,7 +105,7 @@ program
.description('Advanced search with Triple Intelligence™ (interactive if no query)')
.option('-k, --limit <number>', 'Number of results', '10')
.option('--offset <number>', 'Skip N results (pagination)')
.option('-t, --threshold <number>', 'Similarity threshold (0-1)', '0.7')
.option('-t, --threshold <number>', 'Minimum similarity score (0-1); with --near, the proximity threshold')
.option('--type <types>', 'Filter by type(s) - comma separated')
.option('--where <json>', 'Metadata filters (JSON)')
.option('--near <id>', 'Find items near this ID')
@ -181,6 +177,23 @@ program
.option('--pretty', 'Pretty print JSON')
.action(coreCommands.diagnostics)
// ===== Type Commands =====
program
.command('types')
.description('List all NounType and VerbType values')
.option('--noun', 'Show noun types only')
.option('--verb', 'Show verb types only')
.option('--json', 'Output as JSON')
.action(typesCommand)
program
.command('validate [type]')
.description('Check whether a type string is a valid NounType or VerbType (interactive if no type)')
.option('--verb', 'Validate as a verb type (default: noun type)')
.option('--json', 'Output as JSON')
.action(validateCommand)
// ===== Neural Commands =====
program
@ -222,17 +235,6 @@ program
.option('--children-only', 'Show only child hierarchy')
.action(neuralCommands.hierarchy)
program
.command('path <from> <to>')
.description('Find semantic path between items')
.option('--steps', 'Show step-by-step path')
.option('--max-hops <number>', 'Maximum path length', '5')
.action(() => {
console.log(chalk.yellow('\n⚠️ Semantic path finding coming soon'))
console.log(chalk.dim('This feature requires implementing graph traversal algorithms'))
console.log(chalk.dim('Use "brainy neighbors" and "brainy hierarchy" to explore connections'))
})
program
.command('outliers')
.alias('anomalies')
@ -457,89 +459,26 @@ program
program
.command('storage')
.description('💾 Storage management and cost optimization')
.description('💾 Storage management (filesystem + memory backends)')
.addCommand(
new Command('status')
.description('Show storage status and health')
.option('--detailed', 'Show detailed information')
.option('--quota', 'Show quota information (OPFS)')
.description('Show storage backend, counts, root directory, and writer lock')
.option('--json', 'Output as JSON')
.option('--pretty', 'Pretty-print JSON')
.action((options) => {
storageCommands.status(options)
})
)
.addCommand(
new Command('lifecycle')
.description('Lifecycle policy management')
.addCommand(
new Command('set')
.argument('[config-file]', 'Policy configuration file (JSON)')
.description('Set lifecycle policy (interactive if no file)')
.option('--validate', 'Validate before applying')
.action((configFile, options) => {
storageCommands.lifecycle.set(configFile, options)
})
)
.addCommand(
new Command('get')
.description('Get current lifecycle policy')
.option('-f, --format <type>', 'Output format (json|yaml)', 'json')
.action((options) => {
storageCommands.lifecycle.get(options)
})
)
.addCommand(
new Command('remove')
.description('Remove lifecycle policy')
.action((options) => {
storageCommands.lifecycle.remove(options)
})
)
)
.addCommand(
new Command('compression')
.description('Compression management (FileSystem)')
.addCommand(
new Command('enable')
.description('Enable gzip compression')
.action((options) => {
storageCommands.compression.enable(options)
})
)
.addCommand(
new Command('disable')
.description('Disable compression')
.action((options) => {
storageCommands.compression.disable(options)
})
)
.addCommand(
new Command('status')
.description('Show compression status')
.action((options) => {
storageCommands.compression.status(options)
})
)
)
.addCommand(
new Command('batch-delete')
.argument('<file>', 'File containing entity IDs (one per line)')
.description('Batch delete with retry logic')
.option('--max-retries <n>', 'Maximum retry attempts', '3')
.option('--continue-on-error', 'Continue if some deletes fail')
.description('Delete a list of entities through the public delete path (with retry)')
.option('--max-retries <n>', 'Maximum retry attempts per ID', '3')
.option('--continue-on-error', 'Continue past IDs that still fail after retries')
.action((file, options) => {
storageCommands.batchDelete(file, options)
})
)
.addCommand(
new Command('cost-estimate')
.description('Estimate cloud storage costs')
.option('--provider <type>', 'Cloud provider (aws|gcs|azure|r2)')
.option('--size <gb>', 'Data size in GB')
.option('--operations <n>', 'Monthly operations')
.action((options) => {
storageCommands.costEstimate(options)
})
)
// ===== Data Management Commands =====
@ -742,9 +681,8 @@ program
program
.command('clean')
.description('Clean and optimize database')
.option('--remove-orphans', 'Remove orphaned items')
.option('--rebuild-index', 'Rebuild search index')
.description('Clear the database — ALL entities, relationships, and indexes (asks for confirmation)')
.option('-f, --force', 'Skip confirmation prompt')
.action(utilityCommands.clean)
program
@ -785,17 +723,6 @@ program
.description('Show the store\'s current generation watermark')
.action(snapshotCommands.generation)
// ===== Interactive Mode =====
program
.command('interactive')
.alias('i')
.description('Start interactive REPL mode')
.action(async () => {
const { startInteractiveMode } = await import('./interactive.js')
await startInteractiveMode()
})
// ===== Error Handling =====
program.exitOverride()

653
src/cli/interactive.ts
View file

@ -1,653 +0,0 @@
/**
* Professional Interactive CLI System
*
* Provides consistent, delightful interactive prompts for all commands
* with smart defaults, validation, and helpful examples
*/
import chalk from 'chalk'
import inquirer from 'inquirer'
import ora from 'ora'
import { Brainy } from '../brainy.js'
import { getBrainyVersion } from '../utils/version.js'
// Professional color scheme
export const colors = {
primary: chalk.hex('#3A5F4A'), // Teal (from logo)
success: chalk.hex('#2D4A3A'), // Deep teal
info: chalk.hex('#4A6B5A'), // Medium teal
warning: chalk.hex('#D67441'), // Orange (from logo)
error: chalk.hex('#B85C35'), // Deep orange
brain: chalk.hex('#D67441'), // Brain orange
cream: chalk.hex('#F5E6A3'), // Cream background
dim: chalk.dim,
bold: chalk.bold,
cyan: chalk.cyan,
green: chalk.green,
yellow: chalk.yellow,
red: chalk.red
}
// Icons for consistent visual language
export const icons = {
brain: '🧠',
search: '🔍',
add: '➕',
delete: '🗑️',
update: '🔄',
import: '📥',
export: '📤',
connect: '🔗',
question: '❓',
success: '✅',
error: '❌',
warning: '⚠️',
info: 'ℹ️',
sparkle: '✨',
rocket: '🚀',
thinking: '🤔',
chat: '💬'
}
// Store recent inputs for smart suggestions
const recentInputs = {
searches: [] as string[],
ids: [] as string[],
types: [] as string[],
formats: [] as string[]
}
/**
* Professional prompt wrapper with consistent styling
*/
export async function prompt(config: any): Promise<any> {
// Add consistent styling
if (config.message) {
config.message = colors.cyan(config.message)
}
// Add prefix with appropriate icon
if (!config.prefix) {
config.prefix = colors.dim(' › ')
}
return inquirer.prompt([config])
}
/**
* Interactive prompt for search query with smart features
*/
export async function promptSearchQuery(previousSearches?: string[]): Promise<string> {
console.log(colors.primary(`\n${icons.search} Smart Search\n`))
console.log(colors.dim('Search your neural database with natural language'))
console.log(colors.dim('Examples: "meetings last week", "John from Google", "important documents"'))
const { query } = await prompt({
type: 'input',
name: 'query',
message: 'What would you like to search for?',
validate: (input: string) => {
if (!input.trim()) {
return 'Please enter a search query'
}
return true
},
transformer: (input: string) => {
// Show live character count
const count = input.length
if (count > 100) {
return colors.warning(input)
}
return colors.green(input)
}
})
// Store for future suggestions
if (!recentInputs.searches.includes(query)) {
recentInputs.searches.unshift(query)
recentInputs.searches = recentInputs.searches.slice(0, 10)
}
return query
}
/**
* Interactive prompt for item ID with fuzzy search
*/
export async function promptItemId(
action: string,
brain?: Brainy,
allowMultiple: boolean = false
): Promise<string | string[]> {
console.log(colors.primary(`\n${icons.thinking} Select item to ${action}\n`))
// If we have brain instance, show recent items
let choices: any[] = []
if (brain) {
try {
const recent = await brain.find({
query: '*',
limit: 10
})
choices = recent.map(item => ({
name: `${item.id} - ${(item as any).content?.substring(0, 50) || 'No content'}...`,
value: item.id,
short: item.id
}))
} catch {
// Fallback to manual input
}
}
if (choices.length > 0) {
choices.push(new inquirer.Separator())
choices.push({ name: 'Enter ID manually', value: '__manual__' })
const { selected } = await prompt({
type: allowMultiple ? 'checkbox' : 'list',
name: 'selected',
message: `Select item(s) to ${action}:`,
choices,
pageSize: 10
})
if (selected === '__manual__' || (Array.isArray(selected) && selected.includes('__manual__'))) {
return promptManualId(action, allowMultiple)
}
return selected
} else {
return promptManualId(action, allowMultiple)
}
}
/**
* Manual ID input with validation
*/
async function promptManualId(action: string, allowMultiple: boolean): Promise<string | string[]> {
const { id } = await prompt({
type: 'input',
name: 'id',
message: allowMultiple
? `Enter ID(s) to ${action} (comma-separated):`
: `Enter ID to ${action}:`,
validate: (input: string) => {
if (!input.trim()) {
return `Please enter at least one ID`
}
return true
}
})
if (allowMultiple) {
return id.split(',').map((i: string) => i.trim()).filter(Boolean)
}
return id.trim()
}
/**
* Confirm destructive action with preview
*/
export async function confirmDestructiveAction(
action: string,
items: any[],
showPreview: boolean = true
): Promise<boolean> {
console.log(colors.warning(`\n${icons.warning} Confirmation Required\n`))
if (showPreview && items.length > 0) {
console.log(colors.dim(`You are about to ${action}:`))
items.slice(0, 5).forEach(item => {
console.log(colors.dim(` • ${item.id || item}`))
})
if (items.length > 5) {
console.log(colors.dim(` ... and ${items.length - 5} more`))
}
console.log()
}
const { confirm } = await prompt({
type: 'confirm',
name: 'confirm',
message: colors.warning(`Are you sure you want to ${action}?`),
default: false
})
return confirm
}
/**
* Interactive data input with multiline support
*/
export async function promptDataInput(
action: string = 'add',
currentValue?: string
): Promise<string> {
console.log(colors.primary(`\n${icons.add} ${action === 'add' ? 'Add Data' : 'Update Data'}\n`))
if (currentValue) {
console.log(colors.dim('Current value:'))
console.log(colors.info(` ${currentValue.substring(0, 100)}${currentValue.length > 100 ? '...' : ''}`))
console.log()
}
const { data } = await prompt({
type: 'editor',
name: 'data',
message: 'Enter your data:',
default: currentValue || '',
postfix: '.md',
validate: (input: string) => {
if (!input.trim() && action === 'add') {
return 'Please enter some data'
}
return true
}
})
return data
}
/**
* Interactive metadata input with JSON validation
*/
export async function promptMetadata(
currentMetadata?: any,
suggestions?: string[]
): Promise<any> {
console.log(colors.dim('\nOptional: Add metadata (JSON format)'))
const { addMetadata } = await prompt({
type: 'confirm',
name: 'addMetadata',
message: 'Would you like to add metadata?',
default: false
})
if (!addMetadata) {
return {}
}
// Show field suggestions if available
if (suggestions && suggestions.length > 0) {
console.log(colors.dim('\nAvailable fields:'))
suggestions.forEach(field => {
console.log(colors.dim(` • ${field}`))
})
}
const { metadata } = await prompt({
type: 'editor',
name: 'metadata',
message: 'Enter metadata (JSON):',
default: currentMetadata ? JSON.stringify(currentMetadata, null, 2) : '{\n \n}',
postfix: '.json',
validate: (input: string) => {
try {
JSON.parse(input)
return true
} catch (e) {
return `Invalid JSON: ${e.message}`
}
}
})
return JSON.parse(metadata)
}
/**
* Interactive format selector
*/
export async function promptFormat(
availableFormats: string[],
defaultFormat: string
): Promise<string> {
console.log(colors.primary(`\n${icons.export} Select Format\n`))
const { format } = await prompt({
type: 'list',
name: 'format',
message: 'Choose export format:',
choices: availableFormats.map(f => ({
name: getFormatDescription(f),
value: f,
short: f
})),
default: defaultFormat
})
return format
}
/**
* Get friendly format descriptions
*/
function getFormatDescription(format: string): string {
const descriptions: Record<string, string> = {
json: 'JSON - Universal data interchange',
jsonl: 'JSON Lines - Streaming format',
csv: 'CSV - Spreadsheet compatible',
graphml: 'GraphML - Graph visualization',
dot: 'DOT - Graphviz format',
d3: 'D3.js - Web visualization',
markdown: 'Markdown - Human readable',
yaml: 'YAML - Configuration format'
}
return `${format.toUpperCase()} - ${descriptions[format] || 'Custom format'}`
}
/**
* Interactive file/URL input with validation
*/
export async function promptFileOrUrl(
action: string = 'import'
): Promise<string> {
console.log(colors.primary(`\n${icons.import} ${action === 'import' ? 'Import Source' : 'Export Destination'}\n`))
const { sourceType } = await prompt({
type: 'list',
name: 'sourceType',
message: 'What type of source?',
choices: [
{ name: 'Local file', value: 'file' },
{ name: 'URL', value: 'url' },
{ name: 'Clipboard', value: 'clipboard' },
{ name: 'Direct input', value: 'input' }
]
})
switch (sourceType) {
case 'file':
return promptFilePath(action)
case 'url':
return promptUrl()
case 'clipboard':
// Would need clipboard integration
console.log(colors.warning('Clipboard support coming soon!'))
return promptFilePath(action)
case 'input':
const data = await promptDataInput('import')
// Save to temp file and return path
const tmpFile = `/tmp/brainy-import-${Date.now()}.json`
const { writeFileSync } = await import('node:fs')
writeFileSync(tmpFile, data)
return tmpFile
default:
return ''
}
}
/**
* File path input with autocomplete
*/
async function promptFilePath(action: string): Promise<string> {
const { path } = await prompt({
type: 'input',
name: 'path',
message: `Enter file path to ${action}:`,
validate: async (input: string) => {
if (!input.trim()) {
return 'Please enter a file path'
}
const { existsSync } = await import('node:fs')
if (action === 'import' && !existsSync(input)) {
return `File not found: ${input}`
}
return true
},
// Add file path autocomplete
transformer: (input: string) => {
if (input.startsWith('~/')) {
const home = process.env.HOME || '~'
return colors.green(input.replace('~', home))
}
return colors.green(input)
}
})
return path
}
/**
* URL input with validation
*/
async function promptUrl(): Promise<string> {
const { url } = await prompt({
type: 'input',
name: 'url',
message: 'Enter URL:',
validate: (input: string) => {
try {
new URL(input)
return true
} catch {
return 'Please enter a valid URL'
}
}
})
return url
}
/**
* Interactive relationship builder
*/
export async function promptRelationship(brain?: Brainy): Promise<{
source: string
verb: string
target: string
metadata?: any
}> {
console.log(colors.primary(`\n${icons.connect} Create Relationship\n`))
console.log(colors.dim('Connect two items with a semantic relationship'))
// Get source
const source = await promptItemId('connect from', brain, false) as string
// Get verb/relationship type
const { verb } = await prompt({
type: 'list',
name: 'verb',
message: 'Relationship type:',
choices: [
{ name: 'Works For', value: 'WorksFor' },
{ name: 'Knows', value: 'Knows' },
{ name: 'Created By', value: 'CreatedBy' },
{ name: 'Belongs To', value: 'BelongsTo' },
{ name: 'Uses', value: 'Uses' },
{ name: 'Manages', value: 'Manages' },
{ name: 'Located In', value: 'LocatedIn' },
{ name: 'Related To', value: 'RelatedTo' },
new inquirer.Separator(),
{ name: 'Custom relationship...', value: '__custom__' }
]
})
let finalVerb = verb
if (verb === '__custom__') {
const { customVerb } = await prompt({
type: 'input',
name: 'customVerb',
message: 'Enter custom relationship:',
validate: (input: string) => input.trim() ? true : 'Please enter a relationship'
})
finalVerb = customVerb
}
// Get target
const target = await promptItemId('connect to', brain, false) as string
// Optional metadata
const metadata = await promptMetadata()
return {
source,
verb: finalVerb,
target,
metadata: Object.keys(metadata).length > 0 ? metadata : undefined
}
}
/**
* Smart command suggestions when user types wrong command
*/
export function suggestCommand(input: string, availableCommands: string[]): string[] {
// Simple fuzzy matching without external dependency
// Filter commands that start with or contain the input
const matches = availableCommands
.filter(cmd => cmd.toLowerCase().includes(input.toLowerCase()))
.sort((a, b) => {
// Prefer commands that start with the input
const aStarts = a.toLowerCase().startsWith(input.toLowerCase())
const bStarts = b.toLowerCase().startsWith(input.toLowerCase())
if (aStarts && !bStarts) return -1
if (!aStarts && bStarts) return 1
return 0
})
return matches.slice(0, 3)
}
/**
* Beautiful error display with helpful context
*/
export function showError(error: Error, context?: string): void {
console.log()
console.log(colors.error(`${icons.error} Error`))
if (context) {
console.log(colors.dim(context))
}
console.log(colors.red(error.message))
// Provide helpful suggestions based on error
if (error.message.includes('not found')) {
console.log(colors.dim('\nTip: Use "brainy search" to find items'))
} else if (error.message.includes('network') || error.message.includes('fetch')) {
console.log(colors.dim('\nTip: Check your internet connection'))
} else if (error.message.includes('permission')) {
console.log(colors.dim('\nTip: Check file permissions or run with appropriate access'))
}
}
/**
* Progress indicator for long operations
*/
export class ProgressTracker {
private spinner: any
private startTime: number
constructor(message: string) {
this.spinner = ora({
text: message,
color: 'cyan',
spinner: 'dots'
}).start()
this.startTime = Date.now()
}
update(message: string, count?: number, total?: number): void {
if (count && total) {
const percent = Math.round((count / total) * 100)
const elapsed = ((Date.now() - this.startTime) / 1000).toFixed(1)
this.spinner.text = `${message} (${percent}% - ${elapsed}s)`
} else {
this.spinner.text = message
}
}
succeed(message?: string): void {
const elapsed = ((Date.now() - this.startTime) / 1000).toFixed(1)
this.spinner.succeed(message ? `${message} (${elapsed}s)` : `Done (${elapsed}s)`)
}
fail(message?: string): void {
this.spinner.fail(message || 'Failed')
}
stop(): void {
this.spinner.stop()
}
}
/**
* Welcome message for interactive mode
*/
export function showWelcome(): void {
console.clear()
console.log(colors.primary(`
╔══════════════════════════════════════════════╗
║ ║
║ ${icons.brain} BRAINY - Neural Intelligence ║
║ Your AI-Powered Second Brain ║
║ ║
╚══════════════════════════════════════════════╝
`))
console.log(colors.dim(`Version ${getBrainyVersion()} • Type "help" for commands`))
console.log()
}
/**
* Interactive command selector for beginners
*/
export async function promptCommand(): Promise<string> {
const { command } = await prompt({
type: 'list',
name: 'command',
message: 'What would you like to do?',
choices: [
{ name: `${icons.add} Add data to your brain`, value: 'add' },
{ name: `${icons.search} Search your knowledge`, value: 'search' },
{ name: `${icons.chat} Chat with your data`, value: 'chat' },
{ name: `${icons.update} Update existing data`, value: 'update' },
{ name: `${icons.delete} Delete data`, value: 'delete' },
{ name: `${icons.connect} Create relationships`, value: 'relate' },
{ name: `${icons.import} Import from file`, value: 'import' },
{ name: `${icons.export} Export your brain`, value: 'export' },
new inquirer.Separator(),
{ name: `${icons.brain} Neural operations`, value: 'neural' },
{ name: `${icons.info} View statistics`, value: 'status' },
{ name: 'Exit', value: 'exit' }
],
pageSize: 15
})
return command
}
/**
* Start interactive REPL mode
*/
export async function startInteractiveMode() {
console.log(chalk.cyan('\n🧠 Brainy Interactive Mode\n'))
console.log(chalk.yellow('Interactive REPL mode coming soon\n'))
console.log(chalk.dim('Use specific commands for now: brainy add, brainy search, etc.'))
process.exit(0)
}
/**
* Export all interactive components
*/
export default {
colors,
icons,
prompt,
promptSearchQuery,
promptItemId,
confirmDestructiveAction,
promptDataInput,
promptMetadata,
promptFormat,
promptFileOrUrl,
promptRelationship,
suggestCommand,
showError,
ProgressTracker,
showWelcome,
promptCommand,
startInteractiveMode
}

2
src/config/storageAutoConfig.ts
View file

@ -2,7 +2,7 @@
* @module config/storageAutoConfig
* @description Storage configuration auto-detection for Brainy 8.0.
*
* Brainy 8.0 ships **two storage adapters** per `BR-BRAINY-80-STORAGE-SIMPLIFY`:
* Brainy 8.0 ships **two storage adapters**:
* - `FILESYSTEM` — persistent on-disk storage (Node, Bun, Deno).
* - `MEMORY` — in-memory, ephemeral.
*

2
src/coreTypes.ts
View file

@ -386,7 +386,7 @@ export interface HNSWVerbWithMetadata {
targetId: string
// SUBTYPE — optional per-product sub-classification within a VerbType (e.g. a
// `Manages` relationship might have subtype 'direct' vs 'dotted-line'; a `RelatedTo`
// `ReportsTo` relationship might have subtype 'direct' vs 'dotted-line'; a `RelatedTo`
// edge might carry 'spouse' / 'sibling' / 'colleague'). Flat string (no hierarchy) —
// consumers decide the vocabulary. Indexed and rolled up into per-VerbType statistics
// so it's queryable (`getRelations({ verb, subtype })`) and aggregable

4
src/graph/graphAdjacencyIndex.ts
View file

@ -766,8 +766,8 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
// Note: LSM-trees will be recreated from storage via their own initialization
// Verb data will be loaded on-demand via UnifiedCache
// Brainy 8.0: storage is always local (filesystem or memory) per
// BR-BRAINY-80-STORAGE-SIMPLIFY. Load all verbs at once.
// Brainy 8.0: storage is always local (filesystem or memory — the
// cloud adapters were removed). Load all verbs at once.
const storageType = this.storage?.constructor.name || ''
let totalVerbs = 0

4
src/hnsw/hnswIndex.ts
View file

@ -1261,8 +1261,8 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
}
// Step 4: Load all HNSW nodes at once. Brainy 8.0 ships filesystem +
// memory storage only (per BR-BRAINY-80-STORAGE-SIMPLIFY); the
// cloud-pagination rebuild path was deleted alongside the cloud adapters.
// memory storage only; the cloud-pagination rebuild path was deleted
// alongside the cloud adapters.
const storageType = this.storage?.constructor.name || ''
let loadedCount = 0
let totalCount: number | undefined = undefined

3
src/index.ts
View file

@ -28,6 +28,9 @@ export type {
RelateParams,
FindParams,
SubtypeRegistry,
FillSubtypeRule,
FillSubtypeRules,
FillSubtypesResult,
AggregateDefinition,
AggregateMetricDef,
AggregateSource,

5
src/neural/signals/ExactMatchSignal.ts
View file

@ -8,9 +8,8 @@
* 2. O(1) metadata hints (column names, file structure)
* 3. Format-specific intelligence (Excel, CSV, PDF, YAML, DOCX)
*
* This is the WORKSHOP BUG FIX - finds explicit relationships via exact matching
*
* PRODUCTION-READY: No TODOs, no mocks, real implementation
* Finds explicit relationships via exact matching — added after a consumer
* report of extraction missing explicitly-named relationships.
*/
import type { Brainy } from '../../brainy.js'

33
src/storage/baseStorage.ts
View file

@ -1528,10 +1528,16 @@ export abstract class BaseStorage extends BaseStorageAdapter {
const { limit, offset = 0, filter } = options
const collectedNouns: HNSWNounWithMetadata[] = []
// Collect ONE item past the requested window so `hasMore` is decidable:
// stopping exactly at offset+limit cannot distinguish "page full, nothing
// after it" from "page full, more behind it" — which made hasMore
// permanently false and silently truncated every multi-page walk
// (audit/migrateField/fillSubtypes, graph verb-id recovery, count rebuilds).
const targetCount = offset + limit
const peekCount = targetCount + 1
// Iterate by shards (0x00-0xFF) instead of types
for (let shard = 0; shard < 256 && collectedNouns.length < targetCount; shard++) {
for (let shard = 0; shard < 256 && collectedNouns.length < peekCount; shard++) {
const shardHex = shard.toString(16).padStart(2, '0')
const shardDir = `entities/nouns/${shardHex}`
@ -1539,7 +1545,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
const nounFiles = await this.listCanonicalObjects(shardDir)
for (const nounPath of nounFiles) {
if (collectedNouns.length >= targetCount) break
if (collectedNouns.length >= peekCount) break
if (!nounPath.includes('/vectors.json')) continue
try {
@ -1597,7 +1603,8 @@ export abstract class BaseStorage extends BaseStorageAdapter {
}
}
// Apply pagination
// Apply pagination. The peeked extra item (if any) is dropped by the slice;
// its existence is exactly what makes hasMore true.
const paginatedNouns = collectedNouns.slice(offset, offset + limit)
const hasMore = collectedNouns.length > targetCount
@ -1647,7 +1654,11 @@ export abstract class BaseStorage extends BaseStorageAdapter {
const { limit, offset = 0, filter } = options // cursor intentionally not extracted (not yet implemented)
const collectedVerbs: HNSWVerbWithMetadata[] = []
const targetCount = offset + limit // Early termination target
// Same peek-one-past-the-window strategy as getNounsWithPagination — see
// the comment there. Without the extra item, hasMore is undecidable and
// was permanently false (silent truncation of every multi-page walk).
const targetCount = offset + limit // Requested window end
const peekCount = targetCount + 1 // Early termination target (window + 1 peek)
// Prepare filter sets for efficient lookup
const filterVerbTypes = filter?.verbType
@ -1668,7 +1679,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
: null
// Iterate by shards (0x00-0xFF) instead of types - single pass!
for (let shard = 0; shard < 256 && collectedVerbs.length < targetCount; shard++) {
for (let shard = 0; shard < 256 && collectedVerbs.length < peekCount; shard++) {
const shardHex = shard.toString(16).padStart(2, '0')
const shardDir = `entities/verbs/${shardHex}`
@ -1676,7 +1687,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
const verbFiles = await this.listCanonicalObjects(shardDir)
for (const verbPath of verbFiles) {
if (collectedVerbs.length >= targetCount) break
if (collectedVerbs.length >= peekCount) break
if (!verbPath.includes('/vectors.json')) continue
try {
@ -1737,9 +1748,11 @@ export abstract class BaseStorage extends BaseStorageAdapter {
}
}
// Apply pagination (Efficient slicing after early termination)
// Apply pagination. The peeked extra item (if any) is dropped by the slice;
// its existence is exactly what makes hasMore true. `>` (not `>=`) keeps the
// exact-boundary case (total == offset+limit) from looping forever.
const paginatedVerbs = collectedVerbs.slice(offset, offset + limit)
const hasMore = collectedVerbs.length > targetCount // Fixed >= to > (was causing infinite loop)
const hasMore = collectedVerbs.length > targetCount
return {
items: paginatedVerbs,
@ -3235,8 +3248,8 @@ export abstract class BaseStorage extends BaseStorageAdapter {
* Before this override the Uint32Array counters were only persisted
* on a heuristic schedule inside `saveNoun_internal` (first-of-type or
* every-100th), which left readers seeing stale counts after a clean
* writer flush — the same failure mode that BR-FIND-WHERE-ZERO surfaced
* via `brain.stats()`.
* writer flush — the same silent-stale failure mode that once produced
* zero counts from `brain.stats()`.
*/
public async flushCounts(): Promise<void> {
await super.flushCounts()

2
src/storage/storageFactory.ts
View file

@ -8,7 +8,7 @@
* workloads.
*
* Cloud-storage adapters (GCS, S3, R2, Azure) and the browser-only OPFS
* adapter were removed in 8.0 per `BR-BRAINY-80-STORAGE-SIMPLIFY`. The path
* adapter were removed in 8.0. The path
* forward for cloud backup is operator tooling: persist locally with
* `db.persist()`, sync the resulting on-disk artefact with `gsutil` /
* `aws s3 cp` / `rclone` / `azcopy`. This is the standard pattern every

87
src/types/brainy.types.ts
View file

@ -76,7 +76,7 @@ export interface Relation<T = any> {
/** Relationship type classification (VerbType enum) */
type: VerbType
/**
* Per-product sub-classification within the VerbType (e.g. a `Manages`
* Per-product sub-classification within the VerbType (e.g. a `ReportsTo`
* relationship might have `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo`
* edge might carry `'spouse'` / `'sibling'` / `'colleague'`). Flat string, no
* hierarchy. Top-level standard field — indexed on the fast path and rolled into
@ -208,6 +208,68 @@ export interface SubtypeRegistry {
// Intentionally empty. Consumers extend via declaration merging.
}
/**
* A single back-fill rule for `brain.fillSubtypes()`.
*
* - A **literal string** assigns that subtype to every matching entry that
* lacks one (`'general'` — a blanket default).
* - A **function** receives the full entry and returns the subtype to assign,
* or `undefined` to leave the entry untouched (it is counted as `skipped`
* so a later run with a stricter rule can pick it up). Functions can derive
* the subtype from existing fields, e.g. `(e) => e.metadata?.kind`.
*
* @typeParam E - The entry shape the rule sees: `Entity<T>` for NounType keys,
* `Relation<T>` for VerbType keys.
*/
export type FillSubtypeRule<E> = string | ((entry: E) => string | undefined)
/**
* Rule map for `brain.fillSubtypes()` — the 8.0 subtype migration helper.
*
* Keys are `NounType` values (entity rules) and/or `VerbType` values
* (relationship rules); the two vocabularies don't overlap, so a single map
* covers both sides. Each value is a {@link FillSubtypeRule}: a literal
* subtype string or a function deriving one from the entry.
*
* @example
* ```ts
* await brain.fillSubtypes({
* [NounType.Person]: (e) => e.metadata?.kind ?? 'unspecified',
* [NounType.Thing]: 'general', // literal default
* [VerbType.RelatedTo]: 'unspecified' // relationship rule
* })
* ```
*/
export type FillSubtypeRules<T = any> = {
[K in NounType]?: FillSubtypeRule<Entity<T>>
} & {
[K in VerbType]?: FillSubtypeRule<Relation<T>>
}
/**
* Summary returned by `brain.fillSubtypes()`.
*
* After a run, `skipped` is exactly the remaining migration debt — re-running
* `brain.audit()` reports the same entries. Entries that already carry a
* subtype count toward `scanned` only (they are not debt, so they are neither
* `filled` nor `skipped`).
*/
export interface FillSubtypesResult {
/** Total entries examined (entities + relationships). */
scanned: number
/** Entries that received a subtype during this run. */
filled: number
/**
* Entries still missing a subtype after the run — either their type has no
* rule in the map, or their rule function returned `undefined`/empty.
*/
skipped: number
/** Per-entry write failures (`fillSubtypes` continues past individual errors). */
errors: Array<{ id: string; error: string }>
/** Fill counts grouped by NounType/VerbType key (only types with fills appear). */
byType: Record<string, number>
}
export interface AddParams<T = any> {
/** Content to embed and store. Strings are auto-embedded; objects are JSON-stringified for embedding. */
data: any | Vector
@ -282,7 +344,7 @@ export interface RelateParams<T = any> {
/** Relationship type classification (required) */
type: VerbType
/**
* Per-product sub-classification within the VerbType (e.g. a `Manages`
* Per-product sub-classification within the VerbType (e.g. a `ReportsTo`
* relationship might have `subtype: 'direct'` or `'dotted-line'`; a `RelatedTo`
* edge might carry `'spouse'` or `'colleague'`). Flat string, no hierarchy.
* Indexed and rolled up into per-VerbType statistics for fast filtering
@ -1239,16 +1301,21 @@ export interface BrainyConfig {
/**
* Brain-wide subtype enforcement mode.
*
* Opt-in in 7.30.0 (default: `false`); becomes the default in 8.0.0.
* **Default-on since 8.0.0** (`undefined` resolves to `true`; in 7.30.x this
* was an opt-in flag defaulting to `false`).
*
* - `false` / `undefined` (default): no brain-wide check. Per-type rules
* registered via `brain.requireSubtype(type, options)` still apply.
* - `true`: every `add()` / `addMany()` / `update()` / `relate()` /
* `relateMany()` / `updateRelation()` rejects writes where the entity's
* NounType (or relationship's VerbType) has no non-empty `subtype` value.
* - `{ except: [NounType.Thing, NounType.Custom] }`: same as `true`, but
* the listed types are allowed through without a subtype. Use for genuine
* - `true` / `undefined` (default): every `add()` / `addMany()` / `update()` /
* `relate()` / `relateMany()` / `updateRelation()` rejects writes where the
* entity's NounType (or relationship's VerbType) has no non-empty `subtype`
* value. Brainy's own VFS infrastructure writes (`metadata.isVFSEntity` /
* `metadata.isVFS`) bypass the check.
* - `{ except: [NounType.Thing, NounType.Custom] }`: same as `true`, but the
* listed types are allowed through without a subtype. Use for genuine
* catch-all types where no subtype makes sense.
* - `false`: disable the brain-wide check entirely. Last-resort escape hatch
* for opening pre-8.0 data — run `brain.audit()` to find the gaps, back-fill
* with `brain.fillSubtypes(rules)`, then remove the opt-out so the default
* enforcement protects new writes.
*
* Per-type registrations always compose with the brain-wide flag — a type
* registered with `requireSubtype(type, { required: true })` is always

16
src/utils/metadataIndex.ts
View file

@ -748,9 +748,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
*
* If neither the column store nor a sparse index covers the field, the
* function throws `BrainyError(FIELD_NOT_INDEXED)`. Returning `[]` for a
* genuinely unindexed field was the bug class `BR-FIND-WHERE-ZERO`
* tracked — a silent empty result indistinguishable from "the data
* really isn't there."
* genuinely unindexed field was a long-standing silent-empty bug class —
* an empty result indistinguishable from "the data really isn't there."
*/
private async getIdsFromChunks(field: string, value: any): Promise<string[]> {
// Load sparse index via UnifiedCache (lazy loading)
@ -1200,7 +1199,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// - Chunked Sparse Index: ~50 values per chunk, lazy-loaded
// - UnifiedCache LRU: Only hot chunks in memory
//
// Future: Bloom filter hybrid for unlimited words (see .strategy/BILLION-SCALE-PLAN.md)
// A Bloom-filter hybrid could lift the per-entity word cap entirely if
// full-document indexing at billion-entity scale ever becomes a need.
const textContent = this.extractTextContent(data)
if (textContent) {
const MAX_WORDS_PER_ENTITY = 5000 // Handles articles/chapters, memory-safe at scale
@ -1600,8 +1600,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
* Throws `BrainyError(FIELD_NOT_INDEXED)` if the field has no entries in
* either store. Callers in find()-evaluation catch this and translate to
* an empty result with a logged warning. The throw aligns the production
* `find()` path with the `brain.explain()` diagnostic, so the silent-
* empty bug class (BR-FIND-WHERE-ZERO) is no longer possible.
* `find()` path with the `brain.explain()` diagnostic, so a silently
* empty result for an unindexed field is no longer possible.
*/
async getIds(field: string, value: any): Promise<string[]> {
// Track exact query for field statistics
@ -2866,8 +2866,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
*
* Prior implementation read from `this.fieldIndexes` + lazy-loaded sparse
* indices, which silently returned `0` entries for any workspace written
* after sparse-index writes were deleted in commit `11be039`. That defect
* is what `BR-FIND-WHERE-ZERO` tracked.
* after sparse-index writes were deleted in commit `11be039`. That
* silent-zero defect is why this reads the column store first.
*/
async getStats(): Promise<MetadataIndexStats> {
const entityCount = this.idMapper.size

3
src/utils/paramValidation.ts
View file

@ -124,7 +124,8 @@ const getContainerMemoryLimit = (): number | null => {
* 1.5 KB + standard fields + metadata). On a 900 MB free-memory box this
* capped `limit` at 9_000, breaking common safety-cap call patterns like
* `find({ type, where, limit: 10_000 })` that typically return 10-500
* entities. Surfaced as `BR-MAXLIMIT-9000` in PLATFORM-HANDOFF.md.
* entities. Reported by a production consumer whose 10K safety-cap
* queries started failing after the cap landed.
* - 7.30.2+: `25` (assumes 25 KB per result). Generous over typical (7-10 KB),
* comfortably under the worst case (~20 KB with large metadata blobs).
* Same 900 MB box now gives ~36_000 — typical 10_000 limits pass silently,

2
tests/integration/strict-mode-self-test.test.ts
View file

@ -226,7 +226,7 @@ describe('Brainy strict-mode self-test (7.30.1)', () => {
const report = await brain.audit()
expect(report.total).toBeGreaterThanOrEqual(1)
expect(report.entitiesWithoutSubtype['person']).toBe(1)
expect(report.recommendation).toMatch(/Migrate via/)
expect(report.recommendation).toMatch(/fillSubtypes/)
})
it('excludes VFS entities by default (they bypass enforcement anyway)', async () => {

338
tests/unit/brainy/fill-subtypes.test.ts Normal file
View file

@ -0,0 +1,338 @@
/**
* @module tests/unit/brainy/fill-subtypes
* @description Unit tests for `brain.fillSubtypes(rules)` — the 8.0 subtype
* migration helper. Proves the full contract: literal and function rules,
* entity + relationship fills through one rule map, never overwriting an
* existing subtype, function rules declining entries (counted as skipped),
* VFS-marker exclusion, the `{ scanned, filled, skipped, errors, byType }`
* report, `_rev` bumping through the real update path, idempotent re-runs,
* per-entry error collection, and fail-fast rule-map validation.
*/
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
import { Brainy } from '../../../src/brainy'
import { NounType, VerbType } from '../../../src/types/graphTypes'
import { createTestConfig } from '../../helpers/test-factory'
describe('Brainy.fillSubtypes()', () => {
let brain: Brainy
beforeEach(async () => {
// requireSubtype: false so the tests can create the pre-8.0 migration
// debt (entities/relationships without subtype) that fillSubtypes exists
// to clear.
brain = new Brainy(createTestConfig())
await brain.init()
})
afterEach(async () => {
await brain.close()
})
describe('literal rules', () => {
it('fills missing subtypes with a literal default', async () => {
const a = await brain.add({ type: NounType.Person, data: 'Alice' })
const b = await brain.add({ type: NounType.Person, data: 'Bob' })
const report = await brain.fillSubtypes({
[NounType.Person]: 'unspecified'
})
expect(report.filled).toBe(2)
expect(report.byType).toEqual({ person: 2 })
expect((await brain.get(a))!.subtype).toBe('unspecified')
expect((await brain.get(b))!.subtype).toBe('unspecified')
})
it('never overwrites an existing subtype', async () => {
const employee = await brain.add({
type: NounType.Person,
subtype: 'employee',
data: 'Has a subtype already'
})
const blank = await brain.add({ type: NounType.Person, data: 'No subtype' })
const report = await brain.fillSubtypes({
[NounType.Person]: 'unspecified'
})
expect(report.filled).toBe(1)
expect((await brain.get(employee))!.subtype).toBe('employee') // untouched
expect((await brain.get(blank))!.subtype).toBe('unspecified')
})
it('leaves types without a rule untouched and counts them as skipped', async () => {
await brain.add({ type: NounType.Person, data: 'Person without rule coverage' })
const doc = await brain.add({ type: NounType.Document, data: 'Doc gets filled' })
const report = await brain.fillSubtypes({
[NounType.Document]: 'general'
})
expect(report.filled).toBe(1)
expect(report.skipped).toBe(1) // the Person — still migration debt
expect((await brain.get(doc))!.subtype).toBe('general')
const audit = await brain.audit()
expect(audit.total).toBe(report.skipped) // skipped IS the remaining debt
})
})
describe('function rules', () => {
it('derives the subtype from entity fields', async () => {
const vendor = await brain.add({
type: NounType.Person,
data: 'Vendor person',
metadata: { kind: 'vendor' }
})
const fallback = await brain.add({ type: NounType.Person, data: 'No kind field' })
const report = await brain.fillSubtypes({
[NounType.Person]: (e) => (e.metadata as { kind?: string })?.kind ?? 'unspecified'
})
expect(report.filled).toBe(2)
expect((await brain.get(vendor))!.subtype).toBe('vendor')
expect((await brain.get(fallback))!.subtype).toBe('unspecified')
})
it('treats undefined returns as declines (selective fill, counted as skipped)', async () => {
const classified = await brain.add({
type: NounType.Person,
data: 'Classifiable',
metadata: { department: 'engineering' }
})
const unclassified = await brain.add({ type: NounType.Person, data: 'Not classifiable' })
const report = await brain.fillSubtypes({
[NounType.Person]: (e) =>
(e.metadata as { department?: string })?.department ? 'employee' : undefined
})
expect(report.filled).toBe(1)
expect(report.skipped).toBe(1)
expect((await brain.get(classified))!.subtype).toBe('employee')
expect((await brain.get(unclassified))!.subtype).toBeUndefined()
})
it('treats empty-string returns as declines (an empty subtype would fail enforcement)', async () => {
await brain.add({ type: NounType.Person, data: 'Rule returns empty string' })
const report = await brain.fillSubtypes({
[NounType.Person]: () => ''
})
expect(report.filled).toBe(0)
expect(report.skipped).toBe(1)
})
it('collects per-entry rule errors and continues the pass', async () => {
const poisoned = await brain.add({
type: NounType.Person,
data: 'Rule throws on this one',
metadata: { poison: true }
})
const healthy = await brain.add({ type: NounType.Person, data: 'Rule works on this one' })
const report = await brain.fillSubtypes({
[NounType.Person]: (e) => {
if ((e.metadata as { poison?: boolean })?.poison) {
throw new Error('cannot classify poisoned entity')
}
return 'unspecified'
}
})
expect(report.filled).toBe(1)
expect(report.errors).toHaveLength(1)
expect(report.errors[0].id).toBe(poisoned)
expect(report.errors[0].error).toMatch(/cannot classify/)
expect((await brain.get(healthy))!.subtype).toBe('unspecified')
})
})
describe('relationship rules (VerbType keys)', () => {
it('fills missing relationship subtypes through the same rule map', async () => {
const a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'A' })
const b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'B' })
const relId = await brain.relate({ from: a, to: b, type: VerbType.RelatedTo })
const labeled = await brain.relate({
from: b,
to: a,
type: VerbType.RelatedTo,
subtype: 'colleague'
})
const report = await brain.fillSubtypes({
[VerbType.RelatedTo]: 'unspecified'
})
expect(report.filled).toBe(1)
expect(report.byType).toEqual({ relatedTo: 1 })
const relations = await brain.getRelations({ from: a })
expect(relations.find((r) => r.id === relId)!.subtype).toBe('unspecified')
const reverse = await brain.getRelations({ from: b })
expect(reverse.find((r) => r.id === labeled)!.subtype).toBe('colleague') // untouched
})
it('passes the full Relation shape to relationship rule functions', async () => {
const a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'A' })
const b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'B' })
await brain.relate({
from: a,
to: b,
type: VerbType.ReportsTo,
metadata: { dotted: true }
})
const report = await brain.fillSubtypes({
[VerbType.ReportsTo]: (r) => {
// The rule sees the public Relation shape — endpoints included.
expect(r.from).toBe(a)
expect(r.to).toBe(b)
return (r.metadata as { dotted?: boolean })?.dotted ? 'dotted-line' : 'direct'
}
})
expect(report.filled).toBe(1)
const relations = await brain.getRelations({ from: a })
expect(relations[0].subtype).toBe('dotted-line')
})
it('handles entity and relationship rules in a single pass', async () => {
const a = await brain.add({ type: NounType.Person, data: 'A' })
const b = await brain.add({ type: NounType.Person, data: 'B' })
await brain.relate({ from: a, to: b, type: VerbType.RelatedTo })
const report = await brain.fillSubtypes({
[NounType.Person]: 'unspecified',
[VerbType.RelatedTo]: 'unspecified'
})
expect(report.filled).toBe(3)
expect(report.byType).toEqual({ person: 2, relatedTo: 1 })
const audit = await brain.audit()
expect(audit.total).toBe(0)
})
})
describe('report + write semantics', () => {
it('bumps _rev through the real update path', async () => {
const id = await brain.add({ type: NounType.Person, data: 'Rev check' })
const before = await brain.get(id)
await brain.fillSubtypes({ [NounType.Person]: 'unspecified' })
const after = await brain.get(id)
expect(after!._rev).toBe((before!._rev ?? 1) + 1)
})
it('is idempotent — a re-run scans but fills nothing', async () => {
await brain.add({ type: NounType.Person, data: 'Fill once' })
const first = await brain.fillSubtypes({ [NounType.Person]: 'unspecified' })
expect(first.filled).toBe(1)
const second = await brain.fillSubtypes({ [NounType.Person]: 'unspecified' })
expect(second.filled).toBe(0)
expect(second.skipped).toBe(0)
expect(second.scanned).toBeGreaterThan(0)
expect(second.errors).toEqual([])
})
it('excludes VFS-marked entries by default, fills them with includeVFS: true', async () => {
// VFS markers bypass subtype enforcement, so marked entries are not
// migration debt. Fabricate one explicitly (public consumers shouldn't
// set this marker; the test exercises the exclusion contract).
const vfsLike = await brain.add({
type: NounType.Document,
data: 'Infrastructure-marked entry',
metadata: { isVFS: true }
})
const defaultRun = await brain.fillSubtypes({ [NounType.Document]: 'general' })
expect(defaultRun.filled).toBe(0)
expect((await brain.get(vfsLike))!.subtype).toBeUndefined()
const inclusiveRun = await brain.fillSubtypes(
{ [NounType.Document]: 'general' },
{ includeVFS: true }
)
expect(inclusiveRun.filled).toBeGreaterThanOrEqual(1)
expect((await brain.get(vfsLike))!.subtype).toBe('general')
})
it('reports progress after each batch', async () => {
await brain.add({ type: NounType.Person, data: 'P1' })
await brain.add({ type: NounType.Person, data: 'P2' })
await brain.add({ type: NounType.Person, data: 'P3' })
const snapshots: Array<{ scanned: number; filled: number; skipped: number }> = []
await brain.fillSubtypes(
{ [NounType.Person]: 'unspecified' },
{
batchSize: 1,
onProgress: (p) => snapshots.push({ ...p })
}
)
expect(snapshots.length).toBeGreaterThanOrEqual(3)
const last = snapshots[snapshots.length - 1]
expect(last.filled).toBe(3)
})
it('clears the debt brain.audit() reports', async () => {
await brain.add({ type: NounType.Person, data: 'Debt 1' })
await brain.add({ type: NounType.Document, data: 'Debt 2' })
const before = await brain.audit()
expect(before.total).toBe(2)
await brain.fillSubtypes({
[NounType.Person]: 'unspecified',
[NounType.Document]: 'general'
})
const after = await brain.audit()
expect(after.total).toBe(0)
expect(after.recommendation).toMatch(/strict-mode-ready/)
})
})
describe('rule-map validation (fail fast, before touching data)', () => {
it('rejects an empty rules map', async () => {
await expect(brain.fillSubtypes({})).rejects.toThrow(/rules map is empty/)
})
it('rejects keys that are not a NounType or VerbType', async () => {
await expect(
brain.fillSubtypes({ 'not-a-type': 'whatever' } as never)
).rejects.toThrow(/'not-a-type' is not a valid NounType or VerbType/)
})
it('rejects empty-string literal rules', async () => {
await expect(
brain.fillSubtypes({ [NounType.Person]: '' })
).rejects.toThrow(/empty string/)
})
it('rejects rule values that are neither string nor function', async () => {
await expect(
brain.fillSubtypes({ [NounType.Person]: 42 } as never)
).rejects.toThrow(/must be a subtype string or a function/)
})
it('does not write anything when validation fails', async () => {
const id = await brain.add({ type: NounType.Person, data: 'Untouched on failure' })
await expect(
brain.fillSubtypes({
[NounType.Person]: 'unspecified',
'bogus-type': 'x'
} as never)
).rejects.toThrow(/bogus-type/)
expect((await brain.get(id))!.subtype).toBeUndefined()
})
})
})