feat(8.0): brain.fillSubtypes migration helper + pre-RC1 gap closure

- brain.fillSubtypes(rules): idempotent subtype back-fill for pre-8.0 data.
  One rule per NounType/VerbType (literal default or per-entry function);
  fills only entries still missing a subtype through the public update()/
  updateRelation() paths; returns { scanned, filled, skipped, errors, byType }.
  Full unit suite in tests/unit/brainy/fill-subtypes.test.ts.
- Fix getNouns/getVerbs pagination hasMore (peek one past the window) —
  was permanently false, silently truncating every multi-page walk.
- find({ near }) without near.id now throws a teaching error instead of an
  opaque storage sharding failure; CLI --threshold without --near applies a
  plain score floor.
- CLI init/close audit: every one-shot command init()s, close()s, and exits
  explicitly; delete the unmaintained interactive REPL; replace the cloud-era
  storage subcommands with status/batch-delete; new types/validate commands.
- requireSubtype JSDoc now documents the 8.0 default-on contract; audit()
  recommendation points at fillSubtypes.
- Docs: data-storage-architecture rewritten to the real 8.0 on-disk layout;
  README storage section reflects filesystem+memory and snapshots; eli5 and
  SEMANTIC_VFS /as-of/ semantics corrected; internal tracker IDs and
  .strategy references scrubbed from published files.
This commit is contained in:
David Snelling 2026-06-11 10:42:34 -07:00
parent 9b0f4acd5b
commit c44678390e
30 changed files with 1517 additions and 3226 deletions

View file

@ -278,11 +278,16 @@ brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
**Enforce the pairing.** Register a vocabulary per type or turn on brain-wide strict mode to ensure every entity AND relationship has both `type` AND `subtype`: **Enforce the pairing.** Register a vocabulary per type or turn on brain-wide strict mode to ensure every entity AND relationship has both `type` AND `subtype`:
```javascript ```javascript
// Per-type rule with vocabulary // Per-type rule with a closed vocabulary
brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true }) brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true })
// Or brain-wide strict mode // 8.0 default: every write requires a subtype. Exempt genuine catch-all types…
const brain = new Brainy({ requireSubtype: true }) const brain = new Brainy({ requireSubtype: { except: [NounType.Thing] } })
// …or opt out while migrating pre-8.0 data, then audit and back-fill:
const legacy = new Brainy({ requireSubtype: false })
await legacy.audit() // gaps, grouped by type
await legacy.fillSubtypes({ [NounType.Person]: 'unspecified' }) // close them
``` ```
For other facets you want counted (`status`, `source`, `role`), register them with `brain.trackField(name)`. Renaming an existing convention to `subtype`? Use `brain.migrateField({from, to, entityKind: 'both'})` to walk nouns AND verbs in one pass. Full guide: **[Subtypes & Facets](docs/guides/subtypes-and-facets.md)**. For other facets you want counted (`status`, `source`, `role`), register them with `brain.trackField(name)`. Renaming an existing convention to `subtype`? Use `brain.migrateField({from, to, entityKind: 'both'})` to walk nouns AND verbs in one pass. Full guide: **[Subtypes & Facets](docs/guides/subtypes-and-facets.md)**.
@ -291,7 +296,7 @@ For other facets you want counted (`status`, `source`, `role`), register them wi
--- ---
## Storage: Memory to Cloud ## Storage: Memory and Filesystem
The same API at every scale. Change one config line to go from prototype to production. The same API at every scale. Change one config line to go from prototype to production.
@ -301,28 +306,31 @@ The same API at every scale. Change one config line to go from prototype to prod
const brain = new Brainy() const brain = new Brainy()
``` ```
### Production — Filesystem with Compression ### Production — Filesystem (gzip compression on by default)
```javascript ```javascript
const brain = new Brainy({ const brain = new Brainy({
storage: { type: 'filesystem', path: './data', compression: true } storage: { type: 'filesystem', rootDirectory: './data' }
}) })
``` ```
### Cloud — S3, GCS, Azure, Cloudflare R2 ### Backups and Portability — Snapshots
```javascript ```javascript
const brain = new Brainy({ const db = brain.now()
storage: { await db.persist('/backups/2026-06-11') // instant hard-link snapshot
type: 's3', await db.release()
s3Storage: { bucketName: 'my-knowledge-base', region: 'us-east-1' }
} const snapshot = await Brainy.load('/backups/2026-06-11') // read-only Db
}) const hits = await snapshot.search('quarterly invoices')
await snapshot.release()
``` ```
A snapshot directory is self-contained — copy it to another machine, open it with `Brainy.load()`, or restore it wholesale with `brain.restore(path, { confirm: true })`.
Performance benchmarks and capacity planning in **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)**. Performance benchmarks and capacity planning in **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)**.
**[Cloud Deployment Guide](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** | **[Capacity Planning](docs/operations/capacity-planning.md)** **[Capacity Planning](docs/operations/capacity-planning.md)**
--- ---
@ -371,10 +379,8 @@ Performance benchmarks and capacity planning in **[docs/PERFORMANCE.md](docs/PER
### Operations ### Operations
- **[Cloud Deployment](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** — AWS, GCS, Azure
- **[Capacity Planning](docs/operations/capacity-planning.md)** — Memory, storage, and scaling - **[Capacity Planning](docs/operations/capacity-planning.md)** — Memory, storage, and scaling
- **[Performance](docs/PERFORMANCE.md)** — Benchmarks and architecture details - **[Performance](docs/PERFORMANCE.md)** — Benchmarks and architecture details
- Cost Optimization: **[AWS S3](docs/operations/cost-optimization-aws-s3.md)** | **[GCS](docs/operations/cost-optimization-gcs.md)** | **[Azure](docs/operations/cost-optimization-azure.md)** | **[R2](docs/operations/cost-optimization-cloudflare-r2.md)**
--- ---
@ -387,7 +393,7 @@ bun install @soulcraft/brainy # Bun — best performance
npm install @soulcraft/brainy # Node.js — fully supported npm install @soulcraft/brainy # Node.js — fully supported
``` ```
> **Deprecation Notice:** Browser support (OPFS, Web Workers, WASM embeddings) is deprecated in v7.10.0 and will be removed in v8.0.0. Brainy v8+ will be server-only. > Brainy 8.0 is server-only. Browser support (OPFS storage, Web Workers, in-browser WASM embeddings) was removed in 8.0 — the 7.x line remains available on npm if you need it.
## Single-Writer Model ## Single-Writer Model

View file

@ -167,8 +167,12 @@ this rename; only the API surface moved.
Escape hatches: `requireSubtype: false` (last-resort opt-out for legacy Escape hatches: `requireSubtype: false` (last-resort opt-out for legacy
data) or `requireSubtype: { except: [NounType.Thing, ...] }` (per-type data) or `requireSubtype: { except: [NounType.Thing, ...] }` (per-type
allowlist). Per-type rules registered via `brain.requireSubtype(type, opts)` allowlist). Per-type rules registered via `brain.requireSubtype(type, opts)`
still compose. `brain.audit()` reports entries missing a subtype and still compose. `brain.audit()` reports entries missing a subtype and the new
`brain.migrateField()` backfills them. `brain.fillSubtypes(rules)` migration helper backfills them — one rule per
NounType/VerbType (literal default or per-entry function), applied only to
entries still missing a subtype, returning
`{ scanned, filled, skipped, errors, byType }`. Idempotent: re-running fills
nothing, so a crashed run is resumed by running it again.
- **Verb ids are Brainy-generated UUIDs — by contract.** `relate()` now throws - **Verb ids are Brainy-generated UUIDs — by contract.** `relate()` now throws
a teaching error if a caller passes an `id` field a teaching error if a caller passes an `id` field
(`src/utils/paramValidation.ts:526`). In 7.x a supplied id was silently (`src/utils/paramValidation.ts:526`). In 7.x a supplied id was silently
@ -270,8 +274,10 @@ The full cost model is in
`getHistory` / `streamHistory` / `versions` / `data()` per the table above. `getHistory` / `streamHistory` / `versions` / `data()` per the table above.
6. **Subtypes:** if your data predates subtype discipline, start with 6. **Subtypes:** if your data predates subtype discipline, start with
`requireSubtype: false`, run `brain.audit()`, backfill with `requireSubtype: false`, run `brain.audit()`, backfill with
`brain.migrateField()`, then remove the opt-out so the 8.0 default `brain.fillSubtypes(rules)` (one rule per type — a literal default or a
enforcement protects you going forward. function deriving the subtype from each entry), re-run `audit()` until
`total === 0`, then remove the opt-out so the 8.0 default enforcement
protects you going forward.
7. **CLI scripts:** `fork` / `branch` / `checkout` / `migrate` 7. **CLI scripts:** `fork` / `branch` / `checkout` / `migrate`
`snapshot <path>` / `restore <path>` / `history` / `generation`. `snapshot <path>` / `restore <path>` / `history` / `generation`.
8. **Plugin authors:** apply the contract renames and the BigInt graph 8. **Plugin authors:** apply the contract renames and the BigInt graph
@ -553,8 +559,7 @@ alongside every other metadata field on the existing write paths.
`_rev`, `ifRev`, `RevisionConflictError`, and `ifAbsent` all survive the 8.0 Db `_rev`, `ifRev`, `RevisionConflictError`, and `ifAbsent` all survive the 8.0 Db
redesign unchanged. 8.0 layers `brain.transact(tx, { ifAtGeneration })` for redesign unchanged. 8.0 layers `brain.transact(tx, { ifAtGeneration })` for
whole-tx CAS on top of the same per-entity mechanism — per-entity for single-record whole-tx CAS on top of the same per-entity mechanism — per-entity for single-record
patterns, generation-based for "did the world move under me." See patterns, generation-based for "did the world move under me."
`.strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md` § C-6 (internal).
--- ---
@ -672,8 +677,7 @@ in `validateFindParams()`, both fire before any storage/index/Cortex call. Corte
The new `docs/guides/find-limits.md` calls out that 8.0's Datomic-style `Db.find()` The new `docs/guides/find-limits.md` calls out that 8.0's Datomic-style `Db.find()`
may tighten per-call limits; that's a Brainy 8.0 / Cortex 3.0 coordination point may tighten per-call limits; that's a Brainy 8.0 / Cortex 3.0 coordination point
documented in `.strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md` (open question #4 covers whose rollout staging now also covers query limits.
the rollout staging, which now also applies to query limits).
### What consumers should do ### What consumers should do
@ -830,8 +834,7 @@ Brainy before any native call.
- **Native `audit()` proxy.** For billion-scale brains, `audit()` walks every entity (O(N)). - **Native `audit()` proxy.** For billion-scale brains, `audit()` walks every entity (O(N)).
A native implementation reading from a "null-subtype" bitmap in the column store would be A native implementation reading from a "null-subtype" bitmap in the column store would be
O(buckets). Listed as the 6th open question in the O(buckets).
[Brainy 8.0 spec doc](`/media/dpsifr/storage/home/Projects/brainy/.strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md`).
- **Strict-mode parity test.** Cortex should mirror Brainy's new - **Strict-mode parity test.** Cortex should mirror Brainy's new
`tests/integration/strict-mode-self-test.test.ts` against their native paths to catch any `tests/integration/strict-mode-self-test.test.ts` against their native paths to catch any
latent bug where native writes bypass JS validation. latent bug where native writes bypass JS validation.

File diff suppressed because it is too large Load diff

View file

@ -86,9 +86,8 @@ Benefits:
`FileSystemStorage` is the only one in-tree, but Cortex's `FileSystemStorage` is the only one in-tree, but Cortex's
`MmapFileSystemStorage` inherits from it — interface relocation would `MmapFileSystemStorage` inherits from it — interface relocation would
ripple through the plugin ecosystem. ripple through the plugin ecosystem.
- The current state works. Real failure modes - The current state works. The real failure modes seen in the field
(BR-CX-INTERFACE-GAP) were build/install artifacts, not type-system were build/install artifacts, not type-system failures.
failures.
- 7.22.0 just shipped a clean fix. Stacking another refactor before - 7.22.0 just shipped a clean fix. Stacking another refactor before
consumers absorb it adds churn without urgency. consumers absorb it adds churn without urgency.
- The `hasStorageMethod()` guard accomplishes the same runtime safety the - The `hasStorageMethod()` guard accomplishes the same runtime safety the

View file

@ -133,7 +133,7 @@ Brainy is the only row with every box checked. And it runs all of them in a sing
### One library, any scale ### One library, any scale
Brainy scales from a single laptop to billions of entities without changing a line of code. Small datasets live in memory. Larger ones spill to disk. At cloud scale, Brainy uses S3-compatible storage and automatically shards across nodes — the same API the whole way. Up to ten billion entities is fully implemented today. Brainy scales from a quick experiment to serious production datasets without changing a line of code. Small datasets live entirely in memory. Larger ones spill to disk, where Brainy shards and compresses everything automatically. Need a backup or a copy? Snapshots are instant — the same API the whole way.
Add Cortex and you also unlock memory-mapped storage — aggregate state lives directly in the operating system's memory with zero serialization overhead, as fast as the hardware allows. Add Cortex and you also unlock memory-mapped storage — aggregate state lives directly in the operating system's memory with zero serialization overhead, as fast as the hardware allows.

View file

@ -47,16 +47,18 @@ const authFiles = await vfs.readdir('/by-concept/authentication')
const aliceFiles = await vfs.readdir('/by-author/alice') const aliceFiles = await vfs.readdir('/by-author/alice')
``` ```
### 2. **Time Travel** ### 2. **Change Tracking by Date**
See your codebase as it existed at any point: List the files that were modified on any given day:
```typescript ```typescript
// Code from March 15th // Files that changed on March 15th
const snapshot = await vfs.readdir('/as-of/2024-03-15') const changed = await vfs.readdir('/as-of/2024-03-15')
// Compare with today // Everything under /src right now
const current = await vfs.readdir('/src') const current = await vfs.readdir('/src')
``` ```
`/as-of/<date>` selects by *modification date* — it reads the files' current content, not historical versions. For true point-in-time queries over entity state, use the Db API (`brain.asOf(generation)`).
### 3. **Knowledge Graph Navigation** ### 3. **Knowledge Graph Navigation**
Navigate by semantic relationships: Navigate by semantic relationships:
```typescript ```typescript
@ -118,13 +120,14 @@ await vfs.stat('/by-author/alice/config.ts')
### 4. By Time (Temporal) ✅ **Production** ### 4. By Time (Temporal) ✅ **Production**
```typescript ```typescript
await vfs.readdir('/as-of/2024-03-15') await vfs.readdir('/as-of/2024-03-15')
// Files modified on March 15, 2024 // Files modified on March 15, 2024 (24-hour window)
await vfs.readFile('/as-of/2024-03-15/src/auth.ts') await vfs.readFile('/as-of/2024-03-15/auth.ts')
// Read auth.ts as it existed that day // Current content of auth.ts, addressed by modification date —
// the path only resolves if auth.ts was modified that day
``` ```
**How it works:** Tracks `modified` timestamp. Uses B-tree range queries (`greaterEqual`/`lessEqual`) for O(log n) performance. **How it works:** Tracks the `modified` timestamp on every file and runs a range query (`greaterEqual`/`lessEqual`) over one 24-hour window for O(log n) performance. The VFS does not store historical file contents — `/as-of/` filters by *when a file last changed*; reads return the current bytes. For point-in-time state, use the Db API (`brain.asOf(generation)`).
**Status:** ✅ Fully implemented and tested at 10K file scale **Status:** ✅ Fully implemented and tested at 10K file scale
@ -222,7 +225,7 @@ console.log(authFiles)
// ['login.ts', 'signup.ts', 'oauth.ts'] // ['login.ts', 'signup.ts', 'oauth.ts']
``` ```
### Example 2: Time Travel ### Example 2: Changes by Day
```typescript ```typescript
// See what changed today // See what changed today
const today = new Date().toISOString().split('T')[0] const today = new Date().toISOString().split('T')[0]
@ -232,8 +235,8 @@ const todaysFiles = await vfs.readdir(`/as-of/${today}`)
const yesterday = new Date(Date.now() - 86400000).toISOString().split('T')[0] const yesterday = new Date(Date.now() - 86400000).toISOString().split('T')[0]
const yesterdaysFiles = await vfs.readdir(`/as-of/${yesterday}`) const yesterdaysFiles = await vfs.readdir(`/as-of/${yesterday}`)
const newFiles = todaysFiles.filter(f => !yesterdaysFiles.includes(f)) const onlyToday = todaysFiles.filter(f => !yesterdaysFiles.includes(f))
console.log('New files today:', newFiles) console.log('Changed today (untouched yesterday):', onlyToday)
``` ```
### Example 3: Graph Navigation ### Example 3: Graph Navigation
@ -432,25 +435,17 @@ await vfs.writeFile(path, code, {
}) })
``` ```
### 3. Optimize for Your Scale ### 3. Combine Dimensions
```typescript ```typescript
// For < 100K files: Post-filtering is fine // Find security files Alice changed on a given day
// For > 100K files: Use flattened indexes // (each /as-of/<date> path covers exactly that one day)
// Force index refresh after bulk operations
await brain.storage.rebuildIndexes()
```
### 4. Combine Dimensions
```typescript
// Find security files Alice worked on this week
const aliceFiles = await vfs.readdir('/by-author/alice') const aliceFiles = await vfs.readdir('/by-author/alice')
const securityFiles = await vfs.readdir('/by-tag/security') const securityFiles = await vfs.readdir('/by-tag/security')
const thisWeek = await vfs.readdir(`/as-of/${weekAgo}`) const changedThatDay = await vfs.readdir('/as-of/2024-03-15')
const intersection = aliceFiles const intersection = aliceFiles
.filter(f => securityFiles.includes(f)) .filter(f => securityFiles.includes(f))
.filter(f => thisWeek.includes(f)) .filter(f => changedThatDay.includes(f))
``` ```
--- ---
@ -469,14 +464,14 @@ console.log(entity.metadata.concepts)
### Slow Queries on Large Datasets ### Slow Queries on Large Datasets
```typescript ```typescript
// Check if indexes are built // Check if indexes are built and populated
const stats = await brain.storage.getIndexStats() const stats = await brain.getIndexStats()
console.log(stats) console.log(stats)
// Rebuild if needed
await brain.storage.rebuildIndexes()
``` ```
If an index looks empty or inconsistent, rebuild from raw storage with the CLI
(stop the live writer first): `brainy inspect repair <data-dir>`.
### Semantic Path Returns Empty ### Semantic Path Returns Empty
```typescript ```typescript
// Check if metadata exists // Check if metadata exists

View file

@ -94,7 +94,10 @@ import {
BatchResult, BatchResult,
BrainyConfig, BrainyConfig,
BrainyStats, BrainyStats,
ScoreExplanation ScoreExplanation,
FillSubtypeRule,
FillSubtypeRules,
FillSubtypesResult
} from './types/brainy.types.js' } from './types/brainy.types.js'
import { NounType, VerbType, TypeUtils } from './types/graphTypes.js' import { NounType, VerbType, TypeUtils } from './types/graphTypes.js'
import { BrainyInterface } from './types/brainyInterface.js' import { BrainyInterface } from './types/brainyInterface.js'
@ -2885,7 +2888,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* }) * })
* *
* @example Lock down management relationships * @example Lock down management relationships
* brain.requireSubtype(VerbType.Manages, { * brain.requireSubtype(VerbType.ReportsTo, {
* values: ['direct', 'dotted-line'], * values: ['direct', 'dotted-line'],
* required: true * required: true
* }) * })
@ -7330,10 +7333,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* @param subtype - Optional specific subtype string for O(1) point count * @param subtype - Optional specific subtype string for O(1) point count
* *
* @example * @example
* brain.counts.byRelationshipSubtype(VerbType.Manages) * brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
* // → { direct: 12, 'dotted-line': 3 } * // → { direct: 12, 'dotted-line': 3 }
* *
* brain.counts.byRelationshipSubtype(VerbType.Manages, 'direct') * brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct')
* // → 12 * // → 12
*/ */
byRelationshipSubtype: (verb: VerbType, subtype?: string): number | Record<string, number> => { byRelationshipSubtype: (verb: VerbType, subtype?: string): number | Record<string, number> => {
@ -7361,7 +7364,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* `topSubtypes` for verbs. * `topSubtypes` for verbs.
* *
* @example * @example
* brain.counts.topRelationshipSubtypes(VerbType.Manages, 5) * brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 5)
* // → [['direct', 12], ['dotted-line', 3]] * // → [['direct', 12], ['dotted-line', 3]]
*/ */
topRelationshipSubtypes: (verb: VerbType, n: number = 10): Array<[string, number]> => { topRelationshipSubtypes: (verb: VerbType, n: number = 10): Array<[string, number]> => {
@ -7533,7 +7536,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* @returns Sorted list of distinct subtype strings (empty if none) * @returns Sorted list of distinct subtype strings (empty if none)
* *
* @example * @example
* const variants = brain.relationshipSubtypesOf(VerbType.Manages) * const variants = brain.relationshipSubtypesOf(VerbType.ReportsTo)
* // → ['direct', 'dotted-line'] * // → ['direct', 'dotted-line']
*/ */
relationshipSubtypesOf(verb: VerbType): string[] { relationshipSubtypesOf(verb: VerbType): string[] {
@ -7550,17 +7553,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
/** /**
* Find entities and relationships missing a `subtype` value, grouped by type. * Find entities and relationships missing a `subtype` value, grouped by type.
* *
* The diagnostic pair to `migrateField()` / `fillSubtypes()` answers the * The diagnostic pair to `fillSubtypes()` / `migrateField()` answers the
* question "what would break if I enabled strict subtype enforcement?". Run * question "what would strict subtype enforcement reject?". 8.0 makes
* this before adopting an SDK that registers `requireSubtype()` rules on * `requireSubtype: true` the default, so run this when opening a pre-8.0
* common NounTypes, or before upgrading to Brainy 8.0 (which makes * brain (with `requireSubtype: false` as the temporary escape hatch), then
* `requireSubtype: true` the default). * back-fill the reported gaps with `fillSubtypes(rules)`.
* *
* Streams the brain via the same paginated `storage.getNouns()` / * Streams the brain via the same paginated `storage.getNouns()` /
* `storage.getVerbs()` pattern `migrateField()` uses safe for large brains * `storage.getVerbs()` pattern `fillSubtypes()` uses safe for large brains
* but linear in `O(N)`. Cortex 3.0+ may proxy this through the native * but linear in `O(N)`. A native index provider may serve this from a
* column-store null-subtype bitmap for sub-linear performance on billion-scale * column-store null-subtype bitmap in the future for sub-linear performance
* brains (tracked in `CTX-SUBTYPE-8.0-CONTRACT`). * on billion-scale brains.
* *
* @param options.includeVFS - When `false` (default), entities marked with * @param options.includeVFS - When `false` (default), entities marked with
* `metadata.isVFSEntity` or `metadata.isVFS` are excluded from the report * `metadata.isVFSEntity` or `metadata.isVFS` are excluded from the report
@ -7570,7 +7573,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* @param options.onProgress - Optional progress callback invoked after each batch. * @param options.onProgress - Optional progress callback invoked after each batch.
* @returns Report with per-type counts of entities/relationships without a * @returns Report with per-type counts of entities/relationships without a
* subtype, plus the overall total and a one-line recommendation pointing at * subtype, plus the overall total and a one-line recommendation pointing at
* `migrateField()` (7.x) or `fillSubtypes()` (8.0). * `fillSubtypes()`.
* *
* @example Find pre-existing gaps before turning on strict mode * @example Find pre-existing gaps before turning on strict mode
* const report = await brain.audit() * const report = await brain.audit()
@ -7655,7 +7658,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
const recommendation = missingSubtype === 0 const recommendation = missingSubtype === 0
? 'No subtype gaps detected — this brain is strict-mode-ready.' ? 'No subtype gaps detected — this brain is strict-mode-ready.'
: 'Found ' + missingSubtype + ' entries without subtype. Migrate via `brain.migrateField()` (7.x) — or wait for `brain.fillSubtypes()` (8.0) which closes the same gap with caller-supplied rules.' : 'Found ' + missingSubtype + ' entries without subtype. Back-fill with `brain.fillSubtypes(rules)` — one rule per NounType/VerbType, literal default or per-entry function. To lift an existing field into `subtype` instead, use `brain.migrateField({ from, to: \'subtype\' })`.'
return { return {
entitiesWithoutSubtype, entitiesWithoutSubtype,
@ -7666,6 +7669,262 @@ export class Brainy<T = any> implements BrainyInterface<T> {
} }
} }
/**
* Back-fill missing `subtype` values across the whole brain the 8.0
* migration helper for data written before subtype became required.
*
* 8.0 enforces a non-empty `subtype` on every write by default
* (`requireSubtype: true`). A brain created on 7.x typically carries entities
* and relationships without one; this method clears that debt in a single
* idempotent pass so the opt-out (`requireSubtype: false`) can be removed.
* The intended upgrade flow:
*
* 1. Open the brain with `requireSubtype: false` (temporary escape hatch).
* 2. `await brain.audit()` see what's missing, grouped by type.
* 3. `await brain.fillSubtypes(rules)` back-fill with one rule per type.
* 4. Re-run `audit()` until `total === 0`, then drop the opt-out.
*
* **Rules.** One rule per NounType (entities) and/or VerbType
* (relationships) the two vocabularies don't overlap, so a single map
* covers both sides. A rule is either a literal subtype string (blanket
* default) or a function deriving the subtype from the entry; functions
* subsume `where`-style filtering by returning `undefined` for entries they
* decline (those stay untouched and count as `skipped`, so a later run with
* a stricter rule can pick them up).
*
* **What is never touched:** entries that already carry a non-empty
* `subtype` (re-running is a no-op on them), and unless
* `includeVFS: true` Brainy's own VFS infrastructure entries
* (`metadata.isVFSEntity` / `metadata.isVFS`), which bypass enforcement
* anyway and are not migration debt.
*
* **Write strategy (deliberate):** each fill goes through the public
* `update()` / `updateRelation()` paths, so every write is individually
* atomic (storage record + indexes + subtype rollups commit together) and
* bumps the entry's `_rev` like any other update. The pass is *not* one
* whole-brain transaction: a migration over millions of entries inside a
* single transaction would hold an unbounded working set and turn one bad
* entry into an all-or-nothing failure. Idempotence is the recovery model
* a crashed or partially-failed run is resumed safely by re-running, because
* only entries still missing a subtype are written.
*
* Streams via the same paginated `storage.getNouns()` / `storage.getVerbs()`
* walk `audit()` uses `O(N)` but constant memory. The noun pass is skipped
* entirely when the map has no NounType rules, and vice versa.
*
* @param rules - Map of NounType/VerbType literal subtype or rule function.
* Must contain at least one valid type key; invalid keys, empty-string
* literals, and non-string/non-function values throw before any data is
* touched.
* @param options.includeVFS - Also fill VFS infrastructure entries (default
* `false` they bypass enforcement and don't need a subtype).
* @param options.batchSize - Pagination batch size (default `200`).
* @param options.onProgress - Optional callback invoked after each batch.
* @returns `{ scanned, filled, skipped, errors, byType }` see
* {@link FillSubtypesResult}. After a clean run, `skipped` equals the
* remaining `audit().total`.
* @throws If the brain is read-only, the rule map is empty/malformed, or a
* key is not a valid NounType/VerbType. Per-entry write failures do NOT
* throw they are collected in `errors` and the pass continues.
*
* @example Back-fill entities and relationships in one pass
* const report = await brain.fillSubtypes({
* [NounType.Person]: (e) => e.metadata?.kind ?? 'unspecified',
* [NounType.Document]: 'general',
* [VerbType.RelatedTo]: 'unspecified'
* })
* // → { scanned: 5200, filled: 1429, skipped: 0, errors: [], byType: { person: 800, document: 600, relatedTo: 29 } }
*
* @example Selective fill decline entries a rule can't classify
* await brain.fillSubtypes({
* [NounType.Person]: (e) => e.metadata?.department ? 'employee' : undefined
* })
* // Persons without a department stay untouched (counted as skipped).
*
* @since 8.0.0
*/
async fillSubtypes(
rules: FillSubtypeRules<T>,
options: {
includeVFS?: boolean
batchSize?: number
onProgress?: (progress: { scanned: number; filled: number; skipped: number }) => void
} = {}
): Promise<FillSubtypesResult> {
this.assertWritable('fillSubtypes')
await this.ensureInitialized()
// Validate the rule map up front — fail fast on shape errors before any
// data is touched.
if (!rules || typeof rules !== 'object' || Array.isArray(rules)) {
throw new Error(
'fillSubtypes: rules must be a map of NounType/VerbType → subtype string or rule function'
)
}
const nounTypeValues = new Set<string>(Object.values(NounType))
const verbTypeValues = new Set<string>(Object.values(VerbType))
const nounRules = new Map<string, FillSubtypeRule<Entity<T>>>()
const verbRules = new Map<string, FillSubtypeRule<Relation<T>>>()
for (const [key, rule] of Object.entries(rules)) {
if (rule === undefined) continue
if (typeof rule !== 'string' && typeof rule !== 'function') {
throw new Error(
`fillSubtypes: rule for '${key}' must be a subtype string or a function (got ${typeof rule})`
)
}
if (typeof rule === 'string' && rule.length === 0) {
throw new Error(
`fillSubtypes: rule for '${key}' is an empty string — a subtype must be non-empty`
)
}
if (nounTypeValues.has(key)) {
nounRules.set(key, rule as FillSubtypeRule<Entity<T>>)
} else if (verbTypeValues.has(key)) {
verbRules.set(key, rule as FillSubtypeRule<Relation<T>>)
} else {
throw new Error(`fillSubtypes: '${key}' is not a valid NounType or VerbType`)
}
}
if (nounRules.size === 0 && verbRules.size === 0) {
throw new Error(
'fillSubtypes: rules map is empty — provide at least one NounType or VerbType rule'
)
}
const includeVFS = options.includeVFS === true
const batchSize = Math.max(1, options.batchSize ?? 200)
let scanned = 0
let filled = 0
let skipped = 0
const errors: Array<{ id: string; error: string }> = []
const byType: Record<string, number> = {}
const reportProgress = (): void => {
if (options.onProgress) options.onProgress({ scanned, filled, skipped })
}
// Normalize a rule's output: only a non-empty string is a fill; anything
// else (undefined, empty string) is a decline. An empty subtype would fail
// the very strict-mode check this helper exists to satisfy.
const normalize = (value: string | undefined): string | undefined =>
typeof value === 'string' && value.length > 0 ? value : undefined
// Pass 1: entities (skipped entirely when the map has no NounType rules).
if (nounRules.size > 0) {
let offset = 0
while (true) {
const page = await this.storage.getNouns({ pagination: { offset, limit: batchSize } })
if (page.items.length === 0) break
for (const noun of page.items) {
scanned++
// VFS infrastructure entries bypass enforcement via their marker, so
// they're not migration debt — leave them alone unless asked.
if (
!includeVFS &&
(noun.metadata?.isVFSEntity === true || noun.metadata?.isVFS === true)
) {
continue
}
if (typeof noun.subtype === 'string' && noun.subtype.length > 0) continue // already filled — never overwrite
const rule = nounRules.get(noun.type)
if (rule === undefined) {
skipped++
continue
}
try {
let subtype: string | undefined
if (typeof rule === 'function') {
const entity = await this.convertNounToEntity(noun)
subtype = normalize(rule(entity))
} else {
subtype = rule
}
if (subtype === undefined) {
skipped++
continue
}
await this.update({ id: noun.id, subtype })
filled++
byType[noun.type] = (byType[noun.type] || 0) + 1
} catch (err) {
errors.push({
id: noun.id,
error: err instanceof Error ? err.message : String(err)
})
}
}
reportProgress()
if (!page.hasMore) break
offset += page.items.length
}
}
// Pass 2: relationships (skipped entirely when the map has no VerbType rules).
if (verbRules.size > 0) {
let offset = 0
while (true) {
const page = await this.storage.getVerbs({ pagination: { offset, limit: batchSize } })
if (page.items.length === 0) break
for (const verb of page.items) {
scanned++
if (
!includeVFS &&
(verb.metadata?.isVFSEntity === true || verb.metadata?.isVFS === true)
) {
continue
}
if (typeof verb.subtype === 'string' && verb.subtype.length > 0) continue // already filled — never overwrite
const rule = verbRules.get(verb.verb)
if (rule === undefined) {
skipped++
continue
}
try {
let subtype: string | undefined
if (typeof rule === 'function') {
// Project the stored verb onto the public Relation<T> shape the
// rule function is typed against.
const relation: Relation<T> = {
id: verb.id,
from: verb.sourceId,
to: verb.targetId,
type: verb.verb,
weight: verb.weight,
data: verb.data,
metadata: verb.metadata as T,
service: verb.service,
createdAt: verb.createdAt,
updatedAt: verb.updatedAt,
confidence: verb.confidence
}
subtype = normalize(rule(relation))
} else {
subtype = rule
}
if (subtype === undefined) {
skipped++
continue
}
await this.updateRelation({ id: verb.id, subtype })
filled++
byType[verb.verb] = (byType[verb.verb] || 0) + 1
} catch (err) {
errors.push({
id: verb.id,
error: err instanceof Error ? err.message : String(err)
})
}
}
reportProgress()
if (!page.hasMore) break
offset += page.items.length
}
}
return { scanned, filled, skipped, errors, byType }
}
/** /**
* Stream-and-rewrite a field across every entity in the brain. * Stream-and-rewrite a field across every entity in the brain.
* *
@ -8845,6 +9104,16 @@ export class Brainy<T = any> implements BrainyInterface<T> {
private async executeProximitySearch(params: FindParams<T>): Promise<Result<T>[]> { private async executeProximitySearch(params: FindParams<T>): Promise<Result<T>[]> {
if (!params.near) return [] if (!params.near) return []
// Teaching error: without an anchor id the constraint is meaningless, and
// letting it fall through produces an opaque storage-layer sharding error.
if (!params.near.id) {
throw new Error(
"find({ near }): 'near.id' is required — pass the entity to search around, " +
'e.g. near: { id, threshold }. To impose a minimum score on plain semantic ' +
'results, filter on result.score instead.'
)
}
const nearEntity = await this.get(params.near.id) const nearEntity = await this.get(params.near.id)
if (!nearEntity) return [] if (!nearEntity) return []
@ -9680,7 +9949,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
*/ */
private normalizeConfig(config?: BrainyConfig): Required<BrainyConfig> { private normalizeConfig(config?: BrainyConfig): Required<BrainyConfig> {
// Validate storage configuration. Brainy 8.0 ships two adapters only — // Validate storage configuration. Brainy 8.0 ships two adapters only —
// FileSystemStorage and MemoryStorage — per BR-BRAINY-80-STORAGE-SIMPLIFY. // FileSystemStorage and MemoryStorage (cloud + OPFS adapters were removed).
// Cloud backup remains supported via operator tooling (db.persist() + // Cloud backup remains supported via operator tooling (db.persist() +
// gsutil / aws s3 cp / rclone / azcopy). Pre-constructed adapter // gsutil / aws s3 cp / rclone / azcopy). Pre-constructed adapter
// instances bypass the type check (they ARE the storage). // instances bypass the type check (they ARE the storage).

View file

@ -132,6 +132,7 @@ export const coreCommands = {
const spinner = ora('Adding to neural database...').start() const spinner = ora('Adding to neural database...').start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
let metadata: any = {} let metadata: any = {}
if (options.metadata) { if (options.metadata) {
@ -204,6 +205,12 @@ export const coreCommands = {
} else { } else {
formatOutput({ id: result, metadata, confidence: addParams.confidence, weight: addParams.weight }, options) formatOutput({ id: result, metadata, confidence: addParams.confidence, weight: addParams.weight }, options)
} }
// close() releases the writer lock and indexes, but global timers
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
// alive. CLI commands are one-shot — exit explicitly.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
if (spinner) spinner.fail('Failed to add data') if (spinner) spinner.fail('Failed to add data')
console.error(chalk.red('Failed to add data:', error.message)) console.error(chalk.red('Failed to add data:', error.message))
@ -280,6 +287,7 @@ export const coreCommands = {
const spinner = ora('Searching with Triple Intelligence™...').start() const spinner = ora('Searching with Triple Intelligence™...').start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
// Build comprehensive search params // Build comprehensive search params
const searchParams: any = { const searchParams: any = {
@ -292,11 +300,6 @@ export const coreCommands = {
searchParams.offset = parseInt(options.offset) searchParams.offset = parseInt(options.offset)
} }
// Vector Intelligence - similarity threshold
if (options.threshold) {
searchParams.near = { threshold: parseFloat(options.threshold) }
}
// Metadata Intelligence - type filtering // Metadata Intelligence - type filtering
if (options.type) { if (options.type) {
const types = options.type.split(',').map(t => t.trim()) const types = options.type.split(',').map(t => t.trim())
@ -314,7 +317,9 @@ export const coreCommands = {
} }
} }
// Vector Intelligence - proximity search // Vector Intelligence - proximity search around an anchor entity.
// `near` requires an id; a bare --threshold (no --near) is applied as a
// plain score floor on the fused results after find() returns.
if (options.near) { if (options.near) {
searchParams.near = { searchParams.near = {
id: options.near, id: options.near,
@ -371,7 +376,14 @@ export const coreCommands = {
} }
} }
const results = await brain.find(searchParams) let results = await brain.find(searchParams)
// Without --near there is no proximity anchor; apply --threshold as a
// minimum-score filter on the fused results instead.
if (!options.near && options.threshold) {
const minScore = parseFloat(options.threshold)
results = results.filter((r) => r.score === undefined || r.score >= minScore)
}
spinner.succeed(`Found ${results.length} results`) spinner.succeed(`Found ${results.length} results`)
@ -457,6 +469,10 @@ export const coreCommands = {
} else { } else {
formatOutput(results, options) formatOutput(results, options)
} }
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
if (spinner) spinner.fail('Search failed') if (spinner) spinner.fail('Search failed')
console.error(chalk.red('Search failed:', error.message)) console.error(chalk.red('Search failed:', error.message))
@ -496,6 +512,7 @@ export const coreCommands = {
const spinner = ora('Fetching item...').start() const spinner = ora('Fetching item...').start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
// Try to get the item // Try to get the item
const item = await brain.get(id) const item = await brain.get(id)
@ -529,6 +546,10 @@ export const coreCommands = {
} else { } else {
formatOutput(item, options) formatOutput(item, options)
} }
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
if (spinner) spinner.fail('Failed to get item') if (spinner) spinner.fail('Failed to get item')
console.error(chalk.red('Failed to get item:', error.message)) console.error(chalk.red('Failed to get item:', error.message))
@ -589,6 +610,7 @@ export const coreCommands = {
const spinner = ora('Creating relationship...').start() const spinner = ora('Creating relationship...').start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
let metadata: any = {} let metadata: any = {}
if (options.metadata) { if (options.metadata) {
@ -627,6 +649,10 @@ export const coreCommands = {
} else { } else {
formatOutput({ id: result, source, verb, target, metadata }, options) formatOutput({ id: result, source, verb, target, metadata }, options)
} }
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
if (spinner) spinner.fail('Failed to create relationship') if (spinner) spinner.fail('Failed to create relationship')
console.error(chalk.red('Failed to create relationship:', error.message)) console.error(chalk.red('Failed to create relationship:', error.message))
@ -683,6 +709,7 @@ export const coreCommands = {
spinner = ora('Updating entity...').start() spinner = ora('Updating entity...').start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
// Get existing entity first // Get existing entity first
const existing = await brain.get(id) const existing = await brain.get(id)
@ -731,6 +758,10 @@ export const coreCommands = {
} else { } else {
formatOutput({ id, updated: true }, options) formatOutput({ id, updated: true }, options)
} }
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
if (spinner) spinner.fail('Failed to update entity') if (spinner) spinner.fail('Failed to update entity')
console.error(chalk.red('Update failed:', error.message)) console.error(chalk.red('Update failed:', error.message))
@ -784,6 +815,7 @@ export const coreCommands = {
spinner = ora('Deleting entity...').start() spinner = ora('Deleting entity...').start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
await brain.delete(id) await brain.delete(id)
@ -794,6 +826,10 @@ export const coreCommands = {
} else { } else {
formatOutput({ id, deleted: true }, options) formatOutput({ id, deleted: true }, options)
} }
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
if (spinner) spinner.fail('Failed to delete entity') if (spinner) spinner.fail('Failed to delete entity')
console.error(chalk.red('Delete failed:', error.message)) console.error(chalk.red('Delete failed:', error.message))
@ -846,6 +882,7 @@ export const coreCommands = {
spinner = ora('Removing relationship...').start() spinner = ora('Removing relationship...').start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
await brain.unrelate(id) await brain.unrelate(id)
@ -856,6 +893,10 @@ export const coreCommands = {
} else { } else {
formatOutput({ id, removed: true }, options) formatOutput({ id, removed: true }, options)
} }
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
if (spinner) spinner.fail('Failed to remove relationship') if (spinner) spinner.fail('Failed to remove relationship')
console.error(chalk.red('Unrelate failed:', error.message)) console.error(chalk.red('Unrelate failed:', error.message))
@ -875,7 +916,9 @@ export const coreCommands = {
if (options.json) { if (options.json) {
formatOutput(diag, options) formatOutput(diag, options)
return // One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} }
console.log(chalk.bold('\nBrainy Diagnostics')) console.log(chalk.bold('\nBrainy Diagnostics'))
@ -907,6 +950,10 @@ export const coreCommands = {
console.log(` Metadata: ${diag.indexes.metadata.type} (initialized: ${diag.indexes.metadata.initialized})`) console.log(` Metadata: ${diag.indexes.metadata.type} (initialized: ${diag.indexes.metadata.initialized})`)
console.log(` Graph: ${diag.indexes.graph.type} (initialized: ${diag.indexes.graph.initialized}, wired: ${diag.indexes.graph.wiredToStorage})`) console.log(` Graph: ${diag.indexes.graph.type} (initialized: ${diag.indexes.graph.initialized}, wired: ${diag.indexes.graph.wiredToStorage})`)
console.log() console.log()
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
console.error(chalk.red('Diagnostics failed: ' + error.message)) console.error(chalk.red('Diagnostics failed: ' + error.message))
process.exit(1) process.exit(1)

View file

@ -152,6 +152,7 @@ export const importCommands = {
spinner = ora('Initializing import...').start() spinner = ora('Initializing import...').start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
// Handle different source types // Handle different source types
let result: any let result: any
@ -401,6 +402,12 @@ export const importCommands = {
} else if (options.json) { } else if (options.json) {
formatOutput(result, options) formatOutput(result, options)
} }
// close() releases the writer lock and indexes, but global timers
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
// alive. CLI commands are one-shot — exit explicitly.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
if (spinner) spinner.fail('Import failed') if (spinner) spinner.fail('Import failed')
console.error(chalk.red('Import failed:', error.message)) console.error(chalk.red('Import failed:', error.message))
@ -472,9 +479,10 @@ export const importCommands = {
spinner = ora('Initializing VFS import...').start() spinner = ora('Initializing VFS import...').start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
// Get VFS // Get VFS
const vfs = await brain.vfs const vfs = brain.vfs
// Load DirectoryImporter // Load DirectoryImporter
const { DirectoryImporter } = await import('../../vfs/importers/DirectoryImporter.js') const { DirectoryImporter } = await import('../../vfs/importers/DirectoryImporter.js')
@ -536,6 +544,10 @@ export const importCommands = {
} else if (options.json) { } else if (options.json) {
formatOutput(result, options) formatOutput(result, options)
} }
// One-shot command — see import() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
if (spinner) spinner.fail('VFS import failed') if (spinner) spinner.fail('VFS import failed')
console.error(chalk.red('VFS import failed:', error.message)) console.error(chalk.red('VFS import failed:', error.message))

View file

@ -41,6 +41,7 @@ export const insightsCommands = {
try { try {
const brain = getBrainy() const brain = getBrainy()
await brain.init()
// Get insights from Brainy // Get insights from Brainy
const insights = await brain.insights() const insights = await brain.insights()
@ -102,6 +103,12 @@ export const insightsCommands = {
} else { } else {
formatOutput(insights, options) formatOutput(insights, options)
} }
// close() releases the writer lock and indexes, but global timers
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
// alive. CLI commands are one-shot — exit explicitly.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
spinner.fail('Failed to get insights') spinner.fail('Failed to get insights')
console.error(chalk.red('Insights failed:', error.message)) console.error(chalk.red('Insights failed:', error.message))
@ -120,6 +127,7 @@ export const insightsCommands = {
try { try {
const brain = getBrainy() const brain = getBrainy()
await brain.init()
// Get available fields from metadata index // Get available fields from metadata index
const fields = await brain.getAvailableFields() const fields = await brain.getAvailableFields()
@ -166,6 +174,10 @@ export const insightsCommands = {
})) }))
formatOutput(fieldsWithStats, options) formatOutput(fieldsWithStats, options)
} }
// One-shot command — see insights() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
spinner.fail('Failed to get fields') spinner.fail('Failed to get fields')
console.error(chalk.red('Fields analysis failed:', error.message)) console.error(chalk.red('Fields analysis failed:', error.message))
@ -186,6 +198,7 @@ export const insightsCommands = {
if (!field) { if (!field) {
spinner = ora('Getting available fields...').start() spinner = ora('Getting available fields...').start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
const availableFields = await brain.getAvailableFields() const availableFields = await brain.getAvailableFields()
spinner.stop() spinner.stop()
@ -202,6 +215,7 @@ export const insightsCommands = {
spinner = ora(`Getting values for field: ${field}...`).start() spinner = ora(`Getting values for field: ${field}...`).start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
const values = await brain.getFieldValues(field) const values = await brain.getFieldValues(field)
const limit = options.limit ? parseInt(options.limit) : 100 const limit = options.limit ? parseInt(options.limit) : 100
@ -242,6 +256,10 @@ export const insightsCommands = {
} else { } else {
formatOutput({ field, values, count: values.length }, options) formatOutput({ field, values, count: values.length }, options)
} }
// One-shot command — see insights() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
if (spinner) spinner.fail('Failed to get field values') if (spinner) spinner.fail('Failed to get field values')
console.error(chalk.red('Field values failed:', error.message)) console.error(chalk.red('Field values failed:', error.message))
@ -289,6 +307,7 @@ export const insightsCommands = {
spinner = ora('Analyzing optimal query plan...').start() spinner = ora('Analyzing optimal query plan...').start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
const plan = await brain.getOptimalQueryPlan(filters) const plan = await brain.getOptimalQueryPlan(filters)
@ -328,6 +347,10 @@ export const insightsCommands = {
} else { } else {
formatOutput({ filters, plan }, options) formatOutput({ filters, plan }, options)
} }
// One-shot command — see insights() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
if (spinner) spinner.fail('Failed to generate query plan') if (spinner) spinner.fail('Failed to generate query plan')
console.error(chalk.red('Query plan failed:', error.message)) console.error(chalk.red('Query plan failed:', error.message))

View file

@ -1,7 +1,11 @@
/** /**
* 🧠 Neural Similarity API Commands * @module cli/commands/neural
* * @description Neural CLI commands: semantic similarity, clustering,
* CLI interface for semantic similarity, clustering, and neural operations * hierarchy, neighbors, outlier detection, and visualization data export.
* Registered in `src/cli/index.ts` as `similar` / `cluster` / `related` /
* `hierarchy` / `outliers` / `visualize`. Each command is one-shot: it
* initializes the shared Brainy instance, runs the neural operation, then
* closes the store and exits explicitly.
*/ */
import inquirer from 'inquirer' import inquirer from 'inquirer'
@ -25,136 +29,13 @@ interface CommandArguments {
_: string[]; _: string[];
} }
export const neuralCommand = { let brainyInstance: Brainy | null = null
command: 'neural [action]',
describe: '🧠 Neural similarity and clustering operations',
builder: (yargs: any) => {
return yargs
.positional('action', {
describe: 'Neural operation to perform',
type: 'string',
choices: ['similar', 'clusters', 'hierarchy', 'neighbors', 'path', 'outliers', 'visualize']
})
.option('id', {
describe: 'Item ID for similarity operations',
type: 'string',
alias: 'i'
})
.option('query', {
describe: 'Query text for similarity search',
type: 'string',
alias: 'q'
})
.option('threshold', {
describe: 'Similarity threshold (0-1)',
type: 'number',
default: 0.7,
alias: 't'
})
.option('format', {
describe: 'Output format',
type: 'string',
choices: ['json', 'table', 'tree', 'graph'],
default: 'table',
alias: 'f'
})
.option('output', {
describe: 'Output file path',
type: 'string',
alias: 'o'
})
.option('limit', {
describe: 'Maximum number of results',
type: 'number',
default: 10,
alias: 'l'
})
.option('algorithm', {
describe: 'Clustering algorithm',
type: 'string',
choices: ['hierarchical', 'kmeans', 'dbscan', 'auto'],
default: 'auto',
alias: 'a'
})
.option('dimensions', {
describe: 'Visualization dimensions (2 or 3)',
type: 'number',
choices: [2, 3],
default: 2,
alias: 'd'
})
.option('explain', {
describe: 'Include detailed explanations',
type: 'boolean',
default: false,
alias: 'e'
})
},
handler: async (argv: CommandArguments) => {
console.log(chalk.cyan('\n🧠 NEURAL SIMILARITY API'))
console.log(chalk.gray('━'.repeat(50)))
// Initialize Brainy and Neural API const getBrainy = (): Brainy => {
const brain = new Brainy() if (!brainyInstance) {
const neural = brain.neural() brainyInstance = new Brainy()
try {
const action = argv.action || await promptForAction()
switch (action) {
case 'similar':
await handleSimilarCommand(neural, argv)
break
case 'clusters':
await handleClustersCommand(neural, argv)
break
case 'hierarchy':
await handleHierarchyCommand(neural, argv)
break
case 'neighbors':
await handleNeighborsCommand(neural, argv)
break
case 'path':
console.log(chalk.yellow('\n⚠ Semantic path finding coming soon'))
console.log(chalk.dim('This feature requires implementing graph traversal algorithms'))
console.log(chalk.dim('Use "neighbors" and "hierarchy" commands to explore connections'))
break
case 'outliers':
await handleOutliersCommand(neural, argv)
break
case 'visualize':
await handleVisualizeCommand(neural, argv)
break
default:
console.log(chalk.red(`❌ Unknown action: ${action}`))
showHelp()
}
} catch (error) {
console.error(chalk.red('💥 Error:'), error instanceof Error ? error.message : error)
process.exit(1)
}
} }
} return brainyInstance
async function promptForAction(): Promise<string> {
const answer = await inquirer.prompt([{
type: 'list',
name: 'action',
message: 'Choose a neural operation:',
choices: [
{ name: '🔗 Calculate similarity between items', value: 'similar' },
{ name: '🎯 Find semantic clusters', value: 'clusters' },
{ name: '🌳 Show item hierarchy', value: 'hierarchy' },
{ name: '🕸️ Find semantic neighbors', value: 'neighbors' },
{ name: '🛣️ Find semantic path between items (coming soon)', value: 'path', disabled: true },
{ name: '🚨 Detect outliers', value: 'outliers' },
{ name: '📊 Generate visualization data', value: 'visualize' }
]
}])
return answer.action
} }
async function handleSimilarCommand(neural: any, argv: CommandArguments): Promise<void> { async function handleSimilarCommand(neural: any, argv: CommandArguments): Promise<void> {
@ -436,63 +317,6 @@ async function handleNeighborsCommand(neural: any, argv: CommandArguments): Prom
} }
} }
async function handlePathCommand(neural: any, argv: CommandArguments): Promise<void> {
const spinner = ora('🛣️ Finding semantic path...').start()
try {
let fromId: string, toId: string
if (argv._ && argv._.length >= 3) {
fromId = argv._[1]
toId = argv._[2]
} else {
spinner.stop()
const answers = await inquirer.prompt([
{
type: 'input',
name: 'from',
message: 'From item ID:',
validate: (input: string) => input.length > 0
},
{
type: 'input',
name: 'to',
message: 'To item ID:',
validate: (input: string) => input.length > 0
}
])
fromId = answers.from
toId = answers.to
spinner.start()
}
const path = await neural.semanticPath(fromId, toId)
if (path.length === 0) {
spinner.warn('🚫 No semantic path found')
console.log(`No path found between ${chalk.cyan(fromId)} and ${chalk.cyan(toId)}`)
} else {
spinner.succeed(`✅ Found path with ${path.length} hops`)
console.log(`\n🛣 Semantic Path from ${chalk.cyan(fromId)} to ${chalk.cyan(toId)}:`)
console.log(`${chalk.cyan(fromId)} (start)`)
path.forEach((hop, index) => {
console.log(`${' '.repeat(index + 1)}${(hop.similarity * 100).toFixed(1)}%`)
console.log(`${' '.repeat(index + 1)}${hop.id} (hop ${hop.hop})`)
})
}
if (argv.output) {
await saveToFile(argv.output, path, argv.format!)
}
} catch (error) {
spinner.fail('💥 Failed to find path')
throw error
}
}
async function handleOutliersCommand(neural: any, argv: CommandArguments): Promise<void> { async function handleOutliersCommand(neural: any, argv: CommandArguments): Promise<void> {
const spinner = ora('🚨 Detecting semantic outliers...').start() const spinner = ora('🚨 Detecting semantic outliers...').start()
@ -592,32 +416,32 @@ function formatAsTable(data: any): string {
return JSON.stringify(data, null, 2) return JSON.stringify(data, null, 2)
} }
function showHelp(): void { /**
console.log('\n🧠 Neural Similarity API Commands:') * @description Run a neural CLI handler with the one-shot lifecycle every
console.log('') * Brainy command follows: init work `close()` explicit `process.exit`.
console.log(' brainy neural similar <item1> <item2> Calculate similarity') * close() releases the writer lock and indexes, but global timers
console.log(' brainy neural clusters Find semantic clusters') * (UnifiedCache bookkeeping, PathResolver stats) keep the event loop alive,
console.log(' brainy neural hierarchy <id> Show item hierarchy') * so one-shot commands must exit explicitly.
console.log(' brainy neural neighbors <id> Find semantic neighbors') * @param work - The handler body, given the initialized neural API.
console.log(' brainy neural path <from> <to> Find semantic path') */
console.log(' brainy neural outliers Detect outliers') async function runNeuralCommand(work: (neural: ReturnType<Brainy['neural']>) => Promise<void>): Promise<void> {
console.log(' brainy neural visualize Generate visualization data') try {
console.log('') const brain = getBrainy()
console.log('Options:') await brain.init()
console.log(' --threshold, -t Similarity threshold (0-1)')
console.log(' --format, -f Output format (json|table|tree|graph)') await work(brain.neural())
console.log(' --output, -o Save to file')
console.log(' --limit, -l Maximum results') await brain.close()
console.log(' --explain, -e Include explanations') process.exit(0)
console.log('') } catch (error) {
console.error(chalk.red('💥 Error:'), error instanceof Error ? error.message : error)
process.exit(1)
}
} }
// Commander-compatible wrappers // Commander-compatible wrappers
export const neuralCommands = { export const neuralCommands = {
async similar(a?: string, b?: string, options?: any) { async similar(a?: string, b?: string, options?: any) {
const brain = new Brainy()
const neural = brain.neural()
// Build argv-style object for handler // Build argv-style object for handler
const argv: CommandArguments = { const argv: CommandArguments = {
_: ['neural', 'similar', a || '', b || ''].filter(x => x), _: ['neural', 'similar', a || '', b || ''].filter(x => x),
@ -627,13 +451,10 @@ export const neuralCommands = {
...options ...options
} }
await handleSimilarCommand(neural, argv) await runNeuralCommand((neural) => handleSimilarCommand(neural, argv))
}, },
async cluster(options?: any) { async cluster(options?: any) {
const brain = new Brainy()
const neural = brain.neural()
const argv: CommandArguments = { const argv: CommandArguments = {
_: ['neural', 'cluster'], _: ['neural', 'cluster'],
algorithm: options?.algorithm || 'hierarchical', algorithm: options?.algorithm || 'hierarchical',
@ -643,26 +464,20 @@ export const neuralCommands = {
...options ...options
} }
await handleClustersCommand(neural, argv) await runNeuralCommand((neural) => handleClustersCommand(neural, argv))
}, },
async hierarchy(id?: string, options?: any) { async hierarchy(id?: string, options?: any) {
const brain = new Brainy()
const neural = brain.neural()
const argv: CommandArguments = { const argv: CommandArguments = {
_: ['neural', 'hierarchy', id || ''].filter(x => x), _: ['neural', 'hierarchy', id || ''].filter(x => x),
id, id,
...options ...options
} }
await handleHierarchyCommand(neural, argv) await runNeuralCommand((neural) => handleHierarchyCommand(neural, argv))
}, },
async related(id?: string, options?: any) { async related(id?: string, options?: any) {
const brain = new Brainy()
const neural = brain.neural()
const argv: CommandArguments = { const argv: CommandArguments = {
_: ['neural', 'related', id || ''].filter(x => x), _: ['neural', 'related', id || ''].filter(x => x),
id, id,
@ -671,13 +486,10 @@ export const neuralCommands = {
...options ...options
} }
await handleNeighborsCommand(neural, argv) await runNeuralCommand((neural) => handleNeighborsCommand(neural, argv))
}, },
async outliers(options?: any) { async outliers(options?: any) {
const brain = new Brainy()
const neural = brain.neural()
const argv: CommandArguments = { const argv: CommandArguments = {
_: ['neural', 'outliers'], _: ['neural', 'outliers'],
threshold: options?.threshold ? parseFloat(options.threshold) : 0.3, threshold: options?.threshold ? parseFloat(options.threshold) : 0.3,
@ -685,13 +497,10 @@ export const neuralCommands = {
...options ...options
} }
await handleOutliersCommand(neural, argv) await runNeuralCommand((neural) => handleOutliersCommand(neural, argv))
}, },
async visualize(options?: any) { async visualize(options?: any) {
const brain = new Brainy()
const neural = brain.neural()
const argv: CommandArguments = { const argv: CommandArguments = {
_: ['neural', 'visualize'], _: ['neural', 'visualize'],
format: options?.format || 'json', format: options?.format || 'json',
@ -701,8 +510,6 @@ export const neuralCommands = {
...options ...options
} }
await handleVisualizeCommand(neural, argv) await runNeuralCommand((neural) => handleVisualizeCommand(neural, argv))
} }
} }
export default neuralCommand

View file

@ -62,6 +62,7 @@ export const nlpCommands = {
spinner = ora('Extracting entities with neural NLP...').start() spinner = ora('Extracting entities with neural NLP...').start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
// Extract entities using Brainy's neural entity extractor // Extract entities using Brainy's neural entity extractor
const entities = await brain.extract(text) const entities = await brain.extract(text)
@ -105,6 +106,12 @@ export const nlpCommands = {
} else { } else {
formatOutput(entities, options) formatOutput(entities, options)
} }
// close() releases the writer lock and indexes, but global timers
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
// alive. CLI commands are one-shot — exit explicitly.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
if (spinner) spinner.fail('Entity extraction failed') if (spinner) spinner.fail('Entity extraction failed')
console.error(chalk.red('Extraction failed:', error.message)) console.error(chalk.red('Extraction failed:', error.message))
@ -147,6 +154,7 @@ export const nlpCommands = {
spinner = ora('Extracting concepts with neural analysis...').start() spinner = ora('Extracting concepts with neural analysis...').start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
const confidence = options.threshold ? parseFloat(options.threshold) : 0.5 const confidence = options.threshold ? parseFloat(options.threshold) : 0.5
const concepts = await brain.extractConcepts(text, { confidence }) const concepts = await brain.extractConcepts(text, { confidence })
@ -171,6 +179,10 @@ export const nlpCommands = {
} else { } else {
formatOutput(concepts, options) formatOutput(concepts, options)
} }
// One-shot command — see extract() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
if (spinner) spinner.fail('Concept extraction failed') if (spinner) spinner.fail('Concept extraction failed')
console.error(chalk.red('Extraction failed:', error.message)) console.error(chalk.red('Extraction failed:', error.message))
@ -201,6 +213,7 @@ export const nlpCommands = {
spinner = ora('Analyzing text with neural NLP...').start() spinner = ora('Analyzing text with neural NLP...').start()
const brain = getBrainy() const brain = getBrainy()
await brain.init()
// Run both entity extraction and concept extraction // Run both entity extraction and concept extraction
const [entities, concepts] = await Promise.all([ const [entities, concepts] = await Promise.all([
@ -265,6 +278,10 @@ export const nlpCommands = {
concepts concepts
}, options) }, options)
} }
// One-shot command — see extract() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
if (spinner) spinner.fail('Analysis failed') if (spinner) spinner.fail('Analysis failed')
console.error(chalk.red('Analysis failed:', error.message)) console.error(chalk.red('Analysis failed:', error.message))

File diff suppressed because it is too large Load diff

View file

@ -22,8 +22,7 @@ interface StatsOptions extends UtilityOptions {
} }
interface CleanOptions extends UtilityOptions { interface CleanOptions extends UtilityOptions {
removeOrphans?: boolean force?: boolean
rebuildIndex?: boolean
} }
interface BenchmarkOptions extends UtilityOptions { interface BenchmarkOptions extends UtilityOptions {
@ -63,6 +62,7 @@ export const utilityCommands = {
try { try {
const brain = getBrainy() const brain = getBrainy()
await brain.init()
const nounCount = await brain.getNounCount() const nounCount = await brain.getNounCount()
const verbCount = await brain.getVerbCount() const verbCount = await brain.getVerbCount()
const memUsage = process.memoryUsage() const memUsage = process.memoryUsage()
@ -77,7 +77,11 @@ export const utilityCommands = {
if (options.json) { if (options.json) {
formatOutput(stats, options) formatOutput(stats, options)
return // close() releases the writer lock and indexes, but global timers
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
// alive. CLI commands are one-shot — exit explicitly.
await brain.close()
process.exit(0)
} }
console.log(chalk.cyan('\n📊 Database Statistics\n')) console.log(chalk.cyan('\n📊 Database Statistics\n'))
@ -112,7 +116,10 @@ export const utilityCommands = {
) )
console.log(memTable.toString()) console.log(memTable.toString())
// One-shot command — see the --json branch for why the explicit exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
spinner.fail('Failed to gather statistics') spinner.fail('Failed to gather statistics')
console.error(chalk.red(error.message)) console.error(chalk.red(error.message))
@ -121,30 +128,33 @@ export const utilityCommands = {
}, },
/** /**
* Clean and optimize database * Clear the database (all entities, relationships, and indexes).
* Destructive asks for confirmation unless --force is passed.
*/ */
async clean(options: CleanOptions) { async clean(options: CleanOptions) {
const spinner = ora('Cleaning database...').start() let spinner: ReturnType<typeof ora> | null = null
try { try {
const brain = getBrainy() // Destructive operation — confirm first (skipped with --force).
if (!options.force) {
const inquirer = (await import('inquirer')).default
const { confirm } = await inquirer.prompt([{
type: 'confirm',
name: 'confirm',
message: chalk.yellow('⚠️ Permanently delete ALL data (entities, relationships, indexes)?'),
default: false
}])
// For now, only support full clear if (!confirm) {
// removeOrphans and rebuildIndex would require new Brainy APIs console.log(chalk.yellow('Clean cancelled'))
if (options.removeOrphans || options.rebuildIndex) { process.exit(0)
spinner.warn('Advanced cleanup options not yet implemented') }
console.log(chalk.yellow('\n⚠ Advanced cleanup features coming soon:'))
console.log(chalk.dim(' • --remove-orphans: Remove disconnected items'))
console.log(chalk.dim(' • --rebuild-index: Rebuild vector index'))
console.log(chalk.dim('\nUse "brainy clean" without options to clear the database'))
return
} }
// Show warning before clearing const brain = getBrainy()
console.log(chalk.yellow('\n⚠ WARNING: This will permanently delete ALL data!'))
// Clear all data (entities, relationships, and every index) // Clear all data (entities, relationships, and every index)
spinner.text = 'Clearing all data...' spinner = ora('Clearing all data...').start()
await brain.init() await brain.init()
await brain.clear() await brain.clear()
@ -163,7 +173,7 @@ export const utilityCommands = {
await brain.close() await brain.close()
process.exit(0) process.exit(0)
} catch (error: any) { } catch (error: any) {
spinner.fail('Cleanup failed') if (spinner) spinner.fail('Cleanup failed')
console.error(chalk.red(error.message)) console.error(chalk.red(error.message))
process.exit(1) process.exit(1)
} }
@ -185,7 +195,8 @@ export const utilityCommands = {
try { try {
const brain = getBrainy() const brain = getBrainy()
await brain.init()
// Benchmark different operations // Benchmark different operations
const benchmarks = [ const benchmarks = [
{ name: 'add', enabled: operations === 'all' || operations.includes('add') }, { name: 'add', enabled: operations === 'all' || operations.includes('add') },
@ -205,7 +216,8 @@ export const utilityCommands = {
switch (bench.name) { switch (bench.name) {
case 'add': case 'add':
await brain.add({ data: `Test item ${i}`, type: NounType.Thing, metadata: { benchmark: true } }) // 8.0 requires a subtype on every write by default.
await brain.add({ data: `Test item ${i}`, type: NounType.Thing, subtype: 'benchmark', metadata: { benchmark: true } })
break break
case 'search': case 'search':
await brain.find({ query: 'test', limit: 10 }) await brain.find({ query: 'test', limit: 10 })
@ -284,7 +296,10 @@ export const utilityCommands = {
} else { } else {
formatOutput(results, options) formatOutput(results, options)
} }
// One-shot command — see stats() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
console.error(chalk.red('Benchmark failed:'), error.message) console.error(chalk.red('Benchmark failed:'), error.message)
process.exit(1) process.exit(1)

View file

@ -70,6 +70,12 @@ export const vfsCommands = {
} else { } else {
formatOutput({ path, content: buffer.toString(), size: buffer.length }, options) formatOutput({ path, content: buffer.toString(), size: buffer.length }, options)
} }
// close() releases the writer lock and indexes, but global timers
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
// alive. CLI commands are one-shot — exit explicitly.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
spinner.fail('Failed to read file') spinner.fail('Failed to read file')
console.error(chalk.red(error.message)) console.error(chalk.red(error.message))
@ -109,6 +115,10 @@ export const vfsCommands = {
} else { } else {
formatOutput({ path, size: Buffer.byteLength(data) }, options) formatOutput({ path, size: Buffer.byteLength(data) }, options)
} }
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
spinner.fail('Failed to write file') spinner.fail('Failed to write file')
console.error(chalk.red(error.message)) console.error(chalk.red(error.message))
@ -132,7 +142,9 @@ export const vfsCommands = {
if (!options.json) { if (!options.json) {
if (!Array.isArray(entries) || entries.length === 0) { if (!Array.isArray(entries) || entries.length === 0) {
console.log(chalk.yellow('Directory is empty')) console.log(chalk.yellow('Directory is empty'))
return // One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} }
if (options.long) { if (options.long) {
@ -171,6 +183,10 @@ export const vfsCommands = {
} else { } else {
formatOutput(entries, options) formatOutput(entries, options)
} }
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
spinner.fail('Failed to list directory') spinner.fail('Failed to list directory')
console.error(chalk.red(error.message)) console.error(chalk.red(error.message))
@ -202,6 +218,10 @@ export const vfsCommands = {
} else { } else {
formatOutput(stats, options) formatOutput(stats, options)
} }
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
spinner.fail('Failed to get stats') spinner.fail('Failed to get stats')
console.error(chalk.red(error.message)) console.error(chalk.red(error.message))
@ -227,6 +247,10 @@ export const vfsCommands = {
} else { } else {
formatOutput({ path, created: true }, options) formatOutput({ path, created: true }, options)
} }
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
spinner.fail('Failed to create directory') spinner.fail('Failed to create directory')
console.error(chalk.red(error.message)) console.error(chalk.red(error.message))
@ -258,12 +282,16 @@ export const vfsCommands = {
} else { } else {
formatOutput({ path, removed: true }, options) formatOutput({ path, removed: true }, options)
} }
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
spinner.fail('Failed to remove') spinner.fail('Failed to remove')
console.error(chalk.red(error.message)) console.error(chalk.red(error.message))
if (!options.force) { // --force tolerates the failure but the process is still one-shot:
process.exit(1) // exit cleanly (0) instead of falling off the event loop.
} process.exit(options.force ? 0 : 1)
} }
}, },
@ -303,6 +331,10 @@ export const vfsCommands = {
} else { } else {
formatOutput(results, options) formatOutput(results, options)
} }
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
spinner.fail('Search failed') spinner.fail('Search failed')
console.error(chalk.red(error.message)) console.error(chalk.red(error.message))
@ -343,6 +375,10 @@ export const vfsCommands = {
} else { } else {
formatOutput(results, options) formatOutput(results, options)
} }
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
spinner.fail('Failed to find similar files') spinner.fail('Failed to find similar files')
console.error(chalk.red(error.message)) console.error(chalk.red(error.message))
@ -371,6 +407,10 @@ export const vfsCommands = {
} else { } else {
formatOutput(tree, options) formatOutput(tree, options)
} }
// One-shot command — see read() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) { } catch (error: any) {
spinner.fail('Failed to build tree') spinner.fail('Failed to build tree')
console.error(chalk.red(error.message)) console.error(chalk.red(error.message))

View file

@ -19,6 +19,7 @@ import { insightsCommands } from './commands/insights.js'
import { importCommands } from './commands/import.js' import { importCommands } from './commands/import.js'
import { snapshotCommands } from './commands/snapshot.js' import { snapshotCommands } from './commands/snapshot.js'
import { inspectCommands } from './commands/inspect.js' import { inspectCommands } from './commands/inspect.js'
import { types as typesCommand, validate as validateCommand } from './commands/types.js'
import { readFileSync } from 'fs' import { readFileSync } from 'fs'
import { fileURLToPath } from 'url' import { fileURLToPath } from 'url'
import { dirname, join } from 'path' import { dirname, join } from 'path'
@ -70,14 +71,9 @@ ${chalk.cyan('Examples:')}
$ brainy vfs similar /code/Button.tsx $ brainy vfs similar /code/Button.tsx
${chalk.dim('# Storage management')} ${chalk.dim('# Storage management')}
$ brainy storage status --quota $ brainy storage status
$ brainy storage lifecycle set ${chalk.dim('# Interactive mode')}
$ brainy storage cost-estimate
$ brainy storage batch-delete old-ids.txt $ brainy storage batch-delete old-ids.txt
${chalk.dim('# Interactive mode')}
$ brainy interactive
${chalk.cyan('Documentation:')} ${chalk.cyan('Documentation:')}
${chalk.dim('Full docs:')} https://github.com/soulcraftlabs/brainy ${chalk.dim('Full docs:')} https://github.com/soulcraftlabs/brainy
${chalk.dim('Report issues:')} https://github.com/soulcraftlabs/brainy/issues ${chalk.dim('Report issues:')} https://github.com/soulcraftlabs/brainy/issues
@ -109,7 +105,7 @@ program
.description('Advanced search with Triple Intelligence™ (interactive if no query)') .description('Advanced search with Triple Intelligence™ (interactive if no query)')
.option('-k, --limit <number>', 'Number of results', '10') .option('-k, --limit <number>', 'Number of results', '10')
.option('--offset <number>', 'Skip N results (pagination)') .option('--offset <number>', 'Skip N results (pagination)')
.option('-t, --threshold <number>', 'Similarity threshold (0-1)', '0.7') .option('-t, --threshold <number>', 'Minimum similarity score (0-1); with --near, the proximity threshold')
.option('--type <types>', 'Filter by type(s) - comma separated') .option('--type <types>', 'Filter by type(s) - comma separated')
.option('--where <json>', 'Metadata filters (JSON)') .option('--where <json>', 'Metadata filters (JSON)')
.option('--near <id>', 'Find items near this ID') .option('--near <id>', 'Find items near this ID')
@ -181,6 +177,23 @@ program
.option('--pretty', 'Pretty print JSON') .option('--pretty', 'Pretty print JSON')
.action(coreCommands.diagnostics) .action(coreCommands.diagnostics)
// ===== Type Commands =====
program
.command('types')
.description('List all NounType and VerbType values')
.option('--noun', 'Show noun types only')
.option('--verb', 'Show verb types only')
.option('--json', 'Output as JSON')
.action(typesCommand)
program
.command('validate [type]')
.description('Check whether a type string is a valid NounType or VerbType (interactive if no type)')
.option('--verb', 'Validate as a verb type (default: noun type)')
.option('--json', 'Output as JSON')
.action(validateCommand)
// ===== Neural Commands ===== // ===== Neural Commands =====
program program
@ -222,17 +235,6 @@ program
.option('--children-only', 'Show only child hierarchy') .option('--children-only', 'Show only child hierarchy')
.action(neuralCommands.hierarchy) .action(neuralCommands.hierarchy)
program
.command('path <from> <to>')
.description('Find semantic path between items')
.option('--steps', 'Show step-by-step path')
.option('--max-hops <number>', 'Maximum path length', '5')
.action(() => {
console.log(chalk.yellow('\n⚠ Semantic path finding coming soon'))
console.log(chalk.dim('This feature requires implementing graph traversal algorithms'))
console.log(chalk.dim('Use "brainy neighbors" and "brainy hierarchy" to explore connections'))
})
program program
.command('outliers') .command('outliers')
.alias('anomalies') .alias('anomalies')
@ -457,89 +459,26 @@ program
program program
.command('storage') .command('storage')
.description('💾 Storage management and cost optimization') .description('💾 Storage management (filesystem + memory backends)')
.addCommand( .addCommand(
new Command('status') new Command('status')
.description('Show storage status and health') .description('Show storage backend, counts, root directory, and writer lock')
.option('--detailed', 'Show detailed information') .option('--json', 'Output as JSON')
.option('--quota', 'Show quota information (OPFS)') .option('--pretty', 'Pretty-print JSON')
.action((options) => { .action((options) => {
storageCommands.status(options) storageCommands.status(options)
}) })
) )
.addCommand(
new Command('lifecycle')
.description('Lifecycle policy management')
.addCommand(
new Command('set')
.argument('[config-file]', 'Policy configuration file (JSON)')
.description('Set lifecycle policy (interactive if no file)')
.option('--validate', 'Validate before applying')
.action((configFile, options) => {
storageCommands.lifecycle.set(configFile, options)
})
)
.addCommand(
new Command('get')
.description('Get current lifecycle policy')
.option('-f, --format <type>', 'Output format (json|yaml)', 'json')
.action((options) => {
storageCommands.lifecycle.get(options)
})
)
.addCommand(
new Command('remove')
.description('Remove lifecycle policy')
.action((options) => {
storageCommands.lifecycle.remove(options)
})
)
)
.addCommand(
new Command('compression')
.description('Compression management (FileSystem)')
.addCommand(
new Command('enable')
.description('Enable gzip compression')
.action((options) => {
storageCommands.compression.enable(options)
})
)
.addCommand(
new Command('disable')
.description('Disable compression')
.action((options) => {
storageCommands.compression.disable(options)
})
)
.addCommand(
new Command('status')
.description('Show compression status')
.action((options) => {
storageCommands.compression.status(options)
})
)
)
.addCommand( .addCommand(
new Command('batch-delete') new Command('batch-delete')
.argument('<file>', 'File containing entity IDs (one per line)') .argument('<file>', 'File containing entity IDs (one per line)')
.description('Batch delete with retry logic') .description('Delete a list of entities through the public delete path (with retry)')
.option('--max-retries <n>', 'Maximum retry attempts', '3') .option('--max-retries <n>', 'Maximum retry attempts per ID', '3')
.option('--continue-on-error', 'Continue if some deletes fail') .option('--continue-on-error', 'Continue past IDs that still fail after retries')
.action((file, options) => { .action((file, options) => {
storageCommands.batchDelete(file, options) storageCommands.batchDelete(file, options)
}) })
) )
.addCommand(
new Command('cost-estimate')
.description('Estimate cloud storage costs')
.option('--provider <type>', 'Cloud provider (aws|gcs|azure|r2)')
.option('--size <gb>', 'Data size in GB')
.option('--operations <n>', 'Monthly operations')
.action((options) => {
storageCommands.costEstimate(options)
})
)
// ===== Data Management Commands ===== // ===== Data Management Commands =====
@ -742,9 +681,8 @@ program
program program
.command('clean') .command('clean')
.description('Clean and optimize database') .description('Clear the database — ALL entities, relationships, and indexes (asks for confirmation)')
.option('--remove-orphans', 'Remove orphaned items') .option('-f, --force', 'Skip confirmation prompt')
.option('--rebuild-index', 'Rebuild search index')
.action(utilityCommands.clean) .action(utilityCommands.clean)
program program
@ -785,17 +723,6 @@ program
.description('Show the store\'s current generation watermark') .description('Show the store\'s current generation watermark')
.action(snapshotCommands.generation) .action(snapshotCommands.generation)
// ===== Interactive Mode =====
program
.command('interactive')
.alias('i')
.description('Start interactive REPL mode')
.action(async () => {
const { startInteractiveMode } = await import('./interactive.js')
await startInteractiveMode()
})
// ===== Error Handling ===== // ===== Error Handling =====
program.exitOverride() program.exitOverride()

View file

@ -1,653 +0,0 @@
/**
* Professional Interactive CLI System
*
* Provides consistent, delightful interactive prompts for all commands
* with smart defaults, validation, and helpful examples
*/
import chalk from 'chalk'
import inquirer from 'inquirer'
import ora from 'ora'
import { Brainy } from '../brainy.js'
import { getBrainyVersion } from '../utils/version.js'
// Professional color scheme
export const colors = {
primary: chalk.hex('#3A5F4A'), // Teal (from logo)
success: chalk.hex('#2D4A3A'), // Deep teal
info: chalk.hex('#4A6B5A'), // Medium teal
warning: chalk.hex('#D67441'), // Orange (from logo)
error: chalk.hex('#B85C35'), // Deep orange
brain: chalk.hex('#D67441'), // Brain orange
cream: chalk.hex('#F5E6A3'), // Cream background
dim: chalk.dim,
bold: chalk.bold,
cyan: chalk.cyan,
green: chalk.green,
yellow: chalk.yellow,
red: chalk.red
}
// Icons for consistent visual language
export const icons = {
brain: '🧠',
search: '🔍',
add: '',
delete: '🗑️',
update: '🔄',
import: '📥',
export: '📤',
connect: '🔗',
question: '❓',
success: '✅',
error: '❌',
warning: '⚠️',
info: '',
sparkle: '✨',
rocket: '🚀',
thinking: '🤔',
chat: '💬'
}
// Store recent inputs for smart suggestions
const recentInputs = {
searches: [] as string[],
ids: [] as string[],
types: [] as string[],
formats: [] as string[]
}
/**
* Professional prompt wrapper with consistent styling
*/
export async function prompt(config: any): Promise<any> {
// Add consistent styling
if (config.message) {
config.message = colors.cyan(config.message)
}
// Add prefix with appropriate icon
if (!config.prefix) {
config.prefix = colors.dim(' ')
}
return inquirer.prompt([config])
}
/**
* Interactive prompt for search query with smart features
*/
export async function promptSearchQuery(previousSearches?: string[]): Promise<string> {
console.log(colors.primary(`\n${icons.search} Smart Search\n`))
console.log(colors.dim('Search your neural database with natural language'))
console.log(colors.dim('Examples: "meetings last week", "John from Google", "important documents"'))
const { query } = await prompt({
type: 'input',
name: 'query',
message: 'What would you like to search for?',
validate: (input: string) => {
if (!input.trim()) {
return 'Please enter a search query'
}
return true
},
transformer: (input: string) => {
// Show live character count
const count = input.length
if (count > 100) {
return colors.warning(input)
}
return colors.green(input)
}
})
// Store for future suggestions
if (!recentInputs.searches.includes(query)) {
recentInputs.searches.unshift(query)
recentInputs.searches = recentInputs.searches.slice(0, 10)
}
return query
}
/**
* Interactive prompt for item ID with fuzzy search
*/
export async function promptItemId(
action: string,
brain?: Brainy,
allowMultiple: boolean = false
): Promise<string | string[]> {
console.log(colors.primary(`\n${icons.thinking} Select item to ${action}\n`))
// If we have brain instance, show recent items
let choices: any[] = []
if (brain) {
try {
const recent = await brain.find({
query: '*',
limit: 10
})
choices = recent.map(item => ({
name: `${item.id} - ${(item as any).content?.substring(0, 50) || 'No content'}...`,
value: item.id,
short: item.id
}))
} catch {
// Fallback to manual input
}
}
if (choices.length > 0) {
choices.push(new inquirer.Separator())
choices.push({ name: 'Enter ID manually', value: '__manual__' })
const { selected } = await prompt({
type: allowMultiple ? 'checkbox' : 'list',
name: 'selected',
message: `Select item(s) to ${action}:`,
choices,
pageSize: 10
})
if (selected === '__manual__' || (Array.isArray(selected) && selected.includes('__manual__'))) {
return promptManualId(action, allowMultiple)
}
return selected
} else {
return promptManualId(action, allowMultiple)
}
}
/**
* Manual ID input with validation
*/
async function promptManualId(action: string, allowMultiple: boolean): Promise<string | string[]> {
const { id } = await prompt({
type: 'input',
name: 'id',
message: allowMultiple
? `Enter ID(s) to ${action} (comma-separated):`
: `Enter ID to ${action}:`,
validate: (input: string) => {
if (!input.trim()) {
return `Please enter at least one ID`
}
return true
}
})
if (allowMultiple) {
return id.split(',').map((i: string) => i.trim()).filter(Boolean)
}
return id.trim()
}
/**
* Confirm destructive action with preview
*/
export async function confirmDestructiveAction(
action: string,
items: any[],
showPreview: boolean = true
): Promise<boolean> {
console.log(colors.warning(`\n${icons.warning} Confirmation Required\n`))
if (showPreview && items.length > 0) {
console.log(colors.dim(`You are about to ${action}:`))
items.slice(0, 5).forEach(item => {
console.log(colors.dim(`${item.id || item}`))
})
if (items.length > 5) {
console.log(colors.dim(` ... and ${items.length - 5} more`))
}
console.log()
}
const { confirm } = await prompt({
type: 'confirm',
name: 'confirm',
message: colors.warning(`Are you sure you want to ${action}?`),
default: false
})
return confirm
}
/**
* Interactive data input with multiline support
*/
export async function promptDataInput(
action: string = 'add',
currentValue?: string
): Promise<string> {
console.log(colors.primary(`\n${icons.add} ${action === 'add' ? 'Add Data' : 'Update Data'}\n`))
if (currentValue) {
console.log(colors.dim('Current value:'))
console.log(colors.info(` ${currentValue.substring(0, 100)}${currentValue.length > 100 ? '...' : ''}`))
console.log()
}
const { data } = await prompt({
type: 'editor',
name: 'data',
message: 'Enter your data:',
default: currentValue || '',
postfix: '.md',
validate: (input: string) => {
if (!input.trim() && action === 'add') {
return 'Please enter some data'
}
return true
}
})
return data
}
/**
* Interactive metadata input with JSON validation
*/
export async function promptMetadata(
currentMetadata?: any,
suggestions?: string[]
): Promise<any> {
console.log(colors.dim('\nOptional: Add metadata (JSON format)'))
const { addMetadata } = await prompt({
type: 'confirm',
name: 'addMetadata',
message: 'Would you like to add metadata?',
default: false
})
if (!addMetadata) {
return {}
}
// Show field suggestions if available
if (suggestions && suggestions.length > 0) {
console.log(colors.dim('\nAvailable fields:'))
suggestions.forEach(field => {
console.log(colors.dim(`${field}`))
})
}
const { metadata } = await prompt({
type: 'editor',
name: 'metadata',
message: 'Enter metadata (JSON):',
default: currentMetadata ? JSON.stringify(currentMetadata, null, 2) : '{\n \n}',
postfix: '.json',
validate: (input: string) => {
try {
JSON.parse(input)
return true
} catch (e) {
return `Invalid JSON: ${e.message}`
}
}
})
return JSON.parse(metadata)
}
/**
* Interactive format selector
*/
export async function promptFormat(
availableFormats: string[],
defaultFormat: string
): Promise<string> {
console.log(colors.primary(`\n${icons.export} Select Format\n`))
const { format } = await prompt({
type: 'list',
name: 'format',
message: 'Choose export format:',
choices: availableFormats.map(f => ({
name: getFormatDescription(f),
value: f,
short: f
})),
default: defaultFormat
})
return format
}
/**
* Get friendly format descriptions
*/
function getFormatDescription(format: string): string {
const descriptions: Record<string, string> = {
json: 'JSON - Universal data interchange',
jsonl: 'JSON Lines - Streaming format',
csv: 'CSV - Spreadsheet compatible',
graphml: 'GraphML - Graph visualization',
dot: 'DOT - Graphviz format',
d3: 'D3.js - Web visualization',
markdown: 'Markdown - Human readable',
yaml: 'YAML - Configuration format'
}
return `${format.toUpperCase()} - ${descriptions[format] || 'Custom format'}`
}
/**
* Interactive file/URL input with validation
*/
export async function promptFileOrUrl(
action: string = 'import'
): Promise<string> {
console.log(colors.primary(`\n${icons.import} ${action === 'import' ? 'Import Source' : 'Export Destination'}\n`))
const { sourceType } = await prompt({
type: 'list',
name: 'sourceType',
message: 'What type of source?',
choices: [
{ name: 'Local file', value: 'file' },
{ name: 'URL', value: 'url' },
{ name: 'Clipboard', value: 'clipboard' },
{ name: 'Direct input', value: 'input' }
]
})
switch (sourceType) {
case 'file':
return promptFilePath(action)
case 'url':
return promptUrl()
case 'clipboard':
// Would need clipboard integration
console.log(colors.warning('Clipboard support coming soon!'))
return promptFilePath(action)
case 'input':
const data = await promptDataInput('import')
// Save to temp file and return path
const tmpFile = `/tmp/brainy-import-${Date.now()}.json`
const { writeFileSync } = await import('node:fs')
writeFileSync(tmpFile, data)
return tmpFile
default:
return ''
}
}
/**
* File path input with autocomplete
*/
async function promptFilePath(action: string): Promise<string> {
const { path } = await prompt({
type: 'input',
name: 'path',
message: `Enter file path to ${action}:`,
validate: async (input: string) => {
if (!input.trim()) {
return 'Please enter a file path'
}
const { existsSync } = await import('node:fs')
if (action === 'import' && !existsSync(input)) {
return `File not found: ${input}`
}
return true
},
// Add file path autocomplete
transformer: (input: string) => {
if (input.startsWith('~/')) {
const home = process.env.HOME || '~'
return colors.green(input.replace('~', home))
}
return colors.green(input)
}
})
return path
}
/**
* URL input with validation
*/
async function promptUrl(): Promise<string> {
const { url } = await prompt({
type: 'input',
name: 'url',
message: 'Enter URL:',
validate: (input: string) => {
try {
new URL(input)
return true
} catch {
return 'Please enter a valid URL'
}
}
})
return url
}
/**
* Interactive relationship builder
*/
export async function promptRelationship(brain?: Brainy): Promise<{
source: string
verb: string
target: string
metadata?: any
}> {
console.log(colors.primary(`\n${icons.connect} Create Relationship\n`))
console.log(colors.dim('Connect two items with a semantic relationship'))
// Get source
const source = await promptItemId('connect from', brain, false) as string
// Get verb/relationship type
const { verb } = await prompt({
type: 'list',
name: 'verb',
message: 'Relationship type:',
choices: [
{ name: 'Works For', value: 'WorksFor' },
{ name: 'Knows', value: 'Knows' },
{ name: 'Created By', value: 'CreatedBy' },
{ name: 'Belongs To', value: 'BelongsTo' },
{ name: 'Uses', value: 'Uses' },
{ name: 'Manages', value: 'Manages' },
{ name: 'Located In', value: 'LocatedIn' },
{ name: 'Related To', value: 'RelatedTo' },
new inquirer.Separator(),
{ name: 'Custom relationship...', value: '__custom__' }
]
})
let finalVerb = verb
if (verb === '__custom__') {
const { customVerb } = await prompt({
type: 'input',
name: 'customVerb',
message: 'Enter custom relationship:',
validate: (input: string) => input.trim() ? true : 'Please enter a relationship'
})
finalVerb = customVerb
}
// Get target
const target = await promptItemId('connect to', brain, false) as string
// Optional metadata
const metadata = await promptMetadata()
return {
source,
verb: finalVerb,
target,
metadata: Object.keys(metadata).length > 0 ? metadata : undefined
}
}
/**
* Smart command suggestions when user types wrong command
*/
export function suggestCommand(input: string, availableCommands: string[]): string[] {
// Simple fuzzy matching without external dependency
// Filter commands that start with or contain the input
const matches = availableCommands
.filter(cmd => cmd.toLowerCase().includes(input.toLowerCase()))
.sort((a, b) => {
// Prefer commands that start with the input
const aStarts = a.toLowerCase().startsWith(input.toLowerCase())
const bStarts = b.toLowerCase().startsWith(input.toLowerCase())
if (aStarts && !bStarts) return -1
if (!aStarts && bStarts) return 1
return 0
})
return matches.slice(0, 3)
}
/**
* Beautiful error display with helpful context
*/
export function showError(error: Error, context?: string): void {
console.log()
console.log(colors.error(`${icons.error} Error`))
if (context) {
console.log(colors.dim(context))
}
console.log(colors.red(error.message))
// Provide helpful suggestions based on error
if (error.message.includes('not found')) {
console.log(colors.dim('\nTip: Use "brainy search" to find items'))
} else if (error.message.includes('network') || error.message.includes('fetch')) {
console.log(colors.dim('\nTip: Check your internet connection'))
} else if (error.message.includes('permission')) {
console.log(colors.dim('\nTip: Check file permissions or run with appropriate access'))
}
}
/**
* Progress indicator for long operations
*/
export class ProgressTracker {
private spinner: any
private startTime: number
constructor(message: string) {
this.spinner = ora({
text: message,
color: 'cyan',
spinner: 'dots'
}).start()
this.startTime = Date.now()
}
update(message: string, count?: number, total?: number): void {
if (count && total) {
const percent = Math.round((count / total) * 100)
const elapsed = ((Date.now() - this.startTime) / 1000).toFixed(1)
this.spinner.text = `${message} (${percent}% - ${elapsed}s)`
} else {
this.spinner.text = message
}
}
succeed(message?: string): void {
const elapsed = ((Date.now() - this.startTime) / 1000).toFixed(1)
this.spinner.succeed(message ? `${message} (${elapsed}s)` : `Done (${elapsed}s)`)
}
fail(message?: string): void {
this.spinner.fail(message || 'Failed')
}
stop(): void {
this.spinner.stop()
}
}
/**
* Welcome message for interactive mode
*/
export function showWelcome(): void {
console.clear()
console.log(colors.primary(`
${icons.brain} BRAINY - Neural Intelligence
Your AI-Powered Second Brain
`))
console.log(colors.dim(`Version ${getBrainyVersion()} • Type "help" for commands`))
console.log()
}
/**
* Interactive command selector for beginners
*/
export async function promptCommand(): Promise<string> {
const { command } = await prompt({
type: 'list',
name: 'command',
message: 'What would you like to do?',
choices: [
{ name: `${icons.add} Add data to your brain`, value: 'add' },
{ name: `${icons.search} Search your knowledge`, value: 'search' },
{ name: `${icons.chat} Chat with your data`, value: 'chat' },
{ name: `${icons.update} Update existing data`, value: 'update' },
{ name: `${icons.delete} Delete data`, value: 'delete' },
{ name: `${icons.connect} Create relationships`, value: 'relate' },
{ name: `${icons.import} Import from file`, value: 'import' },
{ name: `${icons.export} Export your brain`, value: 'export' },
new inquirer.Separator(),
{ name: `${icons.brain} Neural operations`, value: 'neural' },
{ name: `${icons.info} View statistics`, value: 'status' },
{ name: 'Exit', value: 'exit' }
],
pageSize: 15
})
return command
}
/**
* Start interactive REPL mode
*/
export async function startInteractiveMode() {
console.log(chalk.cyan('\n🧠 Brainy Interactive Mode\n'))
console.log(chalk.yellow('Interactive REPL mode coming soon\n'))
console.log(chalk.dim('Use specific commands for now: brainy add, brainy search, etc.'))
process.exit(0)
}
/**
* Export all interactive components
*/
export default {
colors,
icons,
prompt,
promptSearchQuery,
promptItemId,
confirmDestructiveAction,
promptDataInput,
promptMetadata,
promptFormat,
promptFileOrUrl,
promptRelationship,
suggestCommand,
showError,
ProgressTracker,
showWelcome,
promptCommand,
startInteractiveMode
}

View file

@ -2,7 +2,7 @@
* @module config/storageAutoConfig * @module config/storageAutoConfig
* @description Storage configuration auto-detection for Brainy 8.0. * @description Storage configuration auto-detection for Brainy 8.0.
* *
* Brainy 8.0 ships **two storage adapters** per `BR-BRAINY-80-STORAGE-SIMPLIFY`: * Brainy 8.0 ships **two storage adapters**:
* - `FILESYSTEM` persistent on-disk storage (Node, Bun, Deno). * - `FILESYSTEM` persistent on-disk storage (Node, Bun, Deno).
* - `MEMORY` in-memory, ephemeral. * - `MEMORY` in-memory, ephemeral.
* *

View file

@ -386,7 +386,7 @@ export interface HNSWVerbWithMetadata {
targetId: string targetId: string
// SUBTYPE — optional per-product sub-classification within a VerbType (e.g. a // SUBTYPE — optional per-product sub-classification within a VerbType (e.g. a
// `Manages` relationship might have subtype 'direct' vs 'dotted-line'; a `RelatedTo` // `ReportsTo` relationship might have subtype 'direct' vs 'dotted-line'; a `RelatedTo`
// edge might carry 'spouse' / 'sibling' / 'colleague'). Flat string (no hierarchy) — // edge might carry 'spouse' / 'sibling' / 'colleague'). Flat string (no hierarchy) —
// consumers decide the vocabulary. Indexed and rolled up into per-VerbType statistics // consumers decide the vocabulary. Indexed and rolled up into per-VerbType statistics
// so it's queryable (`getRelations({ verb, subtype })`) and aggregable // so it's queryable (`getRelations({ verb, subtype })`) and aggregable

View file

@ -766,8 +766,8 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
// Note: LSM-trees will be recreated from storage via their own initialization // Note: LSM-trees will be recreated from storage via their own initialization
// Verb data will be loaded on-demand via UnifiedCache // Verb data will be loaded on-demand via UnifiedCache
// Brainy 8.0: storage is always local (filesystem or memory) per // Brainy 8.0: storage is always local (filesystem or memory — the
// BR-BRAINY-80-STORAGE-SIMPLIFY. Load all verbs at once. // cloud adapters were removed). Load all verbs at once.
const storageType = this.storage?.constructor.name || '' const storageType = this.storage?.constructor.name || ''
let totalVerbs = 0 let totalVerbs = 0

View file

@ -1261,8 +1261,8 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
} }
// Step 4: Load all HNSW nodes at once. Brainy 8.0 ships filesystem + // Step 4: Load all HNSW nodes at once. Brainy 8.0 ships filesystem +
// memory storage only (per BR-BRAINY-80-STORAGE-SIMPLIFY); the // memory storage only; the cloud-pagination rebuild path was deleted
// cloud-pagination rebuild path was deleted alongside the cloud adapters. // alongside the cloud adapters.
const storageType = this.storage?.constructor.name || '' const storageType = this.storage?.constructor.name || ''
let loadedCount = 0 let loadedCount = 0
let totalCount: number | undefined = undefined let totalCount: number | undefined = undefined

View file

@ -28,6 +28,9 @@ export type {
RelateParams, RelateParams,
FindParams, FindParams,
SubtypeRegistry, SubtypeRegistry,
FillSubtypeRule,
FillSubtypeRules,
FillSubtypesResult,
AggregateDefinition, AggregateDefinition,
AggregateMetricDef, AggregateMetricDef,
AggregateSource, AggregateSource,

View file

@ -8,9 +8,8 @@
* 2. O(1) metadata hints (column names, file structure) * 2. O(1) metadata hints (column names, file structure)
* 3. Format-specific intelligence (Excel, CSV, PDF, YAML, DOCX) * 3. Format-specific intelligence (Excel, CSV, PDF, YAML, DOCX)
* *
* This is the WORKSHOP BUG FIX - finds explicit relationships via exact matching * Finds explicit relationships via exact matching added after a consumer
* * report of extraction missing explicitly-named relationships.
* PRODUCTION-READY: No TODOs, no mocks, real implementation
*/ */
import type { Brainy } from '../../brainy.js' import type { Brainy } from '../../brainy.js'

View file

@ -1528,10 +1528,16 @@ export abstract class BaseStorage extends BaseStorageAdapter {
const { limit, offset = 0, filter } = options const { limit, offset = 0, filter } = options
const collectedNouns: HNSWNounWithMetadata[] = [] const collectedNouns: HNSWNounWithMetadata[] = []
// Collect ONE item past the requested window so `hasMore` is decidable:
// stopping exactly at offset+limit cannot distinguish "page full, nothing
// after it" from "page full, more behind it" — which made hasMore
// permanently false and silently truncated every multi-page walk
// (audit/migrateField/fillSubtypes, graph verb-id recovery, count rebuilds).
const targetCount = offset + limit const targetCount = offset + limit
const peekCount = targetCount + 1
// Iterate by shards (0x00-0xFF) instead of types // Iterate by shards (0x00-0xFF) instead of types
for (let shard = 0; shard < 256 && collectedNouns.length < targetCount; shard++) { for (let shard = 0; shard < 256 && collectedNouns.length < peekCount; shard++) {
const shardHex = shard.toString(16).padStart(2, '0') const shardHex = shard.toString(16).padStart(2, '0')
const shardDir = `entities/nouns/${shardHex}` const shardDir = `entities/nouns/${shardHex}`
@ -1539,7 +1545,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
const nounFiles = await this.listCanonicalObjects(shardDir) const nounFiles = await this.listCanonicalObjects(shardDir)
for (const nounPath of nounFiles) { for (const nounPath of nounFiles) {
if (collectedNouns.length >= targetCount) break if (collectedNouns.length >= peekCount) break
if (!nounPath.includes('/vectors.json')) continue if (!nounPath.includes('/vectors.json')) continue
try { try {
@ -1597,7 +1603,8 @@ export abstract class BaseStorage extends BaseStorageAdapter {
} }
} }
// Apply pagination // Apply pagination. The peeked extra item (if any) is dropped by the slice;
// its existence is exactly what makes hasMore true.
const paginatedNouns = collectedNouns.slice(offset, offset + limit) const paginatedNouns = collectedNouns.slice(offset, offset + limit)
const hasMore = collectedNouns.length > targetCount const hasMore = collectedNouns.length > targetCount
@ -1647,7 +1654,11 @@ export abstract class BaseStorage extends BaseStorageAdapter {
const { limit, offset = 0, filter } = options // cursor intentionally not extracted (not yet implemented) const { limit, offset = 0, filter } = options // cursor intentionally not extracted (not yet implemented)
const collectedVerbs: HNSWVerbWithMetadata[] = [] const collectedVerbs: HNSWVerbWithMetadata[] = []
const targetCount = offset + limit // Early termination target // Same peek-one-past-the-window strategy as getNounsWithPagination — see
// the comment there. Without the extra item, hasMore is undecidable and
// was permanently false (silent truncation of every multi-page walk).
const targetCount = offset + limit // Requested window end
const peekCount = targetCount + 1 // Early termination target (window + 1 peek)
// Prepare filter sets for efficient lookup // Prepare filter sets for efficient lookup
const filterVerbTypes = filter?.verbType const filterVerbTypes = filter?.verbType
@ -1668,7 +1679,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
: null : null
// Iterate by shards (0x00-0xFF) instead of types - single pass! // Iterate by shards (0x00-0xFF) instead of types - single pass!
for (let shard = 0; shard < 256 && collectedVerbs.length < targetCount; shard++) { for (let shard = 0; shard < 256 && collectedVerbs.length < peekCount; shard++) {
const shardHex = shard.toString(16).padStart(2, '0') const shardHex = shard.toString(16).padStart(2, '0')
const shardDir = `entities/verbs/${shardHex}` const shardDir = `entities/verbs/${shardHex}`
@ -1676,7 +1687,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
const verbFiles = await this.listCanonicalObjects(shardDir) const verbFiles = await this.listCanonicalObjects(shardDir)
for (const verbPath of verbFiles) { for (const verbPath of verbFiles) {
if (collectedVerbs.length >= targetCount) break if (collectedVerbs.length >= peekCount) break
if (!verbPath.includes('/vectors.json')) continue if (!verbPath.includes('/vectors.json')) continue
try { try {
@ -1737,9 +1748,11 @@ export abstract class BaseStorage extends BaseStorageAdapter {
} }
} }
// Apply pagination (Efficient slicing after early termination) // Apply pagination. The peeked extra item (if any) is dropped by the slice;
// its existence is exactly what makes hasMore true. `>` (not `>=`) keeps the
// exact-boundary case (total == offset+limit) from looping forever.
const paginatedVerbs = collectedVerbs.slice(offset, offset + limit) const paginatedVerbs = collectedVerbs.slice(offset, offset + limit)
const hasMore = collectedVerbs.length > targetCount // Fixed >= to > (was causing infinite loop) const hasMore = collectedVerbs.length > targetCount
return { return {
items: paginatedVerbs, items: paginatedVerbs,
@ -3235,8 +3248,8 @@ export abstract class BaseStorage extends BaseStorageAdapter {
* Before this override the Uint32Array counters were only persisted * Before this override the Uint32Array counters were only persisted
* on a heuristic schedule inside `saveNoun_internal` (first-of-type or * on a heuristic schedule inside `saveNoun_internal` (first-of-type or
* every-100th), which left readers seeing stale counts after a clean * every-100th), which left readers seeing stale counts after a clean
* writer flush the same failure mode that BR-FIND-WHERE-ZERO surfaced * writer flush the same silent-stale failure mode that once produced
* via `brain.stats()`. * zero counts from `brain.stats()`.
*/ */
public async flushCounts(): Promise<void> { public async flushCounts(): Promise<void> {
await super.flushCounts() await super.flushCounts()

View file

@ -8,7 +8,7 @@
* workloads. * workloads.
* *
* Cloud-storage adapters (GCS, S3, R2, Azure) and the browser-only OPFS * Cloud-storage adapters (GCS, S3, R2, Azure) and the browser-only OPFS
* adapter were removed in 8.0 per `BR-BRAINY-80-STORAGE-SIMPLIFY`. The path * adapter were removed in 8.0. The path
* forward for cloud backup is operator tooling: persist locally with * forward for cloud backup is operator tooling: persist locally with
* `db.persist()`, sync the resulting on-disk artefact with `gsutil` / * `db.persist()`, sync the resulting on-disk artefact with `gsutil` /
* `aws s3 cp` / `rclone` / `azcopy`. This is the standard pattern every * `aws s3 cp` / `rclone` / `azcopy`. This is the standard pattern every

View file

@ -76,7 +76,7 @@ export interface Relation<T = any> {
/** Relationship type classification (VerbType enum) */ /** Relationship type classification (VerbType enum) */
type: VerbType type: VerbType
/** /**
* Per-product sub-classification within the VerbType (e.g. a `Manages` * Per-product sub-classification within the VerbType (e.g. a `ReportsTo`
* relationship might have `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` * relationship might have `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo`
* edge might carry `'spouse'` / `'sibling'` / `'colleague'`). Flat string, no * edge might carry `'spouse'` / `'sibling'` / `'colleague'`). Flat string, no
* hierarchy. Top-level standard field indexed on the fast path and rolled into * hierarchy. Top-level standard field indexed on the fast path and rolled into
@ -208,6 +208,68 @@ export interface SubtypeRegistry {
// Intentionally empty. Consumers extend via declaration merging. // Intentionally empty. Consumers extend via declaration merging.
} }
/**
* A single back-fill rule for `brain.fillSubtypes()`.
*
* - A **literal string** assigns that subtype to every matching entry that
* lacks one (`'general'` a blanket default).
* - A **function** receives the full entry and returns the subtype to assign,
* or `undefined` to leave the entry untouched (it is counted as `skipped`
* so a later run with a stricter rule can pick it up). Functions can derive
* the subtype from existing fields, e.g. `(e) => e.metadata?.kind`.
*
* @typeParam E - The entry shape the rule sees: `Entity<T>` for NounType keys,
* `Relation<T>` for VerbType keys.
*/
export type FillSubtypeRule<E> = string | ((entry: E) => string | undefined)
/**
* Rule map for `brain.fillSubtypes()` the 8.0 subtype migration helper.
*
* Keys are `NounType` values (entity rules) and/or `VerbType` values
* (relationship rules); the two vocabularies don't overlap, so a single map
* covers both sides. Each value is a {@link FillSubtypeRule}: a literal
* subtype string or a function deriving one from the entry.
*
* @example
* ```ts
* await brain.fillSubtypes({
* [NounType.Person]: (e) => e.metadata?.kind ?? 'unspecified',
* [NounType.Thing]: 'general', // literal default
* [VerbType.RelatedTo]: 'unspecified' // relationship rule
* })
* ```
*/
export type FillSubtypeRules<T = any> = {
[K in NounType]?: FillSubtypeRule<Entity<T>>
} & {
[K in VerbType]?: FillSubtypeRule<Relation<T>>
}
/**
* Summary returned by `brain.fillSubtypes()`.
*
* After a run, `skipped` is exactly the remaining migration debt re-running
* `brain.audit()` reports the same entries. Entries that already carry a
* subtype count toward `scanned` only (they are not debt, so they are neither
* `filled` nor `skipped`).
*/
export interface FillSubtypesResult {
/** Total entries examined (entities + relationships). */
scanned: number
/** Entries that received a subtype during this run. */
filled: number
/**
* Entries still missing a subtype after the run either their type has no
* rule in the map, or their rule function returned `undefined`/empty.
*/
skipped: number
/** Per-entry write failures (`fillSubtypes` continues past individual errors). */
errors: Array<{ id: string; error: string }>
/** Fill counts grouped by NounType/VerbType key (only types with fills appear). */
byType: Record<string, number>
}
export interface AddParams<T = any> { export interface AddParams<T = any> {
/** Content to embed and store. Strings are auto-embedded; objects are JSON-stringified for embedding. */ /** Content to embed and store. Strings are auto-embedded; objects are JSON-stringified for embedding. */
data: any | Vector data: any | Vector
@ -282,7 +344,7 @@ export interface RelateParams<T = any> {
/** Relationship type classification (required) */ /** Relationship type classification (required) */
type: VerbType type: VerbType
/** /**
* Per-product sub-classification within the VerbType (e.g. a `Manages` * Per-product sub-classification within the VerbType (e.g. a `ReportsTo`
* relationship might have `subtype: 'direct'` or `'dotted-line'`; a `RelatedTo` * relationship might have `subtype: 'direct'` or `'dotted-line'`; a `RelatedTo`
* edge might carry `'spouse'` or `'colleague'`). Flat string, no hierarchy. * edge might carry `'spouse'` or `'colleague'`). Flat string, no hierarchy.
* Indexed and rolled up into per-VerbType statistics for fast filtering * Indexed and rolled up into per-VerbType statistics for fast filtering
@ -1239,16 +1301,21 @@ export interface BrainyConfig {
/** /**
* Brain-wide subtype enforcement mode. * Brain-wide subtype enforcement mode.
* *
* Opt-in in 7.30.0 (default: `false`); becomes the default in 8.0.0. * **Default-on since 8.0.0** (`undefined` resolves to `true`; in 7.30.x this
* was an opt-in flag defaulting to `false`).
* *
* - `false` / `undefined` (default): no brain-wide check. Per-type rules * - `true` / `undefined` (default): every `add()` / `addMany()` / `update()` /
* registered via `brain.requireSubtype(type, options)` still apply. * `relate()` / `relateMany()` / `updateRelation()` rejects writes where the
* - `true`: every `add()` / `addMany()` / `update()` / `relate()` / * entity's NounType (or relationship's VerbType) has no non-empty `subtype`
* `relateMany()` / `updateRelation()` rejects writes where the entity's * value. Brainy's own VFS infrastructure writes (`metadata.isVFSEntity` /
* NounType (or relationship's VerbType) has no non-empty `subtype` value. * `metadata.isVFS`) bypass the check.
* - `{ except: [NounType.Thing, NounType.Custom] }`: same as `true`, but * - `{ except: [NounType.Thing, NounType.Custom] }`: same as `true`, but the
* the listed types are allowed through without a subtype. Use for genuine * listed types are allowed through without a subtype. Use for genuine
* catch-all types where no subtype makes sense. * catch-all types where no subtype makes sense.
* - `false`: disable the brain-wide check entirely. Last-resort escape hatch
* for opening pre-8.0 data run `brain.audit()` to find the gaps, back-fill
* with `brain.fillSubtypes(rules)`, then remove the opt-out so the default
* enforcement protects new writes.
* *
* Per-type registrations always compose with the brain-wide flag a type * Per-type registrations always compose with the brain-wide flag a type
* registered with `requireSubtype(type, { required: true })` is always * registered with `requireSubtype(type, { required: true })` is always

View file

@ -748,9 +748,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
* *
* If neither the column store nor a sparse index covers the field, the * If neither the column store nor a sparse index covers the field, the
* function throws `BrainyError(FIELD_NOT_INDEXED)`. Returning `[]` for a * function throws `BrainyError(FIELD_NOT_INDEXED)`. Returning `[]` for a
* genuinely unindexed field was the bug class `BR-FIND-WHERE-ZERO` * genuinely unindexed field was a long-standing silent-empty bug class
* tracked a silent empty result indistinguishable from "the data * an empty result indistinguishable from "the data really isn't there."
* really isn't there."
*/ */
private async getIdsFromChunks(field: string, value: any): Promise<string[]> { private async getIdsFromChunks(field: string, value: any): Promise<string[]> {
// Load sparse index via UnifiedCache (lazy loading) // Load sparse index via UnifiedCache (lazy loading)
@ -1200,7 +1199,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// - Chunked Sparse Index: ~50 values per chunk, lazy-loaded // - Chunked Sparse Index: ~50 values per chunk, lazy-loaded
// - UnifiedCache LRU: Only hot chunks in memory // - UnifiedCache LRU: Only hot chunks in memory
// //
// Future: Bloom filter hybrid for unlimited words (see .strategy/BILLION-SCALE-PLAN.md) // A Bloom-filter hybrid could lift the per-entity word cap entirely if
// full-document indexing at billion-entity scale ever becomes a need.
const textContent = this.extractTextContent(data) const textContent = this.extractTextContent(data)
if (textContent) { if (textContent) {
const MAX_WORDS_PER_ENTITY = 5000 // Handles articles/chapters, memory-safe at scale const MAX_WORDS_PER_ENTITY = 5000 // Handles articles/chapters, memory-safe at scale
@ -1600,8 +1600,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
* Throws `BrainyError(FIELD_NOT_INDEXED)` if the field has no entries in * Throws `BrainyError(FIELD_NOT_INDEXED)` if the field has no entries in
* either store. Callers in find()-evaluation catch this and translate to * either store. Callers in find()-evaluation catch this and translate to
* an empty result with a logged warning. The throw aligns the production * an empty result with a logged warning. The throw aligns the production
* `find()` path with the `brain.explain()` diagnostic, so the silent- * `find()` path with the `brain.explain()` diagnostic, so a silently
* empty bug class (BR-FIND-WHERE-ZERO) is no longer possible. * empty result for an unindexed field is no longer possible.
*/ */
async getIds(field: string, value: any): Promise<string[]> { async getIds(field: string, value: any): Promise<string[]> {
// Track exact query for field statistics // Track exact query for field statistics
@ -2866,8 +2866,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
* *
* Prior implementation read from `this.fieldIndexes` + lazy-loaded sparse * Prior implementation read from `this.fieldIndexes` + lazy-loaded sparse
* indices, which silently returned `0` entries for any workspace written * indices, which silently returned `0` entries for any workspace written
* after sparse-index writes were deleted in commit `11be039`. That defect * after sparse-index writes were deleted in commit `11be039`. That
* is what `BR-FIND-WHERE-ZERO` tracked. * silent-zero defect is why this reads the column store first.
*/ */
async getStats(): Promise<MetadataIndexStats> { async getStats(): Promise<MetadataIndexStats> {
const entityCount = this.idMapper.size const entityCount = this.idMapper.size

View file

@ -124,7 +124,8 @@ const getContainerMemoryLimit = (): number | null => {
* 1.5 KB + standard fields + metadata). On a 900 MB free-memory box this * 1.5 KB + standard fields + metadata). On a 900 MB free-memory box this
* capped `limit` at 9_000, breaking common safety-cap call patterns like * capped `limit` at 9_000, breaking common safety-cap call patterns like
* `find({ type, where, limit: 10_000 })` that typically return 10-500 * `find({ type, where, limit: 10_000 })` that typically return 10-500
* entities. Surfaced as `BR-MAXLIMIT-9000` in PLATFORM-HANDOFF.md. * entities. Reported by a production consumer whose 10K safety-cap
* queries started failing after the cap landed.
* - 7.30.2+: `25` (assumes 25 KB per result). Generous over typical (7-10 KB), * - 7.30.2+: `25` (assumes 25 KB per result). Generous over typical (7-10 KB),
* comfortably under the worst case (~20 KB with large metadata blobs). * comfortably under the worst case (~20 KB with large metadata blobs).
* Same 900 MB box now gives ~36_000 typical 10_000 limits pass silently, * Same 900 MB box now gives ~36_000 typical 10_000 limits pass silently,

View file

@ -226,7 +226,7 @@ describe('Brainy strict-mode self-test (7.30.1)', () => {
const report = await brain.audit() const report = await brain.audit()
expect(report.total).toBeGreaterThanOrEqual(1) expect(report.total).toBeGreaterThanOrEqual(1)
expect(report.entitiesWithoutSubtype['person']).toBe(1) expect(report.entitiesWithoutSubtype['person']).toBe(1)
expect(report.recommendation).toMatch(/Migrate via/) expect(report.recommendation).toMatch(/fillSubtypes/)
}) })
it('excludes VFS entities by default (they bypass enforcement anyway)', async () => { it('excludes VFS entities by default (they bypass enforcement anyway)', async () => {

View file

@ -0,0 +1,338 @@
/**
* @module tests/unit/brainy/fill-subtypes
* @description Unit tests for `brain.fillSubtypes(rules)` the 8.0 subtype
* migration helper. Proves the full contract: literal and function rules,
* entity + relationship fills through one rule map, never overwriting an
* existing subtype, function rules declining entries (counted as skipped),
* VFS-marker exclusion, the `{ scanned, filled, skipped, errors, byType }`
* report, `_rev` bumping through the real update path, idempotent re-runs,
* per-entry error collection, and fail-fast rule-map validation.
*/
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
import { Brainy } from '../../../src/brainy'
import { NounType, VerbType } from '../../../src/types/graphTypes'
import { createTestConfig } from '../../helpers/test-factory'
describe('Brainy.fillSubtypes()', () => {
let brain: Brainy
beforeEach(async () => {
// requireSubtype: false so the tests can create the pre-8.0 migration
// debt (entities/relationships without subtype) that fillSubtypes exists
// to clear.
brain = new Brainy(createTestConfig())
await brain.init()
})
afterEach(async () => {
await brain.close()
})
describe('literal rules', () => {
it('fills missing subtypes with a literal default', async () => {
const a = await brain.add({ type: NounType.Person, data: 'Alice' })
const b = await brain.add({ type: NounType.Person, data: 'Bob' })
const report = await brain.fillSubtypes({
[NounType.Person]: 'unspecified'
})
expect(report.filled).toBe(2)
expect(report.byType).toEqual({ person: 2 })
expect((await brain.get(a))!.subtype).toBe('unspecified')
expect((await brain.get(b))!.subtype).toBe('unspecified')
})
it('never overwrites an existing subtype', async () => {
const employee = await brain.add({
type: NounType.Person,
subtype: 'employee',
data: 'Has a subtype already'
})
const blank = await brain.add({ type: NounType.Person, data: 'No subtype' })
const report = await brain.fillSubtypes({
[NounType.Person]: 'unspecified'
})
expect(report.filled).toBe(1)
expect((await brain.get(employee))!.subtype).toBe('employee') // untouched
expect((await brain.get(blank))!.subtype).toBe('unspecified')
})
it('leaves types without a rule untouched and counts them as skipped', async () => {
await brain.add({ type: NounType.Person, data: 'Person without rule coverage' })
const doc = await brain.add({ type: NounType.Document, data: 'Doc gets filled' })
const report = await brain.fillSubtypes({
[NounType.Document]: 'general'
})
expect(report.filled).toBe(1)
expect(report.skipped).toBe(1) // the Person — still migration debt
expect((await brain.get(doc))!.subtype).toBe('general')
const audit = await brain.audit()
expect(audit.total).toBe(report.skipped) // skipped IS the remaining debt
})
})
describe('function rules', () => {
it('derives the subtype from entity fields', async () => {
const vendor = await brain.add({
type: NounType.Person,
data: 'Vendor person',
metadata: { kind: 'vendor' }
})
const fallback = await brain.add({ type: NounType.Person, data: 'No kind field' })
const report = await brain.fillSubtypes({
[NounType.Person]: (e) => (e.metadata as { kind?: string })?.kind ?? 'unspecified'
})
expect(report.filled).toBe(2)
expect((await brain.get(vendor))!.subtype).toBe('vendor')
expect((await brain.get(fallback))!.subtype).toBe('unspecified')
})
it('treats undefined returns as declines (selective fill, counted as skipped)', async () => {
const classified = await brain.add({
type: NounType.Person,
data: 'Classifiable',
metadata: { department: 'engineering' }
})
const unclassified = await brain.add({ type: NounType.Person, data: 'Not classifiable' })
const report = await brain.fillSubtypes({
[NounType.Person]: (e) =>
(e.metadata as { department?: string })?.department ? 'employee' : undefined
})
expect(report.filled).toBe(1)
expect(report.skipped).toBe(1)
expect((await brain.get(classified))!.subtype).toBe('employee')
expect((await brain.get(unclassified))!.subtype).toBeUndefined()
})
it('treats empty-string returns as declines (an empty subtype would fail enforcement)', async () => {
await brain.add({ type: NounType.Person, data: 'Rule returns empty string' })
const report = await brain.fillSubtypes({
[NounType.Person]: () => ''
})
expect(report.filled).toBe(0)
expect(report.skipped).toBe(1)
})
it('collects per-entry rule errors and continues the pass', async () => {
const poisoned = await brain.add({
type: NounType.Person,
data: 'Rule throws on this one',
metadata: { poison: true }
})
const healthy = await brain.add({ type: NounType.Person, data: 'Rule works on this one' })
const report = await brain.fillSubtypes({
[NounType.Person]: (e) => {
if ((e.metadata as { poison?: boolean })?.poison) {
throw new Error('cannot classify poisoned entity')
}
return 'unspecified'
}
})
expect(report.filled).toBe(1)
expect(report.errors).toHaveLength(1)
expect(report.errors[0].id).toBe(poisoned)
expect(report.errors[0].error).toMatch(/cannot classify/)
expect((await brain.get(healthy))!.subtype).toBe('unspecified')
})
})
describe('relationship rules (VerbType keys)', () => {
it('fills missing relationship subtypes through the same rule map', async () => {
const a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'A' })
const b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'B' })
const relId = await brain.relate({ from: a, to: b, type: VerbType.RelatedTo })
const labeled = await brain.relate({
from: b,
to: a,
type: VerbType.RelatedTo,
subtype: 'colleague'
})
const report = await brain.fillSubtypes({
[VerbType.RelatedTo]: 'unspecified'
})
expect(report.filled).toBe(1)
expect(report.byType).toEqual({ relatedTo: 1 })
const relations = await brain.getRelations({ from: a })
expect(relations.find((r) => r.id === relId)!.subtype).toBe('unspecified')
const reverse = await brain.getRelations({ from: b })
expect(reverse.find((r) => r.id === labeled)!.subtype).toBe('colleague') // untouched
})
it('passes the full Relation shape to relationship rule functions', async () => {
const a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'A' })
const b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'B' })
await brain.relate({
from: a,
to: b,
type: VerbType.ReportsTo,
metadata: { dotted: true }
})
const report = await brain.fillSubtypes({
[VerbType.ReportsTo]: (r) => {
// The rule sees the public Relation shape — endpoints included.
expect(r.from).toBe(a)
expect(r.to).toBe(b)
return (r.metadata as { dotted?: boolean })?.dotted ? 'dotted-line' : 'direct'
}
})
expect(report.filled).toBe(1)
const relations = await brain.getRelations({ from: a })
expect(relations[0].subtype).toBe('dotted-line')
})
it('handles entity and relationship rules in a single pass', async () => {
const a = await brain.add({ type: NounType.Person, data: 'A' })
const b = await brain.add({ type: NounType.Person, data: 'B' })
await brain.relate({ from: a, to: b, type: VerbType.RelatedTo })
const report = await brain.fillSubtypes({
[NounType.Person]: 'unspecified',
[VerbType.RelatedTo]: 'unspecified'
})
expect(report.filled).toBe(3)
expect(report.byType).toEqual({ person: 2, relatedTo: 1 })
const audit = await brain.audit()
expect(audit.total).toBe(0)
})
})
describe('report + write semantics', () => {
it('bumps _rev through the real update path', async () => {
const id = await brain.add({ type: NounType.Person, data: 'Rev check' })
const before = await brain.get(id)
await brain.fillSubtypes({ [NounType.Person]: 'unspecified' })
const after = await brain.get(id)
expect(after!._rev).toBe((before!._rev ?? 1) + 1)
})
it('is idempotent — a re-run scans but fills nothing', async () => {
await brain.add({ type: NounType.Person, data: 'Fill once' })
const first = await brain.fillSubtypes({ [NounType.Person]: 'unspecified' })
expect(first.filled).toBe(1)
const second = await brain.fillSubtypes({ [NounType.Person]: 'unspecified' })
expect(second.filled).toBe(0)
expect(second.skipped).toBe(0)
expect(second.scanned).toBeGreaterThan(0)
expect(second.errors).toEqual([])
})
it('excludes VFS-marked entries by default, fills them with includeVFS: true', async () => {
// VFS markers bypass subtype enforcement, so marked entries are not
// migration debt. Fabricate one explicitly (public consumers shouldn't
// set this marker; the test exercises the exclusion contract).
const vfsLike = await brain.add({
type: NounType.Document,
data: 'Infrastructure-marked entry',
metadata: { isVFS: true }
})
const defaultRun = await brain.fillSubtypes({ [NounType.Document]: 'general' })
expect(defaultRun.filled).toBe(0)
expect((await brain.get(vfsLike))!.subtype).toBeUndefined()
const inclusiveRun = await brain.fillSubtypes(
{ [NounType.Document]: 'general' },
{ includeVFS: true }
)
expect(inclusiveRun.filled).toBeGreaterThanOrEqual(1)
expect((await brain.get(vfsLike))!.subtype).toBe('general')
})
it('reports progress after each batch', async () => {
await brain.add({ type: NounType.Person, data: 'P1' })
await brain.add({ type: NounType.Person, data: 'P2' })
await brain.add({ type: NounType.Person, data: 'P3' })
const snapshots: Array<{ scanned: number; filled: number; skipped: number }> = []
await brain.fillSubtypes(
{ [NounType.Person]: 'unspecified' },
{
batchSize: 1,
onProgress: (p) => snapshots.push({ ...p })
}
)
expect(snapshots.length).toBeGreaterThanOrEqual(3)
const last = snapshots[snapshots.length - 1]
expect(last.filled).toBe(3)
})
it('clears the debt brain.audit() reports', async () => {
await brain.add({ type: NounType.Person, data: 'Debt 1' })
await brain.add({ type: NounType.Document, data: 'Debt 2' })
const before = await brain.audit()
expect(before.total).toBe(2)
await brain.fillSubtypes({
[NounType.Person]: 'unspecified',
[NounType.Document]: 'general'
})
const after = await brain.audit()
expect(after.total).toBe(0)
expect(after.recommendation).toMatch(/strict-mode-ready/)
})
})
describe('rule-map validation (fail fast, before touching data)', () => {
it('rejects an empty rules map', async () => {
await expect(brain.fillSubtypes({})).rejects.toThrow(/rules map is empty/)
})
it('rejects keys that are not a NounType or VerbType', async () => {
await expect(
brain.fillSubtypes({ 'not-a-type': 'whatever' } as never)
).rejects.toThrow(/'not-a-type' is not a valid NounType or VerbType/)
})
it('rejects empty-string literal rules', async () => {
await expect(
brain.fillSubtypes({ [NounType.Person]: '' })
).rejects.toThrow(/empty string/)
})
it('rejects rule values that are neither string nor function', async () => {
await expect(
brain.fillSubtypes({ [NounType.Person]: 42 } as never)
).rejects.toThrow(/must be a subtype string or a function/)
})
it('does not write anything when validation fails', async () => {
const id = await brain.add({ type: NounType.Person, data: 'Untouched on failure' })
await expect(
brain.fillSubtypes({
[NounType.Person]: 'unspecified',
'bogus-type': 'x'
} as never)
).rejects.toThrow(/bogus-type/)
expect((await brain.get(id))!.subtype).toBeUndefined()
})
})
})