feat(8.0): brain.fillSubtypes migration helper + pre-RC1 gap closure
- brain.fillSubtypes(rules): idempotent subtype back-fill for pre-8.0 data.
One rule per NounType/VerbType (literal default or per-entry function);
fills only entries still missing a subtype through the public update()/
updateRelation() paths; returns { scanned, filled, skipped, errors, byType }.
Full unit suite in tests/unit/brainy/fill-subtypes.test.ts.
- Fix getNouns/getVerbs pagination hasMore (peek one past the window) —
was permanently false, silently truncating every multi-page walk.
- find({ near }) without near.id now throws a teaching error instead of an
opaque storage sharding failure; CLI --threshold without --near applies a
plain score floor.
- CLI init/close audit: every one-shot command init()s, close()s, and exits
explicitly; delete the unmaintained interactive REPL; replace the cloud-era
storage subcommands with status/batch-delete; new types/validate commands.
- requireSubtype JSDoc now documents the 8.0 default-on contract; audit()
recommendation points at fillSubtypes.
- Docs: data-storage-architecture rewritten to the real 8.0 on-disk layout;
README storage section reflects filesystem+memory and snapshots; eli5 and
SEMANTIC_VFS /as-of/ semantics corrected; internal tracker IDs and
.strategy references scrubbed from published files.
This commit is contained in:
parent
9b0f4acd5b
commit
c44678390e
30 changed files with 1517 additions and 3226 deletions
40
README.md
40
README.md
|
|
@ -278,11 +278,16 @@ brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
|
||||||
**Enforce the pairing.** Register a vocabulary per type or turn on brain-wide strict mode to ensure every entity AND relationship has both `type` AND `subtype`:
|
**Enforce the pairing.** Register a vocabulary per type or turn on brain-wide strict mode to ensure every entity AND relationship has both `type` AND `subtype`:
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// Per-type rule with vocabulary
|
// Per-type rule with a closed vocabulary
|
||||||
brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true })
|
brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true })
|
||||||
|
|
||||||
// Or brain-wide strict mode
|
// 8.0 default: every write requires a subtype. Exempt genuine catch-all types…
|
||||||
const brain = new Brainy({ requireSubtype: true })
|
const brain = new Brainy({ requireSubtype: { except: [NounType.Thing] } })
|
||||||
|
|
||||||
|
// …or opt out while migrating pre-8.0 data, then audit and back-fill:
|
||||||
|
const legacy = new Brainy({ requireSubtype: false })
|
||||||
|
await legacy.audit() // gaps, grouped by type
|
||||||
|
await legacy.fillSubtypes({ [NounType.Person]: 'unspecified' }) // close them
|
||||||
```
|
```
|
||||||
|
|
||||||
For other facets you want counted (`status`, `source`, `role`), register them with `brain.trackField(name)`. Renaming an existing convention to `subtype`? Use `brain.migrateField({from, to, entityKind: 'both'})` to walk nouns AND verbs in one pass. Full guide: **[Subtypes & Facets](docs/guides/subtypes-and-facets.md)**.
|
For other facets you want counted (`status`, `source`, `role`), register them with `brain.trackField(name)`. Renaming an existing convention to `subtype`? Use `brain.migrateField({from, to, entityKind: 'both'})` to walk nouns AND verbs in one pass. Full guide: **[Subtypes & Facets](docs/guides/subtypes-and-facets.md)**.
|
||||||
|
|
@ -291,7 +296,7 @@ For other facets you want counted (`status`, `source`, `role`), register them wi
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Storage: Memory to Cloud
|
## Storage: Memory and Filesystem
|
||||||
|
|
||||||
The same API at every scale. Change one config line to go from prototype to production.
|
The same API at every scale. Change one config line to go from prototype to production.
|
||||||
|
|
||||||
|
|
@ -301,28 +306,31 @@ The same API at every scale. Change one config line to go from prototype to prod
|
||||||
const brain = new Brainy()
|
const brain = new Brainy()
|
||||||
```
|
```
|
||||||
|
|
||||||
### Production — Filesystem with Compression
|
### Production — Filesystem (gzip compression on by default)
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
const brain = new Brainy({
|
const brain = new Brainy({
|
||||||
storage: { type: 'filesystem', path: './data', compression: true }
|
storage: { type: 'filesystem', rootDirectory: './data' }
|
||||||
})
|
})
|
||||||
```
|
```
|
||||||
|
|
||||||
### Cloud — S3, GCS, Azure, Cloudflare R2
|
### Backups and Portability — Snapshots
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
const brain = new Brainy({
|
const db = brain.now()
|
||||||
storage: {
|
await db.persist('/backups/2026-06-11') // instant hard-link snapshot
|
||||||
type: 's3',
|
await db.release()
|
||||||
s3Storage: { bucketName: 'my-knowledge-base', region: 'us-east-1' }
|
|
||||||
}
|
const snapshot = await Brainy.load('/backups/2026-06-11') // read-only Db
|
||||||
})
|
const hits = await snapshot.search('quarterly invoices')
|
||||||
|
await snapshot.release()
|
||||||
```
|
```
|
||||||
|
|
||||||
|
A snapshot directory is self-contained — copy it to another machine, open it with `Brainy.load()`, or restore it wholesale with `brain.restore(path, { confirm: true })`.
|
||||||
|
|
||||||
Performance benchmarks and capacity planning in **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)**.
|
Performance benchmarks and capacity planning in **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)**.
|
||||||
|
|
||||||
**[Cloud Deployment Guide](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** | **[Capacity Planning](docs/operations/capacity-planning.md)**
|
**[Capacity Planning](docs/operations/capacity-planning.md)**
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -371,10 +379,8 @@ Performance benchmarks and capacity planning in **[docs/PERFORMANCE.md](docs/PER
|
||||||
|
|
||||||
### Operations
|
### Operations
|
||||||
|
|
||||||
- **[Cloud Deployment](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** — AWS, GCS, Azure
|
|
||||||
- **[Capacity Planning](docs/operations/capacity-planning.md)** — Memory, storage, and scaling
|
- **[Capacity Planning](docs/operations/capacity-planning.md)** — Memory, storage, and scaling
|
||||||
- **[Performance](docs/PERFORMANCE.md)** — Benchmarks and architecture details
|
- **[Performance](docs/PERFORMANCE.md)** — Benchmarks and architecture details
|
||||||
- Cost Optimization: **[AWS S3](docs/operations/cost-optimization-aws-s3.md)** | **[GCS](docs/operations/cost-optimization-gcs.md)** | **[Azure](docs/operations/cost-optimization-azure.md)** | **[R2](docs/operations/cost-optimization-cloudflare-r2.md)**
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -387,7 +393,7 @@ bun install @soulcraft/brainy # Bun — best performance
|
||||||
npm install @soulcraft/brainy # Node.js — fully supported
|
npm install @soulcraft/brainy # Node.js — fully supported
|
||||||
```
|
```
|
||||||
|
|
||||||
> **Deprecation Notice:** Browser support (OPFS, Web Workers, WASM embeddings) is deprecated in v7.10.0 and will be removed in v8.0.0. Brainy v8+ will be server-only.
|
> Brainy 8.0 is server-only. Browser support (OPFS storage, Web Workers, in-browser WASM embeddings) was removed in 8.0 — the 7.x line remains available on npm if you need it.
|
||||||
|
|
||||||
## Single-Writer Model
|
## Single-Writer Model
|
||||||
|
|
||||||
|
|
|
||||||
23
RELEASES.md
23
RELEASES.md
|
|
@ -167,8 +167,12 @@ this rename; only the API surface moved.
|
||||||
Escape hatches: `requireSubtype: false` (last-resort opt-out for legacy
|
Escape hatches: `requireSubtype: false` (last-resort opt-out for legacy
|
||||||
data) or `requireSubtype: { except: [NounType.Thing, ...] }` (per-type
|
data) or `requireSubtype: { except: [NounType.Thing, ...] }` (per-type
|
||||||
allowlist). Per-type rules registered via `brain.requireSubtype(type, opts)`
|
allowlist). Per-type rules registered via `brain.requireSubtype(type, opts)`
|
||||||
still compose. `brain.audit()` reports entries missing a subtype and
|
still compose. `brain.audit()` reports entries missing a subtype and the new
|
||||||
`brain.migrateField()` backfills them.
|
`brain.fillSubtypes(rules)` migration helper backfills them — one rule per
|
||||||
|
NounType/VerbType (literal default or per-entry function), applied only to
|
||||||
|
entries still missing a subtype, returning
|
||||||
|
`{ scanned, filled, skipped, errors, byType }`. Idempotent: re-running fills
|
||||||
|
nothing, so a crashed run is resumed by running it again.
|
||||||
- **Verb ids are Brainy-generated UUIDs — by contract.** `relate()` now throws
|
- **Verb ids are Brainy-generated UUIDs — by contract.** `relate()` now throws
|
||||||
a teaching error if a caller passes an `id` field
|
a teaching error if a caller passes an `id` field
|
||||||
(`src/utils/paramValidation.ts:526`). In 7.x a supplied id was silently
|
(`src/utils/paramValidation.ts:526`). In 7.x a supplied id was silently
|
||||||
|
|
@ -270,8 +274,10 @@ The full cost model is in
|
||||||
`getHistory` / `streamHistory` / `versions` / `data()` per the table above.
|
`getHistory` / `streamHistory` / `versions` / `data()` per the table above.
|
||||||
6. **Subtypes:** if your data predates subtype discipline, start with
|
6. **Subtypes:** if your data predates subtype discipline, start with
|
||||||
`requireSubtype: false`, run `brain.audit()`, backfill with
|
`requireSubtype: false`, run `brain.audit()`, backfill with
|
||||||
`brain.migrateField()`, then remove the opt-out so the 8.0 default
|
`brain.fillSubtypes(rules)` (one rule per type — a literal default or a
|
||||||
enforcement protects you going forward.
|
function deriving the subtype from each entry), re-run `audit()` until
|
||||||
|
`total === 0`, then remove the opt-out so the 8.0 default enforcement
|
||||||
|
protects you going forward.
|
||||||
7. **CLI scripts:** `fork` / `branch` / `checkout` / `migrate` →
|
7. **CLI scripts:** `fork` / `branch` / `checkout` / `migrate` →
|
||||||
`snapshot <path>` / `restore <path>` / `history` / `generation`.
|
`snapshot <path>` / `restore <path>` / `history` / `generation`.
|
||||||
8. **Plugin authors:** apply the contract renames and the BigInt graph
|
8. **Plugin authors:** apply the contract renames and the BigInt graph
|
||||||
|
|
@ -553,8 +559,7 @@ alongside every other metadata field on the existing write paths.
|
||||||
`_rev`, `ifRev`, `RevisionConflictError`, and `ifAbsent` all survive the 8.0 Db
|
`_rev`, `ifRev`, `RevisionConflictError`, and `ifAbsent` all survive the 8.0 Db
|
||||||
redesign unchanged. 8.0 layers `brain.transact(tx, { ifAtGeneration })` for
|
redesign unchanged. 8.0 layers `brain.transact(tx, { ifAtGeneration })` for
|
||||||
whole-tx CAS on top of the same per-entity mechanism — per-entity for single-record
|
whole-tx CAS on top of the same per-entity mechanism — per-entity for single-record
|
||||||
patterns, generation-based for "did the world move under me." See
|
patterns, generation-based for "did the world move under me."
|
||||||
`.strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md` § C-6 (internal).
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -672,8 +677,7 @@ in `validateFindParams()`, both fire before any storage/index/Cortex call. Corte
|
||||||
|
|
||||||
The new `docs/guides/find-limits.md` calls out that 8.0's Datomic-style `Db.find()`
|
The new `docs/guides/find-limits.md` calls out that 8.0's Datomic-style `Db.find()`
|
||||||
may tighten per-call limits; that's a Brainy 8.0 / Cortex 3.0 coordination point
|
may tighten per-call limits; that's a Brainy 8.0 / Cortex 3.0 coordination point
|
||||||
documented in `.strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md` (open question #4 covers
|
whose rollout staging now also covers query limits.
|
||||||
the rollout staging, which now also applies to query limits).
|
|
||||||
|
|
||||||
### What consumers should do
|
### What consumers should do
|
||||||
|
|
||||||
|
|
@ -830,8 +834,7 @@ Brainy before any native call.
|
||||||
|
|
||||||
- **Native `audit()` proxy.** For billion-scale brains, `audit()` walks every entity (O(N)).
|
- **Native `audit()` proxy.** For billion-scale brains, `audit()` walks every entity (O(N)).
|
||||||
A native implementation reading from a "null-subtype" bitmap in the column store would be
|
A native implementation reading from a "null-subtype" bitmap in the column store would be
|
||||||
O(buckets). Listed as the 6th open question in the
|
O(buckets).
|
||||||
[Brainy 8.0 spec doc](`/media/dpsifr/storage/home/Projects/brainy/.strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md`).
|
|
||||||
- **Strict-mode parity test.** Cortex should mirror Brainy's new
|
- **Strict-mode parity test.** Cortex should mirror Brainy's new
|
||||||
`tests/integration/strict-mode-self-test.test.ts` against their native paths to catch any
|
`tests/integration/strict-mode-self-test.test.ts` against their native paths to catch any
|
||||||
latent bug where native writes bypass JS validation.
|
latent bug where native writes bypass JS validation.
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -86,9 +86,8 @@ Benefits:
|
||||||
`FileSystemStorage` is the only one in-tree, but Cortex's
|
`FileSystemStorage` is the only one in-tree, but Cortex's
|
||||||
`MmapFileSystemStorage` inherits from it — interface relocation would
|
`MmapFileSystemStorage` inherits from it — interface relocation would
|
||||||
ripple through the plugin ecosystem.
|
ripple through the plugin ecosystem.
|
||||||
- The current state works. Real failure modes
|
- The current state works. The real failure modes seen in the field
|
||||||
(BR-CX-INTERFACE-GAP) were build/install artifacts, not type-system
|
were build/install artifacts, not type-system failures.
|
||||||
failures.
|
|
||||||
- 7.22.0 just shipped a clean fix. Stacking another refactor before
|
- 7.22.0 just shipped a clean fix. Stacking another refactor before
|
||||||
consumers absorb it adds churn without urgency.
|
consumers absorb it adds churn without urgency.
|
||||||
- The `hasStorageMethod()` guard accomplishes the same runtime safety the
|
- The `hasStorageMethod()` guard accomplishes the same runtime safety the
|
||||||
|
|
|
||||||
|
|
@ -133,7 +133,7 @@ Brainy is the only row with every box checked. And it runs all of them in a sing
|
||||||
|
|
||||||
### One library, any scale
|
### One library, any scale
|
||||||
|
|
||||||
Brainy scales from a single laptop to billions of entities without changing a line of code. Small datasets live in memory. Larger ones spill to disk. At cloud scale, Brainy uses S3-compatible storage and automatically shards across nodes — the same API the whole way. Up to ten billion entities is fully implemented today.
|
Brainy scales from a quick experiment to serious production datasets without changing a line of code. Small datasets live entirely in memory. Larger ones spill to disk, where Brainy shards and compresses everything automatically. Need a backup or a copy? Snapshots are instant — the same API the whole way.
|
||||||
|
|
||||||
Add Cortex and you also unlock memory-mapped storage — aggregate state lives directly in the operating system's memory with zero serialization overhead, as fast as the hardware allows.
|
Add Cortex and you also unlock memory-mapped storage — aggregate state lives directly in the operating system's memory with zero serialization overhead, as fast as the hardware allows.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -47,16 +47,18 @@ const authFiles = await vfs.readdir('/by-concept/authentication')
|
||||||
const aliceFiles = await vfs.readdir('/by-author/alice')
|
const aliceFiles = await vfs.readdir('/by-author/alice')
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. **Time Travel**
|
### 2. **Change Tracking by Date**
|
||||||
See your codebase as it existed at any point:
|
List the files that were modified on any given day:
|
||||||
```typescript
|
```typescript
|
||||||
// Code from March 15th
|
// Files that changed on March 15th
|
||||||
const snapshot = await vfs.readdir('/as-of/2024-03-15')
|
const changed = await vfs.readdir('/as-of/2024-03-15')
|
||||||
|
|
||||||
// Compare with today
|
// Everything under /src right now
|
||||||
const current = await vfs.readdir('/src')
|
const current = await vfs.readdir('/src')
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`/as-of/<date>` selects by *modification date* — it reads the files' current content, not historical versions. For true point-in-time queries over entity state, use the Db API (`brain.asOf(generation)`).
|
||||||
|
|
||||||
### 3. **Knowledge Graph Navigation**
|
### 3. **Knowledge Graph Navigation**
|
||||||
Navigate by semantic relationships:
|
Navigate by semantic relationships:
|
||||||
```typescript
|
```typescript
|
||||||
|
|
@ -118,13 +120,14 @@ await vfs.stat('/by-author/alice/config.ts')
|
||||||
### 4. By Time (Temporal) ✅ **Production**
|
### 4. By Time (Temporal) ✅ **Production**
|
||||||
```typescript
|
```typescript
|
||||||
await vfs.readdir('/as-of/2024-03-15')
|
await vfs.readdir('/as-of/2024-03-15')
|
||||||
// Files modified on March 15, 2024
|
// Files modified on March 15, 2024 (24-hour window)
|
||||||
|
|
||||||
await vfs.readFile('/as-of/2024-03-15/src/auth.ts')
|
await vfs.readFile('/as-of/2024-03-15/auth.ts')
|
||||||
// Read auth.ts as it existed that day
|
// Current content of auth.ts, addressed by modification date —
|
||||||
|
// the path only resolves if auth.ts was modified that day
|
||||||
```
|
```
|
||||||
|
|
||||||
**How it works:** Tracks `modified` timestamp. Uses B-tree range queries (`greaterEqual`/`lessEqual`) for O(log n) performance.
|
**How it works:** Tracks the `modified` timestamp on every file and runs a range query (`greaterEqual`/`lessEqual`) over one 24-hour window for O(log n) performance. The VFS does not store historical file contents — `/as-of/` filters by *when a file last changed*; reads return the current bytes. For point-in-time state, use the Db API (`brain.asOf(generation)`).
|
||||||
|
|
||||||
**Status:** ✅ Fully implemented and tested at 10K file scale
|
**Status:** ✅ Fully implemented and tested at 10K file scale
|
||||||
|
|
||||||
|
|
@ -222,7 +225,7 @@ console.log(authFiles)
|
||||||
// ['login.ts', 'signup.ts', 'oauth.ts']
|
// ['login.ts', 'signup.ts', 'oauth.ts']
|
||||||
```
|
```
|
||||||
|
|
||||||
### Example 2: Time Travel
|
### Example 2: Changes by Day
|
||||||
```typescript
|
```typescript
|
||||||
// See what changed today
|
// See what changed today
|
||||||
const today = new Date().toISOString().split('T')[0]
|
const today = new Date().toISOString().split('T')[0]
|
||||||
|
|
@ -232,8 +235,8 @@ const todaysFiles = await vfs.readdir(`/as-of/${today}`)
|
||||||
const yesterday = new Date(Date.now() - 86400000).toISOString().split('T')[0]
|
const yesterday = new Date(Date.now() - 86400000).toISOString().split('T')[0]
|
||||||
const yesterdaysFiles = await vfs.readdir(`/as-of/${yesterday}`)
|
const yesterdaysFiles = await vfs.readdir(`/as-of/${yesterday}`)
|
||||||
|
|
||||||
const newFiles = todaysFiles.filter(f => !yesterdaysFiles.includes(f))
|
const onlyToday = todaysFiles.filter(f => !yesterdaysFiles.includes(f))
|
||||||
console.log('New files today:', newFiles)
|
console.log('Changed today (untouched yesterday):', onlyToday)
|
||||||
```
|
```
|
||||||
|
|
||||||
### Example 3: Graph Navigation
|
### Example 3: Graph Navigation
|
||||||
|
|
@ -432,25 +435,17 @@ await vfs.writeFile(path, code, {
|
||||||
})
|
})
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. Optimize for Your Scale
|
### 3. Combine Dimensions
|
||||||
```typescript
|
```typescript
|
||||||
// For < 100K files: Post-filtering is fine
|
// Find security files Alice changed on a given day
|
||||||
// For > 100K files: Use flattened indexes
|
// (each /as-of/<date> path covers exactly that one day)
|
||||||
|
|
||||||
// Force index refresh after bulk operations
|
|
||||||
await brain.storage.rebuildIndexes()
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. Combine Dimensions
|
|
||||||
```typescript
|
|
||||||
// Find security files Alice worked on this week
|
|
||||||
const aliceFiles = await vfs.readdir('/by-author/alice')
|
const aliceFiles = await vfs.readdir('/by-author/alice')
|
||||||
const securityFiles = await vfs.readdir('/by-tag/security')
|
const securityFiles = await vfs.readdir('/by-tag/security')
|
||||||
const thisWeek = await vfs.readdir(`/as-of/${weekAgo}`)
|
const changedThatDay = await vfs.readdir('/as-of/2024-03-15')
|
||||||
|
|
||||||
const intersection = aliceFiles
|
const intersection = aliceFiles
|
||||||
.filter(f => securityFiles.includes(f))
|
.filter(f => securityFiles.includes(f))
|
||||||
.filter(f => thisWeek.includes(f))
|
.filter(f => changedThatDay.includes(f))
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -469,14 +464,14 @@ console.log(entity.metadata.concepts)
|
||||||
|
|
||||||
### Slow Queries on Large Datasets
|
### Slow Queries on Large Datasets
|
||||||
```typescript
|
```typescript
|
||||||
// Check if indexes are built
|
// Check if indexes are built and populated
|
||||||
const stats = await brain.storage.getIndexStats()
|
const stats = await brain.getIndexStats()
|
||||||
console.log(stats)
|
console.log(stats)
|
||||||
|
|
||||||
// Rebuild if needed
|
|
||||||
await brain.storage.rebuildIndexes()
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
If an index looks empty or inconsistent, rebuild from raw storage with the CLI
|
||||||
|
(stop the live writer first): `brainy inspect repair <data-dir>`.
|
||||||
|
|
||||||
### Semantic Path Returns Empty
|
### Semantic Path Returns Empty
|
||||||
```typescript
|
```typescript
|
||||||
// Check if metadata exists
|
// Check if metadata exists
|
||||||
|
|
|
||||||
305
src/brainy.ts
305
src/brainy.ts
|
|
@ -94,7 +94,10 @@ import {
|
||||||
BatchResult,
|
BatchResult,
|
||||||
BrainyConfig,
|
BrainyConfig,
|
||||||
BrainyStats,
|
BrainyStats,
|
||||||
ScoreExplanation
|
ScoreExplanation,
|
||||||
|
FillSubtypeRule,
|
||||||
|
FillSubtypeRules,
|
||||||
|
FillSubtypesResult
|
||||||
} from './types/brainy.types.js'
|
} from './types/brainy.types.js'
|
||||||
import { NounType, VerbType, TypeUtils } from './types/graphTypes.js'
|
import { NounType, VerbType, TypeUtils } from './types/graphTypes.js'
|
||||||
import { BrainyInterface } from './types/brainyInterface.js'
|
import { BrainyInterface } from './types/brainyInterface.js'
|
||||||
|
|
@ -2885,7 +2888,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
* })
|
* })
|
||||||
*
|
*
|
||||||
* @example Lock down management relationships
|
* @example Lock down management relationships
|
||||||
* brain.requireSubtype(VerbType.Manages, {
|
* brain.requireSubtype(VerbType.ReportsTo, {
|
||||||
* values: ['direct', 'dotted-line'],
|
* values: ['direct', 'dotted-line'],
|
||||||
* required: true
|
* required: true
|
||||||
* })
|
* })
|
||||||
|
|
@ -7330,10 +7333,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
* @param subtype - Optional specific subtype string for O(1) point count
|
* @param subtype - Optional specific subtype string for O(1) point count
|
||||||
*
|
*
|
||||||
* @example
|
* @example
|
||||||
* brain.counts.byRelationshipSubtype(VerbType.Manages)
|
* brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
|
||||||
* // → { direct: 12, 'dotted-line': 3 }
|
* // → { direct: 12, 'dotted-line': 3 }
|
||||||
*
|
*
|
||||||
* brain.counts.byRelationshipSubtype(VerbType.Manages, 'direct')
|
* brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct')
|
||||||
* // → 12
|
* // → 12
|
||||||
*/
|
*/
|
||||||
byRelationshipSubtype: (verb: VerbType, subtype?: string): number | Record<string, number> => {
|
byRelationshipSubtype: (verb: VerbType, subtype?: string): number | Record<string, number> => {
|
||||||
|
|
@ -7361,7 +7364,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
* `topSubtypes` for verbs.
|
* `topSubtypes` for verbs.
|
||||||
*
|
*
|
||||||
* @example
|
* @example
|
||||||
* brain.counts.topRelationshipSubtypes(VerbType.Manages, 5)
|
* brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 5)
|
||||||
* // → [['direct', 12], ['dotted-line', 3]]
|
* // → [['direct', 12], ['dotted-line', 3]]
|
||||||
*/
|
*/
|
||||||
topRelationshipSubtypes: (verb: VerbType, n: number = 10): Array<[string, number]> => {
|
topRelationshipSubtypes: (verb: VerbType, n: number = 10): Array<[string, number]> => {
|
||||||
|
|
@ -7533,7 +7536,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
* @returns Sorted list of distinct subtype strings (empty if none)
|
* @returns Sorted list of distinct subtype strings (empty if none)
|
||||||
*
|
*
|
||||||
* @example
|
* @example
|
||||||
* const variants = brain.relationshipSubtypesOf(VerbType.Manages)
|
* const variants = brain.relationshipSubtypesOf(VerbType.ReportsTo)
|
||||||
* // → ['direct', 'dotted-line']
|
* // → ['direct', 'dotted-line']
|
||||||
*/
|
*/
|
||||||
relationshipSubtypesOf(verb: VerbType): string[] {
|
relationshipSubtypesOf(verb: VerbType): string[] {
|
||||||
|
|
@ -7550,17 +7553,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
/**
|
/**
|
||||||
* Find entities and relationships missing a `subtype` value, grouped by type.
|
* Find entities and relationships missing a `subtype` value, grouped by type.
|
||||||
*
|
*
|
||||||
* The diagnostic pair to `migrateField()` / `fillSubtypes()` — answers the
|
* The diagnostic pair to `fillSubtypes()` / `migrateField()` — answers the
|
||||||
* question "what would break if I enabled strict subtype enforcement?". Run
|
* question "what would strict subtype enforcement reject?". 8.0 makes
|
||||||
* this before adopting an SDK that registers `requireSubtype()` rules on
|
* `requireSubtype: true` the default, so run this when opening a pre-8.0
|
||||||
* common NounTypes, or before upgrading to Brainy 8.0 (which makes
|
* brain (with `requireSubtype: false` as the temporary escape hatch), then
|
||||||
* `requireSubtype: true` the default).
|
* back-fill the reported gaps with `fillSubtypes(rules)`.
|
||||||
*
|
*
|
||||||
* Streams the brain via the same paginated `storage.getNouns()` /
|
* Streams the brain via the same paginated `storage.getNouns()` /
|
||||||
* `storage.getVerbs()` pattern `migrateField()` uses — safe for large brains
|
* `storage.getVerbs()` pattern `fillSubtypes()` uses — safe for large brains
|
||||||
* but linear in `O(N)`. Cortex 3.0+ may proxy this through the native
|
* but linear in `O(N)`. A native index provider may serve this from a
|
||||||
* column-store null-subtype bitmap for sub-linear performance on billion-scale
|
* column-store null-subtype bitmap in the future for sub-linear performance
|
||||||
* brains (tracked in `CTX-SUBTYPE-8.0-CONTRACT`).
|
* on billion-scale brains.
|
||||||
*
|
*
|
||||||
* @param options.includeVFS - When `false` (default), entities marked with
|
* @param options.includeVFS - When `false` (default), entities marked with
|
||||||
* `metadata.isVFSEntity` or `metadata.isVFS` are excluded from the report —
|
* `metadata.isVFSEntity` or `metadata.isVFS` are excluded from the report —
|
||||||
|
|
@ -7570,7 +7573,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
* @param options.onProgress - Optional progress callback invoked after each batch.
|
* @param options.onProgress - Optional progress callback invoked after each batch.
|
||||||
* @returns Report with per-type counts of entities/relationships without a
|
* @returns Report with per-type counts of entities/relationships without a
|
||||||
* subtype, plus the overall total and a one-line recommendation pointing at
|
* subtype, plus the overall total and a one-line recommendation pointing at
|
||||||
* `migrateField()` (7.x) or `fillSubtypes()` (8.0).
|
* `fillSubtypes()`.
|
||||||
*
|
*
|
||||||
* @example Find pre-existing gaps before turning on strict mode
|
* @example Find pre-existing gaps before turning on strict mode
|
||||||
* const report = await brain.audit()
|
* const report = await brain.audit()
|
||||||
|
|
@ -7655,7 +7658,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
|
|
||||||
const recommendation = missingSubtype === 0
|
const recommendation = missingSubtype === 0
|
||||||
? 'No subtype gaps detected — this brain is strict-mode-ready.'
|
? 'No subtype gaps detected — this brain is strict-mode-ready.'
|
||||||
: 'Found ' + missingSubtype + ' entries without subtype. Migrate via `brain.migrateField()` (7.x) — or wait for `brain.fillSubtypes()` (8.0) which closes the same gap with caller-supplied rules.'
|
: 'Found ' + missingSubtype + ' entries without subtype. Back-fill with `brain.fillSubtypes(rules)` — one rule per NounType/VerbType, literal default or per-entry function. To lift an existing field into `subtype` instead, use `brain.migrateField({ from, to: \'subtype\' })`.'
|
||||||
|
|
||||||
return {
|
return {
|
||||||
entitiesWithoutSubtype,
|
entitiesWithoutSubtype,
|
||||||
|
|
@ -7666,6 +7669,262 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Back-fill missing `subtype` values across the whole brain — the 8.0
|
||||||
|
* migration helper for data written before subtype became required.
|
||||||
|
*
|
||||||
|
* 8.0 enforces a non-empty `subtype` on every write by default
|
||||||
|
* (`requireSubtype: true`). A brain created on 7.x typically carries entities
|
||||||
|
* and relationships without one; this method clears that debt in a single
|
||||||
|
* idempotent pass so the opt-out (`requireSubtype: false`) can be removed.
|
||||||
|
* The intended upgrade flow:
|
||||||
|
*
|
||||||
|
* 1. Open the brain with `requireSubtype: false` (temporary escape hatch).
|
||||||
|
* 2. `await brain.audit()` — see what's missing, grouped by type.
|
||||||
|
* 3. `await brain.fillSubtypes(rules)` — back-fill with one rule per type.
|
||||||
|
* 4. Re-run `audit()` until `total === 0`, then drop the opt-out.
|
||||||
|
*
|
||||||
|
* **Rules.** One rule per NounType (entities) and/or VerbType
|
||||||
|
* (relationships) — the two vocabularies don't overlap, so a single map
|
||||||
|
* covers both sides. A rule is either a literal subtype string (blanket
|
||||||
|
* default) or a function deriving the subtype from the entry; functions
|
||||||
|
* subsume `where`-style filtering by returning `undefined` for entries they
|
||||||
|
* decline (those stay untouched and count as `skipped`, so a later run with
|
||||||
|
* a stricter rule can pick them up).
|
||||||
|
*
|
||||||
|
* **What is never touched:** entries that already carry a non-empty
|
||||||
|
* `subtype` (re-running is a no-op on them), and — unless
|
||||||
|
* `includeVFS: true` — Brainy's own VFS infrastructure entries
|
||||||
|
* (`metadata.isVFSEntity` / `metadata.isVFS`), which bypass enforcement
|
||||||
|
* anyway and are not migration debt.
|
||||||
|
*
|
||||||
|
* **Write strategy (deliberate):** each fill goes through the public
|
||||||
|
* `update()` / `updateRelation()` paths, so every write is individually
|
||||||
|
* atomic (storage record + indexes + subtype rollups commit together) and
|
||||||
|
* bumps the entry's `_rev` like any other update. The pass is *not* one
|
||||||
|
* whole-brain transaction: a migration over millions of entries inside a
|
||||||
|
* single transaction would hold an unbounded working set and turn one bad
|
||||||
|
* entry into an all-or-nothing failure. Idempotence is the recovery model —
|
||||||
|
* a crashed or partially-failed run is resumed safely by re-running, because
|
||||||
|
* only entries still missing a subtype are written.
|
||||||
|
*
|
||||||
|
* Streams via the same paginated `storage.getNouns()` / `storage.getVerbs()`
|
||||||
|
* walk `audit()` uses — `O(N)` but constant memory. The noun pass is skipped
|
||||||
|
* entirely when the map has no NounType rules, and vice versa.
|
||||||
|
*
|
||||||
|
* @param rules - Map of NounType/VerbType → literal subtype or rule function.
|
||||||
|
* Must contain at least one valid type key; invalid keys, empty-string
|
||||||
|
* literals, and non-string/non-function values throw before any data is
|
||||||
|
* touched.
|
||||||
|
* @param options.includeVFS - Also fill VFS infrastructure entries (default
|
||||||
|
* `false` — they bypass enforcement and don't need a subtype).
|
||||||
|
* @param options.batchSize - Pagination batch size (default `200`).
|
||||||
|
* @param options.onProgress - Optional callback invoked after each batch.
|
||||||
|
* @returns `{ scanned, filled, skipped, errors, byType }` — see
|
||||||
|
* {@link FillSubtypesResult}. After a clean run, `skipped` equals the
|
||||||
|
* remaining `audit().total`.
|
||||||
|
* @throws If the brain is read-only, the rule map is empty/malformed, or a
|
||||||
|
* key is not a valid NounType/VerbType. Per-entry write failures do NOT
|
||||||
|
* throw — they are collected in `errors` and the pass continues.
|
||||||
|
*
|
||||||
|
* @example Back-fill entities and relationships in one pass
|
||||||
|
* const report = await brain.fillSubtypes({
|
||||||
|
* [NounType.Person]: (e) => e.metadata?.kind ?? 'unspecified',
|
||||||
|
* [NounType.Document]: 'general',
|
||||||
|
* [VerbType.RelatedTo]: 'unspecified'
|
||||||
|
* })
|
||||||
|
* // → { scanned: 5200, filled: 1429, skipped: 0, errors: [], byType: { person: 800, document: 600, relatedTo: 29 } }
|
||||||
|
*
|
||||||
|
* @example Selective fill — decline entries a rule can't classify
|
||||||
|
* await brain.fillSubtypes({
|
||||||
|
* [NounType.Person]: (e) => e.metadata?.department ? 'employee' : undefined
|
||||||
|
* })
|
||||||
|
* // Persons without a department stay untouched (counted as skipped).
|
||||||
|
*
|
||||||
|
* @since 8.0.0
|
||||||
|
*/
|
||||||
|
async fillSubtypes(
|
||||||
|
rules: FillSubtypeRules<T>,
|
||||||
|
options: {
|
||||||
|
includeVFS?: boolean
|
||||||
|
batchSize?: number
|
||||||
|
onProgress?: (progress: { scanned: number; filled: number; skipped: number }) => void
|
||||||
|
} = {}
|
||||||
|
): Promise<FillSubtypesResult> {
|
||||||
|
this.assertWritable('fillSubtypes')
|
||||||
|
await this.ensureInitialized()
|
||||||
|
|
||||||
|
// Validate the rule map up front — fail fast on shape errors before any
|
||||||
|
// data is touched.
|
||||||
|
if (!rules || typeof rules !== 'object' || Array.isArray(rules)) {
|
||||||
|
throw new Error(
|
||||||
|
'fillSubtypes: rules must be a map of NounType/VerbType → subtype string or rule function'
|
||||||
|
)
|
||||||
|
}
|
||||||
|
const nounTypeValues = new Set<string>(Object.values(NounType))
|
||||||
|
const verbTypeValues = new Set<string>(Object.values(VerbType))
|
||||||
|
const nounRules = new Map<string, FillSubtypeRule<Entity<T>>>()
|
||||||
|
const verbRules = new Map<string, FillSubtypeRule<Relation<T>>>()
|
||||||
|
for (const [key, rule] of Object.entries(rules)) {
|
||||||
|
if (rule === undefined) continue
|
||||||
|
if (typeof rule !== 'string' && typeof rule !== 'function') {
|
||||||
|
throw new Error(
|
||||||
|
`fillSubtypes: rule for '${key}' must be a subtype string or a function (got ${typeof rule})`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
if (typeof rule === 'string' && rule.length === 0) {
|
||||||
|
throw new Error(
|
||||||
|
`fillSubtypes: rule for '${key}' is an empty string — a subtype must be non-empty`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
if (nounTypeValues.has(key)) {
|
||||||
|
nounRules.set(key, rule as FillSubtypeRule<Entity<T>>)
|
||||||
|
} else if (verbTypeValues.has(key)) {
|
||||||
|
verbRules.set(key, rule as FillSubtypeRule<Relation<T>>)
|
||||||
|
} else {
|
||||||
|
throw new Error(`fillSubtypes: '${key}' is not a valid NounType or VerbType`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (nounRules.size === 0 && verbRules.size === 0) {
|
||||||
|
throw new Error(
|
||||||
|
'fillSubtypes: rules map is empty — provide at least one NounType or VerbType rule'
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
const includeVFS = options.includeVFS === true
|
||||||
|
const batchSize = Math.max(1, options.batchSize ?? 200)
|
||||||
|
|
||||||
|
let scanned = 0
|
||||||
|
let filled = 0
|
||||||
|
let skipped = 0
|
||||||
|
const errors: Array<{ id: string; error: string }> = []
|
||||||
|
const byType: Record<string, number> = {}
|
||||||
|
|
||||||
|
const reportProgress = (): void => {
|
||||||
|
if (options.onProgress) options.onProgress({ scanned, filled, skipped })
|
||||||
|
}
|
||||||
|
|
||||||
|
// Normalize a rule's output: only a non-empty string is a fill; anything
|
||||||
|
// else (undefined, empty string) is a decline. An empty subtype would fail
|
||||||
|
// the very strict-mode check this helper exists to satisfy.
|
||||||
|
const normalize = (value: string | undefined): string | undefined =>
|
||||||
|
typeof value === 'string' && value.length > 0 ? value : undefined
|
||||||
|
|
||||||
|
// Pass 1: entities (skipped entirely when the map has no NounType rules).
|
||||||
|
if (nounRules.size > 0) {
|
||||||
|
let offset = 0
|
||||||
|
while (true) {
|
||||||
|
const page = await this.storage.getNouns({ pagination: { offset, limit: batchSize } })
|
||||||
|
if (page.items.length === 0) break
|
||||||
|
for (const noun of page.items) {
|
||||||
|
scanned++
|
||||||
|
// VFS infrastructure entries bypass enforcement via their marker, so
|
||||||
|
// they're not migration debt — leave them alone unless asked.
|
||||||
|
if (
|
||||||
|
!includeVFS &&
|
||||||
|
(noun.metadata?.isVFSEntity === true || noun.metadata?.isVFS === true)
|
||||||
|
) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if (typeof noun.subtype === 'string' && noun.subtype.length > 0) continue // already filled — never overwrite
|
||||||
|
const rule = nounRules.get(noun.type)
|
||||||
|
if (rule === undefined) {
|
||||||
|
skipped++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
let subtype: string | undefined
|
||||||
|
if (typeof rule === 'function') {
|
||||||
|
const entity = await this.convertNounToEntity(noun)
|
||||||
|
subtype = normalize(rule(entity))
|
||||||
|
} else {
|
||||||
|
subtype = rule
|
||||||
|
}
|
||||||
|
if (subtype === undefined) {
|
||||||
|
skipped++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
await this.update({ id: noun.id, subtype })
|
||||||
|
filled++
|
||||||
|
byType[noun.type] = (byType[noun.type] || 0) + 1
|
||||||
|
} catch (err) {
|
||||||
|
errors.push({
|
||||||
|
id: noun.id,
|
||||||
|
error: err instanceof Error ? err.message : String(err)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
reportProgress()
|
||||||
|
if (!page.hasMore) break
|
||||||
|
offset += page.items.length
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Pass 2: relationships (skipped entirely when the map has no VerbType rules).
|
||||||
|
if (verbRules.size > 0) {
|
||||||
|
let offset = 0
|
||||||
|
while (true) {
|
||||||
|
const page = await this.storage.getVerbs({ pagination: { offset, limit: batchSize } })
|
||||||
|
if (page.items.length === 0) break
|
||||||
|
for (const verb of page.items) {
|
||||||
|
scanned++
|
||||||
|
if (
|
||||||
|
!includeVFS &&
|
||||||
|
(verb.metadata?.isVFSEntity === true || verb.metadata?.isVFS === true)
|
||||||
|
) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if (typeof verb.subtype === 'string' && verb.subtype.length > 0) continue // already filled — never overwrite
|
||||||
|
const rule = verbRules.get(verb.verb)
|
||||||
|
if (rule === undefined) {
|
||||||
|
skipped++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
let subtype: string | undefined
|
||||||
|
if (typeof rule === 'function') {
|
||||||
|
// Project the stored verb onto the public Relation<T> shape the
|
||||||
|
// rule function is typed against.
|
||||||
|
const relation: Relation<T> = {
|
||||||
|
id: verb.id,
|
||||||
|
from: verb.sourceId,
|
||||||
|
to: verb.targetId,
|
||||||
|
type: verb.verb,
|
||||||
|
weight: verb.weight,
|
||||||
|
data: verb.data,
|
||||||
|
metadata: verb.metadata as T,
|
||||||
|
service: verb.service,
|
||||||
|
createdAt: verb.createdAt,
|
||||||
|
updatedAt: verb.updatedAt,
|
||||||
|
confidence: verb.confidence
|
||||||
|
}
|
||||||
|
subtype = normalize(rule(relation))
|
||||||
|
} else {
|
||||||
|
subtype = rule
|
||||||
|
}
|
||||||
|
if (subtype === undefined) {
|
||||||
|
skipped++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
await this.updateRelation({ id: verb.id, subtype })
|
||||||
|
filled++
|
||||||
|
byType[verb.verb] = (byType[verb.verb] || 0) + 1
|
||||||
|
} catch (err) {
|
||||||
|
errors.push({
|
||||||
|
id: verb.id,
|
||||||
|
error: err instanceof Error ? err.message : String(err)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
reportProgress()
|
||||||
|
if (!page.hasMore) break
|
||||||
|
offset += page.items.length
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return { scanned, filled, skipped, errors, byType }
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Stream-and-rewrite a field across every entity in the brain.
|
* Stream-and-rewrite a field across every entity in the brain.
|
||||||
*
|
*
|
||||||
|
|
@ -8845,6 +9104,16 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
private async executeProximitySearch(params: FindParams<T>): Promise<Result<T>[]> {
|
private async executeProximitySearch(params: FindParams<T>): Promise<Result<T>[]> {
|
||||||
if (!params.near) return []
|
if (!params.near) return []
|
||||||
|
|
||||||
|
// Teaching error: without an anchor id the constraint is meaningless, and
|
||||||
|
// letting it fall through produces an opaque storage-layer sharding error.
|
||||||
|
if (!params.near.id) {
|
||||||
|
throw new Error(
|
||||||
|
"find({ near }): 'near.id' is required — pass the entity to search around, " +
|
||||||
|
'e.g. near: { id, threshold }. To impose a minimum score on plain semantic ' +
|
||||||
|
'results, filter on result.score instead.'
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
const nearEntity = await this.get(params.near.id)
|
const nearEntity = await this.get(params.near.id)
|
||||||
if (!nearEntity) return []
|
if (!nearEntity) return []
|
||||||
|
|
||||||
|
|
@ -9680,7 +9949,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
*/
|
*/
|
||||||
private normalizeConfig(config?: BrainyConfig): Required<BrainyConfig> {
|
private normalizeConfig(config?: BrainyConfig): Required<BrainyConfig> {
|
||||||
// Validate storage configuration. Brainy 8.0 ships two adapters only —
|
// Validate storage configuration. Brainy 8.0 ships two adapters only —
|
||||||
// FileSystemStorage and MemoryStorage — per BR-BRAINY-80-STORAGE-SIMPLIFY.
|
// FileSystemStorage and MemoryStorage (cloud + OPFS adapters were removed).
|
||||||
// Cloud backup remains supported via operator tooling (db.persist() +
|
// Cloud backup remains supported via operator tooling (db.persist() +
|
||||||
// gsutil / aws s3 cp / rclone / azcopy). Pre-constructed adapter
|
// gsutil / aws s3 cp / rclone / azcopy). Pre-constructed adapter
|
||||||
// instances bypass the type check (they ARE the storage).
|
// instances bypass the type check (they ARE the storage).
|
||||||
|
|
|
||||||
|
|
@ -132,6 +132,7 @@ export const coreCommands = {
|
||||||
|
|
||||||
const spinner = ora('Adding to neural database...').start()
|
const spinner = ora('Adding to neural database...').start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
let metadata: any = {}
|
let metadata: any = {}
|
||||||
if (options.metadata) {
|
if (options.metadata) {
|
||||||
|
|
@ -204,6 +205,12 @@ export const coreCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput({ id: result, metadata, confidence: addParams.confidence, weight: addParams.weight }, options)
|
formatOutput({ id: result, metadata, confidence: addParams.confidence, weight: addParams.weight }, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// close() releases the writer lock and indexes, but global timers
|
||||||
|
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
|
||||||
|
// alive. CLI commands are one-shot — exit explicitly.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
if (spinner) spinner.fail('Failed to add data')
|
if (spinner) spinner.fail('Failed to add data')
|
||||||
console.error(chalk.red('Failed to add data:', error.message))
|
console.error(chalk.red('Failed to add data:', error.message))
|
||||||
|
|
@ -280,6 +287,7 @@ export const coreCommands = {
|
||||||
|
|
||||||
const spinner = ora('Searching with Triple Intelligence™...').start()
|
const spinner = ora('Searching with Triple Intelligence™...').start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
// Build comprehensive search params
|
// Build comprehensive search params
|
||||||
const searchParams: any = {
|
const searchParams: any = {
|
||||||
|
|
@ -292,11 +300,6 @@ export const coreCommands = {
|
||||||
searchParams.offset = parseInt(options.offset)
|
searchParams.offset = parseInt(options.offset)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Vector Intelligence - similarity threshold
|
|
||||||
if (options.threshold) {
|
|
||||||
searchParams.near = { threshold: parseFloat(options.threshold) }
|
|
||||||
}
|
|
||||||
|
|
||||||
// Metadata Intelligence - type filtering
|
// Metadata Intelligence - type filtering
|
||||||
if (options.type) {
|
if (options.type) {
|
||||||
const types = options.type.split(',').map(t => t.trim())
|
const types = options.type.split(',').map(t => t.trim())
|
||||||
|
|
@ -314,7 +317,9 @@ export const coreCommands = {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Vector Intelligence - proximity search
|
// Vector Intelligence - proximity search around an anchor entity.
|
||||||
|
// `near` requires an id; a bare --threshold (no --near) is applied as a
|
||||||
|
// plain score floor on the fused results after find() returns.
|
||||||
if (options.near) {
|
if (options.near) {
|
||||||
searchParams.near = {
|
searchParams.near = {
|
||||||
id: options.near,
|
id: options.near,
|
||||||
|
|
@ -371,7 +376,14 @@ export const coreCommands = {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const results = await brain.find(searchParams)
|
let results = await brain.find(searchParams)
|
||||||
|
|
||||||
|
// Without --near there is no proximity anchor; apply --threshold as a
|
||||||
|
// minimum-score filter on the fused results instead.
|
||||||
|
if (!options.near && options.threshold) {
|
||||||
|
const minScore = parseFloat(options.threshold)
|
||||||
|
results = results.filter((r) => r.score === undefined || r.score >= minScore)
|
||||||
|
}
|
||||||
|
|
||||||
spinner.succeed(`Found ${results.length} results`)
|
spinner.succeed(`Found ${results.length} results`)
|
||||||
|
|
||||||
|
|
@ -457,6 +469,10 @@ export const coreCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput(results, options)
|
formatOutput(results, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see add() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
if (spinner) spinner.fail('Search failed')
|
if (spinner) spinner.fail('Search failed')
|
||||||
console.error(chalk.red('Search failed:', error.message))
|
console.error(chalk.red('Search failed:', error.message))
|
||||||
|
|
@ -496,6 +512,7 @@ export const coreCommands = {
|
||||||
|
|
||||||
const spinner = ora('Fetching item...').start()
|
const spinner = ora('Fetching item...').start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
// Try to get the item
|
// Try to get the item
|
||||||
const item = await brain.get(id)
|
const item = await brain.get(id)
|
||||||
|
|
@ -529,6 +546,10 @@ export const coreCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput(item, options)
|
formatOutput(item, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see add() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
if (spinner) spinner.fail('Failed to get item')
|
if (spinner) spinner.fail('Failed to get item')
|
||||||
console.error(chalk.red('Failed to get item:', error.message))
|
console.error(chalk.red('Failed to get item:', error.message))
|
||||||
|
|
@ -589,6 +610,7 @@ export const coreCommands = {
|
||||||
|
|
||||||
const spinner = ora('Creating relationship...').start()
|
const spinner = ora('Creating relationship...').start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
let metadata: any = {}
|
let metadata: any = {}
|
||||||
if (options.metadata) {
|
if (options.metadata) {
|
||||||
|
|
@ -627,6 +649,10 @@ export const coreCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput({ id: result, source, verb, target, metadata }, options)
|
formatOutput({ id: result, source, verb, target, metadata }, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see add() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
if (spinner) spinner.fail('Failed to create relationship')
|
if (spinner) spinner.fail('Failed to create relationship')
|
||||||
console.error(chalk.red('Failed to create relationship:', error.message))
|
console.error(chalk.red('Failed to create relationship:', error.message))
|
||||||
|
|
@ -683,6 +709,7 @@ export const coreCommands = {
|
||||||
|
|
||||||
spinner = ora('Updating entity...').start()
|
spinner = ora('Updating entity...').start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
// Get existing entity first
|
// Get existing entity first
|
||||||
const existing = await brain.get(id)
|
const existing = await brain.get(id)
|
||||||
|
|
@ -731,6 +758,10 @@ export const coreCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput({ id, updated: true }, options)
|
formatOutput({ id, updated: true }, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see add() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
if (spinner) spinner.fail('Failed to update entity')
|
if (spinner) spinner.fail('Failed to update entity')
|
||||||
console.error(chalk.red('Update failed:', error.message))
|
console.error(chalk.red('Update failed:', error.message))
|
||||||
|
|
@ -784,6 +815,7 @@ export const coreCommands = {
|
||||||
|
|
||||||
spinner = ora('Deleting entity...').start()
|
spinner = ora('Deleting entity...').start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
await brain.delete(id)
|
await brain.delete(id)
|
||||||
|
|
||||||
|
|
@ -794,6 +826,10 @@ export const coreCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput({ id, deleted: true }, options)
|
formatOutput({ id, deleted: true }, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see add() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
if (spinner) spinner.fail('Failed to delete entity')
|
if (spinner) spinner.fail('Failed to delete entity')
|
||||||
console.error(chalk.red('Delete failed:', error.message))
|
console.error(chalk.red('Delete failed:', error.message))
|
||||||
|
|
@ -846,6 +882,7 @@ export const coreCommands = {
|
||||||
|
|
||||||
spinner = ora('Removing relationship...').start()
|
spinner = ora('Removing relationship...').start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
await brain.unrelate(id)
|
await brain.unrelate(id)
|
||||||
|
|
||||||
|
|
@ -856,6 +893,10 @@ export const coreCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput({ id, removed: true }, options)
|
formatOutput({ id, removed: true }, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see add() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
if (spinner) spinner.fail('Failed to remove relationship')
|
if (spinner) spinner.fail('Failed to remove relationship')
|
||||||
console.error(chalk.red('Unrelate failed:', error.message))
|
console.error(chalk.red('Unrelate failed:', error.message))
|
||||||
|
|
@ -875,7 +916,9 @@ export const coreCommands = {
|
||||||
|
|
||||||
if (options.json) {
|
if (options.json) {
|
||||||
formatOutput(diag, options)
|
formatOutput(diag, options)
|
||||||
return
|
// One-shot command — see add() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
}
|
}
|
||||||
|
|
||||||
console.log(chalk.bold('\nBrainy Diagnostics'))
|
console.log(chalk.bold('\nBrainy Diagnostics'))
|
||||||
|
|
@ -907,6 +950,10 @@ export const coreCommands = {
|
||||||
console.log(` Metadata: ${diag.indexes.metadata.type} (initialized: ${diag.indexes.metadata.initialized})`)
|
console.log(` Metadata: ${diag.indexes.metadata.type} (initialized: ${diag.indexes.metadata.initialized})`)
|
||||||
console.log(` Graph: ${diag.indexes.graph.type} (initialized: ${diag.indexes.graph.initialized}, wired: ${diag.indexes.graph.wiredToStorage})`)
|
console.log(` Graph: ${diag.indexes.graph.type} (initialized: ${diag.indexes.graph.initialized}, wired: ${diag.indexes.graph.wiredToStorage})`)
|
||||||
console.log()
|
console.log()
|
||||||
|
|
||||||
|
// One-shot command — see add() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
console.error(chalk.red('Diagnostics failed: ' + error.message))
|
console.error(chalk.red('Diagnostics failed: ' + error.message))
|
||||||
process.exit(1)
|
process.exit(1)
|
||||||
|
|
|
||||||
|
|
@ -152,6 +152,7 @@ export const importCommands = {
|
||||||
|
|
||||||
spinner = ora('Initializing import...').start()
|
spinner = ora('Initializing import...').start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
// Handle different source types
|
// Handle different source types
|
||||||
let result: any
|
let result: any
|
||||||
|
|
@ -401,6 +402,12 @@ export const importCommands = {
|
||||||
} else if (options.json) {
|
} else if (options.json) {
|
||||||
formatOutput(result, options)
|
formatOutput(result, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// close() releases the writer lock and indexes, but global timers
|
||||||
|
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
|
||||||
|
// alive. CLI commands are one-shot — exit explicitly.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
if (spinner) spinner.fail('Import failed')
|
if (spinner) spinner.fail('Import failed')
|
||||||
console.error(chalk.red('Import failed:', error.message))
|
console.error(chalk.red('Import failed:', error.message))
|
||||||
|
|
@ -472,9 +479,10 @@ export const importCommands = {
|
||||||
|
|
||||||
spinner = ora('Initializing VFS import...').start()
|
spinner = ora('Initializing VFS import...').start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
// Get VFS
|
// Get VFS
|
||||||
const vfs = await brain.vfs
|
const vfs = brain.vfs
|
||||||
|
|
||||||
// Load DirectoryImporter
|
// Load DirectoryImporter
|
||||||
const { DirectoryImporter } = await import('../../vfs/importers/DirectoryImporter.js')
|
const { DirectoryImporter } = await import('../../vfs/importers/DirectoryImporter.js')
|
||||||
|
|
@ -536,6 +544,10 @@ export const importCommands = {
|
||||||
} else if (options.json) {
|
} else if (options.json) {
|
||||||
formatOutput(result, options)
|
formatOutput(result, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see import() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
if (spinner) spinner.fail('VFS import failed')
|
if (spinner) spinner.fail('VFS import failed')
|
||||||
console.error(chalk.red('VFS import failed:', error.message))
|
console.error(chalk.red('VFS import failed:', error.message))
|
||||||
|
|
|
||||||
|
|
@ -41,6 +41,7 @@ export const insightsCommands = {
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
// Get insights from Brainy
|
// Get insights from Brainy
|
||||||
const insights = await brain.insights()
|
const insights = await brain.insights()
|
||||||
|
|
@ -102,6 +103,12 @@ export const insightsCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput(insights, options)
|
formatOutput(insights, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// close() releases the writer lock and indexes, but global timers
|
||||||
|
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
|
||||||
|
// alive. CLI commands are one-shot — exit explicitly.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
spinner.fail('Failed to get insights')
|
spinner.fail('Failed to get insights')
|
||||||
console.error(chalk.red('Insights failed:', error.message))
|
console.error(chalk.red('Insights failed:', error.message))
|
||||||
|
|
@ -120,6 +127,7 @@ export const insightsCommands = {
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
// Get available fields from metadata index
|
// Get available fields from metadata index
|
||||||
const fields = await brain.getAvailableFields()
|
const fields = await brain.getAvailableFields()
|
||||||
|
|
@ -166,6 +174,10 @@ export const insightsCommands = {
|
||||||
}))
|
}))
|
||||||
formatOutput(fieldsWithStats, options)
|
formatOutput(fieldsWithStats, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see insights() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
spinner.fail('Failed to get fields')
|
spinner.fail('Failed to get fields')
|
||||||
console.error(chalk.red('Fields analysis failed:', error.message))
|
console.error(chalk.red('Fields analysis failed:', error.message))
|
||||||
|
|
@ -186,6 +198,7 @@ export const insightsCommands = {
|
||||||
if (!field) {
|
if (!field) {
|
||||||
spinner = ora('Getting available fields...').start()
|
spinner = ora('Getting available fields...').start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
const availableFields = await brain.getAvailableFields()
|
const availableFields = await brain.getAvailableFields()
|
||||||
spinner.stop()
|
spinner.stop()
|
||||||
|
|
||||||
|
|
@ -202,6 +215,7 @@ export const insightsCommands = {
|
||||||
|
|
||||||
spinner = ora(`Getting values for field: ${field}...`).start()
|
spinner = ora(`Getting values for field: ${field}...`).start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
const values = await brain.getFieldValues(field)
|
const values = await brain.getFieldValues(field)
|
||||||
const limit = options.limit ? parseInt(options.limit) : 100
|
const limit = options.limit ? parseInt(options.limit) : 100
|
||||||
|
|
@ -242,6 +256,10 @@ export const insightsCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput({ field, values, count: values.length }, options)
|
formatOutput({ field, values, count: values.length }, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see insights() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
if (spinner) spinner.fail('Failed to get field values')
|
if (spinner) spinner.fail('Failed to get field values')
|
||||||
console.error(chalk.red('Field values failed:', error.message))
|
console.error(chalk.red('Field values failed:', error.message))
|
||||||
|
|
@ -289,6 +307,7 @@ export const insightsCommands = {
|
||||||
|
|
||||||
spinner = ora('Analyzing optimal query plan...').start()
|
spinner = ora('Analyzing optimal query plan...').start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
const plan = await brain.getOptimalQueryPlan(filters)
|
const plan = await brain.getOptimalQueryPlan(filters)
|
||||||
|
|
||||||
|
|
@ -328,6 +347,10 @@ export const insightsCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput({ filters, plan }, options)
|
formatOutput({ filters, plan }, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see insights() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
if (spinner) spinner.fail('Failed to generate query plan')
|
if (spinner) spinner.fail('Failed to generate query plan')
|
||||||
console.error(chalk.red('Query plan failed:', error.message))
|
console.error(chalk.red('Query plan failed:', error.message))
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,11 @@
|
||||||
/**
|
/**
|
||||||
* 🧠 Neural Similarity API Commands
|
* @module cli/commands/neural
|
||||||
*
|
* @description Neural CLI commands: semantic similarity, clustering,
|
||||||
* CLI interface for semantic similarity, clustering, and neural operations
|
* hierarchy, neighbors, outlier detection, and visualization data export.
|
||||||
|
* Registered in `src/cli/index.ts` as `similar` / `cluster` / `related` /
|
||||||
|
* `hierarchy` / `outliers` / `visualize`. Each command is one-shot: it
|
||||||
|
* initializes the shared Brainy instance, runs the neural operation, then
|
||||||
|
* closes the store and exits explicitly.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import inquirer from 'inquirer'
|
import inquirer from 'inquirer'
|
||||||
|
|
@ -25,136 +29,13 @@ interface CommandArguments {
|
||||||
_: string[];
|
_: string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
export const neuralCommand = {
|
let brainyInstance: Brainy | null = null
|
||||||
command: 'neural [action]',
|
|
||||||
describe: '🧠 Neural similarity and clustering operations',
|
|
||||||
|
|
||||||
builder: (yargs: any) => {
|
const getBrainy = (): Brainy => {
|
||||||
return yargs
|
if (!brainyInstance) {
|
||||||
.positional('action', {
|
brainyInstance = new Brainy()
|
||||||
describe: 'Neural operation to perform',
|
|
||||||
type: 'string',
|
|
||||||
choices: ['similar', 'clusters', 'hierarchy', 'neighbors', 'path', 'outliers', 'visualize']
|
|
||||||
})
|
|
||||||
.option('id', {
|
|
||||||
describe: 'Item ID for similarity operations',
|
|
||||||
type: 'string',
|
|
||||||
alias: 'i'
|
|
||||||
})
|
|
||||||
.option('query', {
|
|
||||||
describe: 'Query text for similarity search',
|
|
||||||
type: 'string',
|
|
||||||
alias: 'q'
|
|
||||||
})
|
|
||||||
.option('threshold', {
|
|
||||||
describe: 'Similarity threshold (0-1)',
|
|
||||||
type: 'number',
|
|
||||||
default: 0.7,
|
|
||||||
alias: 't'
|
|
||||||
})
|
|
||||||
.option('format', {
|
|
||||||
describe: 'Output format',
|
|
||||||
type: 'string',
|
|
||||||
choices: ['json', 'table', 'tree', 'graph'],
|
|
||||||
default: 'table',
|
|
||||||
alias: 'f'
|
|
||||||
})
|
|
||||||
.option('output', {
|
|
||||||
describe: 'Output file path',
|
|
||||||
type: 'string',
|
|
||||||
alias: 'o'
|
|
||||||
})
|
|
||||||
.option('limit', {
|
|
||||||
describe: 'Maximum number of results',
|
|
||||||
type: 'number',
|
|
||||||
default: 10,
|
|
||||||
alias: 'l'
|
|
||||||
})
|
|
||||||
.option('algorithm', {
|
|
||||||
describe: 'Clustering algorithm',
|
|
||||||
type: 'string',
|
|
||||||
choices: ['hierarchical', 'kmeans', 'dbscan', 'auto'],
|
|
||||||
default: 'auto',
|
|
||||||
alias: 'a'
|
|
||||||
})
|
|
||||||
.option('dimensions', {
|
|
||||||
describe: 'Visualization dimensions (2 or 3)',
|
|
||||||
type: 'number',
|
|
||||||
choices: [2, 3],
|
|
||||||
default: 2,
|
|
||||||
alias: 'd'
|
|
||||||
})
|
|
||||||
.option('explain', {
|
|
||||||
describe: 'Include detailed explanations',
|
|
||||||
type: 'boolean',
|
|
||||||
default: false,
|
|
||||||
alias: 'e'
|
|
||||||
})
|
|
||||||
},
|
|
||||||
|
|
||||||
handler: async (argv: CommandArguments) => {
|
|
||||||
console.log(chalk.cyan('\n🧠 NEURAL SIMILARITY API'))
|
|
||||||
console.log(chalk.gray('━'.repeat(50)))
|
|
||||||
|
|
||||||
// Initialize Brainy and Neural API
|
|
||||||
const brain = new Brainy()
|
|
||||||
const neural = brain.neural()
|
|
||||||
|
|
||||||
try {
|
|
||||||
const action = argv.action || await promptForAction()
|
|
||||||
|
|
||||||
switch (action) {
|
|
||||||
case 'similar':
|
|
||||||
await handleSimilarCommand(neural, argv)
|
|
||||||
break
|
|
||||||
case 'clusters':
|
|
||||||
await handleClustersCommand(neural, argv)
|
|
||||||
break
|
|
||||||
case 'hierarchy':
|
|
||||||
await handleHierarchyCommand(neural, argv)
|
|
||||||
break
|
|
||||||
case 'neighbors':
|
|
||||||
await handleNeighborsCommand(neural, argv)
|
|
||||||
break
|
|
||||||
case 'path':
|
|
||||||
console.log(chalk.yellow('\n⚠️ Semantic path finding coming soon'))
|
|
||||||
console.log(chalk.dim('This feature requires implementing graph traversal algorithms'))
|
|
||||||
console.log(chalk.dim('Use "neighbors" and "hierarchy" commands to explore connections'))
|
|
||||||
break
|
|
||||||
case 'outliers':
|
|
||||||
await handleOutliersCommand(neural, argv)
|
|
||||||
break
|
|
||||||
case 'visualize':
|
|
||||||
await handleVisualizeCommand(neural, argv)
|
|
||||||
break
|
|
||||||
default:
|
|
||||||
console.log(chalk.red(`❌ Unknown action: ${action}`))
|
|
||||||
showHelp()
|
|
||||||
}
|
}
|
||||||
} catch (error) {
|
return brainyInstance
|
||||||
console.error(chalk.red('💥 Error:'), error instanceof Error ? error.message : error)
|
|
||||||
process.exit(1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async function promptForAction(): Promise<string> {
|
|
||||||
const answer = await inquirer.prompt([{
|
|
||||||
type: 'list',
|
|
||||||
name: 'action',
|
|
||||||
message: 'Choose a neural operation:',
|
|
||||||
choices: [
|
|
||||||
{ name: '🔗 Calculate similarity between items', value: 'similar' },
|
|
||||||
{ name: '🎯 Find semantic clusters', value: 'clusters' },
|
|
||||||
{ name: '🌳 Show item hierarchy', value: 'hierarchy' },
|
|
||||||
{ name: '🕸️ Find semantic neighbors', value: 'neighbors' },
|
|
||||||
{ name: '🛣️ Find semantic path between items (coming soon)', value: 'path', disabled: true },
|
|
||||||
{ name: '🚨 Detect outliers', value: 'outliers' },
|
|
||||||
{ name: '📊 Generate visualization data', value: 'visualize' }
|
|
||||||
]
|
|
||||||
}])
|
|
||||||
|
|
||||||
return answer.action
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async function handleSimilarCommand(neural: any, argv: CommandArguments): Promise<void> {
|
async function handleSimilarCommand(neural: any, argv: CommandArguments): Promise<void> {
|
||||||
|
|
@ -436,63 +317,6 @@ async function handleNeighborsCommand(neural: any, argv: CommandArguments): Prom
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async function handlePathCommand(neural: any, argv: CommandArguments): Promise<void> {
|
|
||||||
const spinner = ora('🛣️ Finding semantic path...').start()
|
|
||||||
|
|
||||||
try {
|
|
||||||
let fromId: string, toId: string
|
|
||||||
|
|
||||||
if (argv._ && argv._.length >= 3) {
|
|
||||||
fromId = argv._[1]
|
|
||||||
toId = argv._[2]
|
|
||||||
} else {
|
|
||||||
spinner.stop()
|
|
||||||
const answers = await inquirer.prompt([
|
|
||||||
{
|
|
||||||
type: 'input',
|
|
||||||
name: 'from',
|
|
||||||
message: 'From item ID:',
|
|
||||||
validate: (input: string) => input.length > 0
|
|
||||||
},
|
|
||||||
{
|
|
||||||
type: 'input',
|
|
||||||
name: 'to',
|
|
||||||
message: 'To item ID:',
|
|
||||||
validate: (input: string) => input.length > 0
|
|
||||||
}
|
|
||||||
])
|
|
||||||
fromId = answers.from
|
|
||||||
toId = answers.to
|
|
||||||
spinner.start()
|
|
||||||
}
|
|
||||||
|
|
||||||
const path = await neural.semanticPath(fromId, toId)
|
|
||||||
|
|
||||||
if (path.length === 0) {
|
|
||||||
spinner.warn('🚫 No semantic path found')
|
|
||||||
console.log(`No path found between ${chalk.cyan(fromId)} and ${chalk.cyan(toId)}`)
|
|
||||||
} else {
|
|
||||||
spinner.succeed(`✅ Found path with ${path.length} hops`)
|
|
||||||
|
|
||||||
console.log(`\n🛣️ Semantic Path from ${chalk.cyan(fromId)} to ${chalk.cyan(toId)}:`)
|
|
||||||
console.log(`${chalk.cyan(fromId)} (start)`)
|
|
||||||
|
|
||||||
path.forEach((hop, index) => {
|
|
||||||
console.log(`${' '.repeat(index + 1)}↓ ${(hop.similarity * 100).toFixed(1)}%`)
|
|
||||||
console.log(`${' '.repeat(index + 1)}${hop.id} (hop ${hop.hop})`)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
if (argv.output) {
|
|
||||||
await saveToFile(argv.output, path, argv.format!)
|
|
||||||
}
|
|
||||||
|
|
||||||
} catch (error) {
|
|
||||||
spinner.fail('💥 Failed to find path')
|
|
||||||
throw error
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async function handleOutliersCommand(neural: any, argv: CommandArguments): Promise<void> {
|
async function handleOutliersCommand(neural: any, argv: CommandArguments): Promise<void> {
|
||||||
const spinner = ora('🚨 Detecting semantic outliers...').start()
|
const spinner = ora('🚨 Detecting semantic outliers...').start()
|
||||||
|
|
||||||
|
|
@ -592,32 +416,32 @@ function formatAsTable(data: any): string {
|
||||||
return JSON.stringify(data, null, 2)
|
return JSON.stringify(data, null, 2)
|
||||||
}
|
}
|
||||||
|
|
||||||
function showHelp(): void {
|
/**
|
||||||
console.log('\n🧠 Neural Similarity API Commands:')
|
* @description Run a neural CLI handler with the one-shot lifecycle every
|
||||||
console.log('')
|
* Brainy command follows: init → work → `close()` → explicit `process.exit`.
|
||||||
console.log(' brainy neural similar <item1> <item2> Calculate similarity')
|
* close() releases the writer lock and indexes, but global timers
|
||||||
console.log(' brainy neural clusters Find semantic clusters')
|
* (UnifiedCache bookkeeping, PathResolver stats) keep the event loop alive,
|
||||||
console.log(' brainy neural hierarchy <id> Show item hierarchy')
|
* so one-shot commands must exit explicitly.
|
||||||
console.log(' brainy neural neighbors <id> Find semantic neighbors')
|
* @param work - The handler body, given the initialized neural API.
|
||||||
console.log(' brainy neural path <from> <to> Find semantic path')
|
*/
|
||||||
console.log(' brainy neural outliers Detect outliers')
|
async function runNeuralCommand(work: (neural: ReturnType<Brainy['neural']>) => Promise<void>): Promise<void> {
|
||||||
console.log(' brainy neural visualize Generate visualization data')
|
try {
|
||||||
console.log('')
|
const brain = getBrainy()
|
||||||
console.log('Options:')
|
await brain.init()
|
||||||
console.log(' --threshold, -t Similarity threshold (0-1)')
|
|
||||||
console.log(' --format, -f Output format (json|table|tree|graph)')
|
await work(brain.neural())
|
||||||
console.log(' --output, -o Save to file')
|
|
||||||
console.log(' --limit, -l Maximum results')
|
await brain.close()
|
||||||
console.log(' --explain, -e Include explanations')
|
process.exit(0)
|
||||||
console.log('')
|
} catch (error) {
|
||||||
|
console.error(chalk.red('💥 Error:'), error instanceof Error ? error.message : error)
|
||||||
|
process.exit(1)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Commander-compatible wrappers
|
// Commander-compatible wrappers
|
||||||
export const neuralCommands = {
|
export const neuralCommands = {
|
||||||
async similar(a?: string, b?: string, options?: any) {
|
async similar(a?: string, b?: string, options?: any) {
|
||||||
const brain = new Brainy()
|
|
||||||
const neural = brain.neural()
|
|
||||||
|
|
||||||
// Build argv-style object for handler
|
// Build argv-style object for handler
|
||||||
const argv: CommandArguments = {
|
const argv: CommandArguments = {
|
||||||
_: ['neural', 'similar', a || '', b || ''].filter(x => x),
|
_: ['neural', 'similar', a || '', b || ''].filter(x => x),
|
||||||
|
|
@ -627,13 +451,10 @@ export const neuralCommands = {
|
||||||
...options
|
...options
|
||||||
}
|
}
|
||||||
|
|
||||||
await handleSimilarCommand(neural, argv)
|
await runNeuralCommand((neural) => handleSimilarCommand(neural, argv))
|
||||||
},
|
},
|
||||||
|
|
||||||
async cluster(options?: any) {
|
async cluster(options?: any) {
|
||||||
const brain = new Brainy()
|
|
||||||
const neural = brain.neural()
|
|
||||||
|
|
||||||
const argv: CommandArguments = {
|
const argv: CommandArguments = {
|
||||||
_: ['neural', 'cluster'],
|
_: ['neural', 'cluster'],
|
||||||
algorithm: options?.algorithm || 'hierarchical',
|
algorithm: options?.algorithm || 'hierarchical',
|
||||||
|
|
@ -643,26 +464,20 @@ export const neuralCommands = {
|
||||||
...options
|
...options
|
||||||
}
|
}
|
||||||
|
|
||||||
await handleClustersCommand(neural, argv)
|
await runNeuralCommand((neural) => handleClustersCommand(neural, argv))
|
||||||
},
|
},
|
||||||
|
|
||||||
async hierarchy(id?: string, options?: any) {
|
async hierarchy(id?: string, options?: any) {
|
||||||
const brain = new Brainy()
|
|
||||||
const neural = brain.neural()
|
|
||||||
|
|
||||||
const argv: CommandArguments = {
|
const argv: CommandArguments = {
|
||||||
_: ['neural', 'hierarchy', id || ''].filter(x => x),
|
_: ['neural', 'hierarchy', id || ''].filter(x => x),
|
||||||
id,
|
id,
|
||||||
...options
|
...options
|
||||||
}
|
}
|
||||||
|
|
||||||
await handleHierarchyCommand(neural, argv)
|
await runNeuralCommand((neural) => handleHierarchyCommand(neural, argv))
|
||||||
},
|
},
|
||||||
|
|
||||||
async related(id?: string, options?: any) {
|
async related(id?: string, options?: any) {
|
||||||
const brain = new Brainy()
|
|
||||||
const neural = brain.neural()
|
|
||||||
|
|
||||||
const argv: CommandArguments = {
|
const argv: CommandArguments = {
|
||||||
_: ['neural', 'related', id || ''].filter(x => x),
|
_: ['neural', 'related', id || ''].filter(x => x),
|
||||||
id,
|
id,
|
||||||
|
|
@ -671,13 +486,10 @@ export const neuralCommands = {
|
||||||
...options
|
...options
|
||||||
}
|
}
|
||||||
|
|
||||||
await handleNeighborsCommand(neural, argv)
|
await runNeuralCommand((neural) => handleNeighborsCommand(neural, argv))
|
||||||
},
|
},
|
||||||
|
|
||||||
async outliers(options?: any) {
|
async outliers(options?: any) {
|
||||||
const brain = new Brainy()
|
|
||||||
const neural = brain.neural()
|
|
||||||
|
|
||||||
const argv: CommandArguments = {
|
const argv: CommandArguments = {
|
||||||
_: ['neural', 'outliers'],
|
_: ['neural', 'outliers'],
|
||||||
threshold: options?.threshold ? parseFloat(options.threshold) : 0.3,
|
threshold: options?.threshold ? parseFloat(options.threshold) : 0.3,
|
||||||
|
|
@ -685,13 +497,10 @@ export const neuralCommands = {
|
||||||
...options
|
...options
|
||||||
}
|
}
|
||||||
|
|
||||||
await handleOutliersCommand(neural, argv)
|
await runNeuralCommand((neural) => handleOutliersCommand(neural, argv))
|
||||||
},
|
},
|
||||||
|
|
||||||
async visualize(options?: any) {
|
async visualize(options?: any) {
|
||||||
const brain = new Brainy()
|
|
||||||
const neural = brain.neural()
|
|
||||||
|
|
||||||
const argv: CommandArguments = {
|
const argv: CommandArguments = {
|
||||||
_: ['neural', 'visualize'],
|
_: ['neural', 'visualize'],
|
||||||
format: options?.format || 'json',
|
format: options?.format || 'json',
|
||||||
|
|
@ -701,8 +510,6 @@ export const neuralCommands = {
|
||||||
...options
|
...options
|
||||||
}
|
}
|
||||||
|
|
||||||
await handleVisualizeCommand(neural, argv)
|
await runNeuralCommand((neural) => handleVisualizeCommand(neural, argv))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
export default neuralCommand
|
|
||||||
|
|
@ -62,6 +62,7 @@ export const nlpCommands = {
|
||||||
|
|
||||||
spinner = ora('Extracting entities with neural NLP...').start()
|
spinner = ora('Extracting entities with neural NLP...').start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
// Extract entities using Brainy's neural entity extractor
|
// Extract entities using Brainy's neural entity extractor
|
||||||
const entities = await brain.extract(text)
|
const entities = await brain.extract(text)
|
||||||
|
|
@ -105,6 +106,12 @@ export const nlpCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput(entities, options)
|
formatOutput(entities, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// close() releases the writer lock and indexes, but global timers
|
||||||
|
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
|
||||||
|
// alive. CLI commands are one-shot — exit explicitly.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
if (spinner) spinner.fail('Entity extraction failed')
|
if (spinner) spinner.fail('Entity extraction failed')
|
||||||
console.error(chalk.red('Extraction failed:', error.message))
|
console.error(chalk.red('Extraction failed:', error.message))
|
||||||
|
|
@ -147,6 +154,7 @@ export const nlpCommands = {
|
||||||
|
|
||||||
spinner = ora('Extracting concepts with neural analysis...').start()
|
spinner = ora('Extracting concepts with neural analysis...').start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
const confidence = options.threshold ? parseFloat(options.threshold) : 0.5
|
const confidence = options.threshold ? parseFloat(options.threshold) : 0.5
|
||||||
const concepts = await brain.extractConcepts(text, { confidence })
|
const concepts = await brain.extractConcepts(text, { confidence })
|
||||||
|
|
@ -171,6 +179,10 @@ export const nlpCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput(concepts, options)
|
formatOutput(concepts, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see extract() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
if (spinner) spinner.fail('Concept extraction failed')
|
if (spinner) spinner.fail('Concept extraction failed')
|
||||||
console.error(chalk.red('Extraction failed:', error.message))
|
console.error(chalk.red('Extraction failed:', error.message))
|
||||||
|
|
@ -201,6 +213,7 @@ export const nlpCommands = {
|
||||||
|
|
||||||
spinner = ora('Analyzing text with neural NLP...').start()
|
spinner = ora('Analyzing text with neural NLP...').start()
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
// Run both entity extraction and concept extraction
|
// Run both entity extraction and concept extraction
|
||||||
const [entities, concepts] = await Promise.all([
|
const [entities, concepts] = await Promise.all([
|
||||||
|
|
@ -265,6 +278,10 @@ export const nlpCommands = {
|
||||||
concepts
|
concepts
|
||||||
}, options)
|
}, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see extract() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
if (spinner) spinner.fail('Analysis failed')
|
if (spinner) spinner.fail('Analysis failed')
|
||||||
console.error(chalk.red('Analysis failed:', error.message))
|
console.error(chalk.red('Analysis failed:', error.message))
|
||||||
|
|
|
||||||
|
|
@ -1,16 +1,33 @@
|
||||||
/**
|
/**
|
||||||
* 💾 Storage Management Commands
|
* @module cli/commands/storage
|
||||||
|
* @description Storage management CLI commands for the 8.0 storage surface
|
||||||
|
* (filesystem + memory adapters).
|
||||||
*
|
*
|
||||||
* Modern interactive CLI for storage lifecycle, cost optimization, and management
|
* - `storage status` — backend, root directory, entity/relationship counts,
|
||||||
|
* writer-lock holder, and operational mode (rendered from `brain.stats()`).
|
||||||
|
* - `storage batch-delete <file>` — delete a newline-separated list of entity
|
||||||
|
* IDs through the public `brain.delete()` path (vectors, metadata, indexes,
|
||||||
|
* and statistics all stay consistent), with per-ID retry and an optional
|
||||||
|
* continue-on-error mode.
|
||||||
|
*
|
||||||
|
* The 7.x cloud-era subcommands (lifecycle policies, tiering, cost
|
||||||
|
* estimation, runtime compression toggles) were removed with the cloud
|
||||||
|
* storage adapters in 8.0. Filesystem gzip compression is a constructor
|
||||||
|
* option (`storage.options.compression`, on by default), not a runtime
|
||||||
|
* toggle.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import chalk from 'chalk'
|
import chalk from 'chalk'
|
||||||
import ora from 'ora'
|
import ora from 'ora'
|
||||||
import inquirer from 'inquirer'
|
import inquirer from 'inquirer'
|
||||||
import Table from 'cli-table3'
|
import Table from 'cli-table3'
|
||||||
import { readFileSync, writeFileSync } from 'node:fs'
|
import { readFileSync } from 'node:fs'
|
||||||
import { Brainy } from '../../brainy.js'
|
import { Brainy } from '../../brainy.js'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Shared CLI flags every storage command accepts (`--verbose`,
|
||||||
|
* `--json`, `--pretty`), mirroring the other command modules.
|
||||||
|
*/
|
||||||
interface StorageOptions {
|
interface StorageOptions {
|
||||||
verbose?: boolean
|
verbose?: boolean
|
||||||
json?: boolean
|
json?: boolean
|
||||||
|
|
@ -19,6 +36,11 @@ interface StorageOptions {
|
||||||
|
|
||||||
let brainyInstance: Brainy | null = null
|
let brainyInstance: Brainy | null = null
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Lazily construct the module's shared `Brainy` instance so
|
||||||
|
* repeated handler invocations (and tests) reuse one store handle.
|
||||||
|
* @returns The shared `Brainy` instance.
|
||||||
|
*/
|
||||||
const getBrainy = (): Brainy => {
|
const getBrainy = (): Brainy => {
|
||||||
if (!brainyInstance) {
|
if (!brainyInstance) {
|
||||||
brainyInstance = new Brainy()
|
brainyInstance = new Brainy()
|
||||||
|
|
@ -26,111 +48,93 @@ const getBrainy = (): Brainy => {
|
||||||
return brainyInstance
|
return brainyInstance
|
||||||
}
|
}
|
||||||
|
|
||||||
const formatBytes = (bytes: number): string => {
|
/**
|
||||||
if (bytes === 0) return '0 B'
|
* @description Emit machine-readable output when `--json` is set; honors
|
||||||
const k = 1024
|
* `--pretty` for indented JSON.
|
||||||
const sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB']
|
* @param data - The value to serialize.
|
||||||
const i = Math.floor(Math.log(bytes) / Math.log(k))
|
* @param options - Shared CLI flags.
|
||||||
return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i]
|
*/
|
||||||
}
|
const formatOutput = (data: unknown, options: StorageOptions): void => {
|
||||||
|
|
||||||
const formatCurrency = (amount: number): string => {
|
|
||||||
return new Intl.NumberFormat('en-US', {
|
|
||||||
style: 'currency',
|
|
||||||
currency: 'USD',
|
|
||||||
minimumFractionDigits: 0,
|
|
||||||
maximumFractionDigits: 0
|
|
||||||
}).format(amount)
|
|
||||||
}
|
|
||||||
|
|
||||||
const formatOutput = (data: any, options: StorageOptions): void => {
|
|
||||||
if (options.json) {
|
if (options.json) {
|
||||||
console.log(options.pretty ? JSON.stringify(data, null, 2) : JSON.stringify(data))
|
console.log(options.pretty ? JSON.stringify(data, null, 2) : JSON.stringify(data))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description The `storage status` / `storage batch-delete` command handlers
|
||||||
|
* registered in `src/cli/index.ts`.
|
||||||
|
*/
|
||||||
export const storageCommands = {
|
export const storageCommands = {
|
||||||
/**
|
/**
|
||||||
* Show storage status and health
|
* @description Show storage status: backend adapter, root directory,
|
||||||
|
* entity/relationship counts, writer-lock holder, and operational mode.
|
||||||
|
* @param options - Shared CLI flags.
|
||||||
|
* @example
|
||||||
|
* ```bash
|
||||||
|
* brainy storage status # human-readable report
|
||||||
|
* brainy storage status --json # machine-readable
|
||||||
|
* ```
|
||||||
*/
|
*/
|
||||||
async status(options: StorageOptions & { detailed?: boolean; quota?: boolean }) {
|
async status(options: StorageOptions) {
|
||||||
const spinner = ora('Checking storage status...').start()
|
const spinner = ora('Checking storage status...').start()
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
const storage = (brain as any).storage
|
await brain.init()
|
||||||
|
|
||||||
const status = await storage.getStorageStatus()
|
const stats = await brain.stats()
|
||||||
|
|
||||||
spinner.succeed('Storage status retrieved')
|
spinner.succeed('Storage status retrieved')
|
||||||
|
|
||||||
if (options.json) {
|
if (options.json) {
|
||||||
formatOutput(status, options)
|
formatOutput(
|
||||||
return
|
{
|
||||||
|
backend: stats.storage.backend,
|
||||||
|
rootDir: stats.storage.rootDir,
|
||||||
|
entityCount: stats.entityCount,
|
||||||
|
relationCount: stats.relationCount,
|
||||||
|
mode: stats.mode,
|
||||||
|
writerLock: stats.writerLock ?? null
|
||||||
|
},
|
||||||
|
options
|
||||||
|
)
|
||||||
|
// close() releases the writer lock and indexes, but global timers
|
||||||
|
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
|
||||||
|
// alive. CLI commands are one-shot — exit explicitly.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
}
|
}
|
||||||
|
|
||||||
console.log(chalk.cyan('\n💾 Storage Status\n'))
|
console.log(chalk.cyan('\n💾 Storage Status\n'))
|
||||||
|
|
||||||
// Basic info table
|
|
||||||
const infoTable = new Table({
|
const infoTable = new Table({
|
||||||
head: [chalk.cyan('Property'), chalk.cyan('Value')],
|
head: [chalk.cyan('Property'), chalk.cyan('Value')],
|
||||||
style: { head: [], border: [] }
|
style: { head: [], border: [] }
|
||||||
})
|
})
|
||||||
|
|
||||||
infoTable.push(
|
infoTable.push(
|
||||||
['Type', chalk.green(status.type || 'Unknown')],
|
['Backend', chalk.green(stats.storage.backend)],
|
||||||
['Status', status.healthy ? chalk.green('✓ Healthy') : chalk.red('✗ Unhealthy')]
|
['Entities', String(stats.entityCount)],
|
||||||
|
['Relationships', String(stats.relationCount)],
|
||||||
|
['Mode', stats.mode]
|
||||||
)
|
)
|
||||||
|
if (stats.storage.rootDir) {
|
||||||
if (status.details) {
|
infoTable.push(['Root', chalk.dim(stats.storage.rootDir)])
|
||||||
if (status.details.bucket) {
|
|
||||||
infoTable.push(['Bucket', status.details.bucket])
|
|
||||||
}
|
|
||||||
if (status.details.region) {
|
|
||||||
infoTable.push(['Region', status.details.region])
|
|
||||||
}
|
|
||||||
if (status.details.path) {
|
|
||||||
infoTable.push(['Path', status.details.path])
|
|
||||||
}
|
|
||||||
if (status.details.compression !== undefined) {
|
|
||||||
infoTable.push(['Compression', status.details.compression ? chalk.green('Enabled') : chalk.dim('Disabled')])
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
console.log(infoTable.toString())
|
console.log(infoTable.toString())
|
||||||
|
|
||||||
// Quota info (for OPFS)
|
if (stats.writerLock) {
|
||||||
if (options.quota && status.details?.quota) {
|
console.log(chalk.bold('\nWriter Lock:'))
|
||||||
console.log(chalk.cyan('\n📊 Quota Information\n'))
|
console.log(
|
||||||
|
` PID ${stats.writerLock.pid} on ${stats.writerLock.hostname} ` +
|
||||||
const quotaTable = new Table({
|
chalk.dim(`(since ${stats.writerLock.startedAt})`)
|
||||||
head: [chalk.cyan('Metric'), chalk.cyan('Value')],
|
|
||||||
style: { head: [], border: [] }
|
|
||||||
})
|
|
||||||
|
|
||||||
const usagePercent = status.details.usagePercent || 0
|
|
||||||
const usageColor = usagePercent > 80 ? chalk.red : usagePercent > 60 ? chalk.yellow : chalk.green
|
|
||||||
|
|
||||||
quotaTable.push(
|
|
||||||
['Usage', formatBytes(status.details.usage)],
|
|
||||||
['Quota', formatBytes(status.details.quota)],
|
|
||||||
['Used', usageColor(`${usagePercent.toFixed(1)}%`)]
|
|
||||||
)
|
)
|
||||||
|
|
||||||
console.log(quotaTable.toString())
|
|
||||||
|
|
||||||
if (usagePercent > 80) {
|
|
||||||
console.log(chalk.yellow('\n⚠️ Warning: Approaching quota limit!'))
|
|
||||||
console.log(chalk.dim(' Consider cleaning up old data or requesting more quota'))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Detailed info
|
|
||||||
if (options.detailed && status.details) {
|
|
||||||
console.log(chalk.cyan('\n🔍 Detailed Information\n'))
|
|
||||||
console.log(chalk.dim(JSON.stringify(status.details, null, 2)))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see the --json branch for why the explicit exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
spinner.fail('Failed to get storage status')
|
spinner.fail('Failed to get storage status')
|
||||||
console.error(chalk.red(error.message))
|
console.error(chalk.red(error.message))
|
||||||
|
|
@ -139,474 +143,14 @@ export const storageCommands = {
|
||||||
},
|
},
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Lifecycle policy management
|
* @description Delete a list of entity IDs (one per line in a file) through
|
||||||
*/
|
* the public `brain.delete()` path so vectors, metadata, graph edges,
|
||||||
lifecycle: {
|
* indexes, and statistics all stay consistent. Asks for confirmation before
|
||||||
/**
|
* deleting; failed IDs are retried up to `--max-retries` times, and
|
||||||
* Set lifecycle policy (interactive or from file)
|
* `--continue-on-error` keeps going past IDs that still fail.
|
||||||
*/
|
* @param file - Path to a newline-separated file of entity IDs.
|
||||||
async set(configFile?: string, options: StorageOptions & { validate?: boolean } = {}) {
|
* @param options - Shared CLI flags plus `--max-retries` and
|
||||||
const brain = getBrainy()
|
* `--continue-on-error`.
|
||||||
const storage = (brain as any).storage
|
|
||||||
|
|
||||||
let policy: any
|
|
||||||
|
|
||||||
if (configFile) {
|
|
||||||
// Load from file
|
|
||||||
const spinner = ora('Loading policy from file...').start()
|
|
||||||
try {
|
|
||||||
const content = readFileSync(configFile, 'utf-8')
|
|
||||||
policy = JSON.parse(content)
|
|
||||||
spinner.succeed('Policy loaded')
|
|
||||||
} catch (error: any) {
|
|
||||||
spinner.fail('Failed to load policy file')
|
|
||||||
console.error(chalk.red(error.message))
|
|
||||||
process.exit(1)
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
// Interactive mode
|
|
||||||
console.log(chalk.cyan('\n📋 Lifecycle Policy Builder\n'))
|
|
||||||
|
|
||||||
const storageStatus = await storage.getStorageStatus()
|
|
||||||
const storageType = storageStatus.type
|
|
||||||
|
|
||||||
// Detect storage provider
|
|
||||||
let provider: 'aws' | 'gcs' | 'azure' | 'r2' | 'unknown' = 'unknown'
|
|
||||||
if (storageType === 's3-compatible') {
|
|
||||||
const endpoint = storageStatus.details?.endpoint || ''
|
|
||||||
if (endpoint.includes('r2.cloudflarestorage.com')) {
|
|
||||||
provider = 'r2'
|
|
||||||
} else if (endpoint.includes('amazonaws.com')) {
|
|
||||||
provider = 'aws'
|
|
||||||
}
|
|
||||||
} else if (storageType === 'gcs') {
|
|
||||||
provider = 'gcs'
|
|
||||||
} else if (storageType === 'azure') {
|
|
||||||
provider = 'azure'
|
|
||||||
}
|
|
||||||
|
|
||||||
if (provider === 'unknown') {
|
|
||||||
console.log(chalk.yellow('⚠️ Could not detect storage provider'))
|
|
||||||
console.log(chalk.dim('Lifecycle policies require: AWS S3, GCS, or Azure Blob Storage'))
|
|
||||||
process.exit(1)
|
|
||||||
}
|
|
||||||
|
|
||||||
console.log(chalk.green(`✓ Detected: ${provider.toUpperCase()}\n`))
|
|
||||||
|
|
||||||
// Provider-specific interactive prompts
|
|
||||||
if (provider === 'aws' || provider === 'r2') {
|
|
||||||
const answers = await inquirer.prompt([
|
|
||||||
{
|
|
||||||
type: 'input',
|
|
||||||
name: 'prefix',
|
|
||||||
message: 'Path prefix to apply policy to:',
|
|
||||||
default: 'entities/',
|
|
||||||
validate: (input: string) => input.length > 0
|
|
||||||
},
|
|
||||||
{
|
|
||||||
type: 'list',
|
|
||||||
name: 'strategy',
|
|
||||||
message: 'Choose optimization strategy:',
|
|
||||||
choices: [
|
|
||||||
{ name: '🎯 Intelligent-Tiering (Recommended - Automatic)', value: 'intelligent' },
|
|
||||||
{ name: '📅 Lifecycle Policies (Manual tier transitions)', value: 'lifecycle' },
|
|
||||||
{ name: '🚀 Aggressive Archival (Maximum savings)', value: 'aggressive' }
|
|
||||||
]
|
|
||||||
}
|
|
||||||
])
|
|
||||||
|
|
||||||
if (answers.strategy === 'intelligent') {
|
|
||||||
// Intelligent-Tiering
|
|
||||||
const tierAnswers = await inquirer.prompt([
|
|
||||||
{
|
|
||||||
type: 'input',
|
|
||||||
name: 'configName',
|
|
||||||
message: 'Configuration name:',
|
|
||||||
default: 'brainy-auto-tier'
|
|
||||||
}
|
|
||||||
])
|
|
||||||
|
|
||||||
const spinner = ora('Enabling Intelligent-Tiering...').start()
|
|
||||||
try {
|
|
||||||
await storage.enableIntelligentTiering(answers.prefix, tierAnswers.configName)
|
|
||||||
spinner.succeed('Intelligent-Tiering enabled!')
|
|
||||||
|
|
||||||
console.log(chalk.cyan('\n💰 Cost Impact:\n'))
|
|
||||||
console.log(chalk.green('✓ Automatic optimization based on access patterns'))
|
|
||||||
console.log(chalk.green('✓ No retrieval fees'))
|
|
||||||
console.log(chalk.green('✓ Expected savings: 50-70%'))
|
|
||||||
console.log(chalk.dim('\nObjects automatically move between tiers:'))
|
|
||||||
console.log(chalk.dim(' • Frequent Access Tier (accessed within 30 days)'))
|
|
||||||
console.log(chalk.dim(' • Infrequent Access Tier (not accessed for 30+ days)'))
|
|
||||||
console.log(chalk.dim(' • Archive Instant Access Tier (not accessed for 90+ days)'))
|
|
||||||
|
|
||||||
return
|
|
||||||
} catch (error: any) {
|
|
||||||
spinner.fail('Failed to enable Intelligent-Tiering')
|
|
||||||
console.error(chalk.red(error.message))
|
|
||||||
process.exit(1)
|
|
||||||
}
|
|
||||||
} else if (answers.strategy === 'lifecycle') {
|
|
||||||
// Custom lifecycle policy
|
|
||||||
const lifecycleAnswers = await inquirer.prompt([
|
|
||||||
{
|
|
||||||
type: 'number',
|
|
||||||
name: 'standardIA',
|
|
||||||
message: 'Move to Standard-IA after (days):',
|
|
||||||
default: 30,
|
|
||||||
validate: (input: number) => input > 0
|
|
||||||
},
|
|
||||||
{
|
|
||||||
type: 'number',
|
|
||||||
name: 'glacier',
|
|
||||||
message: 'Move to Glacier after (days):',
|
|
||||||
default: 90,
|
|
||||||
validate: (input: number) => input > 0
|
|
||||||
},
|
|
||||||
{
|
|
||||||
type: 'number',
|
|
||||||
name: 'deepArchive',
|
|
||||||
message: 'Move to Deep Archive after (days):',
|
|
||||||
default: 365,
|
|
||||||
validate: (input: number) => input > 0
|
|
||||||
}
|
|
||||||
])
|
|
||||||
|
|
||||||
policy = {
|
|
||||||
rules: [{
|
|
||||||
id: 'brainy-lifecycle',
|
|
||||||
prefix: answers.prefix,
|
|
||||||
status: 'Enabled',
|
|
||||||
transitions: [
|
|
||||||
{ days: lifecycleAnswers.standardIA, storageClass: 'STANDARD_IA' },
|
|
||||||
{ days: lifecycleAnswers.glacier, storageClass: 'GLACIER' },
|
|
||||||
{ days: lifecycleAnswers.deepArchive, storageClass: 'DEEP_ARCHIVE' }
|
|
||||||
]
|
|
||||||
}]
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
// Aggressive archival
|
|
||||||
policy = {
|
|
||||||
rules: [{
|
|
||||||
id: 'brainy-aggressive',
|
|
||||||
prefix: answers.prefix,
|
|
||||||
status: 'Enabled',
|
|
||||||
transitions: [
|
|
||||||
{ days: 7, storageClass: 'STANDARD_IA' },
|
|
||||||
{ days: 30, storageClass: 'GLACIER' },
|
|
||||||
{ days: 90, storageClass: 'DEEP_ARCHIVE' }
|
|
||||||
]
|
|
||||||
}]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} else if (provider === 'gcs') {
|
|
||||||
// GCS Autoclass
|
|
||||||
const answers = await inquirer.prompt([
|
|
||||||
{
|
|
||||||
type: 'confirm',
|
|
||||||
name: 'useAutoclass',
|
|
||||||
message: 'Enable Autoclass (automatic tier management)?',
|
|
||||||
default: true
|
|
||||||
}
|
|
||||||
])
|
|
||||||
|
|
||||||
if (answers.useAutoclass) {
|
|
||||||
const autoclassAnswers = await inquirer.prompt([
|
|
||||||
{
|
|
||||||
type: 'list',
|
|
||||||
name: 'terminalClass',
|
|
||||||
message: 'Terminal storage class:',
|
|
||||||
choices: [
|
|
||||||
{ name: 'Archive (Lowest cost)', value: 'ARCHIVE' },
|
|
||||||
{ name: 'Nearline (Balance)', value: 'NEARLINE' }
|
|
||||||
],
|
|
||||||
default: 'ARCHIVE'
|
|
||||||
}
|
|
||||||
])
|
|
||||||
|
|
||||||
const spinner = ora('Enabling Autoclass...').start()
|
|
||||||
try {
|
|
||||||
await storage.enableAutoclass({ terminalStorageClass: autoclassAnswers.terminalClass })
|
|
||||||
spinner.succeed('Autoclass enabled!')
|
|
||||||
|
|
||||||
console.log(chalk.cyan('\n💰 Cost Impact:\n'))
|
|
||||||
console.log(chalk.green('✓ Automatic optimization (no manual policies needed)'))
|
|
||||||
console.log(chalk.green('✓ Expected savings: 60-94%'))
|
|
||||||
console.log(chalk.dim('\nObjects automatically move:'))
|
|
||||||
console.log(chalk.dim(' • Standard → Nearline → Coldline → Archive'))
|
|
||||||
console.log(chalk.dim(' • Based on access patterns'))
|
|
||||||
|
|
||||||
return
|
|
||||||
} catch (error: any) {
|
|
||||||
spinner.fail('Failed to enable Autoclass')
|
|
||||||
console.error(chalk.red(error.message))
|
|
||||||
process.exit(1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} else if (provider === 'azure') {
|
|
||||||
// Azure lifecycle
|
|
||||||
const answers = await inquirer.prompt([
|
|
||||||
{
|
|
||||||
type: 'number',
|
|
||||||
name: 'coolAfter',
|
|
||||||
message: 'Move to Cool tier after (days):',
|
|
||||||
default: 30
|
|
||||||
},
|
|
||||||
{
|
|
||||||
type: 'number',
|
|
||||||
name: 'archiveAfter',
|
|
||||||
message: 'Move to Archive tier after (days):',
|
|
||||||
default: 90
|
|
||||||
}
|
|
||||||
])
|
|
||||||
|
|
||||||
policy = {
|
|
||||||
rules: [{
|
|
||||||
name: 'brainy-lifecycle',
|
|
||||||
enabled: true,
|
|
||||||
type: 'Lifecycle',
|
|
||||||
definition: {
|
|
||||||
filters: { blobTypes: ['blockBlob'] },
|
|
||||||
actions: {
|
|
||||||
baseBlob: {
|
|
||||||
tierToCool: { daysAfterModificationGreaterThan: answers.coolAfter },
|
|
||||||
tierToArchive: { daysAfterModificationGreaterThan: answers.archiveAfter }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Validate policy
|
|
||||||
if (options.validate && policy) {
|
|
||||||
console.log(chalk.cyan('\n📋 Policy Preview:\n'))
|
|
||||||
console.log(chalk.dim(JSON.stringify(policy, null, 2)))
|
|
||||||
|
|
||||||
const { confirm } = await inquirer.prompt([{
|
|
||||||
type: 'confirm',
|
|
||||||
name: 'confirm',
|
|
||||||
message: 'Apply this policy?',
|
|
||||||
default: true
|
|
||||||
}])
|
|
||||||
|
|
||||||
if (!confirm) {
|
|
||||||
console.log(chalk.yellow('Policy not applied'))
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Apply policy
|
|
||||||
const spinner = ora('Applying lifecycle policy...').start()
|
|
||||||
try {
|
|
||||||
await storage.setLifecyclePolicy(policy)
|
|
||||||
spinner.succeed('Lifecycle policy applied!')
|
|
||||||
|
|
||||||
// Calculate estimated savings
|
|
||||||
if (!options.json) {
|
|
||||||
console.log(chalk.cyan('\n💰 Estimated Annual Savings:\n'))
|
|
||||||
|
|
||||||
const savingsTable = new Table({
|
|
||||||
head: [chalk.cyan('Scale'), chalk.cyan('Before'), chalk.cyan('After'), chalk.cyan('Savings')],
|
|
||||||
style: { head: [], border: [] }
|
|
||||||
})
|
|
||||||
|
|
||||||
const scenarios = [
|
|
||||||
{ size: 5, before: 1380, after: 59, savings: 1321, percent: 96 },
|
|
||||||
{ size: 50, before: 13800, after: 594, savings: 13206, percent: 96 },
|
|
||||||
{ size: 500, before: 138000, after: 5940, savings: 132060, percent: 96 }
|
|
||||||
]
|
|
||||||
|
|
||||||
scenarios.forEach(s => {
|
|
||||||
savingsTable.push([
|
|
||||||
`${s.size}TB`,
|
|
||||||
formatCurrency(s.before),
|
|
||||||
chalk.green(formatCurrency(s.after)),
|
|
||||||
chalk.green(`${formatCurrency(s.savings)} (${s.percent}%)`)
|
|
||||||
])
|
|
||||||
})
|
|
||||||
|
|
||||||
console.log(savingsTable.toString())
|
|
||||||
console.log(chalk.dim('\n💡 Tip: Monitor costs with: brainy monitor cost --breakdown'))
|
|
||||||
}
|
|
||||||
|
|
||||||
if (options.json) {
|
|
||||||
formatOutput({ success: true, policy }, options)
|
|
||||||
}
|
|
||||||
} catch (error: any) {
|
|
||||||
spinner.fail('Failed to apply lifecycle policy')
|
|
||||||
console.error(chalk.red(error.message))
|
|
||||||
process.exit(1)
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get current lifecycle policy
|
|
||||||
*/
|
|
||||||
async get(options: StorageOptions & { format?: 'json' | 'yaml' } = {}) {
|
|
||||||
const spinner = ora('Retrieving lifecycle policy...').start()
|
|
||||||
|
|
||||||
try {
|
|
||||||
const brain = getBrainy()
|
|
||||||
const storage = (brain as any).storage
|
|
||||||
|
|
||||||
const policy = await storage.getLifecyclePolicy()
|
|
||||||
|
|
||||||
spinner.succeed('Policy retrieved')
|
|
||||||
|
|
||||||
if (options.json || options.format === 'json') {
|
|
||||||
console.log(JSON.stringify(policy, null, 2))
|
|
||||||
} else {
|
|
||||||
console.log(chalk.cyan('\n📋 Current Lifecycle Policy:\n'))
|
|
||||||
console.log(chalk.dim(JSON.stringify(policy, null, 2)))
|
|
||||||
}
|
|
||||||
} catch (error: any) {
|
|
||||||
spinner.fail('Failed to get lifecycle policy')
|
|
||||||
console.error(chalk.red(error.message))
|
|
||||||
process.exit(1)
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Remove lifecycle policy
|
|
||||||
*/
|
|
||||||
async remove(options: StorageOptions) {
|
|
||||||
const { confirm } = await inquirer.prompt([{
|
|
||||||
type: 'confirm',
|
|
||||||
name: 'confirm',
|
|
||||||
message: chalk.yellow('⚠️ Remove lifecycle policy? (This will stop cost optimization)'),
|
|
||||||
default: false
|
|
||||||
}])
|
|
||||||
|
|
||||||
if (!confirm) {
|
|
||||||
console.log(chalk.yellow('Policy not removed'))
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
const spinner = ora('Removing lifecycle policy...').start()
|
|
||||||
|
|
||||||
try {
|
|
||||||
const brain = getBrainy()
|
|
||||||
const storage = (brain as any).storage
|
|
||||||
|
|
||||||
await storage.removeLifecyclePolicy()
|
|
||||||
|
|
||||||
spinner.succeed('Lifecycle policy removed')
|
|
||||||
|
|
||||||
if (!options.json) {
|
|
||||||
console.log(chalk.yellow('\n⚠️ Cost optimization disabled'))
|
|
||||||
console.log(chalk.dim(' Storage costs will increase to standard rates'))
|
|
||||||
console.log(chalk.dim(' Run "brainy storage lifecycle set" to re-enable'))
|
|
||||||
}
|
|
||||||
} catch (error: any) {
|
|
||||||
spinner.fail('Failed to remove lifecycle policy')
|
|
||||||
console.error(chalk.red(error.message))
|
|
||||||
process.exit(1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Compression management (FileSystem storage)
|
|
||||||
*/
|
|
||||||
compression: {
|
|
||||||
async enable(options: StorageOptions) {
|
|
||||||
const spinner = ora('Enabling compression...').start()
|
|
||||||
|
|
||||||
try {
|
|
||||||
const brain = getBrainy()
|
|
||||||
const storage = (brain as any).storage
|
|
||||||
|
|
||||||
const status = await storage.getStorageStatus()
|
|
||||||
|
|
||||||
if (status.type !== 'filesystem') {
|
|
||||||
spinner.fail('Compression is only available for FileSystem storage')
|
|
||||||
console.log(chalk.yellow('\n⚠️ Current storage type: ' + status.type))
|
|
||||||
console.log(chalk.dim(' Compression works with: filesystem'))
|
|
||||||
process.exit(1)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Enable compression (would need to update storage config)
|
|
||||||
spinner.succeed('Compression enabled!')
|
|
||||||
|
|
||||||
if (!options.json) {
|
|
||||||
console.log(chalk.cyan('\n📦 Compression Settings:\n'))
|
|
||||||
console.log(chalk.green('✓ Gzip compression enabled'))
|
|
||||||
console.log(chalk.dim(' Expected space savings: 60-80%'))
|
|
||||||
console.log(chalk.dim(' All new files will be compressed'))
|
|
||||||
console.log(chalk.dim('\n💡 Tip: Existing files will be compressed during next write'))
|
|
||||||
}
|
|
||||||
} catch (error: any) {
|
|
||||||
spinner.fail('Failed to enable compression')
|
|
||||||
console.error(chalk.red(error.message))
|
|
||||||
process.exit(1)
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
async disable(options: StorageOptions) {
|
|
||||||
const spinner = ora('Disabling compression...').start()
|
|
||||||
|
|
||||||
try {
|
|
||||||
spinner.succeed('Compression disabled')
|
|
||||||
|
|
||||||
if (!options.json) {
|
|
||||||
console.log(chalk.yellow('\n⚠️ Compression disabled'))
|
|
||||||
console.log(chalk.dim(' Files will no longer be compressed'))
|
|
||||||
console.log(chalk.dim(' Existing compressed files will still be readable'))
|
|
||||||
}
|
|
||||||
} catch (error: any) {
|
|
||||||
spinner.fail('Failed to disable compression')
|
|
||||||
console.error(chalk.red(error.message))
|
|
||||||
process.exit(1)
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
async status(options: StorageOptions) {
|
|
||||||
const spinner = ora('Checking compression status...').start()
|
|
||||||
|
|
||||||
try {
|
|
||||||
const brain = getBrainy()
|
|
||||||
const storage = (brain as any).storage
|
|
||||||
|
|
||||||
const status = await storage.getStorageStatus()
|
|
||||||
|
|
||||||
spinner.succeed('Status retrieved')
|
|
||||||
|
|
||||||
const compressionEnabled = status.details?.compression || false
|
|
||||||
|
|
||||||
if (!options.json) {
|
|
||||||
console.log(chalk.cyan('\n📦 Compression Status:\n'))
|
|
||||||
|
|
||||||
const table = new Table({
|
|
||||||
head: [chalk.cyan('Property'), chalk.cyan('Value')],
|
|
||||||
style: { head: [], border: [] }
|
|
||||||
})
|
|
||||||
|
|
||||||
table.push(
|
|
||||||
['Status', compressionEnabled ? chalk.green('✓ Enabled') : chalk.dim('Disabled')],
|
|
||||||
['Algorithm', compressionEnabled ? 'gzip' : 'None'],
|
|
||||||
['Space Savings', compressionEnabled ? chalk.green('60-80%') : chalk.dim('0%')]
|
|
||||||
)
|
|
||||||
|
|
||||||
console.log(table.toString())
|
|
||||||
|
|
||||||
if (!compressionEnabled) {
|
|
||||||
console.log(chalk.dim('\n💡 Enable compression: brainy storage compression enable'))
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
formatOutput({ enabled: compressionEnabled }, options)
|
|
||||||
}
|
|
||||||
} catch (error: any) {
|
|
||||||
spinner.fail('Failed to check compression status')
|
|
||||||
console.error(chalk.red(error.message))
|
|
||||||
process.exit(1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Batch delete with retry logic
|
|
||||||
*/
|
*/
|
||||||
async batchDelete(
|
async batchDelete(
|
||||||
file: string,
|
file: string,
|
||||||
|
|
@ -615,227 +159,107 @@ export const storageCommands = {
|
||||||
const spinner = ora('Loading entity IDs...').start()
|
const spinner = ora('Loading entity IDs...').start()
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const brain = getBrainy()
|
// Read IDs from file before touching the store.
|
||||||
const storage = (brain as any).storage
|
|
||||||
|
|
||||||
// Read IDs from file
|
|
||||||
const content = readFileSync(file, 'utf-8')
|
const content = readFileSync(file, 'utf-8')
|
||||||
const ids = content.split('\n').filter(line => line.trim())
|
const ids = content
|
||||||
|
.split('\n')
|
||||||
|
.map((line) => line.trim())
|
||||||
|
.filter((line) => line.length > 0)
|
||||||
|
|
||||||
spinner.succeed(`Loaded ${ids.length} entity IDs`)
|
spinner.succeed(`Loaded ${ids.length} entity IDs`)
|
||||||
|
|
||||||
// Confirm
|
if (ids.length === 0) {
|
||||||
const { confirm } = await inquirer.prompt([{
|
console.log(chalk.yellow('No entity IDs found in file — nothing to delete'))
|
||||||
|
process.exit(0)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Confirm — destructive and irreversible.
|
||||||
|
const { confirm } = await inquirer.prompt([
|
||||||
|
{
|
||||||
type: 'confirm',
|
type: 'confirm',
|
||||||
name: 'confirm',
|
name: 'confirm',
|
||||||
message: chalk.yellow(`⚠️ Delete ${ids.length} entities? This cannot be undone.`),
|
message: chalk.yellow(`⚠️ Delete ${ids.length} entities? This cannot be undone.`),
|
||||||
default: false
|
default: false
|
||||||
}])
|
}
|
||||||
|
])
|
||||||
|
|
||||||
if (!confirm) {
|
if (!confirm) {
|
||||||
console.log(chalk.yellow('Deletion cancelled'))
|
console.log(chalk.yellow('Deletion cancelled'))
|
||||||
return
|
process.exit(0)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Generate paths for all entities (vectors + metadata)
|
const brain = getBrainy()
|
||||||
const paths: string[] = []
|
await brain.init()
|
||||||
for (const id of ids) {
|
|
||||||
const shard = id.substring(0, 2)
|
|
||||||
paths.push(`entities/nouns/vectors/${shard}/${id}.json`)
|
|
||||||
paths.push(`entities/nouns/metadata/${shard}/${id}.json`)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Batch delete with progress
|
const maxRetries = options.maxRetries ? parseInt(options.maxRetries, 10) : 3
|
||||||
const deleteSpinner = ora('Deleting entities...').start()
|
|
||||||
|
const deleteSpinner = ora(`Deleting ${ids.length} entities...`).start()
|
||||||
const startTime = Date.now()
|
const startTime = Date.now()
|
||||||
|
let deleted = 0
|
||||||
|
const failed: Array<{ id: string; error: string }> = []
|
||||||
|
|
||||||
|
for (const id of ids) {
|
||||||
|
let lastError: unknown = null
|
||||||
|
let succeeded = false
|
||||||
|
|
||||||
|
for (let attempt = 0; attempt <= maxRetries; attempt++) {
|
||||||
try {
|
try {
|
||||||
await storage.batchDelete(paths, {
|
await brain.delete(id)
|
||||||
maxRetries: options.maxRetries ? parseInt(options.maxRetries) : 3,
|
succeeded = true
|
||||||
continueOnError: options.continueOnError || false
|
break
|
||||||
|
} catch (error) {
|
||||||
|
lastError = error
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (succeeded) {
|
||||||
|
deleted++
|
||||||
|
deleteSpinner.text = `Deleting entities... ${deleted}/${ids.length}`
|
||||||
|
} else {
|
||||||
|
failed.push({
|
||||||
|
id,
|
||||||
|
error: lastError instanceof Error ? lastError.message : String(lastError)
|
||||||
})
|
})
|
||||||
|
if (!options.continueOnError) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
const duration = ((Date.now() - startTime) / 1000).toFixed(1)
|
const duration = ((Date.now() - startTime) / 1000).toFixed(1)
|
||||||
const rate = (ids.length / parseFloat(duration)).toFixed(0)
|
|
||||||
|
|
||||||
deleteSpinner.succeed(`Deleted ${ids.length} entities in ${duration}s (${rate}/sec)`)
|
if (failed.length === 0) {
|
||||||
|
deleteSpinner.succeed(`Deleted ${deleted} entities in ${duration}s`)
|
||||||
|
} else {
|
||||||
|
deleteSpinner.warn(`Deleted ${deleted}/${ids.length} entities in ${duration}s (${failed.length} failed)`)
|
||||||
|
}
|
||||||
|
|
||||||
if (!options.json) {
|
if (!options.json) {
|
||||||
console.log(chalk.green(`\n✓ Batch delete complete`))
|
console.log(chalk.green(`\n✓ Batch delete complete`))
|
||||||
console.log(chalk.dim(` Entities: ${ids.length}`))
|
console.log(chalk.dim(` Deleted: ${deleted}`))
|
||||||
|
console.log(chalk.dim(` Failed: ${failed.length}`))
|
||||||
console.log(chalk.dim(` Duration: ${duration}s`))
|
console.log(chalk.dim(` Duration: ${duration}s`))
|
||||||
console.log(chalk.dim(` Rate: ${rate} entities/sec`))
|
|
||||||
} else {
|
if (failed.length > 0 && options.verbose) {
|
||||||
formatOutput({
|
console.log(chalk.bold('\n⚠️ Failed IDs:'))
|
||||||
deleted: ids.length,
|
for (const f of failed.slice(0, 10)) {
|
||||||
duration: parseFloat(duration),
|
console.log(` ${chalk.dim(f.id)}: ${chalk.red(f.error)}`)
|
||||||
rate: parseFloat(rate)
|
|
||||||
}, options)
|
|
||||||
}
|
}
|
||||||
|
if (failed.length > 10) {
|
||||||
|
console.log(chalk.dim(` ... and ${failed.length - 10} more`))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
formatOutput({ deleted, failed, duration: parseFloat(duration) }, options)
|
||||||
|
}
|
||||||
|
|
||||||
|
// One-shot command — see status() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(failed.length > 0 && !options.continueOnError ? 1 : 0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
deleteSpinner.fail('Batch delete failed')
|
spinner.fail('Batch delete failed')
|
||||||
console.error(chalk.red(error.message))
|
console.error(chalk.red(error.message))
|
||||||
process.exit(1)
|
process.exit(1)
|
||||||
}
|
}
|
||||||
} catch (error: any) {
|
|
||||||
spinner.fail('Failed to load entity IDs')
|
|
||||||
console.error(chalk.red(error.message))
|
|
||||||
process.exit(1)
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Cost estimation tool
|
|
||||||
*/
|
|
||||||
async costEstimate(
|
|
||||||
options: StorageOptions & {
|
|
||||||
provider?: 'aws' | 'gcs' | 'azure' | 'r2'
|
|
||||||
size?: string
|
|
||||||
operations?: string
|
|
||||||
} = {}
|
|
||||||
) {
|
|
||||||
console.log(chalk.cyan('\n💰 Cloud Storage Cost Estimator\n'))
|
|
||||||
|
|
||||||
let provider: string
|
|
||||||
let sizeGB: number
|
|
||||||
let operations: number
|
|
||||||
|
|
||||||
if (!options.provider || !options.size || !options.operations) {
|
|
||||||
// Interactive mode
|
|
||||||
const answers = await inquirer.prompt([
|
|
||||||
{
|
|
||||||
type: 'list',
|
|
||||||
name: 'provider',
|
|
||||||
message: 'Cloud provider:',
|
|
||||||
choices: [
|
|
||||||
{ name: 'AWS S3', value: 'aws' },
|
|
||||||
{ name: 'Google Cloud Storage', value: 'gcs' },
|
|
||||||
{ name: 'Azure Blob Storage', value: 'azure' },
|
|
||||||
{ name: 'Cloudflare R2', value: 'r2' }
|
|
||||||
],
|
|
||||||
when: !options.provider
|
|
||||||
},
|
|
||||||
{
|
|
||||||
type: 'number',
|
|
||||||
name: 'sizeGB',
|
|
||||||
message: 'Total data size (GB):',
|
|
||||||
default: 1000,
|
|
||||||
validate: (input: number) => input > 0,
|
|
||||||
when: !options.size
|
|
||||||
},
|
|
||||||
{
|
|
||||||
type: 'number',
|
|
||||||
name: 'operations',
|
|
||||||
message: 'Monthly operations (reads + writes):',
|
|
||||||
default: 1000000,
|
|
||||||
validate: (input: number) => input >= 0,
|
|
||||||
when: !options.operations
|
|
||||||
}
|
|
||||||
])
|
|
||||||
|
|
||||||
provider = options.provider || answers.provider
|
|
||||||
sizeGB = options.size ? parseFloat(options.size) : answers.sizeGB
|
|
||||||
operations = options.operations ? parseInt(options.operations) : answers.operations
|
|
||||||
} else {
|
|
||||||
provider = options.provider
|
|
||||||
sizeGB = parseFloat(options.size)
|
|
||||||
operations = parseInt(options.operations)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Calculate costs
|
|
||||||
const spinner = ora('Calculating costs...').start()
|
|
||||||
|
|
||||||
// Pricing (2025 estimates)
|
|
||||||
const pricing: Record<string, any> = {
|
|
||||||
aws: {
|
|
||||||
standard: { storage: 0.023, operations: 0.005 },
|
|
||||||
ia: { storage: 0.0125, operations: 0.01 },
|
|
||||||
glacier: { storage: 0.004, operations: 0.05 },
|
|
||||||
deepArchive: { storage: 0.00099, operations: 0.10 }
|
|
||||||
},
|
|
||||||
gcs: {
|
|
||||||
standard: { storage: 0.020, operations: 0.005 },
|
|
||||||
nearline: { storage: 0.010, operations: 0.010 },
|
|
||||||
coldline: { storage: 0.004, operations: 0.050 },
|
|
||||||
archive: { storage: 0.0012, operations: 0.050 }
|
|
||||||
},
|
|
||||||
azure: {
|
|
||||||
hot: { storage: 0.0184, operations: 0.005 },
|
|
||||||
cool: { storage: 0.010, operations: 0.010 },
|
|
||||||
archive: { storage: 0.00099, operations: 0.050 }
|
|
||||||
},
|
|
||||||
r2: {
|
|
||||||
standard: { storage: 0.015, operations: 0.0045 }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const providerPricing = pricing[provider]
|
|
||||||
const results: any = {}
|
|
||||||
|
|
||||||
for (const [tier, prices] of Object.entries(providerPricing)) {
|
|
||||||
const storageCost = sizeGB * (prices as any).storage
|
|
||||||
const opsCost = (operations / 1000000) * (prices as any).operations
|
|
||||||
const monthly = storageCost + opsCost
|
|
||||||
const annual = monthly * 12
|
|
||||||
|
|
||||||
results[tier] = {
|
|
||||||
storage: storageCost,
|
|
||||||
operations: opsCost,
|
|
||||||
monthly,
|
|
||||||
annual
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
spinner.succeed('Cost estimation complete')
|
|
||||||
|
|
||||||
if (!options.json) {
|
|
||||||
console.log(chalk.cyan(`\n💰 Cost Estimate for ${provider.toUpperCase()}\n`))
|
|
||||||
console.log(chalk.dim(`Data Size: ${sizeGB} GB (${formatBytes(sizeGB * 1024 * 1024 * 1024)})`))
|
|
||||||
console.log(chalk.dim(`Operations: ${operations.toLocaleString()}/month\n`))
|
|
||||||
|
|
||||||
const table = new Table({
|
|
||||||
head: [
|
|
||||||
chalk.cyan('Tier'),
|
|
||||||
chalk.cyan('Storage/mo'),
|
|
||||||
chalk.cyan('Ops/mo'),
|
|
||||||
chalk.cyan('Total/mo'),
|
|
||||||
chalk.cyan('Annual')
|
|
||||||
],
|
|
||||||
style: { head: [], border: [] }
|
|
||||||
})
|
|
||||||
|
|
||||||
for (const [tier, costs] of Object.entries(results)) {
|
|
||||||
table.push([
|
|
||||||
tier.toUpperCase(),
|
|
||||||
formatCurrency((costs as any).storage),
|
|
||||||
formatCurrency((costs as any).operations),
|
|
||||||
formatCurrency((costs as any).monthly),
|
|
||||||
chalk.green(formatCurrency((costs as any).annual))
|
|
||||||
])
|
|
||||||
}
|
|
||||||
|
|
||||||
console.log(table.toString())
|
|
||||||
|
|
||||||
// Show savings
|
|
||||||
const tiers = Object.keys(results)
|
|
||||||
if (tiers.length > 1) {
|
|
||||||
const highest = results[tiers[0]]
|
|
||||||
const lowest = results[tiers[tiers.length - 1]]
|
|
||||||
const savings = highest.annual - lowest.annual
|
|
||||||
const savingsPercent = ((savings / highest.annual) * 100).toFixed(0)
|
|
||||||
|
|
||||||
console.log(chalk.cyan('\n💡 Potential Savings:\n'))
|
|
||||||
console.log(chalk.green(` ${formatCurrency(savings)}/year (${savingsPercent}%) by using lifecycle policies`))
|
|
||||||
console.log(chalk.dim(` ${tiers[0].toUpperCase()} → ${tiers[tiers.length - 1].toUpperCase()}`))
|
|
||||||
}
|
|
||||||
|
|
||||||
if (provider === 'r2') {
|
|
||||||
console.log(chalk.cyan('\n✨ R2 Advantage:\n'))
|
|
||||||
console.log(chalk.green(' $0 egress fees (unlimited data transfer out)'))
|
|
||||||
console.log(chalk.dim(' Perfect for high-traffic applications'))
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
formatOutput(results, options)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -22,8 +22,7 @@ interface StatsOptions extends UtilityOptions {
|
||||||
}
|
}
|
||||||
|
|
||||||
interface CleanOptions extends UtilityOptions {
|
interface CleanOptions extends UtilityOptions {
|
||||||
removeOrphans?: boolean
|
force?: boolean
|
||||||
rebuildIndex?: boolean
|
|
||||||
}
|
}
|
||||||
|
|
||||||
interface BenchmarkOptions extends UtilityOptions {
|
interface BenchmarkOptions extends UtilityOptions {
|
||||||
|
|
@ -63,6 +62,7 @@ export const utilityCommands = {
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
const nounCount = await brain.getNounCount()
|
const nounCount = await brain.getNounCount()
|
||||||
const verbCount = await brain.getVerbCount()
|
const verbCount = await brain.getVerbCount()
|
||||||
const memUsage = process.memoryUsage()
|
const memUsage = process.memoryUsage()
|
||||||
|
|
@ -77,7 +77,11 @@ export const utilityCommands = {
|
||||||
|
|
||||||
if (options.json) {
|
if (options.json) {
|
||||||
formatOutput(stats, options)
|
formatOutput(stats, options)
|
||||||
return
|
// close() releases the writer lock and indexes, but global timers
|
||||||
|
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
|
||||||
|
// alive. CLI commands are one-shot — exit explicitly.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
}
|
}
|
||||||
|
|
||||||
console.log(chalk.cyan('\n📊 Database Statistics\n'))
|
console.log(chalk.cyan('\n📊 Database Statistics\n'))
|
||||||
|
|
@ -113,6 +117,9 @@ export const utilityCommands = {
|
||||||
|
|
||||||
console.log(memTable.toString())
|
console.log(memTable.toString())
|
||||||
|
|
||||||
|
// One-shot command — see the --json branch for why the explicit exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
spinner.fail('Failed to gather statistics')
|
spinner.fail('Failed to gather statistics')
|
||||||
console.error(chalk.red(error.message))
|
console.error(chalk.red(error.message))
|
||||||
|
|
@ -121,30 +128,33 @@ export const utilityCommands = {
|
||||||
},
|
},
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Clean and optimize database
|
* Clear the database (all entities, relationships, and indexes).
|
||||||
|
* Destructive — asks for confirmation unless --force is passed.
|
||||||
*/
|
*/
|
||||||
async clean(options: CleanOptions) {
|
async clean(options: CleanOptions) {
|
||||||
const spinner = ora('Cleaning database...').start()
|
let spinner: ReturnType<typeof ora> | null = null
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const brain = getBrainy()
|
// Destructive operation — confirm first (skipped with --force).
|
||||||
|
if (!options.force) {
|
||||||
|
const inquirer = (await import('inquirer')).default
|
||||||
|
const { confirm } = await inquirer.prompt([{
|
||||||
|
type: 'confirm',
|
||||||
|
name: 'confirm',
|
||||||
|
message: chalk.yellow('⚠️ Permanently delete ALL data (entities, relationships, indexes)?'),
|
||||||
|
default: false
|
||||||
|
}])
|
||||||
|
|
||||||
// For now, only support full clear
|
if (!confirm) {
|
||||||
// removeOrphans and rebuildIndex would require new Brainy APIs
|
console.log(chalk.yellow('Clean cancelled'))
|
||||||
if (options.removeOrphans || options.rebuildIndex) {
|
process.exit(0)
|
||||||
spinner.warn('Advanced cleanup options not yet implemented')
|
}
|
||||||
console.log(chalk.yellow('\n⚠️ Advanced cleanup features coming soon:'))
|
|
||||||
console.log(chalk.dim(' • --remove-orphans: Remove disconnected items'))
|
|
||||||
console.log(chalk.dim(' • --rebuild-index: Rebuild vector index'))
|
|
||||||
console.log(chalk.dim('\nUse "brainy clean" without options to clear the database'))
|
|
||||||
return
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Show warning before clearing
|
const brain = getBrainy()
|
||||||
console.log(chalk.yellow('\n⚠️ WARNING: This will permanently delete ALL data!'))
|
|
||||||
|
|
||||||
// Clear all data (entities, relationships, and every index)
|
// Clear all data (entities, relationships, and every index)
|
||||||
spinner.text = 'Clearing all data...'
|
spinner = ora('Clearing all data...').start()
|
||||||
await brain.init()
|
await brain.init()
|
||||||
await brain.clear()
|
await brain.clear()
|
||||||
|
|
||||||
|
|
@ -163,7 +173,7 @@ export const utilityCommands = {
|
||||||
await brain.close()
|
await brain.close()
|
||||||
process.exit(0)
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
spinner.fail('Cleanup failed')
|
if (spinner) spinner.fail('Cleanup failed')
|
||||||
console.error(chalk.red(error.message))
|
console.error(chalk.red(error.message))
|
||||||
process.exit(1)
|
process.exit(1)
|
||||||
}
|
}
|
||||||
|
|
@ -185,6 +195,7 @@ export const utilityCommands = {
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
// Benchmark different operations
|
// Benchmark different operations
|
||||||
const benchmarks = [
|
const benchmarks = [
|
||||||
|
|
@ -205,7 +216,8 @@ export const utilityCommands = {
|
||||||
|
|
||||||
switch (bench.name) {
|
switch (bench.name) {
|
||||||
case 'add':
|
case 'add':
|
||||||
await brain.add({ data: `Test item ${i}`, type: NounType.Thing, metadata: { benchmark: true } })
|
// 8.0 requires a subtype on every write by default.
|
||||||
|
await brain.add({ data: `Test item ${i}`, type: NounType.Thing, subtype: 'benchmark', metadata: { benchmark: true } })
|
||||||
break
|
break
|
||||||
case 'search':
|
case 'search':
|
||||||
await brain.find({ query: 'test', limit: 10 })
|
await brain.find({ query: 'test', limit: 10 })
|
||||||
|
|
@ -285,6 +297,9 @@ export const utilityCommands = {
|
||||||
formatOutput(results, options)
|
formatOutput(results, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see stats() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
console.error(chalk.red('Benchmark failed:'), error.message)
|
console.error(chalk.red('Benchmark failed:'), error.message)
|
||||||
process.exit(1)
|
process.exit(1)
|
||||||
|
|
|
||||||
|
|
@ -70,6 +70,12 @@ export const vfsCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput({ path, content: buffer.toString(), size: buffer.length }, options)
|
formatOutput({ path, content: buffer.toString(), size: buffer.length }, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// close() releases the writer lock and indexes, but global timers
|
||||||
|
// (UnifiedCache bookkeeping, PathResolver stats) keep the event loop
|
||||||
|
// alive. CLI commands are one-shot — exit explicitly.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
spinner.fail('Failed to read file')
|
spinner.fail('Failed to read file')
|
||||||
console.error(chalk.red(error.message))
|
console.error(chalk.red(error.message))
|
||||||
|
|
@ -109,6 +115,10 @@ export const vfsCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput({ path, size: Buffer.byteLength(data) }, options)
|
formatOutput({ path, size: Buffer.byteLength(data) }, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see read() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
spinner.fail('Failed to write file')
|
spinner.fail('Failed to write file')
|
||||||
console.error(chalk.red(error.message))
|
console.error(chalk.red(error.message))
|
||||||
|
|
@ -132,7 +142,9 @@ export const vfsCommands = {
|
||||||
if (!options.json) {
|
if (!options.json) {
|
||||||
if (!Array.isArray(entries) || entries.length === 0) {
|
if (!Array.isArray(entries) || entries.length === 0) {
|
||||||
console.log(chalk.yellow('Directory is empty'))
|
console.log(chalk.yellow('Directory is empty'))
|
||||||
return
|
// One-shot command — see read() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
}
|
}
|
||||||
|
|
||||||
if (options.long) {
|
if (options.long) {
|
||||||
|
|
@ -171,6 +183,10 @@ export const vfsCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput(entries, options)
|
formatOutput(entries, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see read() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
spinner.fail('Failed to list directory')
|
spinner.fail('Failed to list directory')
|
||||||
console.error(chalk.red(error.message))
|
console.error(chalk.red(error.message))
|
||||||
|
|
@ -202,6 +218,10 @@ export const vfsCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput(stats, options)
|
formatOutput(stats, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see read() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
spinner.fail('Failed to get stats')
|
spinner.fail('Failed to get stats')
|
||||||
console.error(chalk.red(error.message))
|
console.error(chalk.red(error.message))
|
||||||
|
|
@ -227,6 +247,10 @@ export const vfsCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput({ path, created: true }, options)
|
formatOutput({ path, created: true }, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see read() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
spinner.fail('Failed to create directory')
|
spinner.fail('Failed to create directory')
|
||||||
console.error(chalk.red(error.message))
|
console.error(chalk.red(error.message))
|
||||||
|
|
@ -258,12 +282,16 @@ export const vfsCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput({ path, removed: true }, options)
|
formatOutput({ path, removed: true }, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see read() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
spinner.fail('Failed to remove')
|
spinner.fail('Failed to remove')
|
||||||
console.error(chalk.red(error.message))
|
console.error(chalk.red(error.message))
|
||||||
if (!options.force) {
|
// --force tolerates the failure but the process is still one-shot:
|
||||||
process.exit(1)
|
// exit cleanly (0) instead of falling off the event loop.
|
||||||
}
|
process.exit(options.force ? 0 : 1)
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
|
||||||
|
|
@ -303,6 +331,10 @@ export const vfsCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput(results, options)
|
formatOutput(results, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see read() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
spinner.fail('Search failed')
|
spinner.fail('Search failed')
|
||||||
console.error(chalk.red(error.message))
|
console.error(chalk.red(error.message))
|
||||||
|
|
@ -343,6 +375,10 @@ export const vfsCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput(results, options)
|
formatOutput(results, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see read() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
spinner.fail('Failed to find similar files')
|
spinner.fail('Failed to find similar files')
|
||||||
console.error(chalk.red(error.message))
|
console.error(chalk.red(error.message))
|
||||||
|
|
@ -371,6 +407,10 @@ export const vfsCommands = {
|
||||||
} else {
|
} else {
|
||||||
formatOutput(tree, options)
|
formatOutput(tree, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One-shot command — see read() for why the explicit close + exit.
|
||||||
|
await brain.close()
|
||||||
|
process.exit(0)
|
||||||
} catch (error: any) {
|
} catch (error: any) {
|
||||||
spinner.fail('Failed to build tree')
|
spinner.fail('Failed to build tree')
|
||||||
console.error(chalk.red(error.message))
|
console.error(chalk.red(error.message))
|
||||||
|
|
|
||||||
131
src/cli/index.ts
131
src/cli/index.ts
|
|
@ -19,6 +19,7 @@ import { insightsCommands } from './commands/insights.js'
|
||||||
import { importCommands } from './commands/import.js'
|
import { importCommands } from './commands/import.js'
|
||||||
import { snapshotCommands } from './commands/snapshot.js'
|
import { snapshotCommands } from './commands/snapshot.js'
|
||||||
import { inspectCommands } from './commands/inspect.js'
|
import { inspectCommands } from './commands/inspect.js'
|
||||||
|
import { types as typesCommand, validate as validateCommand } from './commands/types.js'
|
||||||
import { readFileSync } from 'fs'
|
import { readFileSync } from 'fs'
|
||||||
import { fileURLToPath } from 'url'
|
import { fileURLToPath } from 'url'
|
||||||
import { dirname, join } from 'path'
|
import { dirname, join } from 'path'
|
||||||
|
|
@ -70,14 +71,9 @@ ${chalk.cyan('Examples:')}
|
||||||
$ brainy vfs similar /code/Button.tsx
|
$ brainy vfs similar /code/Button.tsx
|
||||||
|
|
||||||
${chalk.dim('# Storage management')}
|
${chalk.dim('# Storage management')}
|
||||||
$ brainy storage status --quota
|
$ brainy storage status
|
||||||
$ brainy storage lifecycle set ${chalk.dim('# Interactive mode')}
|
|
||||||
$ brainy storage cost-estimate
|
|
||||||
$ brainy storage batch-delete old-ids.txt
|
$ brainy storage batch-delete old-ids.txt
|
||||||
|
|
||||||
${chalk.dim('# Interactive mode')}
|
|
||||||
$ brainy interactive
|
|
||||||
|
|
||||||
${chalk.cyan('Documentation:')}
|
${chalk.cyan('Documentation:')}
|
||||||
${chalk.dim('Full docs:')} https://github.com/soulcraftlabs/brainy
|
${chalk.dim('Full docs:')} https://github.com/soulcraftlabs/brainy
|
||||||
${chalk.dim('Report issues:')} https://github.com/soulcraftlabs/brainy/issues
|
${chalk.dim('Report issues:')} https://github.com/soulcraftlabs/brainy/issues
|
||||||
|
|
@ -109,7 +105,7 @@ program
|
||||||
.description('Advanced search with Triple Intelligence™ (interactive if no query)')
|
.description('Advanced search with Triple Intelligence™ (interactive if no query)')
|
||||||
.option('-k, --limit <number>', 'Number of results', '10')
|
.option('-k, --limit <number>', 'Number of results', '10')
|
||||||
.option('--offset <number>', 'Skip N results (pagination)')
|
.option('--offset <number>', 'Skip N results (pagination)')
|
||||||
.option('-t, --threshold <number>', 'Similarity threshold (0-1)', '0.7')
|
.option('-t, --threshold <number>', 'Minimum similarity score (0-1); with --near, the proximity threshold')
|
||||||
.option('--type <types>', 'Filter by type(s) - comma separated')
|
.option('--type <types>', 'Filter by type(s) - comma separated')
|
||||||
.option('--where <json>', 'Metadata filters (JSON)')
|
.option('--where <json>', 'Metadata filters (JSON)')
|
||||||
.option('--near <id>', 'Find items near this ID')
|
.option('--near <id>', 'Find items near this ID')
|
||||||
|
|
@ -181,6 +177,23 @@ program
|
||||||
.option('--pretty', 'Pretty print JSON')
|
.option('--pretty', 'Pretty print JSON')
|
||||||
.action(coreCommands.diagnostics)
|
.action(coreCommands.diagnostics)
|
||||||
|
|
||||||
|
// ===== Type Commands =====
|
||||||
|
|
||||||
|
program
|
||||||
|
.command('types')
|
||||||
|
.description('List all NounType and VerbType values')
|
||||||
|
.option('--noun', 'Show noun types only')
|
||||||
|
.option('--verb', 'Show verb types only')
|
||||||
|
.option('--json', 'Output as JSON')
|
||||||
|
.action(typesCommand)
|
||||||
|
|
||||||
|
program
|
||||||
|
.command('validate [type]')
|
||||||
|
.description('Check whether a type string is a valid NounType or VerbType (interactive if no type)')
|
||||||
|
.option('--verb', 'Validate as a verb type (default: noun type)')
|
||||||
|
.option('--json', 'Output as JSON')
|
||||||
|
.action(validateCommand)
|
||||||
|
|
||||||
// ===== Neural Commands =====
|
// ===== Neural Commands =====
|
||||||
|
|
||||||
program
|
program
|
||||||
|
|
@ -222,17 +235,6 @@ program
|
||||||
.option('--children-only', 'Show only child hierarchy')
|
.option('--children-only', 'Show only child hierarchy')
|
||||||
.action(neuralCommands.hierarchy)
|
.action(neuralCommands.hierarchy)
|
||||||
|
|
||||||
program
|
|
||||||
.command('path <from> <to>')
|
|
||||||
.description('Find semantic path between items')
|
|
||||||
.option('--steps', 'Show step-by-step path')
|
|
||||||
.option('--max-hops <number>', 'Maximum path length', '5')
|
|
||||||
.action(() => {
|
|
||||||
console.log(chalk.yellow('\n⚠️ Semantic path finding coming soon'))
|
|
||||||
console.log(chalk.dim('This feature requires implementing graph traversal algorithms'))
|
|
||||||
console.log(chalk.dim('Use "brainy neighbors" and "brainy hierarchy" to explore connections'))
|
|
||||||
})
|
|
||||||
|
|
||||||
program
|
program
|
||||||
.command('outliers')
|
.command('outliers')
|
||||||
.alias('anomalies')
|
.alias('anomalies')
|
||||||
|
|
@ -457,89 +459,26 @@ program
|
||||||
|
|
||||||
program
|
program
|
||||||
.command('storage')
|
.command('storage')
|
||||||
.description('💾 Storage management and cost optimization')
|
.description('💾 Storage management (filesystem + memory backends)')
|
||||||
.addCommand(
|
.addCommand(
|
||||||
new Command('status')
|
new Command('status')
|
||||||
.description('Show storage status and health')
|
.description('Show storage backend, counts, root directory, and writer lock')
|
||||||
.option('--detailed', 'Show detailed information')
|
.option('--json', 'Output as JSON')
|
||||||
.option('--quota', 'Show quota information (OPFS)')
|
.option('--pretty', 'Pretty-print JSON')
|
||||||
.action((options) => {
|
.action((options) => {
|
||||||
storageCommands.status(options)
|
storageCommands.status(options)
|
||||||
})
|
})
|
||||||
)
|
)
|
||||||
.addCommand(
|
|
||||||
new Command('lifecycle')
|
|
||||||
.description('Lifecycle policy management')
|
|
||||||
.addCommand(
|
|
||||||
new Command('set')
|
|
||||||
.argument('[config-file]', 'Policy configuration file (JSON)')
|
|
||||||
.description('Set lifecycle policy (interactive if no file)')
|
|
||||||
.option('--validate', 'Validate before applying')
|
|
||||||
.action((configFile, options) => {
|
|
||||||
storageCommands.lifecycle.set(configFile, options)
|
|
||||||
})
|
|
||||||
)
|
|
||||||
.addCommand(
|
|
||||||
new Command('get')
|
|
||||||
.description('Get current lifecycle policy')
|
|
||||||
.option('-f, --format <type>', 'Output format (json|yaml)', 'json')
|
|
||||||
.action((options) => {
|
|
||||||
storageCommands.lifecycle.get(options)
|
|
||||||
})
|
|
||||||
)
|
|
||||||
.addCommand(
|
|
||||||
new Command('remove')
|
|
||||||
.description('Remove lifecycle policy')
|
|
||||||
.action((options) => {
|
|
||||||
storageCommands.lifecycle.remove(options)
|
|
||||||
})
|
|
||||||
)
|
|
||||||
)
|
|
||||||
.addCommand(
|
|
||||||
new Command('compression')
|
|
||||||
.description('Compression management (FileSystem)')
|
|
||||||
.addCommand(
|
|
||||||
new Command('enable')
|
|
||||||
.description('Enable gzip compression')
|
|
||||||
.action((options) => {
|
|
||||||
storageCommands.compression.enable(options)
|
|
||||||
})
|
|
||||||
)
|
|
||||||
.addCommand(
|
|
||||||
new Command('disable')
|
|
||||||
.description('Disable compression')
|
|
||||||
.action((options) => {
|
|
||||||
storageCommands.compression.disable(options)
|
|
||||||
})
|
|
||||||
)
|
|
||||||
.addCommand(
|
|
||||||
new Command('status')
|
|
||||||
.description('Show compression status')
|
|
||||||
.action((options) => {
|
|
||||||
storageCommands.compression.status(options)
|
|
||||||
})
|
|
||||||
)
|
|
||||||
)
|
|
||||||
.addCommand(
|
.addCommand(
|
||||||
new Command('batch-delete')
|
new Command('batch-delete')
|
||||||
.argument('<file>', 'File containing entity IDs (one per line)')
|
.argument('<file>', 'File containing entity IDs (one per line)')
|
||||||
.description('Batch delete with retry logic')
|
.description('Delete a list of entities through the public delete path (with retry)')
|
||||||
.option('--max-retries <n>', 'Maximum retry attempts', '3')
|
.option('--max-retries <n>', 'Maximum retry attempts per ID', '3')
|
||||||
.option('--continue-on-error', 'Continue if some deletes fail')
|
.option('--continue-on-error', 'Continue past IDs that still fail after retries')
|
||||||
.action((file, options) => {
|
.action((file, options) => {
|
||||||
storageCommands.batchDelete(file, options)
|
storageCommands.batchDelete(file, options)
|
||||||
})
|
})
|
||||||
)
|
)
|
||||||
.addCommand(
|
|
||||||
new Command('cost-estimate')
|
|
||||||
.description('Estimate cloud storage costs')
|
|
||||||
.option('--provider <type>', 'Cloud provider (aws|gcs|azure|r2)')
|
|
||||||
.option('--size <gb>', 'Data size in GB')
|
|
||||||
.option('--operations <n>', 'Monthly operations')
|
|
||||||
.action((options) => {
|
|
||||||
storageCommands.costEstimate(options)
|
|
||||||
})
|
|
||||||
)
|
|
||||||
|
|
||||||
// ===== Data Management Commands =====
|
// ===== Data Management Commands =====
|
||||||
|
|
||||||
|
|
@ -742,9 +681,8 @@ program
|
||||||
|
|
||||||
program
|
program
|
||||||
.command('clean')
|
.command('clean')
|
||||||
.description('Clean and optimize database')
|
.description('Clear the database — ALL entities, relationships, and indexes (asks for confirmation)')
|
||||||
.option('--remove-orphans', 'Remove orphaned items')
|
.option('-f, --force', 'Skip confirmation prompt')
|
||||||
.option('--rebuild-index', 'Rebuild search index')
|
|
||||||
.action(utilityCommands.clean)
|
.action(utilityCommands.clean)
|
||||||
|
|
||||||
program
|
program
|
||||||
|
|
@ -785,17 +723,6 @@ program
|
||||||
.description('Show the store\'s current generation watermark')
|
.description('Show the store\'s current generation watermark')
|
||||||
.action(snapshotCommands.generation)
|
.action(snapshotCommands.generation)
|
||||||
|
|
||||||
// ===== Interactive Mode =====
|
|
||||||
|
|
||||||
program
|
|
||||||
.command('interactive')
|
|
||||||
.alias('i')
|
|
||||||
.description('Start interactive REPL mode')
|
|
||||||
.action(async () => {
|
|
||||||
const { startInteractiveMode } = await import('./interactive.js')
|
|
||||||
await startInteractiveMode()
|
|
||||||
})
|
|
||||||
|
|
||||||
// ===== Error Handling =====
|
// ===== Error Handling =====
|
||||||
|
|
||||||
program.exitOverride()
|
program.exitOverride()
|
||||||
|
|
|
||||||
|
|
@ -1,653 +0,0 @@
|
||||||
/**
|
|
||||||
* Professional Interactive CLI System
|
|
||||||
*
|
|
||||||
* Provides consistent, delightful interactive prompts for all commands
|
|
||||||
* with smart defaults, validation, and helpful examples
|
|
||||||
*/
|
|
||||||
|
|
||||||
import chalk from 'chalk'
|
|
||||||
import inquirer from 'inquirer'
|
|
||||||
import ora from 'ora'
|
|
||||||
import { Brainy } from '../brainy.js'
|
|
||||||
import { getBrainyVersion } from '../utils/version.js'
|
|
||||||
|
|
||||||
// Professional color scheme
|
|
||||||
export const colors = {
|
|
||||||
primary: chalk.hex('#3A5F4A'), // Teal (from logo)
|
|
||||||
success: chalk.hex('#2D4A3A'), // Deep teal
|
|
||||||
info: chalk.hex('#4A6B5A'), // Medium teal
|
|
||||||
warning: chalk.hex('#D67441'), // Orange (from logo)
|
|
||||||
error: chalk.hex('#B85C35'), // Deep orange
|
|
||||||
brain: chalk.hex('#D67441'), // Brain orange
|
|
||||||
cream: chalk.hex('#F5E6A3'), // Cream background
|
|
||||||
dim: chalk.dim,
|
|
||||||
bold: chalk.bold,
|
|
||||||
cyan: chalk.cyan,
|
|
||||||
green: chalk.green,
|
|
||||||
yellow: chalk.yellow,
|
|
||||||
red: chalk.red
|
|
||||||
}
|
|
||||||
|
|
||||||
// Icons for consistent visual language
|
|
||||||
export const icons = {
|
|
||||||
brain: '🧠',
|
|
||||||
search: '🔍',
|
|
||||||
add: '➕',
|
|
||||||
delete: '🗑️',
|
|
||||||
update: '🔄',
|
|
||||||
import: '📥',
|
|
||||||
export: '📤',
|
|
||||||
connect: '🔗',
|
|
||||||
question: '❓',
|
|
||||||
success: '✅',
|
|
||||||
error: '❌',
|
|
||||||
warning: '⚠️',
|
|
||||||
info: 'ℹ️',
|
|
||||||
sparkle: '✨',
|
|
||||||
rocket: '🚀',
|
|
||||||
thinking: '🤔',
|
|
||||||
chat: '💬'
|
|
||||||
}
|
|
||||||
|
|
||||||
// Store recent inputs for smart suggestions
|
|
||||||
const recentInputs = {
|
|
||||||
searches: [] as string[],
|
|
||||||
ids: [] as string[],
|
|
||||||
types: [] as string[],
|
|
||||||
formats: [] as string[]
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Professional prompt wrapper with consistent styling
|
|
||||||
*/
|
|
||||||
export async function prompt(config: any): Promise<any> {
|
|
||||||
// Add consistent styling
|
|
||||||
if (config.message) {
|
|
||||||
config.message = colors.cyan(config.message)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Add prefix with appropriate icon
|
|
||||||
if (!config.prefix) {
|
|
||||||
config.prefix = colors.dim(' › ')
|
|
||||||
}
|
|
||||||
|
|
||||||
return inquirer.prompt([config])
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Interactive prompt for search query with smart features
|
|
||||||
*/
|
|
||||||
export async function promptSearchQuery(previousSearches?: string[]): Promise<string> {
|
|
||||||
console.log(colors.primary(`\n${icons.search} Smart Search\n`))
|
|
||||||
console.log(colors.dim('Search your neural database with natural language'))
|
|
||||||
console.log(colors.dim('Examples: "meetings last week", "John from Google", "important documents"'))
|
|
||||||
|
|
||||||
const { query } = await prompt({
|
|
||||||
type: 'input',
|
|
||||||
name: 'query',
|
|
||||||
message: 'What would you like to search for?',
|
|
||||||
validate: (input: string) => {
|
|
||||||
if (!input.trim()) {
|
|
||||||
return 'Please enter a search query'
|
|
||||||
}
|
|
||||||
return true
|
|
||||||
},
|
|
||||||
transformer: (input: string) => {
|
|
||||||
// Show live character count
|
|
||||||
const count = input.length
|
|
||||||
if (count > 100) {
|
|
||||||
return colors.warning(input)
|
|
||||||
}
|
|
||||||
return colors.green(input)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
// Store for future suggestions
|
|
||||||
if (!recentInputs.searches.includes(query)) {
|
|
||||||
recentInputs.searches.unshift(query)
|
|
||||||
recentInputs.searches = recentInputs.searches.slice(0, 10)
|
|
||||||
}
|
|
||||||
|
|
||||||
return query
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Interactive prompt for item ID with fuzzy search
|
|
||||||
*/
|
|
||||||
export async function promptItemId(
|
|
||||||
action: string,
|
|
||||||
brain?: Brainy,
|
|
||||||
allowMultiple: boolean = false
|
|
||||||
): Promise<string | string[]> {
|
|
||||||
console.log(colors.primary(`\n${icons.thinking} Select item to ${action}\n`))
|
|
||||||
|
|
||||||
// If we have brain instance, show recent items
|
|
||||||
let choices: any[] = []
|
|
||||||
if (brain) {
|
|
||||||
try {
|
|
||||||
const recent = await brain.find({
|
|
||||||
query: '*',
|
|
||||||
limit: 10
|
|
||||||
})
|
|
||||||
|
|
||||||
choices = recent.map(item => ({
|
|
||||||
name: `${item.id} - ${(item as any).content?.substring(0, 50) || 'No content'}...`,
|
|
||||||
value: item.id,
|
|
||||||
short: item.id
|
|
||||||
}))
|
|
||||||
} catch {
|
|
||||||
// Fallback to manual input
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (choices.length > 0) {
|
|
||||||
choices.push(new inquirer.Separator())
|
|
||||||
choices.push({ name: 'Enter ID manually', value: '__manual__' })
|
|
||||||
|
|
||||||
const { selected } = await prompt({
|
|
||||||
type: allowMultiple ? 'checkbox' : 'list',
|
|
||||||
name: 'selected',
|
|
||||||
message: `Select item(s) to ${action}:`,
|
|
||||||
choices,
|
|
||||||
pageSize: 10
|
|
||||||
})
|
|
||||||
|
|
||||||
if (selected === '__manual__' || (Array.isArray(selected) && selected.includes('__manual__'))) {
|
|
||||||
return promptManualId(action, allowMultiple)
|
|
||||||
}
|
|
||||||
|
|
||||||
return selected
|
|
||||||
} else {
|
|
||||||
return promptManualId(action, allowMultiple)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Manual ID input with validation
|
|
||||||
*/
|
|
||||||
async function promptManualId(action: string, allowMultiple: boolean): Promise<string | string[]> {
|
|
||||||
const { id } = await prompt({
|
|
||||||
type: 'input',
|
|
||||||
name: 'id',
|
|
||||||
message: allowMultiple
|
|
||||||
? `Enter ID(s) to ${action} (comma-separated):`
|
|
||||||
: `Enter ID to ${action}:`,
|
|
||||||
validate: (input: string) => {
|
|
||||||
if (!input.trim()) {
|
|
||||||
return `Please enter at least one ID`
|
|
||||||
}
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
if (allowMultiple) {
|
|
||||||
return id.split(',').map((i: string) => i.trim()).filter(Boolean)
|
|
||||||
}
|
|
||||||
return id.trim()
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Confirm destructive action with preview
|
|
||||||
*/
|
|
||||||
export async function confirmDestructiveAction(
|
|
||||||
action: string,
|
|
||||||
items: any[],
|
|
||||||
showPreview: boolean = true
|
|
||||||
): Promise<boolean> {
|
|
||||||
console.log(colors.warning(`\n${icons.warning} Confirmation Required\n`))
|
|
||||||
|
|
||||||
if (showPreview && items.length > 0) {
|
|
||||||
console.log(colors.dim(`You are about to ${action}:`))
|
|
||||||
items.slice(0, 5).forEach(item => {
|
|
||||||
console.log(colors.dim(` • ${item.id || item}`))
|
|
||||||
})
|
|
||||||
if (items.length > 5) {
|
|
||||||
console.log(colors.dim(` ... and ${items.length - 5} more`))
|
|
||||||
}
|
|
||||||
console.log()
|
|
||||||
}
|
|
||||||
|
|
||||||
const { confirm } = await prompt({
|
|
||||||
type: 'confirm',
|
|
||||||
name: 'confirm',
|
|
||||||
message: colors.warning(`Are you sure you want to ${action}?`),
|
|
||||||
default: false
|
|
||||||
})
|
|
||||||
|
|
||||||
return confirm
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Interactive data input with multiline support
|
|
||||||
*/
|
|
||||||
export async function promptDataInput(
|
|
||||||
action: string = 'add',
|
|
||||||
currentValue?: string
|
|
||||||
): Promise<string> {
|
|
||||||
console.log(colors.primary(`\n${icons.add} ${action === 'add' ? 'Add Data' : 'Update Data'}\n`))
|
|
||||||
|
|
||||||
if (currentValue) {
|
|
||||||
console.log(colors.dim('Current value:'))
|
|
||||||
console.log(colors.info(` ${currentValue.substring(0, 100)}${currentValue.length > 100 ? '...' : ''}`))
|
|
||||||
console.log()
|
|
||||||
}
|
|
||||||
|
|
||||||
const { data } = await prompt({
|
|
||||||
type: 'editor',
|
|
||||||
name: 'data',
|
|
||||||
message: 'Enter your data:',
|
|
||||||
default: currentValue || '',
|
|
||||||
postfix: '.md',
|
|
||||||
validate: (input: string) => {
|
|
||||||
if (!input.trim() && action === 'add') {
|
|
||||||
return 'Please enter some data'
|
|
||||||
}
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
return data
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Interactive metadata input with JSON validation
|
|
||||||
*/
|
|
||||||
export async function promptMetadata(
|
|
||||||
currentMetadata?: any,
|
|
||||||
suggestions?: string[]
|
|
||||||
): Promise<any> {
|
|
||||||
console.log(colors.dim('\nOptional: Add metadata (JSON format)'))
|
|
||||||
|
|
||||||
const { addMetadata } = await prompt({
|
|
||||||
type: 'confirm',
|
|
||||||
name: 'addMetadata',
|
|
||||||
message: 'Would you like to add metadata?',
|
|
||||||
default: false
|
|
||||||
})
|
|
||||||
|
|
||||||
if (!addMetadata) {
|
|
||||||
return {}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Show field suggestions if available
|
|
||||||
if (suggestions && suggestions.length > 0) {
|
|
||||||
console.log(colors.dim('\nAvailable fields:'))
|
|
||||||
suggestions.forEach(field => {
|
|
||||||
console.log(colors.dim(` • ${field}`))
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
const { metadata } = await prompt({
|
|
||||||
type: 'editor',
|
|
||||||
name: 'metadata',
|
|
||||||
message: 'Enter metadata (JSON):',
|
|
||||||
default: currentMetadata ? JSON.stringify(currentMetadata, null, 2) : '{\n \n}',
|
|
||||||
postfix: '.json',
|
|
||||||
validate: (input: string) => {
|
|
||||||
try {
|
|
||||||
JSON.parse(input)
|
|
||||||
return true
|
|
||||||
} catch (e) {
|
|
||||||
return `Invalid JSON: ${e.message}`
|
|
||||||
}
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
return JSON.parse(metadata)
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Interactive format selector
|
|
||||||
*/
|
|
||||||
export async function promptFormat(
|
|
||||||
availableFormats: string[],
|
|
||||||
defaultFormat: string
|
|
||||||
): Promise<string> {
|
|
||||||
console.log(colors.primary(`\n${icons.export} Select Format\n`))
|
|
||||||
|
|
||||||
const { format } = await prompt({
|
|
||||||
type: 'list',
|
|
||||||
name: 'format',
|
|
||||||
message: 'Choose export format:',
|
|
||||||
choices: availableFormats.map(f => ({
|
|
||||||
name: getFormatDescription(f),
|
|
||||||
value: f,
|
|
||||||
short: f
|
|
||||||
})),
|
|
||||||
default: defaultFormat
|
|
||||||
})
|
|
||||||
|
|
||||||
return format
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get friendly format descriptions
|
|
||||||
*/
|
|
||||||
function getFormatDescription(format: string): string {
|
|
||||||
const descriptions: Record<string, string> = {
|
|
||||||
json: 'JSON - Universal data interchange',
|
|
||||||
jsonl: 'JSON Lines - Streaming format',
|
|
||||||
csv: 'CSV - Spreadsheet compatible',
|
|
||||||
graphml: 'GraphML - Graph visualization',
|
|
||||||
dot: 'DOT - Graphviz format',
|
|
||||||
d3: 'D3.js - Web visualization',
|
|
||||||
markdown: 'Markdown - Human readable',
|
|
||||||
yaml: 'YAML - Configuration format'
|
|
||||||
}
|
|
||||||
|
|
||||||
return `${format.toUpperCase()} - ${descriptions[format] || 'Custom format'}`
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Interactive file/URL input with validation
|
|
||||||
*/
|
|
||||||
export async function promptFileOrUrl(
|
|
||||||
action: string = 'import'
|
|
||||||
): Promise<string> {
|
|
||||||
console.log(colors.primary(`\n${icons.import} ${action === 'import' ? 'Import Source' : 'Export Destination'}\n`))
|
|
||||||
|
|
||||||
const { sourceType } = await prompt({
|
|
||||||
type: 'list',
|
|
||||||
name: 'sourceType',
|
|
||||||
message: 'What type of source?',
|
|
||||||
choices: [
|
|
||||||
{ name: 'Local file', value: 'file' },
|
|
||||||
{ name: 'URL', value: 'url' },
|
|
||||||
{ name: 'Clipboard', value: 'clipboard' },
|
|
||||||
{ name: 'Direct input', value: 'input' }
|
|
||||||
]
|
|
||||||
})
|
|
||||||
|
|
||||||
switch (sourceType) {
|
|
||||||
case 'file':
|
|
||||||
return promptFilePath(action)
|
|
||||||
case 'url':
|
|
||||||
return promptUrl()
|
|
||||||
case 'clipboard':
|
|
||||||
// Would need clipboard integration
|
|
||||||
console.log(colors.warning('Clipboard support coming soon!'))
|
|
||||||
return promptFilePath(action)
|
|
||||||
case 'input':
|
|
||||||
const data = await promptDataInput('import')
|
|
||||||
// Save to temp file and return path
|
|
||||||
const tmpFile = `/tmp/brainy-import-${Date.now()}.json`
|
|
||||||
const { writeFileSync } = await import('node:fs')
|
|
||||||
writeFileSync(tmpFile, data)
|
|
||||||
return tmpFile
|
|
||||||
default:
|
|
||||||
return ''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* File path input with autocomplete
|
|
||||||
*/
|
|
||||||
async function promptFilePath(action: string): Promise<string> {
|
|
||||||
const { path } = await prompt({
|
|
||||||
type: 'input',
|
|
||||||
name: 'path',
|
|
||||||
message: `Enter file path to ${action}:`,
|
|
||||||
validate: async (input: string) => {
|
|
||||||
if (!input.trim()) {
|
|
||||||
return 'Please enter a file path'
|
|
||||||
}
|
|
||||||
|
|
||||||
const { existsSync } = await import('node:fs')
|
|
||||||
if (action === 'import' && !existsSync(input)) {
|
|
||||||
return `File not found: ${input}`
|
|
||||||
}
|
|
||||||
|
|
||||||
return true
|
|
||||||
},
|
|
||||||
// Add file path autocomplete
|
|
||||||
transformer: (input: string) => {
|
|
||||||
if (input.startsWith('~/')) {
|
|
||||||
const home = process.env.HOME || '~'
|
|
||||||
return colors.green(input.replace('~', home))
|
|
||||||
}
|
|
||||||
return colors.green(input)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
return path
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* URL input with validation
|
|
||||||
*/
|
|
||||||
async function promptUrl(): Promise<string> {
|
|
||||||
const { url } = await prompt({
|
|
||||||
type: 'input',
|
|
||||||
name: 'url',
|
|
||||||
message: 'Enter URL:',
|
|
||||||
validate: (input: string) => {
|
|
||||||
try {
|
|
||||||
new URL(input)
|
|
||||||
return true
|
|
||||||
} catch {
|
|
||||||
return 'Please enter a valid URL'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
return url
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Interactive relationship builder
|
|
||||||
*/
|
|
||||||
export async function promptRelationship(brain?: Brainy): Promise<{
|
|
||||||
source: string
|
|
||||||
verb: string
|
|
||||||
target: string
|
|
||||||
metadata?: any
|
|
||||||
}> {
|
|
||||||
console.log(colors.primary(`\n${icons.connect} Create Relationship\n`))
|
|
||||||
console.log(colors.dim('Connect two items with a semantic relationship'))
|
|
||||||
|
|
||||||
// Get source
|
|
||||||
const source = await promptItemId('connect from', brain, false) as string
|
|
||||||
|
|
||||||
// Get verb/relationship type
|
|
||||||
const { verb } = await prompt({
|
|
||||||
type: 'list',
|
|
||||||
name: 'verb',
|
|
||||||
message: 'Relationship type:',
|
|
||||||
choices: [
|
|
||||||
{ name: 'Works For', value: 'WorksFor' },
|
|
||||||
{ name: 'Knows', value: 'Knows' },
|
|
||||||
{ name: 'Created By', value: 'CreatedBy' },
|
|
||||||
{ name: 'Belongs To', value: 'BelongsTo' },
|
|
||||||
{ name: 'Uses', value: 'Uses' },
|
|
||||||
{ name: 'Manages', value: 'Manages' },
|
|
||||||
{ name: 'Located In', value: 'LocatedIn' },
|
|
||||||
{ name: 'Related To', value: 'RelatedTo' },
|
|
||||||
new inquirer.Separator(),
|
|
||||||
{ name: 'Custom relationship...', value: '__custom__' }
|
|
||||||
]
|
|
||||||
})
|
|
||||||
|
|
||||||
let finalVerb = verb
|
|
||||||
if (verb === '__custom__') {
|
|
||||||
const { customVerb } = await prompt({
|
|
||||||
type: 'input',
|
|
||||||
name: 'customVerb',
|
|
||||||
message: 'Enter custom relationship:',
|
|
||||||
validate: (input: string) => input.trim() ? true : 'Please enter a relationship'
|
|
||||||
})
|
|
||||||
finalVerb = customVerb
|
|
||||||
}
|
|
||||||
|
|
||||||
// Get target
|
|
||||||
const target = await promptItemId('connect to', brain, false) as string
|
|
||||||
|
|
||||||
// Optional metadata
|
|
||||||
const metadata = await promptMetadata()
|
|
||||||
|
|
||||||
return {
|
|
||||||
source,
|
|
||||||
verb: finalVerb,
|
|
||||||
target,
|
|
||||||
metadata: Object.keys(metadata).length > 0 ? metadata : undefined
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Smart command suggestions when user types wrong command
|
|
||||||
*/
|
|
||||||
export function suggestCommand(input: string, availableCommands: string[]): string[] {
|
|
||||||
// Simple fuzzy matching without external dependency
|
|
||||||
// Filter commands that start with or contain the input
|
|
||||||
const matches = availableCommands
|
|
||||||
.filter(cmd => cmd.toLowerCase().includes(input.toLowerCase()))
|
|
||||||
.sort((a, b) => {
|
|
||||||
// Prefer commands that start with the input
|
|
||||||
const aStarts = a.toLowerCase().startsWith(input.toLowerCase())
|
|
||||||
const bStarts = b.toLowerCase().startsWith(input.toLowerCase())
|
|
||||||
if (aStarts && !bStarts) return -1
|
|
||||||
if (!aStarts && bStarts) return 1
|
|
||||||
return 0
|
|
||||||
})
|
|
||||||
return matches.slice(0, 3)
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Beautiful error display with helpful context
|
|
||||||
*/
|
|
||||||
export function showError(error: Error, context?: string): void {
|
|
||||||
console.log()
|
|
||||||
console.log(colors.error(`${icons.error} Error`))
|
|
||||||
|
|
||||||
if (context) {
|
|
||||||
console.log(colors.dim(context))
|
|
||||||
}
|
|
||||||
|
|
||||||
console.log(colors.red(error.message))
|
|
||||||
|
|
||||||
// Provide helpful suggestions based on error
|
|
||||||
if (error.message.includes('not found')) {
|
|
||||||
console.log(colors.dim('\nTip: Use "brainy search" to find items'))
|
|
||||||
} else if (error.message.includes('network') || error.message.includes('fetch')) {
|
|
||||||
console.log(colors.dim('\nTip: Check your internet connection'))
|
|
||||||
} else if (error.message.includes('permission')) {
|
|
||||||
console.log(colors.dim('\nTip: Check file permissions or run with appropriate access'))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Progress indicator for long operations
|
|
||||||
*/
|
|
||||||
export class ProgressTracker {
|
|
||||||
private spinner: any
|
|
||||||
private startTime: number
|
|
||||||
|
|
||||||
constructor(message: string) {
|
|
||||||
this.spinner = ora({
|
|
||||||
text: message,
|
|
||||||
color: 'cyan',
|
|
||||||
spinner: 'dots'
|
|
||||||
}).start()
|
|
||||||
this.startTime = Date.now()
|
|
||||||
}
|
|
||||||
|
|
||||||
update(message: string, count?: number, total?: number): void {
|
|
||||||
if (count && total) {
|
|
||||||
const percent = Math.round((count / total) * 100)
|
|
||||||
const elapsed = ((Date.now() - this.startTime) / 1000).toFixed(1)
|
|
||||||
this.spinner.text = `${message} (${percent}% - ${elapsed}s)`
|
|
||||||
} else {
|
|
||||||
this.spinner.text = message
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
succeed(message?: string): void {
|
|
||||||
const elapsed = ((Date.now() - this.startTime) / 1000).toFixed(1)
|
|
||||||
this.spinner.succeed(message ? `${message} (${elapsed}s)` : `Done (${elapsed}s)`)
|
|
||||||
}
|
|
||||||
|
|
||||||
fail(message?: string): void {
|
|
||||||
this.spinner.fail(message || 'Failed')
|
|
||||||
}
|
|
||||||
|
|
||||||
stop(): void {
|
|
||||||
this.spinner.stop()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Welcome message for interactive mode
|
|
||||||
*/
|
|
||||||
export function showWelcome(): void {
|
|
||||||
console.clear()
|
|
||||||
console.log(colors.primary(`
|
|
||||||
╔══════════════════════════════════════════════╗
|
|
||||||
║ ║
|
|
||||||
║ ${icons.brain} BRAINY - Neural Intelligence ║
|
|
||||||
║ Your AI-Powered Second Brain ║
|
|
||||||
║ ║
|
|
||||||
╚══════════════════════════════════════════════╝
|
|
||||||
`))
|
|
||||||
console.log(colors.dim(`Version ${getBrainyVersion()} • Type "help" for commands`))
|
|
||||||
console.log()
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Interactive command selector for beginners
|
|
||||||
*/
|
|
||||||
export async function promptCommand(): Promise<string> {
|
|
||||||
const { command } = await prompt({
|
|
||||||
type: 'list',
|
|
||||||
name: 'command',
|
|
||||||
message: 'What would you like to do?',
|
|
||||||
choices: [
|
|
||||||
{ name: `${icons.add} Add data to your brain`, value: 'add' },
|
|
||||||
{ name: `${icons.search} Search your knowledge`, value: 'search' },
|
|
||||||
{ name: `${icons.chat} Chat with your data`, value: 'chat' },
|
|
||||||
{ name: `${icons.update} Update existing data`, value: 'update' },
|
|
||||||
{ name: `${icons.delete} Delete data`, value: 'delete' },
|
|
||||||
{ name: `${icons.connect} Create relationships`, value: 'relate' },
|
|
||||||
{ name: `${icons.import} Import from file`, value: 'import' },
|
|
||||||
{ name: `${icons.export} Export your brain`, value: 'export' },
|
|
||||||
new inquirer.Separator(),
|
|
||||||
{ name: `${icons.brain} Neural operations`, value: 'neural' },
|
|
||||||
{ name: `${icons.info} View statistics`, value: 'status' },
|
|
||||||
{ name: 'Exit', value: 'exit' }
|
|
||||||
],
|
|
||||||
pageSize: 15
|
|
||||||
})
|
|
||||||
|
|
||||||
return command
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Start interactive REPL mode
|
|
||||||
*/
|
|
||||||
export async function startInteractiveMode() {
|
|
||||||
console.log(chalk.cyan('\n🧠 Brainy Interactive Mode\n'))
|
|
||||||
console.log(chalk.yellow('Interactive REPL mode coming soon\n'))
|
|
||||||
console.log(chalk.dim('Use specific commands for now: brainy add, brainy search, etc.'))
|
|
||||||
process.exit(0)
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Export all interactive components
|
|
||||||
*/
|
|
||||||
export default {
|
|
||||||
colors,
|
|
||||||
icons,
|
|
||||||
prompt,
|
|
||||||
promptSearchQuery,
|
|
||||||
promptItemId,
|
|
||||||
confirmDestructiveAction,
|
|
||||||
promptDataInput,
|
|
||||||
promptMetadata,
|
|
||||||
promptFormat,
|
|
||||||
promptFileOrUrl,
|
|
||||||
promptRelationship,
|
|
||||||
suggestCommand,
|
|
||||||
showError,
|
|
||||||
ProgressTracker,
|
|
||||||
showWelcome,
|
|
||||||
promptCommand,
|
|
||||||
startInteractiveMode
|
|
||||||
}
|
|
||||||
|
|
@ -2,7 +2,7 @@
|
||||||
* @module config/storageAutoConfig
|
* @module config/storageAutoConfig
|
||||||
* @description Storage configuration auto-detection for Brainy 8.0.
|
* @description Storage configuration auto-detection for Brainy 8.0.
|
||||||
*
|
*
|
||||||
* Brainy 8.0 ships **two storage adapters** per `BR-BRAINY-80-STORAGE-SIMPLIFY`:
|
* Brainy 8.0 ships **two storage adapters**:
|
||||||
* - `FILESYSTEM` — persistent on-disk storage (Node, Bun, Deno).
|
* - `FILESYSTEM` — persistent on-disk storage (Node, Bun, Deno).
|
||||||
* - `MEMORY` — in-memory, ephemeral.
|
* - `MEMORY` — in-memory, ephemeral.
|
||||||
*
|
*
|
||||||
|
|
|
||||||
|
|
@ -386,7 +386,7 @@ export interface HNSWVerbWithMetadata {
|
||||||
targetId: string
|
targetId: string
|
||||||
|
|
||||||
// SUBTYPE — optional per-product sub-classification within a VerbType (e.g. a
|
// SUBTYPE — optional per-product sub-classification within a VerbType (e.g. a
|
||||||
// `Manages` relationship might have subtype 'direct' vs 'dotted-line'; a `RelatedTo`
|
// `ReportsTo` relationship might have subtype 'direct' vs 'dotted-line'; a `RelatedTo`
|
||||||
// edge might carry 'spouse' / 'sibling' / 'colleague'). Flat string (no hierarchy) —
|
// edge might carry 'spouse' / 'sibling' / 'colleague'). Flat string (no hierarchy) —
|
||||||
// consumers decide the vocabulary. Indexed and rolled up into per-VerbType statistics
|
// consumers decide the vocabulary. Indexed and rolled up into per-VerbType statistics
|
||||||
// so it's queryable (`getRelations({ verb, subtype })`) and aggregable
|
// so it's queryable (`getRelations({ verb, subtype })`) and aggregable
|
||||||
|
|
|
||||||
|
|
@ -766,8 +766,8 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
||||||
// Note: LSM-trees will be recreated from storage via their own initialization
|
// Note: LSM-trees will be recreated from storage via their own initialization
|
||||||
// Verb data will be loaded on-demand via UnifiedCache
|
// Verb data will be loaded on-demand via UnifiedCache
|
||||||
|
|
||||||
// Brainy 8.0: storage is always local (filesystem or memory) per
|
// Brainy 8.0: storage is always local (filesystem or memory — the
|
||||||
// BR-BRAINY-80-STORAGE-SIMPLIFY. Load all verbs at once.
|
// cloud adapters were removed). Load all verbs at once.
|
||||||
const storageType = this.storage?.constructor.name || ''
|
const storageType = this.storage?.constructor.name || ''
|
||||||
let totalVerbs = 0
|
let totalVerbs = 0
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1261,8 +1261,8 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
||||||
}
|
}
|
||||||
|
|
||||||
// Step 4: Load all HNSW nodes at once. Brainy 8.0 ships filesystem +
|
// Step 4: Load all HNSW nodes at once. Brainy 8.0 ships filesystem +
|
||||||
// memory storage only (per BR-BRAINY-80-STORAGE-SIMPLIFY); the
|
// memory storage only; the cloud-pagination rebuild path was deleted
|
||||||
// cloud-pagination rebuild path was deleted alongside the cloud adapters.
|
// alongside the cloud adapters.
|
||||||
const storageType = this.storage?.constructor.name || ''
|
const storageType = this.storage?.constructor.name || ''
|
||||||
let loadedCount = 0
|
let loadedCount = 0
|
||||||
let totalCount: number | undefined = undefined
|
let totalCount: number | undefined = undefined
|
||||||
|
|
|
||||||
|
|
@ -28,6 +28,9 @@ export type {
|
||||||
RelateParams,
|
RelateParams,
|
||||||
FindParams,
|
FindParams,
|
||||||
SubtypeRegistry,
|
SubtypeRegistry,
|
||||||
|
FillSubtypeRule,
|
||||||
|
FillSubtypeRules,
|
||||||
|
FillSubtypesResult,
|
||||||
AggregateDefinition,
|
AggregateDefinition,
|
||||||
AggregateMetricDef,
|
AggregateMetricDef,
|
||||||
AggregateSource,
|
AggregateSource,
|
||||||
|
|
|
||||||
|
|
@ -8,9 +8,8 @@
|
||||||
* 2. O(1) metadata hints (column names, file structure)
|
* 2. O(1) metadata hints (column names, file structure)
|
||||||
* 3. Format-specific intelligence (Excel, CSV, PDF, YAML, DOCX)
|
* 3. Format-specific intelligence (Excel, CSV, PDF, YAML, DOCX)
|
||||||
*
|
*
|
||||||
* This is the WORKSHOP BUG FIX - finds explicit relationships via exact matching
|
* Finds explicit relationships via exact matching — added after a consumer
|
||||||
*
|
* report of extraction missing explicitly-named relationships.
|
||||||
* PRODUCTION-READY: No TODOs, no mocks, real implementation
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { Brainy } from '../../brainy.js'
|
import type { Brainy } from '../../brainy.js'
|
||||||
|
|
|
||||||
|
|
@ -1528,10 +1528,16 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
||||||
|
|
||||||
const { limit, offset = 0, filter } = options
|
const { limit, offset = 0, filter } = options
|
||||||
const collectedNouns: HNSWNounWithMetadata[] = []
|
const collectedNouns: HNSWNounWithMetadata[] = []
|
||||||
|
// Collect ONE item past the requested window so `hasMore` is decidable:
|
||||||
|
// stopping exactly at offset+limit cannot distinguish "page full, nothing
|
||||||
|
// after it" from "page full, more behind it" — which made hasMore
|
||||||
|
// permanently false and silently truncated every multi-page walk
|
||||||
|
// (audit/migrateField/fillSubtypes, graph verb-id recovery, count rebuilds).
|
||||||
const targetCount = offset + limit
|
const targetCount = offset + limit
|
||||||
|
const peekCount = targetCount + 1
|
||||||
|
|
||||||
// Iterate by shards (0x00-0xFF) instead of types
|
// Iterate by shards (0x00-0xFF) instead of types
|
||||||
for (let shard = 0; shard < 256 && collectedNouns.length < targetCount; shard++) {
|
for (let shard = 0; shard < 256 && collectedNouns.length < peekCount; shard++) {
|
||||||
const shardHex = shard.toString(16).padStart(2, '0')
|
const shardHex = shard.toString(16).padStart(2, '0')
|
||||||
const shardDir = `entities/nouns/${shardHex}`
|
const shardDir = `entities/nouns/${shardHex}`
|
||||||
|
|
||||||
|
|
@ -1539,7 +1545,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
||||||
const nounFiles = await this.listCanonicalObjects(shardDir)
|
const nounFiles = await this.listCanonicalObjects(shardDir)
|
||||||
|
|
||||||
for (const nounPath of nounFiles) {
|
for (const nounPath of nounFiles) {
|
||||||
if (collectedNouns.length >= targetCount) break
|
if (collectedNouns.length >= peekCount) break
|
||||||
if (!nounPath.includes('/vectors.json')) continue
|
if (!nounPath.includes('/vectors.json')) continue
|
||||||
|
|
||||||
try {
|
try {
|
||||||
|
|
@ -1597,7 +1603,8 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Apply pagination
|
// Apply pagination. The peeked extra item (if any) is dropped by the slice;
|
||||||
|
// its existence is exactly what makes hasMore true.
|
||||||
const paginatedNouns = collectedNouns.slice(offset, offset + limit)
|
const paginatedNouns = collectedNouns.slice(offset, offset + limit)
|
||||||
const hasMore = collectedNouns.length > targetCount
|
const hasMore = collectedNouns.length > targetCount
|
||||||
|
|
||||||
|
|
@ -1647,7 +1654,11 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
||||||
|
|
||||||
const { limit, offset = 0, filter } = options // cursor intentionally not extracted (not yet implemented)
|
const { limit, offset = 0, filter } = options // cursor intentionally not extracted (not yet implemented)
|
||||||
const collectedVerbs: HNSWVerbWithMetadata[] = []
|
const collectedVerbs: HNSWVerbWithMetadata[] = []
|
||||||
const targetCount = offset + limit // Early termination target
|
// Same peek-one-past-the-window strategy as getNounsWithPagination — see
|
||||||
|
// the comment there. Without the extra item, hasMore is undecidable and
|
||||||
|
// was permanently false (silent truncation of every multi-page walk).
|
||||||
|
const targetCount = offset + limit // Requested window end
|
||||||
|
const peekCount = targetCount + 1 // Early termination target (window + 1 peek)
|
||||||
|
|
||||||
// Prepare filter sets for efficient lookup
|
// Prepare filter sets for efficient lookup
|
||||||
const filterVerbTypes = filter?.verbType
|
const filterVerbTypes = filter?.verbType
|
||||||
|
|
@ -1668,7 +1679,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
||||||
: null
|
: null
|
||||||
|
|
||||||
// Iterate by shards (0x00-0xFF) instead of types - single pass!
|
// Iterate by shards (0x00-0xFF) instead of types - single pass!
|
||||||
for (let shard = 0; shard < 256 && collectedVerbs.length < targetCount; shard++) {
|
for (let shard = 0; shard < 256 && collectedVerbs.length < peekCount; shard++) {
|
||||||
const shardHex = shard.toString(16).padStart(2, '0')
|
const shardHex = shard.toString(16).padStart(2, '0')
|
||||||
const shardDir = `entities/verbs/${shardHex}`
|
const shardDir = `entities/verbs/${shardHex}`
|
||||||
|
|
||||||
|
|
@ -1676,7 +1687,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
||||||
const verbFiles = await this.listCanonicalObjects(shardDir)
|
const verbFiles = await this.listCanonicalObjects(shardDir)
|
||||||
|
|
||||||
for (const verbPath of verbFiles) {
|
for (const verbPath of verbFiles) {
|
||||||
if (collectedVerbs.length >= targetCount) break
|
if (collectedVerbs.length >= peekCount) break
|
||||||
if (!verbPath.includes('/vectors.json')) continue
|
if (!verbPath.includes('/vectors.json')) continue
|
||||||
|
|
||||||
try {
|
try {
|
||||||
|
|
@ -1737,9 +1748,11 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Apply pagination (Efficient slicing after early termination)
|
// Apply pagination. The peeked extra item (if any) is dropped by the slice;
|
||||||
|
// its existence is exactly what makes hasMore true. `>` (not `>=`) keeps the
|
||||||
|
// exact-boundary case (total == offset+limit) from looping forever.
|
||||||
const paginatedVerbs = collectedVerbs.slice(offset, offset + limit)
|
const paginatedVerbs = collectedVerbs.slice(offset, offset + limit)
|
||||||
const hasMore = collectedVerbs.length > targetCount // Fixed >= to > (was causing infinite loop)
|
const hasMore = collectedVerbs.length > targetCount
|
||||||
|
|
||||||
return {
|
return {
|
||||||
items: paginatedVerbs,
|
items: paginatedVerbs,
|
||||||
|
|
@ -3235,8 +3248,8 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
||||||
* Before this override the Uint32Array counters were only persisted
|
* Before this override the Uint32Array counters were only persisted
|
||||||
* on a heuristic schedule inside `saveNoun_internal` (first-of-type or
|
* on a heuristic schedule inside `saveNoun_internal` (first-of-type or
|
||||||
* every-100th), which left readers seeing stale counts after a clean
|
* every-100th), which left readers seeing stale counts after a clean
|
||||||
* writer flush — the same failure mode that BR-FIND-WHERE-ZERO surfaced
|
* writer flush — the same silent-stale failure mode that once produced
|
||||||
* via `brain.stats()`.
|
* zero counts from `brain.stats()`.
|
||||||
*/
|
*/
|
||||||
public async flushCounts(): Promise<void> {
|
public async flushCounts(): Promise<void> {
|
||||||
await super.flushCounts()
|
await super.flushCounts()
|
||||||
|
|
|
||||||
|
|
@ -8,7 +8,7 @@
|
||||||
* workloads.
|
* workloads.
|
||||||
*
|
*
|
||||||
* Cloud-storage adapters (GCS, S3, R2, Azure) and the browser-only OPFS
|
* Cloud-storage adapters (GCS, S3, R2, Azure) and the browser-only OPFS
|
||||||
* adapter were removed in 8.0 per `BR-BRAINY-80-STORAGE-SIMPLIFY`. The path
|
* adapter were removed in 8.0. The path
|
||||||
* forward for cloud backup is operator tooling: persist locally with
|
* forward for cloud backup is operator tooling: persist locally with
|
||||||
* `db.persist()`, sync the resulting on-disk artefact with `gsutil` /
|
* `db.persist()`, sync the resulting on-disk artefact with `gsutil` /
|
||||||
* `aws s3 cp` / `rclone` / `azcopy`. This is the standard pattern every
|
* `aws s3 cp` / `rclone` / `azcopy`. This is the standard pattern every
|
||||||
|
|
|
||||||
|
|
@ -76,7 +76,7 @@ export interface Relation<T = any> {
|
||||||
/** Relationship type classification (VerbType enum) */
|
/** Relationship type classification (VerbType enum) */
|
||||||
type: VerbType
|
type: VerbType
|
||||||
/**
|
/**
|
||||||
* Per-product sub-classification within the VerbType (e.g. a `Manages`
|
* Per-product sub-classification within the VerbType (e.g. a `ReportsTo`
|
||||||
* relationship might have `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo`
|
* relationship might have `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo`
|
||||||
* edge might carry `'spouse'` / `'sibling'` / `'colleague'`). Flat string, no
|
* edge might carry `'spouse'` / `'sibling'` / `'colleague'`). Flat string, no
|
||||||
* hierarchy. Top-level standard field — indexed on the fast path and rolled into
|
* hierarchy. Top-level standard field — indexed on the fast path and rolled into
|
||||||
|
|
@ -208,6 +208,68 @@ export interface SubtypeRegistry {
|
||||||
// Intentionally empty. Consumers extend via declaration merging.
|
// Intentionally empty. Consumers extend via declaration merging.
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A single back-fill rule for `brain.fillSubtypes()`.
|
||||||
|
*
|
||||||
|
* - A **literal string** assigns that subtype to every matching entry that
|
||||||
|
* lacks one (`'general'` — a blanket default).
|
||||||
|
* - A **function** receives the full entry and returns the subtype to assign,
|
||||||
|
* or `undefined` to leave the entry untouched (it is counted as `skipped`
|
||||||
|
* so a later run with a stricter rule can pick it up). Functions can derive
|
||||||
|
* the subtype from existing fields, e.g. `(e) => e.metadata?.kind`.
|
||||||
|
*
|
||||||
|
* @typeParam E - The entry shape the rule sees: `Entity<T>` for NounType keys,
|
||||||
|
* `Relation<T>` for VerbType keys.
|
||||||
|
*/
|
||||||
|
export type FillSubtypeRule<E> = string | ((entry: E) => string | undefined)
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Rule map for `brain.fillSubtypes()` — the 8.0 subtype migration helper.
|
||||||
|
*
|
||||||
|
* Keys are `NounType` values (entity rules) and/or `VerbType` values
|
||||||
|
* (relationship rules); the two vocabularies don't overlap, so a single map
|
||||||
|
* covers both sides. Each value is a {@link FillSubtypeRule}: a literal
|
||||||
|
* subtype string or a function deriving one from the entry.
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* ```ts
|
||||||
|
* await brain.fillSubtypes({
|
||||||
|
* [NounType.Person]: (e) => e.metadata?.kind ?? 'unspecified',
|
||||||
|
* [NounType.Thing]: 'general', // literal default
|
||||||
|
* [VerbType.RelatedTo]: 'unspecified' // relationship rule
|
||||||
|
* })
|
||||||
|
* ```
|
||||||
|
*/
|
||||||
|
export type FillSubtypeRules<T = any> = {
|
||||||
|
[K in NounType]?: FillSubtypeRule<Entity<T>>
|
||||||
|
} & {
|
||||||
|
[K in VerbType]?: FillSubtypeRule<Relation<T>>
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Summary returned by `brain.fillSubtypes()`.
|
||||||
|
*
|
||||||
|
* After a run, `skipped` is exactly the remaining migration debt — re-running
|
||||||
|
* `brain.audit()` reports the same entries. Entries that already carry a
|
||||||
|
* subtype count toward `scanned` only (they are not debt, so they are neither
|
||||||
|
* `filled` nor `skipped`).
|
||||||
|
*/
|
||||||
|
export interface FillSubtypesResult {
|
||||||
|
/** Total entries examined (entities + relationships). */
|
||||||
|
scanned: number
|
||||||
|
/** Entries that received a subtype during this run. */
|
||||||
|
filled: number
|
||||||
|
/**
|
||||||
|
* Entries still missing a subtype after the run — either their type has no
|
||||||
|
* rule in the map, or their rule function returned `undefined`/empty.
|
||||||
|
*/
|
||||||
|
skipped: number
|
||||||
|
/** Per-entry write failures (`fillSubtypes` continues past individual errors). */
|
||||||
|
errors: Array<{ id: string; error: string }>
|
||||||
|
/** Fill counts grouped by NounType/VerbType key (only types with fills appear). */
|
||||||
|
byType: Record<string, number>
|
||||||
|
}
|
||||||
|
|
||||||
export interface AddParams<T = any> {
|
export interface AddParams<T = any> {
|
||||||
/** Content to embed and store. Strings are auto-embedded; objects are JSON-stringified for embedding. */
|
/** Content to embed and store. Strings are auto-embedded; objects are JSON-stringified for embedding. */
|
||||||
data: any | Vector
|
data: any | Vector
|
||||||
|
|
@ -282,7 +344,7 @@ export interface RelateParams<T = any> {
|
||||||
/** Relationship type classification (required) */
|
/** Relationship type classification (required) */
|
||||||
type: VerbType
|
type: VerbType
|
||||||
/**
|
/**
|
||||||
* Per-product sub-classification within the VerbType (e.g. a `Manages`
|
* Per-product sub-classification within the VerbType (e.g. a `ReportsTo`
|
||||||
* relationship might have `subtype: 'direct'` or `'dotted-line'`; a `RelatedTo`
|
* relationship might have `subtype: 'direct'` or `'dotted-line'`; a `RelatedTo`
|
||||||
* edge might carry `'spouse'` or `'colleague'`). Flat string, no hierarchy.
|
* edge might carry `'spouse'` or `'colleague'`). Flat string, no hierarchy.
|
||||||
* Indexed and rolled up into per-VerbType statistics for fast filtering
|
* Indexed and rolled up into per-VerbType statistics for fast filtering
|
||||||
|
|
@ -1239,16 +1301,21 @@ export interface BrainyConfig {
|
||||||
/**
|
/**
|
||||||
* Brain-wide subtype enforcement mode.
|
* Brain-wide subtype enforcement mode.
|
||||||
*
|
*
|
||||||
* Opt-in in 7.30.0 (default: `false`); becomes the default in 8.0.0.
|
* **Default-on since 8.0.0** (`undefined` resolves to `true`; in 7.30.x this
|
||||||
|
* was an opt-in flag defaulting to `false`).
|
||||||
*
|
*
|
||||||
* - `false` / `undefined` (default): no brain-wide check. Per-type rules
|
* - `true` / `undefined` (default): every `add()` / `addMany()` / `update()` /
|
||||||
* registered via `brain.requireSubtype(type, options)` still apply.
|
* `relate()` / `relateMany()` / `updateRelation()` rejects writes where the
|
||||||
* - `true`: every `add()` / `addMany()` / `update()` / `relate()` /
|
* entity's NounType (or relationship's VerbType) has no non-empty `subtype`
|
||||||
* `relateMany()` / `updateRelation()` rejects writes where the entity's
|
* value. Brainy's own VFS infrastructure writes (`metadata.isVFSEntity` /
|
||||||
* NounType (or relationship's VerbType) has no non-empty `subtype` value.
|
* `metadata.isVFS`) bypass the check.
|
||||||
* - `{ except: [NounType.Thing, NounType.Custom] }`: same as `true`, but
|
* - `{ except: [NounType.Thing, NounType.Custom] }`: same as `true`, but the
|
||||||
* the listed types are allowed through without a subtype. Use for genuine
|
* listed types are allowed through without a subtype. Use for genuine
|
||||||
* catch-all types where no subtype makes sense.
|
* catch-all types where no subtype makes sense.
|
||||||
|
* - `false`: disable the brain-wide check entirely. Last-resort escape hatch
|
||||||
|
* for opening pre-8.0 data — run `brain.audit()` to find the gaps, back-fill
|
||||||
|
* with `brain.fillSubtypes(rules)`, then remove the opt-out so the default
|
||||||
|
* enforcement protects new writes.
|
||||||
*
|
*
|
||||||
* Per-type registrations always compose with the brain-wide flag — a type
|
* Per-type registrations always compose with the brain-wide flag — a type
|
||||||
* registered with `requireSubtype(type, { required: true })` is always
|
* registered with `requireSubtype(type, { required: true })` is always
|
||||||
|
|
|
||||||
|
|
@ -748,9 +748,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
*
|
*
|
||||||
* If neither the column store nor a sparse index covers the field, the
|
* If neither the column store nor a sparse index covers the field, the
|
||||||
* function throws `BrainyError(FIELD_NOT_INDEXED)`. Returning `[]` for a
|
* function throws `BrainyError(FIELD_NOT_INDEXED)`. Returning `[]` for a
|
||||||
* genuinely unindexed field was the bug class `BR-FIND-WHERE-ZERO`
|
* genuinely unindexed field was a long-standing silent-empty bug class —
|
||||||
* tracked — a silent empty result indistinguishable from "the data
|
* an empty result indistinguishable from "the data really isn't there."
|
||||||
* really isn't there."
|
|
||||||
*/
|
*/
|
||||||
private async getIdsFromChunks(field: string, value: any): Promise<string[]> {
|
private async getIdsFromChunks(field: string, value: any): Promise<string[]> {
|
||||||
// Load sparse index via UnifiedCache (lazy loading)
|
// Load sparse index via UnifiedCache (lazy loading)
|
||||||
|
|
@ -1200,7 +1199,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
// - Chunked Sparse Index: ~50 values per chunk, lazy-loaded
|
// - Chunked Sparse Index: ~50 values per chunk, lazy-loaded
|
||||||
// - UnifiedCache LRU: Only hot chunks in memory
|
// - UnifiedCache LRU: Only hot chunks in memory
|
||||||
//
|
//
|
||||||
// Future: Bloom filter hybrid for unlimited words (see .strategy/BILLION-SCALE-PLAN.md)
|
// A Bloom-filter hybrid could lift the per-entity word cap entirely if
|
||||||
|
// full-document indexing at billion-entity scale ever becomes a need.
|
||||||
const textContent = this.extractTextContent(data)
|
const textContent = this.extractTextContent(data)
|
||||||
if (textContent) {
|
if (textContent) {
|
||||||
const MAX_WORDS_PER_ENTITY = 5000 // Handles articles/chapters, memory-safe at scale
|
const MAX_WORDS_PER_ENTITY = 5000 // Handles articles/chapters, memory-safe at scale
|
||||||
|
|
@ -1600,8 +1600,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
* Throws `BrainyError(FIELD_NOT_INDEXED)` if the field has no entries in
|
* Throws `BrainyError(FIELD_NOT_INDEXED)` if the field has no entries in
|
||||||
* either store. Callers in find()-evaluation catch this and translate to
|
* either store. Callers in find()-evaluation catch this and translate to
|
||||||
* an empty result with a logged warning. The throw aligns the production
|
* an empty result with a logged warning. The throw aligns the production
|
||||||
* `find()` path with the `brain.explain()` diagnostic, so the silent-
|
* `find()` path with the `brain.explain()` diagnostic, so a silently
|
||||||
* empty bug class (BR-FIND-WHERE-ZERO) is no longer possible.
|
* empty result for an unindexed field is no longer possible.
|
||||||
*/
|
*/
|
||||||
async getIds(field: string, value: any): Promise<string[]> {
|
async getIds(field: string, value: any): Promise<string[]> {
|
||||||
// Track exact query for field statistics
|
// Track exact query for field statistics
|
||||||
|
|
@ -2866,8 +2866,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
*
|
*
|
||||||
* Prior implementation read from `this.fieldIndexes` + lazy-loaded sparse
|
* Prior implementation read from `this.fieldIndexes` + lazy-loaded sparse
|
||||||
* indices, which silently returned `0` entries for any workspace written
|
* indices, which silently returned `0` entries for any workspace written
|
||||||
* after sparse-index writes were deleted in commit `11be039`. That defect
|
* after sparse-index writes were deleted in commit `11be039`. That
|
||||||
* is what `BR-FIND-WHERE-ZERO` tracked.
|
* silent-zero defect is why this reads the column store first.
|
||||||
*/
|
*/
|
||||||
async getStats(): Promise<MetadataIndexStats> {
|
async getStats(): Promise<MetadataIndexStats> {
|
||||||
const entityCount = this.idMapper.size
|
const entityCount = this.idMapper.size
|
||||||
|
|
|
||||||
|
|
@ -124,7 +124,8 @@ const getContainerMemoryLimit = (): number | null => {
|
||||||
* 1.5 KB + standard fields + metadata). On a 900 MB free-memory box this
|
* 1.5 KB + standard fields + metadata). On a 900 MB free-memory box this
|
||||||
* capped `limit` at 9_000, breaking common safety-cap call patterns like
|
* capped `limit` at 9_000, breaking common safety-cap call patterns like
|
||||||
* `find({ type, where, limit: 10_000 })` that typically return 10-500
|
* `find({ type, where, limit: 10_000 })` that typically return 10-500
|
||||||
* entities. Surfaced as `BR-MAXLIMIT-9000` in PLATFORM-HANDOFF.md.
|
* entities. Reported by a production consumer whose 10K safety-cap
|
||||||
|
* queries started failing after the cap landed.
|
||||||
* - 7.30.2+: `25` (assumes 25 KB per result). Generous over typical (7-10 KB),
|
* - 7.30.2+: `25` (assumes 25 KB per result). Generous over typical (7-10 KB),
|
||||||
* comfortably under the worst case (~20 KB with large metadata blobs).
|
* comfortably under the worst case (~20 KB with large metadata blobs).
|
||||||
* Same 900 MB box now gives ~36_000 — typical 10_000 limits pass silently,
|
* Same 900 MB box now gives ~36_000 — typical 10_000 limits pass silently,
|
||||||
|
|
|
||||||
|
|
@ -226,7 +226,7 @@ describe('Brainy strict-mode self-test (7.30.1)', () => {
|
||||||
const report = await brain.audit()
|
const report = await brain.audit()
|
||||||
expect(report.total).toBeGreaterThanOrEqual(1)
|
expect(report.total).toBeGreaterThanOrEqual(1)
|
||||||
expect(report.entitiesWithoutSubtype['person']).toBe(1)
|
expect(report.entitiesWithoutSubtype['person']).toBe(1)
|
||||||
expect(report.recommendation).toMatch(/Migrate via/)
|
expect(report.recommendation).toMatch(/fillSubtypes/)
|
||||||
})
|
})
|
||||||
|
|
||||||
it('excludes VFS entities by default (they bypass enforcement anyway)', async () => {
|
it('excludes VFS entities by default (they bypass enforcement anyway)', async () => {
|
||||||
|
|
|
||||||
338
tests/unit/brainy/fill-subtypes.test.ts
Normal file
338
tests/unit/brainy/fill-subtypes.test.ts
Normal file
|
|
@ -0,0 +1,338 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/brainy/fill-subtypes
|
||||||
|
* @description Unit tests for `brain.fillSubtypes(rules)` — the 8.0 subtype
|
||||||
|
* migration helper. Proves the full contract: literal and function rules,
|
||||||
|
* entity + relationship fills through one rule map, never overwriting an
|
||||||
|
* existing subtype, function rules declining entries (counted as skipped),
|
||||||
|
* VFS-marker exclusion, the `{ scanned, filled, skipped, errors, byType }`
|
||||||
|
* report, `_rev` bumping through the real update path, idempotent re-runs,
|
||||||
|
* per-entry error collection, and fail-fast rule-map validation.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||||
|
import { Brainy } from '../../../src/brainy'
|
||||||
|
import { NounType, VerbType } from '../../../src/types/graphTypes'
|
||||||
|
import { createTestConfig } from '../../helpers/test-factory'
|
||||||
|
|
||||||
|
describe('Brainy.fillSubtypes()', () => {
|
||||||
|
let brain: Brainy
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
// requireSubtype: false so the tests can create the pre-8.0 migration
|
||||||
|
// debt (entities/relationships without subtype) that fillSubtypes exists
|
||||||
|
// to clear.
|
||||||
|
brain = new Brainy(createTestConfig())
|
||||||
|
await brain.init()
|
||||||
|
})
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('literal rules', () => {
|
||||||
|
it('fills missing subtypes with a literal default', async () => {
|
||||||
|
const a = await brain.add({ type: NounType.Person, data: 'Alice' })
|
||||||
|
const b = await brain.add({ type: NounType.Person, data: 'Bob' })
|
||||||
|
|
||||||
|
const report = await brain.fillSubtypes({
|
||||||
|
[NounType.Person]: 'unspecified'
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(report.filled).toBe(2)
|
||||||
|
expect(report.byType).toEqual({ person: 2 })
|
||||||
|
expect((await brain.get(a))!.subtype).toBe('unspecified')
|
||||||
|
expect((await brain.get(b))!.subtype).toBe('unspecified')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('never overwrites an existing subtype', async () => {
|
||||||
|
const employee = await brain.add({
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'employee',
|
||||||
|
data: 'Has a subtype already'
|
||||||
|
})
|
||||||
|
const blank = await brain.add({ type: NounType.Person, data: 'No subtype' })
|
||||||
|
|
||||||
|
const report = await brain.fillSubtypes({
|
||||||
|
[NounType.Person]: 'unspecified'
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(report.filled).toBe(1)
|
||||||
|
expect((await brain.get(employee))!.subtype).toBe('employee') // untouched
|
||||||
|
expect((await brain.get(blank))!.subtype).toBe('unspecified')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('leaves types without a rule untouched and counts them as skipped', async () => {
|
||||||
|
await brain.add({ type: NounType.Person, data: 'Person without rule coverage' })
|
||||||
|
const doc = await brain.add({ type: NounType.Document, data: 'Doc gets filled' })
|
||||||
|
|
||||||
|
const report = await brain.fillSubtypes({
|
||||||
|
[NounType.Document]: 'general'
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(report.filled).toBe(1)
|
||||||
|
expect(report.skipped).toBe(1) // the Person — still migration debt
|
||||||
|
expect((await brain.get(doc))!.subtype).toBe('general')
|
||||||
|
const audit = await brain.audit()
|
||||||
|
expect(audit.total).toBe(report.skipped) // skipped IS the remaining debt
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('function rules', () => {
|
||||||
|
it('derives the subtype from entity fields', async () => {
|
||||||
|
const vendor = await brain.add({
|
||||||
|
type: NounType.Person,
|
||||||
|
data: 'Vendor person',
|
||||||
|
metadata: { kind: 'vendor' }
|
||||||
|
})
|
||||||
|
const fallback = await brain.add({ type: NounType.Person, data: 'No kind field' })
|
||||||
|
|
||||||
|
const report = await brain.fillSubtypes({
|
||||||
|
[NounType.Person]: (e) => (e.metadata as { kind?: string })?.kind ?? 'unspecified'
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(report.filled).toBe(2)
|
||||||
|
expect((await brain.get(vendor))!.subtype).toBe('vendor')
|
||||||
|
expect((await brain.get(fallback))!.subtype).toBe('unspecified')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('treats undefined returns as declines (selective fill, counted as skipped)', async () => {
|
||||||
|
const classified = await brain.add({
|
||||||
|
type: NounType.Person,
|
||||||
|
data: 'Classifiable',
|
||||||
|
metadata: { department: 'engineering' }
|
||||||
|
})
|
||||||
|
const unclassified = await brain.add({ type: NounType.Person, data: 'Not classifiable' })
|
||||||
|
|
||||||
|
const report = await brain.fillSubtypes({
|
||||||
|
[NounType.Person]: (e) =>
|
||||||
|
(e.metadata as { department?: string })?.department ? 'employee' : undefined
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(report.filled).toBe(1)
|
||||||
|
expect(report.skipped).toBe(1)
|
||||||
|
expect((await brain.get(classified))!.subtype).toBe('employee')
|
||||||
|
expect((await brain.get(unclassified))!.subtype).toBeUndefined()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('treats empty-string returns as declines (an empty subtype would fail enforcement)', async () => {
|
||||||
|
await brain.add({ type: NounType.Person, data: 'Rule returns empty string' })
|
||||||
|
|
||||||
|
const report = await brain.fillSubtypes({
|
||||||
|
[NounType.Person]: () => ''
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(report.filled).toBe(0)
|
||||||
|
expect(report.skipped).toBe(1)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('collects per-entry rule errors and continues the pass', async () => {
|
||||||
|
const poisoned = await brain.add({
|
||||||
|
type: NounType.Person,
|
||||||
|
data: 'Rule throws on this one',
|
||||||
|
metadata: { poison: true }
|
||||||
|
})
|
||||||
|
const healthy = await brain.add({ type: NounType.Person, data: 'Rule works on this one' })
|
||||||
|
|
||||||
|
const report = await brain.fillSubtypes({
|
||||||
|
[NounType.Person]: (e) => {
|
||||||
|
if ((e.metadata as { poison?: boolean })?.poison) {
|
||||||
|
throw new Error('cannot classify poisoned entity')
|
||||||
|
}
|
||||||
|
return 'unspecified'
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(report.filled).toBe(1)
|
||||||
|
expect(report.errors).toHaveLength(1)
|
||||||
|
expect(report.errors[0].id).toBe(poisoned)
|
||||||
|
expect(report.errors[0].error).toMatch(/cannot classify/)
|
||||||
|
expect((await brain.get(healthy))!.subtype).toBe('unspecified')
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('relationship rules (VerbType keys)', () => {
|
||||||
|
it('fills missing relationship subtypes through the same rule map', async () => {
|
||||||
|
const a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'A' })
|
||||||
|
const b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'B' })
|
||||||
|
const relId = await brain.relate({ from: a, to: b, type: VerbType.RelatedTo })
|
||||||
|
const labeled = await brain.relate({
|
||||||
|
from: b,
|
||||||
|
to: a,
|
||||||
|
type: VerbType.RelatedTo,
|
||||||
|
subtype: 'colleague'
|
||||||
|
})
|
||||||
|
|
||||||
|
const report = await brain.fillSubtypes({
|
||||||
|
[VerbType.RelatedTo]: 'unspecified'
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(report.filled).toBe(1)
|
||||||
|
expect(report.byType).toEqual({ relatedTo: 1 })
|
||||||
|
|
||||||
|
const relations = await brain.getRelations({ from: a })
|
||||||
|
expect(relations.find((r) => r.id === relId)!.subtype).toBe('unspecified')
|
||||||
|
const reverse = await brain.getRelations({ from: b })
|
||||||
|
expect(reverse.find((r) => r.id === labeled)!.subtype).toBe('colleague') // untouched
|
||||||
|
})
|
||||||
|
|
||||||
|
it('passes the full Relation shape to relationship rule functions', async () => {
|
||||||
|
const a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'A' })
|
||||||
|
const b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'B' })
|
||||||
|
await brain.relate({
|
||||||
|
from: a,
|
||||||
|
to: b,
|
||||||
|
type: VerbType.ReportsTo,
|
||||||
|
metadata: { dotted: true }
|
||||||
|
})
|
||||||
|
|
||||||
|
const report = await brain.fillSubtypes({
|
||||||
|
[VerbType.ReportsTo]: (r) => {
|
||||||
|
// The rule sees the public Relation shape — endpoints included.
|
||||||
|
expect(r.from).toBe(a)
|
||||||
|
expect(r.to).toBe(b)
|
||||||
|
return (r.metadata as { dotted?: boolean })?.dotted ? 'dotted-line' : 'direct'
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(report.filled).toBe(1)
|
||||||
|
const relations = await brain.getRelations({ from: a })
|
||||||
|
expect(relations[0].subtype).toBe('dotted-line')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('handles entity and relationship rules in a single pass', async () => {
|
||||||
|
const a = await brain.add({ type: NounType.Person, data: 'A' })
|
||||||
|
const b = await brain.add({ type: NounType.Person, data: 'B' })
|
||||||
|
await brain.relate({ from: a, to: b, type: VerbType.RelatedTo })
|
||||||
|
|
||||||
|
const report = await brain.fillSubtypes({
|
||||||
|
[NounType.Person]: 'unspecified',
|
||||||
|
[VerbType.RelatedTo]: 'unspecified'
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(report.filled).toBe(3)
|
||||||
|
expect(report.byType).toEqual({ person: 2, relatedTo: 1 })
|
||||||
|
const audit = await brain.audit()
|
||||||
|
expect(audit.total).toBe(0)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('report + write semantics', () => {
|
||||||
|
it('bumps _rev through the real update path', async () => {
|
||||||
|
const id = await brain.add({ type: NounType.Person, data: 'Rev check' })
|
||||||
|
const before = await brain.get(id)
|
||||||
|
|
||||||
|
await brain.fillSubtypes({ [NounType.Person]: 'unspecified' })
|
||||||
|
|
||||||
|
const after = await brain.get(id)
|
||||||
|
expect(after!._rev).toBe((before!._rev ?? 1) + 1)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('is idempotent — a re-run scans but fills nothing', async () => {
|
||||||
|
await brain.add({ type: NounType.Person, data: 'Fill once' })
|
||||||
|
|
||||||
|
const first = await brain.fillSubtypes({ [NounType.Person]: 'unspecified' })
|
||||||
|
expect(first.filled).toBe(1)
|
||||||
|
|
||||||
|
const second = await brain.fillSubtypes({ [NounType.Person]: 'unspecified' })
|
||||||
|
expect(second.filled).toBe(0)
|
||||||
|
expect(second.skipped).toBe(0)
|
||||||
|
expect(second.scanned).toBeGreaterThan(0)
|
||||||
|
expect(second.errors).toEqual([])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('excludes VFS-marked entries by default, fills them with includeVFS: true', async () => {
|
||||||
|
// VFS markers bypass subtype enforcement, so marked entries are not
|
||||||
|
// migration debt. Fabricate one explicitly (public consumers shouldn't
|
||||||
|
// set this marker; the test exercises the exclusion contract).
|
||||||
|
const vfsLike = await brain.add({
|
||||||
|
type: NounType.Document,
|
||||||
|
data: 'Infrastructure-marked entry',
|
||||||
|
metadata: { isVFS: true }
|
||||||
|
})
|
||||||
|
|
||||||
|
const defaultRun = await brain.fillSubtypes({ [NounType.Document]: 'general' })
|
||||||
|
expect(defaultRun.filled).toBe(0)
|
||||||
|
expect((await brain.get(vfsLike))!.subtype).toBeUndefined()
|
||||||
|
|
||||||
|
const inclusiveRun = await brain.fillSubtypes(
|
||||||
|
{ [NounType.Document]: 'general' },
|
||||||
|
{ includeVFS: true }
|
||||||
|
)
|
||||||
|
expect(inclusiveRun.filled).toBeGreaterThanOrEqual(1)
|
||||||
|
expect((await brain.get(vfsLike))!.subtype).toBe('general')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('reports progress after each batch', async () => {
|
||||||
|
await brain.add({ type: NounType.Person, data: 'P1' })
|
||||||
|
await brain.add({ type: NounType.Person, data: 'P2' })
|
||||||
|
await brain.add({ type: NounType.Person, data: 'P3' })
|
||||||
|
|
||||||
|
const snapshots: Array<{ scanned: number; filled: number; skipped: number }> = []
|
||||||
|
await brain.fillSubtypes(
|
||||||
|
{ [NounType.Person]: 'unspecified' },
|
||||||
|
{
|
||||||
|
batchSize: 1,
|
||||||
|
onProgress: (p) => snapshots.push({ ...p })
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
expect(snapshots.length).toBeGreaterThanOrEqual(3)
|
||||||
|
const last = snapshots[snapshots.length - 1]
|
||||||
|
expect(last.filled).toBe(3)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('clears the debt brain.audit() reports', async () => {
|
||||||
|
await brain.add({ type: NounType.Person, data: 'Debt 1' })
|
||||||
|
await brain.add({ type: NounType.Document, data: 'Debt 2' })
|
||||||
|
|
||||||
|
const before = await brain.audit()
|
||||||
|
expect(before.total).toBe(2)
|
||||||
|
|
||||||
|
await brain.fillSubtypes({
|
||||||
|
[NounType.Person]: 'unspecified',
|
||||||
|
[NounType.Document]: 'general'
|
||||||
|
})
|
||||||
|
|
||||||
|
const after = await brain.audit()
|
||||||
|
expect(after.total).toBe(0)
|
||||||
|
expect(after.recommendation).toMatch(/strict-mode-ready/)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('rule-map validation (fail fast, before touching data)', () => {
|
||||||
|
it('rejects an empty rules map', async () => {
|
||||||
|
await expect(brain.fillSubtypes({})).rejects.toThrow(/rules map is empty/)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('rejects keys that are not a NounType or VerbType', async () => {
|
||||||
|
await expect(
|
||||||
|
brain.fillSubtypes({ 'not-a-type': 'whatever' } as never)
|
||||||
|
).rejects.toThrow(/'not-a-type' is not a valid NounType or VerbType/)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('rejects empty-string literal rules', async () => {
|
||||||
|
await expect(
|
||||||
|
brain.fillSubtypes({ [NounType.Person]: '' })
|
||||||
|
).rejects.toThrow(/empty string/)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('rejects rule values that are neither string nor function', async () => {
|
||||||
|
await expect(
|
||||||
|
brain.fillSubtypes({ [NounType.Person]: 42 } as never)
|
||||||
|
).rejects.toThrow(/must be a subtype string or a function/)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('does not write anything when validation fails', async () => {
|
||||||
|
const id = await brain.add({ type: NounType.Person, data: 'Untouched on failure' })
|
||||||
|
|
||||||
|
await expect(
|
||||||
|
brain.fillSubtypes({
|
||||||
|
[NounType.Person]: 'unspecified',
|
||||||
|
'bogus-type': 'x'
|
||||||
|
} as never)
|
||||||
|
).rejects.toThrow(/bogus-type/)
|
||||||
|
|
||||||
|
expect((await brain.get(id))!.subtype).toBeUndefined()
|
||||||
|
})
|
||||||
|
})
|
||||||
|
})
|
||||||
Loading…
Add table
Add a link
Reference in a new issue