feat(8.0): brain.fillSubtypes migration helper + pre-RC1 gap closure

- brain.fillSubtypes(rules): idempotent subtype back-fill for pre-8.0 data.
  One rule per NounType/VerbType (literal default or per-entry function);
  fills only entries still missing a subtype through the public update()/
  updateRelation() paths; returns { scanned, filled, skipped, errors, byType }.
  Full unit suite in tests/unit/brainy/fill-subtypes.test.ts.
- Fix getNouns/getVerbs pagination hasMore (peek one past the window) —
  was permanently false, silently truncating every multi-page walk.
- find({ near }) without near.id now throws a teaching error instead of an
  opaque storage sharding failure; CLI --threshold without --near applies a
  plain score floor.
- CLI init/close audit: every one-shot command init()s, close()s, and exits
  explicitly; delete the unmaintained interactive REPL; replace the cloud-era
  storage subcommands with status/batch-delete; new types/validate commands.
- requireSubtype JSDoc now documents the 8.0 default-on contract; audit()
  recommendation points at fillSubtypes.
- Docs: data-storage-architecture rewritten to the real 8.0 on-disk layout;
  README storage section reflects filesystem+memory and snapshots; eli5 and
  SEMANTIC_VFS /as-of/ semantics corrected; internal tracker IDs and
  .strategy references scrubbed from published files.
This commit is contained in:
David Snelling 2026-06-11 10:42:34 -07:00
parent 9b0f4acd5b
commit c44678390e
30 changed files with 1517 additions and 3226 deletions

File diff suppressed because it is too large Load diff

View file

@ -86,9 +86,8 @@ Benefits:
`FileSystemStorage` is the only one in-tree, but Cortex's
`MmapFileSystemStorage` inherits from it — interface relocation would
ripple through the plugin ecosystem.
- The current state works. Real failure modes
(BR-CX-INTERFACE-GAP) were build/install artifacts, not type-system
failures.
- The current state works. The real failure modes seen in the field
were build/install artifacts, not type-system failures.
- 7.22.0 just shipped a clean fix. Stacking another refactor before
consumers absorb it adds churn without urgency.
- The `hasStorageMethod()` guard accomplishes the same runtime safety the

View file

@ -133,7 +133,7 @@ Brainy is the only row with every box checked. And it runs all of them in a sing
### One library, any scale
Brainy scales from a single laptop to billions of entities without changing a line of code. Small datasets live in memory. Larger ones spill to disk. At cloud scale, Brainy uses S3-compatible storage and automatically shards across nodes — the same API the whole way. Up to ten billion entities is fully implemented today.
Brainy scales from a quick experiment to serious production datasets without changing a line of code. Small datasets live entirely in memory. Larger ones spill to disk, where Brainy shards and compresses everything automatically. Need a backup or a copy? Snapshots are instant — the same API the whole way.
Add Cortex and you also unlock memory-mapped storage — aggregate state lives directly in the operating system's memory with zero serialization overhead, as fast as the hardware allows.

View file

@ -47,16 +47,18 @@ const authFiles = await vfs.readdir('/by-concept/authentication')
const aliceFiles = await vfs.readdir('/by-author/alice')
```
### 2. **Time Travel**
See your codebase as it existed at any point:
### 2. **Change Tracking by Date**
List the files that were modified on any given day:
```typescript
// Code from March 15th
const snapshot = await vfs.readdir('/as-of/2024-03-15')
// Files that changed on March 15th
const changed = await vfs.readdir('/as-of/2024-03-15')
// Compare with today
// Everything under /src right now
const current = await vfs.readdir('/src')
```
`/as-of/<date>` selects by *modification date* — it reads the files' current content, not historical versions. For true point-in-time queries over entity state, use the Db API (`brain.asOf(generation)`).
### 3. **Knowledge Graph Navigation**
Navigate by semantic relationships:
```typescript
@ -118,13 +120,14 @@ await vfs.stat('/by-author/alice/config.ts')
### 4. By Time (Temporal) ✅ **Production**
```typescript
await vfs.readdir('/as-of/2024-03-15')
// Files modified on March 15, 2024
// Files modified on March 15, 2024 (24-hour window)
await vfs.readFile('/as-of/2024-03-15/src/auth.ts')
// Read auth.ts as it existed that day
await vfs.readFile('/as-of/2024-03-15/auth.ts')
// Current content of auth.ts, addressed by modification date —
// the path only resolves if auth.ts was modified that day
```
**How it works:** Tracks `modified` timestamp. Uses B-tree range queries (`greaterEqual`/`lessEqual`) for O(log n) performance.
**How it works:** Tracks the `modified` timestamp on every file and runs a range query (`greaterEqual`/`lessEqual`) over one 24-hour window for O(log n) performance. The VFS does not store historical file contents — `/as-of/` filters by *when a file last changed*; reads return the current bytes. For point-in-time state, use the Db API (`brain.asOf(generation)`).
**Status:** ✅ Fully implemented and tested at 10K file scale
@ -222,7 +225,7 @@ console.log(authFiles)
// ['login.ts', 'signup.ts', 'oauth.ts']
```
### Example 2: Time Travel
### Example 2: Changes by Day
```typescript
// See what changed today
const today = new Date().toISOString().split('T')[0]
@ -232,8 +235,8 @@ const todaysFiles = await vfs.readdir(`/as-of/${today}`)
const yesterday = new Date(Date.now() - 86400000).toISOString().split('T')[0]
const yesterdaysFiles = await vfs.readdir(`/as-of/${yesterday}`)
const newFiles = todaysFiles.filter(f => !yesterdaysFiles.includes(f))
console.log('New files today:', newFiles)
const onlyToday = todaysFiles.filter(f => !yesterdaysFiles.includes(f))
console.log('Changed today (untouched yesterday):', onlyToday)
```
### Example 3: Graph Navigation
@ -432,25 +435,17 @@ await vfs.writeFile(path, code, {
})
```
### 3. Optimize for Your Scale
### 3. Combine Dimensions
```typescript
// For < 100K files: Post-filtering is fine
// For > 100K files: Use flattened indexes
// Force index refresh after bulk operations
await brain.storage.rebuildIndexes()
```
### 4. Combine Dimensions
```typescript
// Find security files Alice worked on this week
// Find security files Alice changed on a given day
// (each /as-of/<date> path covers exactly that one day)
const aliceFiles = await vfs.readdir('/by-author/alice')
const securityFiles = await vfs.readdir('/by-tag/security')
const thisWeek = await vfs.readdir(`/as-of/${weekAgo}`)
const changedThatDay = await vfs.readdir('/as-of/2024-03-15')
const intersection = aliceFiles
.filter(f => securityFiles.includes(f))
.filter(f => thisWeek.includes(f))
.filter(f => changedThatDay.includes(f))
```
---
@ -469,14 +464,14 @@ console.log(entity.metadata.concepts)
### Slow Queries on Large Datasets
```typescript
// Check if indexes are built
const stats = await brain.storage.getIndexStats()
// Check if indexes are built and populated
const stats = await brain.getIndexStats()
console.log(stats)
// Rebuild if needed
await brain.storage.rebuildIndexes()
```
If an index looks empty or inconsistent, rebuild from raw storage with the CLI
(stop the live writer first): `brainy inspect repair <data-dir>`.
### Semantic Path Returns Empty
```typescript
// Check if metadata exists