From c44678390e9809eefb469a00d174d925b11381fd Mon Sep 17 00:00:00 2001 From: David Snelling Date: Thu, 11 Jun 2026 10:42:34 -0700 Subject: [PATCH] feat(8.0): brain.fillSubtypes migration helper + pre-RC1 gap closure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - brain.fillSubtypes(rules): idempotent subtype back-fill for pre-8.0 data. One rule per NounType/VerbType (literal default or per-entry function); fills only entries still missing a subtype through the public update()/ updateRelation() paths; returns { scanned, filled, skipped, errors, byType }. Full unit suite in tests/unit/brainy/fill-subtypes.test.ts. - Fix getNouns/getVerbs pagination hasMore (peek one past the window) — was permanently false, silently truncating every multi-page walk. - find({ near }) without near.id now throws a teaching error instead of an opaque storage sharding failure; CLI --threshold without --near applies a plain score floor. - CLI init/close audit: every one-shot command init()s, close()s, and exits explicitly; delete the unmaintained interactive REPL; replace the cloud-era storage subcommands with status/batch-delete; new types/validate commands. - requireSubtype JSDoc now documents the 8.0 default-on contract; audit() recommendation points at fillSubtypes. - Docs: data-storage-architecture rewritten to the real 8.0 on-disk layout; README storage section reflects filesystem+memory and snapshots; eli5 and SEMANTIC_VFS /as-of/ semantics corrected; internal tracker IDs and .strategy references scrubbed from published files. --- README.md | 40 +- RELEASES.md | 23 +- .../architecture/data-storage-architecture.md | 1615 +++-------------- .../multiprocess-storage-mixin.md | 5 +- docs/eli5.md | 2 +- docs/vfs/SEMANTIC_VFS.md | 55 +- src/brainy.ts | 305 +++- src/cli/commands/core.ts | 63 +- src/cli/commands/import.ts | 14 +- src/cli/commands/insights.ts | 23 + src/cli/commands/neural.ts | 273 +-- src/cli/commands/nlp.ts | 17 + src/cli/commands/storage.ts | 914 ++-------- src/cli/commands/utility.ts | 61 +- src/cli/commands/vfs.ts | 48 +- src/cli/index.ts | 131 +- src/cli/interactive.ts | 653 ------- src/config/storageAutoConfig.ts | 2 +- src/coreTypes.ts | 2 +- src/graph/graphAdjacencyIndex.ts | 4 +- src/hnsw/hnswIndex.ts | 4 +- src/index.ts | 3 + src/neural/signals/ExactMatchSignal.ts | 5 +- src/storage/baseStorage.ts | 33 +- src/storage/storageFactory.ts | 2 +- src/types/brainy.types.ts | 87 +- src/utils/metadataIndex.ts | 16 +- src/utils/paramValidation.ts | 3 +- .../integration/strict-mode-self-test.test.ts | 2 +- tests/unit/brainy/fill-subtypes.test.ts | 338 ++++ 30 files changed, 1517 insertions(+), 3226 deletions(-) delete mode 100644 src/cli/interactive.ts create mode 100644 tests/unit/brainy/fill-subtypes.test.ts diff --git a/README.md b/README.md index a1c8a8fd..0dd38a86 100644 --- a/README.md +++ b/README.md @@ -278,11 +278,16 @@ brain.counts.byRelationshipSubtype(VerbType.ReportsTo) **Enforce the pairing.** Register a vocabulary per type or turn on brain-wide strict mode to ensure every entity AND relationship has both `type` AND `subtype`: ```javascript -// Per-type rule with vocabulary +// Per-type rule with a closed vocabulary brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true }) -// Or brain-wide strict mode -const brain = new Brainy({ requireSubtype: true }) +// 8.0 default: every write requires a subtype. Exempt genuine catch-all types… +const brain = new Brainy({ requireSubtype: { except: [NounType.Thing] } }) + +// …or opt out while migrating pre-8.0 data, then audit and back-fill: +const legacy = new Brainy({ requireSubtype: false }) +await legacy.audit() // gaps, grouped by type +await legacy.fillSubtypes({ [NounType.Person]: 'unspecified' }) // close them ``` For other facets you want counted (`status`, `source`, `role`), register them with `brain.trackField(name)`. Renaming an existing convention to `subtype`? Use `brain.migrateField({from, to, entityKind: 'both'})` to walk nouns AND verbs in one pass. Full guide: **[Subtypes & Facets](docs/guides/subtypes-and-facets.md)**. @@ -291,7 +296,7 @@ For other facets you want counted (`status`, `source`, `role`), register them wi --- -## Storage: Memory to Cloud +## Storage: Memory and Filesystem The same API at every scale. Change one config line to go from prototype to production. @@ -301,28 +306,31 @@ The same API at every scale. Change one config line to go from prototype to prod const brain = new Brainy() ``` -### Production — Filesystem with Compression +### Production — Filesystem (gzip compression on by default) ```javascript const brain = new Brainy({ - storage: { type: 'filesystem', path: './data', compression: true } + storage: { type: 'filesystem', rootDirectory: './data' } }) ``` -### Cloud — S3, GCS, Azure, Cloudflare R2 +### Backups and Portability — Snapshots ```javascript -const brain = new Brainy({ - storage: { - type: 's3', - s3Storage: { bucketName: 'my-knowledge-base', region: 'us-east-1' } - } -}) +const db = brain.now() +await db.persist('/backups/2026-06-11') // instant hard-link snapshot +await db.release() + +const snapshot = await Brainy.load('/backups/2026-06-11') // read-only Db +const hits = await snapshot.search('quarterly invoices') +await snapshot.release() ``` +A snapshot directory is self-contained — copy it to another machine, open it with `Brainy.load()`, or restore it wholesale with `brain.restore(path, { confirm: true })`. + Performance benchmarks and capacity planning in **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)**. -**[Cloud Deployment Guide](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** | **[Capacity Planning](docs/operations/capacity-planning.md)** +**[Capacity Planning](docs/operations/capacity-planning.md)** --- @@ -371,10 +379,8 @@ Performance benchmarks and capacity planning in **[docs/PERFORMANCE.md](docs/PER ### Operations -- **[Cloud Deployment](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** — AWS, GCS, Azure - **[Capacity Planning](docs/operations/capacity-planning.md)** — Memory, storage, and scaling - **[Performance](docs/PERFORMANCE.md)** — Benchmarks and architecture details -- Cost Optimization: **[AWS S3](docs/operations/cost-optimization-aws-s3.md)** | **[GCS](docs/operations/cost-optimization-gcs.md)** | **[Azure](docs/operations/cost-optimization-azure.md)** | **[R2](docs/operations/cost-optimization-cloudflare-r2.md)** --- @@ -387,7 +393,7 @@ bun install @soulcraft/brainy # Bun — best performance npm install @soulcraft/brainy # Node.js — fully supported ``` -> **Deprecation Notice:** Browser support (OPFS, Web Workers, WASM embeddings) is deprecated in v7.10.0 and will be removed in v8.0.0. Brainy v8+ will be server-only. +> Brainy 8.0 is server-only. Browser support (OPFS storage, Web Workers, in-browser WASM embeddings) was removed in 8.0 — the 7.x line remains available on npm if you need it. ## Single-Writer Model diff --git a/RELEASES.md b/RELEASES.md index 732feafd..59aa50cf 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -167,8 +167,12 @@ this rename; only the API surface moved. Escape hatches: `requireSubtype: false` (last-resort opt-out for legacy data) or `requireSubtype: { except: [NounType.Thing, ...] }` (per-type allowlist). Per-type rules registered via `brain.requireSubtype(type, opts)` - still compose. `brain.audit()` reports entries missing a subtype and - `brain.migrateField()` backfills them. + still compose. `brain.audit()` reports entries missing a subtype and the new + `brain.fillSubtypes(rules)` migration helper backfills them — one rule per + NounType/VerbType (literal default or per-entry function), applied only to + entries still missing a subtype, returning + `{ scanned, filled, skipped, errors, byType }`. Idempotent: re-running fills + nothing, so a crashed run is resumed by running it again. - **Verb ids are Brainy-generated UUIDs — by contract.** `relate()` now throws a teaching error if a caller passes an `id` field (`src/utils/paramValidation.ts:526`). In 7.x a supplied id was silently @@ -270,8 +274,10 @@ The full cost model is in `getHistory` / `streamHistory` / `versions` / `data()` per the table above. 6. **Subtypes:** if your data predates subtype discipline, start with `requireSubtype: false`, run `brain.audit()`, backfill with - `brain.migrateField()`, then remove the opt-out so the 8.0 default - enforcement protects you going forward. + `brain.fillSubtypes(rules)` (one rule per type — a literal default or a + function deriving the subtype from each entry), re-run `audit()` until + `total === 0`, then remove the opt-out so the 8.0 default enforcement + protects you going forward. 7. **CLI scripts:** `fork` / `branch` / `checkout` / `migrate` → `snapshot ` / `restore ` / `history` / `generation`. 8. **Plugin authors:** apply the contract renames and the BigInt graph @@ -553,8 +559,7 @@ alongside every other metadata field on the existing write paths. `_rev`, `ifRev`, `RevisionConflictError`, and `ifAbsent` all survive the 8.0 Db redesign unchanged. 8.0 layers `brain.transact(tx, { ifAtGeneration })` for whole-tx CAS on top of the same per-entity mechanism — per-entity for single-record -patterns, generation-based for "did the world move under me." See -`.strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md` § C-6 (internal). +patterns, generation-based for "did the world move under me." --- @@ -672,8 +677,7 @@ in `validateFindParams()`, both fire before any storage/index/Cortex call. Corte The new `docs/guides/find-limits.md` calls out that 8.0's Datomic-style `Db.find()` may tighten per-call limits; that's a Brainy 8.0 / Cortex 3.0 coordination point -documented in `.strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md` (open question #4 covers -the rollout staging, which now also applies to query limits). +whose rollout staging now also covers query limits. ### What consumers should do @@ -830,8 +834,7 @@ Brainy before any native call. - **Native `audit()` proxy.** For billion-scale brains, `audit()` walks every entity (O(N)). A native implementation reading from a "null-subtype" bitmap in the column store would be - O(buckets). Listed as the 6th open question in the - [Brainy 8.0 spec doc](`/media/dpsifr/storage/home/Projects/brainy/.strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md`). + O(buckets). - **Strict-mode parity test.** Cortex should mirror Brainy's new `tests/integration/strict-mode-self-test.test.ts` against their native paths to catch any latent bug where native writes bypass JS validation. diff --git a/docs/architecture/data-storage-architecture.md b/docs/architecture/data-storage-architecture.md index 2b342ebf..fe0fce60 100644 --- a/docs/architecture/data-storage-architecture.md +++ b/docs/architecture/data-storage-architecture.md @@ -1,1446 +1,385 @@ -# Brainy Data Storage Architecture +# Brainy Data Storage Architecture (8.0) -**Complete file structure reference** +**Complete on-disk reference for the 8.0 layout.** -This document explains how Brainy stores, indexes, and scales data on disk and in memory. +This document describes what a Brainy 8.0 data directory actually contains: the +canonical entity records, the system area, the generational-MVCC bookkeeping, +the column store, the blob area, and the lock files — plus how the in-memory +indexes rebuild from them. The authoritative design records are +[ADR-001 (generational MVCC)](../ADR-001-generational-mvcc.md) and +[index-architecture.md](./index-architecture.md); this document is the on-disk +map that ties them together. -> **8.0 accuracy note (internal):** this reference describes the pre-8.0 -> layout. In 8.0 the copy-on-write subsystem (`_cow/`, branch-scoped -> `branches/{branch}/...` paths) was replaced by generational MVCC: -> canonical entity files live at `entities/nouns/{shard}/{id}/...` with -> history under `_generations/` and the commit watermark in -> `_system/manifest.json`. The authoritative 8.0 records are -> [ADR-001 (generational MVCC)](../ADR-001-generational-mvcc.md) and -> [index-architecture.md](./index-architecture.md); the COW sections below -> are retained only as historical context until this document's full -> rewrite. +8.0 removed the 7.x copy-on-write subsystem (`_cow/`, `branches/{branch}/…` +paths) and the cloud/OPFS storage adapters. The two storage backends are +**filesystem** and **memory**; both speak the same path vocabulary (memory +storage keys its internal map by the identical path strings). --- -## Table of Contents +## 1. Directory Tree -1. [Complete File Structure](#1-complete-file-structure) -2. [What Gets Stored](#2-what-gets-stored) -3. [The 4 Indexes](#3-the-4-indexes) -4. [Sharding Strategy](#4-sharding-strategy) -5. [COW (Copy-on-Write) Architecture](#5-cow-copy-on-write-architecture) -6. [ID-First Storage Architecture](#6-id-first-storage-architecture-v600) -7. [VFS (Virtual File System)](#7-vfs-virtual-file-system) -8. [Storage Backend Mapping](#8-storage-backend-mapping) -9. [Performance Characteristics](#9-performance-characteristics) - ---- - -## 1. Complete File Structure - -### Full Directory Tree +A real 8.0 filesystem store (`rootDirectory`, default `./brainy-data`): ``` -brainy-data/ # Root directory +brainy-data/ │ -├── branches/ # Branch-scoped storage (v5.4.0+, COW always-on) -│ ├── main/ # Main branch (default) -│ │ └── entities/ -│ │ ├── nouns/ -│ │ │ ├── Character/ # Type-first: entities organized by type -│ │ │ │ ├── vectors/ -│ │ │ │ │ ├── 00/ # UUID-based sharding (256 shards) -│ │ │ │ │ │ ├── 001234...uuid.json # Vector + graph connections -│ │ │ │ │ │ └── 00abcd...uuid.json -│ │ │ │ │ ├── 01/ ... fe/ -│ │ │ │ │ └── ff/ -│ │ │ │ └── metadata/ -│ │ │ │ ├── 00/ -│ │ │ │ │ ├── 001234...uuid.json # Business metadata only -│ │ │ │ │ └── 00abcd...uuid.json -│ │ │ │ ├── 01/ ... fe/ -│ │ │ │ └── ff/ -│ │ │ │ -│ │ │ ├── Place/ # Another type -│ │ │ │ ├── vectors/ -│ │ │ │ │ ├── 00/ ... ff/ -│ │ │ │ └── metadata/ -│ │ │ │ ├── 00/ ... ff/ -│ │ │ │ -│ │ │ ├── Concept/ -│ │ │ ├── Organization/ -│ │ │ ├── Event/ -│ │ │ └── [42 total noun types] -│ │ │ -│ │ └── verbs/ -│ │ ├── Knows/ # Type-first for relationships too -│ │ │ ├── vectors/ -│ │ │ │ ├── 00/ ... ff/ -│ │ │ └── metadata/ -│ │ │ ├── 00/ ... ff/ -│ │ │ -│ │ ├── LocatedIn/ -│ │ ├── WorksFor/ -│ │ └── [127 total verb types] -│ │ -│ ├── feature-branch-1/ # Git-like feature branches -│ │ └── entities/ -│ │ └── [same structure as main] -│ │ -│ └── user-workspace-alice/ # User-specific branches -│ └── entities/ -│ └── [same structure as main] +├── entities/ # Canonical records (current state) +│ ├── nouns/ +│ │ └── {shard}/ # 256 shards: first 2 hex chars of the UUID +│ │ └── {id}/ # One directory per entity UUID +│ │ ├── vectors.json.gz # Embedding + HNSW node state +│ │ └── metadata.json.gz # Everything else (type, subtype, data, fields, _rev) +│ └── verbs/ +│ └── {shard}/ +│ └── {id}/ +│ ├── vectors.json.gz # Relationship embedding (when present) +│ └── metadata.json.gz # sourceId, targetId, verb, subtype, weight, data… │ -├── _cow/ # Copy-on-Write version control -│ ├── commits/ # Git-like commit objects -│ │ ├── 00/ -│ │ │ ├── 00a1b2c3...sha256.json # Commit metadata -│ │ │ └── 00d4e5f6...sha256.json -│ │ ├── 01/ ... fe/ -│ │ └── ff/ -│ │ -│ ├── trees/ # Directory snapshots -│ │ ├── 00/ -│ │ │ ├── 00123456...sha256.json # Tree object (directory listing) -│ │ │ └── 00789abc...sha256.json -│ │ ├── 01/ ... fe/ -│ │ └── ff/ -│ │ -│ ├── blobs/ # Content-addressable data storage -│ │ ├── 00/ -│ │ │ ├── 00abcdef...sha256.bin # Deduplicated data blobs -│ │ │ └── 00fedcba...sha256.bin -│ │ ├── 01/ ... fe/ -│ │ └── ff/ -│ │ -│ └── refs/ # Branch pointers (not sharded) -│ ├── heads/ -│ │ ├── main.json # Points to latest commit on main -│ │ ├── feature-branch-1.json -│ │ └── user-workspace-alice.json -│ │ -│ └── tags/ # Version tags -│ ├── v1.0.0.json -│ └── stable.json +├── _system/ # System singletons + bucketed system keys +│ ├── generation.json(.gz) # { generation, updatedAt } — the write watermark +│ ├── manifest.json # { version, generation, … } — MVCC commit point +│ ├── tx-log.jsonl # One line per committed transact() batch (append-only) +│ ├── counts.json # Entity/verb totals +│ ├── type-statistics.json.gz # Per-NounType counts +│ ├── subtype-statistics.json.gz # Per-(NounType, subtype) counts +│ ├── verb-subtype-statistics.json.gz # Per-(VerbType, subtype) counts +│ ├── statistics.json # Aggregate statistics blob (counts, index sizes) +│ ├── hnsw-system.json # Vector-index entry point + max level +│ ├── __metadata_field_registry__.json.gz # Which metadata fields are indexed +│ ├── brainy:entityIdMapper.json.gz # UUID ↔ u64 mapping for native index providers +│ └── idx/ +│ └── {bucket}/ # 256 buckets: FNV-1a hash of the key +│ ├── __metadata_field_index__field_{name}.json.gz # Sparse field indexes +│ ├── __chunk__*.json.gz # Metadata-index roaring-bitmap chunks +│ ├── __sparse_index__*.json.gz# Zone maps + bloom filters +│ └── graph-lsm-verbs-{source|target}-*.json.gz # Graph LSM SSTables + manifest │ -└── _system/ # System metadata (not sharded) - ├── statistics.json # Global statistics - ├── counts.json # Entity/verb counts by type - │ - ├── vector/ # Vector index metadata - │ ├── system.json # Entry point, max level - │ └── nodes/ - │ ├── 00/ - │ │ └── 001234...uuid.json # Per-node graph data - │ ├── 01/ ... fe/ - │ └── ff/ - │ - ├── metadata_indexes/ # Field indexes for filtering - │ ├── __metadata_field_index__status.json - │ ├── __metadata_field_index__category.json - │ ├── __metadata_sorted_index__createdAt.json - │ └── [dynamic based on metadata fields] - │ - └── vfs/ # Virtual File System (v5.0+) - ├── root/ # VFS root directory - │ ├── 00000000-0000-0000-0000-000000000000.json # Root dir entity - │ └── files/ - │ ├── 12345678-...uuid.json # File entities - │ └── 87654321-...uuid.json # Folder entities - │ - └── metadata/ # VFS-specific metadata - └── registry.json # VFS entity registry +├── _generations/ # MVCC history (written ONLY by transact()) +│ └── {N}/ # One directory per committed generation N +│ ├── tx.json # The generation-N delta (immutable) +│ └── prev/ +│ └── {id}.json # Before-image of each touched record (immutable) +│ +├── _column_index/ # Column store manifests (one dir per field) +│ └── {field}/ +│ └── MANIFEST.json.gz # Run list + zone metadata for that column +│ +├── _blobs/ # Binary blob area (`.bin` convention) +│ ├── _column_index/ +│ │ └── {field}/ +│ │ └── L0-000001.bin # Column-store runs (level-0 segments) +│ └── … # VFS file content and other binary blobs +│ +└── locks/ # Process coordination (NEVER snapshotted) + ├── _writer.lock # Single-writer lock: pid, hostname, heartbeat + ├── _flush_requests/ # Reader→writer flush RPC (.req files) + └── _flush_responses/ # Writer acks (.ack files) ``` +Most JSON objects are gzip-compressed (`.json.gz`) — compression is on by +default for filesystem storage (`storage.options.compression`, zlib level 6). +A few hot singletons (`manifest.json`, `counts.json`, `hnsw-system.json`, +`tx-log.jsonl`) are written uncompressed for cheap partial reads and appends. + --- -## 2. What Gets Stored +## 2. Canonical Entity Records -### 2.1 Entities (Nouns) - Split into 2 Files +Each entity (noun) and relationship (verb) is **two files** under one +ID-first directory. The split keeps vector I/O (large, append-mostly) separate +from metadata I/O (small, read-heavy). -Each entity is stored as **2 separate files** for optimal performance. - -#### Vector File -**Location**: `branches/{branch}/entities/nouns/{type}/vectors/{shard}/{uuid}.json` +### Noun vector file — `entities/nouns/{shard}/{id}/vectors.json.gz` ```json { - "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", - "vector": [0.1, 0.2, 0.3, ...], // 384-dimensional embedding - "connections": { // Vector index graph connections - "0": ["uuid1", "uuid2"], // Layer 0 neighbors - "1": ["uuid3", "uuid4"] // Layer 1 neighbors - }, - "level": 2 // Max level for this node + "id": "421d92e7-4241-470a-80f4-4b39414e7a83", + "vector": [-0.139, -0.056, 0.028, "…384 dims…"], + "connections": { "0": ["neighbor-uuid…"] }, + "level": 0 } ``` -**Purpose**: Vector index navigation for semantic search -**Size**: ~4KB per entity (384 dims × 4 bytes × 2.6 overhead) -**Scale**: Millions of entities +The HNSW node state (`connections`, `level`) is persisted with the vector so +the vector index can rebuild without recomputing the graph. -#### Metadata File -**Location**: `branches/{branch}/entities/nouns/{type}/metadata/{shard}/{uuid}.json` +### Noun metadata file — `entities/nouns/{shard}/{id}/metadata.json.gz` ```json { - "type": "Character", - "name": "Alice", - "age": 30, - "occupation": "Software Engineer", - "location": "San Francisco", - "createdAt": 1699564234567, - "customField": "custom value", - "_vfs": { // VFS metadata (if applicable) - "path": "/documents/alice.txt", - "parentId": "parent-uuid", - "isDirectory": false, - "size": 1024 - } + "data": "React is a JavaScript library for building user interfaces", + "noun": "concept", + "subtype": "cli-add", + "createdAt": 1781198053726, + "updatedAt": 1781198053726, + "_rev": 1 } ``` -**Purpose**: Business data and filtering -**Size**: ~1-10KB per entity (varies by metadata complexity) -**Scale**: Millions of entities - ---- - -### 2.2 Relationships (Verbs) - Split into 2 Files - -Each relationship is also stored as **2 separate files**. - -#### Vector File -**Location**: `branches/{branch}/entities/verbs/{type}/vectors/{shard}/{uuid}.json` - -```json -{ - "id": "7b2f5e3c-8d4a-4f1e-9c2b-5a6d7e8f9a0b", - "vector": [0.5, 0.3, 0.7, ...], // Relationship embedding - "connections": { - "0": ["verb-uuid1", "verb-uuid2"] // Verb-to-verb vector connections - }, - "level": 1 -} -``` - -**Purpose**: Relationship similarity for semantic graph queries -**Size**: ~2KB per relationship -**Scale**: Millions of relationships - -#### Metadata File -**Location**: `branches/{branch}/entities/verbs/{type}/metadata/{shard}/{uuid}.json` - -```json -{ - "sourceId": "user-uuid", // Source entity - "targetId": "product-uuid", // Target entity - "type": "Purchased", // Verb type - "weight": 1.0, - "timestamp": 1699564234567, - "metadata": { - "amount": 99.99, - "quantity": 2, - "paymentMethod": "credit_card" - } -} -``` - -**Purpose**: Graph structure (edges) and relationship data -**Size**: ~500 bytes per relationship -**Scale**: Millions of relationships - ---- - -### 2.3 COW (Copy-on-Write) Data - -#### Commit Objects -**Location**: `_cow/commits/{shard}/{sha256}.json` - -```json -{ - "tree": "tree-sha256-hash", // Root tree snapshot - "parent": "parent-commit-sha256", // Previous commit (null for first) - "author": "user@example.com", - "timestamp": 1699564234567, - "message": "Add new characters", - "branch": "main" -} -``` - -**Purpose**: Git-like version history -**Size**: ~300 bytes per commit -**Scale**: Thousands of commits - -#### Tree Objects -**Location**: `_cow/trees/{shard}/{sha256}.json` - -```json -{ - "entries": [ - { - "type": "tree", - "name": "entities/nouns/Character", - "hash": "subtree-sha256-hash" - }, - { - "type": "blob", - "name": "entities/nouns/Character/vectors/00/001234...uuid.json", - "hash": "blob-sha256-hash" - } - ] -} -``` - -**Purpose**: Directory snapshots (like git trees) -**Size**: ~1-50KB per tree (varies by directory size) -**Scale**: Thousands of trees - -#### Blob Objects -**Location**: `_cow/blobs/{shard}/{sha256}.bin` - -``` -[Binary data - deduplicated content] -``` - -**Purpose**: Content-addressable storage (deduplication) -**Size**: Varies (1KB - 1MB typical) -**Scale**: Millions of blobs -**Compression**: Optional zstd compression for >4KB blobs - -#### Refs (Branch Pointers) -**Location**: `_cow/refs/heads/{branch}.json` - -```json -{ - "commit": "latest-commit-sha256-hash", - "updated": 1699564234567 -} -``` - -**Purpose**: Branch head tracking (like git refs) -**Size**: ~100 bytes per ref -**Scale**: Dozens to hundreds of branches - ---- - -### 2.4 System Metadata - -Unlike entities and relationships, system metadata consists of **index files** that enable fast lookups without scanning millions of entities. - -**Purpose**: Fast filtering and range queries -**Scale**: 50-200 files total (NOT per-entity!) -**Location**: `_system/` (not sharded, not branched) - -#### Statistics -**Location**: `_system/statistics.json` - -```json -{ - "nounCount": { - "Character": 50000, - "Place": 30000, - "Concept": 20000 - }, - "verbCount": { - "Knows": 100000, - "LocatedIn": 75000 - }, - "metadataCount": { - "Character": 50000, - "Place": 30000 - }, - "vectorIndexSize": 204800, - "totalNodes": 100000, - "totalEdges": 175000, - "lastUpdated": "2025-11-18T..." -} -``` - -**Purpose**: Global statistics for monitoring and optimization - -#### Counts (Entity Type Counts) -**Location**: `_system/counts.json` - -```json -{ - "nouns": { - "Character": 50000, - "Place": 30000, - "Concept": 20000, - "Organization": 15000 - }, - "verbs": { - "Knows": 100000, - "LocatedIn": 75000, - "WorksFor": 50000 - }, - "total": { - "nouns": 115000, - "verbs": 225000 - }, - "lastUpdated": 1699564234567 -} -``` - -**Purpose**: Fast entity/verb counts by type without scanning storage - -#### Vector Index System Metadata -**Location**: `_system/vector/system.json` - -```json -{ - "entryPointId": "3fa85f64-5717-4562-b3fc-2c963f66afa6", - "maxLevel": 4, - "totalNodes": 100000, - "lastUpdated": 1699564234567 -} -``` - -**Purpose**: Vector index entry point and global parameters - -#### Vector Index Node Data -**Location**: `_system/vector/nodes/{shard}/{uuid}.json` - -```json -{ - "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", - "level": 2, - "connections": { - "0": ["uuid1", "uuid2", "uuid3"], // Layer 0 neighbors - "1": ["uuid4", "uuid5"], // Layer 1 neighbors - "2": ["uuid6"] // Layer 2 neighbors - } -} -``` - -**Purpose**: Per-node vector index graph connections (persisted for fast rebuild) - -#### Field Indexes (Hash Indexes) -**Location**: `_system/metadata_indexes/__metadata_field_index__{field}.json` - -```json -{ - "values": { - "active": 80000, // Count of entities with status=active - "pending": 15000, - "deleted": 5000 - }, - "lastUpdated": 1699564234567 -} -``` - -**Purpose**: Fast exact-match filtering without scanning all entities - -#### Sorted Indexes (Range Queries) -**Location**: `_system/metadata_indexes/__metadata_sorted_index__{field}.json` - -```json -{ - "values": [ - [1704067200000, ["uuid1", "uuid2", "uuid3"]], // Jan 1, 2024 - [1704153600000, ["uuid4", "uuid5"]], // Jan 2, 2024 - [1704240000000, ["uuid6"]] // Jan 3, 2024 - ], - "fieldType": "number" -} -``` - -**Purpose**: Fast range queries (e.g., "created after Jan 1, 2024") - ---- - -## 2.5 Path Construction Algorithm - -Understanding how Brainy constructs storage paths is critical for debugging and optimization. - -### Path Construction Steps - -**For an entity (noun)**: -```typescript -// Given: -const entityId = "3fa85f64-5717-4562-b3fc-2c963f66afa6" -const branch = "main" - -// Step 1: Extract shard from UUID (first 2 hex characters) -const shard = entityId.substring(0, 2) // "3f" - -// Step 2: Construct metadata path (NO TYPE NEEDED!) -const metadataPath = `branches/${branch}/entities/nouns/${shard}/${entityId}/metadata.json` -// Result: "branches/main/entities/nouns/3f/3fa85f64-5717-4562-b3fc-2c963f66afa6/metadata.json" - -// Step 3: Construct vector path -const vectorPath = `branches/${branch}/entities/nouns/${shard}/${entityId}/vector.json` -// Result: "branches/main/entities/nouns/3f/3fa85f64-5717-4562-b3fc-2c963f66afa6/vector.json" - -// Type is IN the metadata, not in the path! -``` - -**For a relationship (verb)**: -```typescript -// Given: -const verbId = "7b2f5e3c-8d4a-4f1e-9c2b-5a6d7e8f9a0b" -const branch = "main" - -// Step 1: Extract shard -const shard = verbId.substring(0, 2) // "7b" - -// Step 2: Construct paths (NO TYPE NEEDED!) -const metadataPath = `branches/${branch}/entities/verbs/${shard}/${verbId}/metadata.json` -const vectorPath = `branches/${branch}/entities/verbs/${shard}/${verbId}/vector.json` -``` - -**For COW objects**: -```typescript -// Commits, trees, blobs use content hash as filename -const commitHash = "00a1b2c3d4e5f6789abcdef0123456789abcdef0123456789abcdef012345678" -const shard = commitHash.substring(0, 2) // "00" -const commitPath = `_cow/commits/${shard}/${commitHash}.json` - -// Refs don't use sharding -const branchRefPath = `_cow/refs/heads/${branch}.json` -const tagRefPath = `_cow/refs/tags/${tagName}.json` -``` - -**For system files**: -```typescript -// System files never use sharding or branching -const statsPath = `_system/statistics.json` -const countsPath = `_system/counts.json` -const vectorSystemPath = `_system/vector/system.json` -const fieldIndexPath = `_system/metadata_indexes/__metadata_field_index__${fieldName}.json` - -// Vector node data IS sharded (entity UUID-based) -const vectorNodePath = `_system/vector/nodes/${shard}/${entityId}.json` -``` - -### Path Patterns Summary - -| Data Type | Path Pattern | Sharded? | Branched? | -|-----------|--------------|----------|-----------| -| **Noun metadata** | `branches/{branch}/entities/nouns/{shard}/{uuid}/metadata.json` | ✅ Yes (UUID) | ✅ Yes | -| **Noun vector** | `branches/{branch}/entities/nouns/{shard}/{uuid}/vector.json` | ✅ Yes (UUID) | ✅ Yes | -| **Verb metadata** | `branches/{branch}/entities/verbs/{shard}/{uuid}/metadata.json` | ✅ Yes (UUID) | ✅ Yes | -| **Verb vector** | `branches/{branch}/entities/verbs/{shard}/{uuid}/vector.json` | ✅ Yes (UUID) | ✅ Yes | -| **COW commit** | `_cow/commits/{shard}/{sha256}.json` | ✅ Yes (SHA) | ❌ No | -| **COW tree** | `_cow/trees/{shard}/{sha256}.json` | ✅ Yes (SHA) | ❌ No | -| **COW blob** | `_cow/blobs/{shard}/{sha256}.bin` | ✅ Yes (SHA) | ❌ No | -| **COW ref** | `_cow/refs/heads/{branch}.json` | ❌ No | ❌ No | -| **Statistics** | `_system/statistics.json` | ❌ No | ❌ No | -| **Counts** | `_system/counts.json` | ❌ No | ❌ No | -| **Vector system** | `_system/vector/system.json` | ❌ No | ❌ No | -| **Vector node** | `_system/vector/nodes/{shard}/{uuid}.json` | ✅ Yes (UUID) | ❌ No | -| **Field index** | `_system/metadata_indexes/__metadata_field_index__{field}.json` | ❌ No | ❌ No | - -### Key Principles - -1. **Shard Extraction**: Always use first 2 hex characters of UUID/SHA-256 -2. **ID-First**: Shard + ID come BEFORE type (type is in metadata) -3. **Branch Isolation**: Only entity data uses branches/ -4. **System Isolation**: System files never use sharding or branching (except vector index nodes) -5. **Content-Addressable**: COW uses SHA-256 hash as filename - ---- - -## 3. The 4 Indexes - -Brainy uses four complementary index systems for different query patterns. - -### 3.1 Vector Index (In-Memory with Lazy Loading) - -**Purpose**: Semantic similarity search -**Location**: RAM (rebuilt from storage on startup) -**Data Structure**: Hierarchical graph of vector connections - -**How It Works**: -1. Loads `branches/{branch}/entities/nouns/{type}/vectors/**/*.json` files -2. Builds vector index graph structure in memory -3. Enables O(log n) approximate nearest neighbor search -4. Vectors loaded on-demand in lazy mode (zero configuration) - -**Performance**: -- Build time: 1-5 seconds per 100K entities -- Query time: 1-10ms for k=10 results (standard mode) -- Query time: 2-15ms for k=10 results (lazy mode, with cache) -- Memory (standard): ~200MB per 100K entities -- Memory (lazy): ~15-33MB per 100K entities (5-10x less!) - -**Automatic Lazy Mode**: Enables automatically when vectors don't fit in UnifiedCache - ---- - -### 3.2 Type Index (Metadata-Based, v6.0.0+) - -**Purpose**: Fast type filtering via metadata index -**Location**: MetadataIndexManager index on `noun` field -**Data Structure**: RoaringBitmap32 per type value - -**How It Works**: -```typescript -// Find all Person entities -const people = await brain.getNouns({ type: 'person' }) - -// Under the hood: -// 1. MetadataIndexManager.getFieldIndex('noun') -// 2. Returns RoaringBitmap32 of IDs where metadata.noun === 'person' -// 3. Batch fetch those IDs using ID-first paths -``` - -**Performance**: -- Type filtering: O(person_count) via metadata index (not O(total_entities)) -- Index lookup: O(1) bitmap intersection -- No filesystem scanning needed -- Works at billion-scale with compressed bitmaps - ---- - -### 3.3 Graph Adjacency Index (In-Memory, LSM-Tree) - -**Purpose**: Navigate relationships (graph queries) -**Location**: RAM (rebuilt from storage on startup) -**Data Structure**: Bidirectional LSM-tree mappings +- `noun` is the NounType. **Type lives in metadata, not in the path** — lookup + by ID is a single path construction, no type needed. +- `subtype` is the per-product sub-classification (required on write by + default in 8.0). +- `_rev` increments on every write and backs `update({ ifRev })` CAS. +- Consumer metadata fields sit alongside the standard ones. + +### Verb files — `entities/verbs/{shard}/{id}/…` + +Same two-file split. The verb metadata record carries the graph edge: +`sourceId`, `targetId`, `verb` (VerbType), `subtype`, `weight`, `data`, +`metadata`, timestamps, `_rev`. Verb IDs are Brainy-generated UUIDs by +contract (8.0 rejects caller-supplied verb ids) because native graph providers +intern the raw UUID bytes as u64 handles. + +### Path construction ```typescript -{ - sourceToTargets: Map>, // "user-uuid" → ["product1", "product2"] - targetToSources: Map> // "product1" → ["user1", "user2"] -} +const shard = id.substring(0, 2) // '42' +const metadataPath = `entities/nouns/${shard}/${id}/metadata.json` +const vectorsPath = `entities/nouns/${shard}/${id}/vectors.json` +// Verbs: same shape under entities/verbs/ ``` -**Example Query**: -```typescript -// Find all products purchased by a user -const verbs = await brain.getVerbsBySource("user-uuid") - -// Find all users who purchased a product -const verbs = await brain.getVerbsByTarget("product-uuid") -``` - -**Performance**: -- Build time: 0.5-2 seconds per 100K relationships -- Query time: <1ms (O(1) lookup) -- Memory: ~100MB per 100K relationships +Implemented in `src/storage/baseStorage.ts` (path generators) and +`src/storage/sharding.ts` (`getShardIdFromUuid`). --- -### 3.4 Metadata Field Indexes (On-Disk) +## 3. The `_system/` Area -**Purpose**: Filter by business fields without loading all entities -**Location**: `_system/metadata_indexes/` -**Data Structure**: Field → Value → IDs mapping +Two kinds of keys live here, resolved by `BaseStorage.parsePath()`: -**Example Query**: -```typescript -// Find all active users -const users = await brain.getNouns({ - filter: { metadata: { status: 'active' } } -}) -// Uses: _system/metadata_indexes/__metadata_field_index__status.json -// Returns: ~1000 IDs in 5ms (vs scanning 1M entities) -``` +1. **Singletons** — well-known keys written at `_system/.json`: + `counts`, `statistics`, `type-statistics`, `hnsw-system`, + `__metadata_field_registry__`, `brainy:entityIdMapper`, plus the MVCC + trio (`generation.json`, `manifest.json`, `tx-log.jsonl`). +2. **Bucketed system keys** — everything else (field indexes, bitmap chunks, + sparse-index segments, graph LSM SSTables) hashes into one of 256 + `_system/idx/{bucket}/` directories via FNV-1a, so no single directory + accumulates unbounded entries. -**Performance**: -- Exact match: O(1) hash lookup -- Range query: O(log n) binary search (sorted indexes) -- Filter time: 5-50ms for 1M entities +Notable singletons: + +| File | Contents | +|------|----------| +| `generation.json` | `{ generation, updatedAt }` — monotonic watermark, bumped by **every** write batch | +| `manifest.json` | MVCC commit point: highest *committed* generation (see §4) | +| `tx-log.jsonl` | One JSON line per committed `transact()` batch: generation, timestamp, `meta` | +| `type-statistics.json.gz` | Per-NounType counts (backs `brain.counts.byType`) | +| `subtype-statistics.json.gz` | `{ counts: { [type]: { [subtype]: n } }, updatedAt }` (contract-bound shape) | +| `verb-subtype-statistics.json.gz` | Same shape for VerbTypes | +| `hnsw-system.json` | `{ entryPointId, maxLevel }` — per-node state lives in each entity's `vectors.json` | +| `brainy:entityIdMapper.json.gz` | UUID ↔ u64 interning table for the BigInt provider contract | +| `__metadata_field_registry__.json.gz` | Registry of indexed metadata field names | --- -## 4. Sharding Strategy +## 4. Generational MVCC (`_generations/` + the `_system` trio) -### 4.1 Why Shard? +Full design in [ADR-001](../ADR-001-generational-mvcc.md). The on-disk shape: -**Filesystem Limitations**: -- Listing 100K files in one directory is slow (`readdir` walks the whole entry list) -- Most filesystems prefer ≤10,000 entries per directory for fast lookups -- Parallel operations across shards beat serial scans +``` +_system/generation.json { generation, updatedAt } atomic tmp+rename +_system/manifest.json { version, generation, … } atomic tmp+rename — THE commit point +_system/tx-log.jsonl one line per committed transact() append-only +_generations/{N}/tx.json the generation-N delta immutable once written +_generations/{N}/prev/{id}.json before-image of {id} immutable once written +``` -**Solution**: Split into 256 shards = ~3,900 files per shard at 1M scale +Commit protocol (writer side): stage before-images and the delta under +`_generations/N/`, fsync, apply the delta to the canonical `entities/…` +records, then atomically rename `manifest.json` to publish generation N. The +tx-log line is appended last (advisory). Crash recovery on open discards any +`_generations/{N}` newer than the manifest. + +Two write classes share the generation clock: + +- **Single-operation writes** (`add`/`update`/`delete`/`relate` outside + `transact()`) bump `generation.json` so watermarks and `_rev` CAS stay sound, + but write **no** history — they are not visible to `db.since()` and remain + visible through earlier pins. +- **`transact()` batches** write the full `_generations/{N}` record and a + tx-log line, and are the unit of time travel (`brain.asOf()`). + +Snapshots (`db.persist(path)`) hard-link the entire store **except `locks/`** +into a self-contained directory openable via `Brainy.load(path)`. --- -### 4.2 How Sharding Works +## 5. Column Store (`_column_index/` + `_blobs/_column_index/`) -**Algorithm**: Extract first 2 hex characters from UUID +The metadata index persists per-field columnar runs for O(log n) range and +membership queries at scale: -``` -UUID: 3fa85f64-5717-4562-b3fc-2c963f66afa6 - ^^ -Shard: 3f -``` +- `_column_index/{field}/MANIFEST.json.gz` — the run list and zone metadata + for one field (`createdAt`, `subtype`, `noun`, `_rev`, consumer fields, + `__words__` for tokenized text…). +- `_blobs/_column_index/{field}/L0-NNNNNN.bin` — the actual level-0 run + segments, stored through the shared `_blobs/.bin` binary convention. -**Properties**: -- **Deterministic**: Same UUID always maps to same shard -- **Uniform**: UUIDs distribute evenly across shards -- **Predictable**: Easy to compute, no randomness -- **Efficient**: Simple string operation (O(1)) - -**Shard Distribution (1M entities)**: -``` -Shard 00: ~3,900 entities -Shard 01: ~3,900 entities -... -Shard fe: ~3,900 entities -Shard ff: ~3,900 entities -Total: 256 shards × 3,900 = ~1,000,000 entities -``` +Sparse per-field indexes, roaring-bitmap chunks, and zone-map/bloom segments +additionally live as bucketed keys under `_system/idx/` (see §3). Which path +serves a given `where` clause is the query planner's decision — inspect it +with `brainy inspect explain --where '…'`. --- -### 4.3 What Gets Sharded vs. Not Sharded +## 6. Blob Area (`_blobs/`) -| Data Type | Sharded? | Path Pattern | -|-----------|----------|--------------| -| **Noun vectors** | ✅ Yes | `branches/{branch}/entities/nouns/{type}/vectors/{shard}/{uuid}.json` | -| **Noun metadata** | ✅ Yes | `branches/{branch}/entities/nouns/{type}/metadata/{shard}/{uuid}.json` | -| **Verb vectors** | ✅ Yes | `branches/{branch}/entities/verbs/{type}/vectors/{shard}/{uuid}.json` | -| **Verb metadata** | ✅ Yes | `branches/{branch}/entities/verbs/{type}/metadata/{shard}/{uuid}.json` | -| **COW commits** | ✅ Yes | `_cow/commits/{shard}/{sha256}.json` | -| **COW trees** | ✅ Yes | `_cow/trees/{shard}/{sha256}.json` | -| **COW blobs** | ✅ Yes | `_cow/blobs/{shard}/{sha256}.bin` | -| **COW refs** | ❌ No | `_cow/refs/heads/{branch}.json` | -| **System metadata** | ❌ No | `_system/statistics.json` | -| **Indexes** | ❌ No | `_system/metadata_indexes/*.json` | +`_blobs/.bin` is the flat binary-blob convention shared by every storage +adapter (`saveBinaryBlob`/`getBinaryBlob` in the storage contract): -**Key Principle**: Shard by **UUID** (entity IDs, commit hashes), not by type or field. +- **VFS file content** — VFS entities are regular nouns (path, ownership, and + timestamps in entity metadata); the file *bytes* are blobs. +- **Column-store runs** (under the `_column_index/` key prefix, §5). +- Any other binary payload an index provider persists. + +Writes use unique temp names + rename, so concurrent writers of the same key +cannot tear each other's blobs. --- -### 4.4 Performance Impact +## 7. Locks (`locks/`) -**Without Sharding (1M entities)**: ``` -List directory: 30 seconds -Find entity: 30 seconds (must list first) -Delete entity: 30 seconds (must list first) +locks/_writer.lock # single-writer lock: { pid, hostname, startedAt, heartbeat, version } +locks/_flush_requests/ # readers drop .req to ask the writer to flush +locks/_flush_responses/ # writer answers with .ack ``` -**With Sharding (1M entities across 256 shards)**: -``` -List directory: 120ms (only ~3,900 files) -Find entity: 150ms (list shard + download) -Delete entity: 150ms (list shard + delete) -``` - -**Speedup**: 200x faster for large datasets +- One **writer** per data directory, enforced at `init()`; stale locks (dead + PID / stale heartbeat) are reclaimed automatically. +- Read-only processes (`Brainy.openReadOnly()`, the `brainy inspect` CLI + family) can ask the live writer to flush via the request/response files, so + out-of-process diagnostics see fresh state. +- `locks/` is excluded from snapshots (`SNAPSHOT_EXCLUDED_TOP_DIRS` in + `src/storage/adapters/fileSystemStorage.ts`). --- -## 5. COW (Copy-on-Write) Architecture +## 8. In-Memory Indexes and What Rebuilds From What -### 5.1 What is COW? +| Index | In memory | Persisted state | Rebuild source | +|-------|-----------|-----------------|----------------| +| **Vector (HNSW)** | Graph of vector connections | `_system/hnsw-system.json` + per-entity `vectors.json` | Walk entity vector files; lazy mode loads structure only and pages vectors on demand | +| **Metadata index** | Field → value bitmaps + column-store readers | `_system/idx/` chunks + `_column_index/` manifests + `_blobs/_column_index/` runs | Loaded directly; full rebuild re-scans entity metadata | +| **Graph adjacency** | sourceId/targetId → verb-id LSM trees | `graph-lsm-verbs-{source,target}-*` SSTables under `_system/idx/` | Loaded from SSTables; full rebuild re-scans verb metadata | +| **Counts/statistics** | Per-type and per-subtype maps | `_system/{type,subtype,verb-subtype}-statistics.json.gz`, `counts.json` | Recomputable by scanning entities (`brainy inspect repair`) | -COW is Brainy's **git-like versioning system** that enables: -- ✅ **Time-travel queries** (query data as it existed at any point in time) -- ✅ **Instant branches** (create lightweight branches in milliseconds) -- ✅ **Efficient forks** (zero-copy duplication via lazy COW) -- ✅ **Deduplication** (identical data stored only once) -- ✅ **Version history** (full audit trail of all changes) - -**Status**: ALWAYS ENABLED - cannot be disabled +A pluggable index provider (the 8.0 plugin contract in +`@soulcraft/brainy/plugin`) may replace any of the JS implementations; the +persisted formats above are contract-bound so JS and native implementations +can interleave on the same directory. --- -### 5.2 COW Directory Structure +## 9. Sharding Strategy -``` -_cow/ -├── commits/ # Commit objects (version history) -├── trees/ # Directory snapshots -├── blobs/ # Content-addressable data storage -└── refs/ # Branch pointers - ├── heads/ # Branch heads (main, feature branches) - └── tags/ # Version tags (v1.0.0, stable, etc.) -``` +**Entities:** first 2 hex characters of the UUID → 256 uniform shards. +Deterministic, configuration-free, and keeps per-directory entry counts low +(at 1M entities: ~3,900 directories per shard). Paginated whole-store walks +(`getNouns`/`getVerbs`) iterate shards `00`–`ff` in order. + +**System keys:** FNV-1a hash of the key → 256 `_system/idx/` buckets. Same +motivation, different keyspace (system keys are not UUIDs). + +**What is never sharded:** the `_system/` singletons, `_generations/{N}` +directories (keyed by generation number), `_column_index/{field}` manifests +(keyed by field name), and `locks/`. --- -### 5.3 How COW Works +## 10. Durability and Atomicity -**When you add data**: -1. Data written to `branches/main/entities/nouns/Character/...` -2. Commit object created in `_cow/commits/{sha}/` -3. Tree objects created for directory structure -4. Blobs created for content (deduplicated by SHA-256) -5. `_cow/refs/heads/main.json` updated to point to new commit - -**When you query `brain.asOf(timestamp)`**: -1. Find commit at specified timestamp -2. Load tree from commit -3. Lazy-load entities from historical tree structure -4. Return read-only view (no writes allowed) - -**When you create a branch**: -1. Copy `_cow/refs/heads/main.json` → `_cow/refs/heads/feature.json` -2. Create `branches/feature/` directory (initially empty) -3. Lazy COW: Only modified files copied, rest shared with main -4. Result: Instant branch creation (milliseconds) +- **Per-object atomicity:** every JSON object and blob is written to a unique + temp file then `rename()`d — readers never observe torn objects. +- **Transaction atomicity:** the `manifest.json` rename is the single commit + point for `transact()` batches (§4); everything staged before it is + discarded by crash recovery if the rename never lands. +- **Compression:** gzip per object (`.json.gz`), transparent to all readers. + Native index providers that mmap binary formats use the uncompressed + `_blobs/` area instead. --- -### 5.4 Deduplication +## 11. `clear()` Semantics -**Content-addressable storage** means identical data is stored only once: - -``` -// Two entities with identical vector data -Entity A: vector = [0.1, 0.2, 0.3, ...] → SHA-256 = abc123... -Entity B: vector = [0.1, 0.2, 0.3, ...] → SHA-256 = abc123... (same!) - -// Only ONE blob stored: -_cow/blobs/ab/abc123...sha256.bin (used by both entities) -``` - -**Deduplication savings**: -- Typical: 10-30% storage reduction -- Forks/branches: 70-90% reduction (shared data not duplicated) -- Identical imports: 95%+ reduction +`brain.clear()` removes all entities, relationships, indexes, statistics, and +MVCC history, then re-resolves every index exactly as `init()` does — +including plugin-provided vector/metadata/id-mapper factories and VFS root +re-creation. The data directory afterwards contains a fresh, empty store (the +writer lock remains held by the running process). --- -## 6. ID-First Storage Architecture +## 12. Common Scenarios -### 6.1 What is ID-First? +### Adding an entity -**ID-first storage** organizes entities by **ID shard only** - no type directories! This eliminates 42-type sequential searches that caused long delays when an entity's type was unknown. - -**Old type-first structure** (v5.12.0): ``` -branches/main/entities/nouns/{TYPE}/metadata/00/001234...uuid.json -# Problem: Requires knowing type OR searching 42 type directories! +brain.add({ data, type, subtype }) +1. Generate UUID → shard = first 2 hex chars +2. Embed data → 384-dim vector +3. Write entities/nouns/{shard}/{id}/vectors.json.gz (vector + HNSW node state) +4. Write entities/nouns/{shard}/{id}/metadata.json.gz (type/subtype/data/fields, _rev: 1) +5. Update in-memory indexes (HNSW insert, metadata index, statistics) +6. Bump _system/generation.json (no _generations/ entry — single-op write) ``` -**NEW ID-first structure**: +### Committing a transaction + ``` -branches/main/entities/nouns/00/001234...uuid/metadata.json -# Direct O(1) lookup - no type needed! +await brain.transact(tx => { tx.add(…); tx.update(…) }) +1. Stage _generations/{N}/prev/{id}.json before-images + tx.json delta; fsync +2. Apply the delta to canonical entities/… records +3. Atomic-rename _system/manifest.json → generation N is committed +4. Append one line to _system/tx-log.jsonl ``` ---- +### Cold start -### 6.2 Benefits of ID-First - -**1. 40x Faster Lookups** -```typescript -// v5.x: Had to search 42 types if type unknown -// Result: 21 seconds (42 types × 500ms) - -// Direct path from ID -const id = '001234...' -const shard = id.substring(0, 2) // '00' -const path = `branches/main/entities/nouns/${shard}/${id}/metadata.json` -// Result: <500ms - 40x faster! ``` - -**2. Simpler Code** -- **Removed 500+ lines** of type cache management -- **No more** nounTypeCache Map tracking -- **No more** persistent type index complexity -- **No more** 42-type fallback search logic - -**3. Billion-Scale Ready** -- Type information stored in **metadata** field (indexed by MetadataIndexManager) -- Type queries still fast via metadata index -- No type cache sync issues in distributed systems - -**4. Clean Architecture** -- One path per ID - no ambiguity -- Predictable storage layout -- Easier to debug and reason about - ---- - -### 6.3 ID-First Path Structure - -**v6.0.0 Path Structure:** -``` -branches/{branch}/entities/nouns/{shard}/{id}/metadata.json -branches/{branch}/entities/nouns/{shard}/{id}/vector.json -branches/{branch}/entities/verbs/{shard}/{id}/metadata.json -branches/{branch}/entities/verbs/{shard}/{id}/vector.json -``` - -**Breakdown**: -- `branches/{branch}`: Branch isolation (main, feature branches, user workspaces) -- `entities/nouns` or `entities/verbs`: Entity vs. relationship -- `{shard}`: UUID-based shard (00-ff, 256 total) - **comes FIRST now!** -- `{id}`: Full entity UUID -- `metadata.json` or `vector.json`: Separate files for metadata vs vectors - -**Key Change:** Shard + ID come **before** type, not after! - ---- - -### 6.4 Type Queries Still Work! - -**How do we filter by type without type directories?** - -The `metadata.noun` field is **indexed by MetadataIndexManager**: - -```typescript -// Find all Person entities - still fast! -const people = await brain.getNouns({ type: 'person' }) - -// Under the hood: -// 1. MetadataIndexManager has index on 'noun' field -// 2. Returns all IDs where metadata.noun === 'person' -// 3. Batch fetch those IDs using ID-first paths -// Result: Still O(person_count), not O(total_entities) -``` - -**Supported Types (unchanged):** -- **42 Noun Types**: Person, Organization, Location, Thing, Concept, Event, Agent, etc. -- **127 Verb Types**: Knows, LocatedIn, WorksFor, HasProperty, etc. -- See [noun-verb-taxonomy.md](./noun-verb-taxonomy.md) for complete list - -**Type is metadata, not storage structure!** - ---- - -## 7. VFS (Virtual File System) - -### 7.1 What is VFS? - -**VFS** lets you store traditional file/folder hierarchies in Brainy's graph database. - -**Example**: -``` -/documents/ - ├── reports/ - │ ├── Q1.pdf (stored as entity) - │ └── Q2.pdf (stored as entity) - └── notes/ - └── meeting.txt (stored as entity) -``` - -Each file/folder is a **regular Brainy entity** with special VFS metadata. - ---- - -### 7.2 VFS Storage Structure - -**File Entity**: -```json -// branches/main/entities/nouns/File/metadata/12/123456...uuid.json -{ - "type": "File", - "name": "Q1.pdf", - "_vfs": { - "path": "/documents/reports/Q1.pdf", - "parentId": "parent-directory-uuid", - "isDirectory": false, - "size": 102400, - "mimeType": "application/pdf", - "createdAt": 1699564234567, - "modifiedAt": 1699564234567 - }, - // Regular metadata fields can coexist - "author": "Alice", - "department": "Finance" -} -``` - -**Directory Entity**: -```json -// branches/main/entities/nouns/Collection/metadata/ab/abcdef...uuid.json -{ - "type": "Collection", - "name": "reports", - "_vfs": { - "path": "/documents/reports", - "parentId": "documents-directory-uuid", - "isDirectory": true, - "childrenIds": ["Q1-uuid", "Q2-uuid"] - } -} -``` - -**Root Directory** (special fixed UUID): -```json -// branches/main/entities/nouns/Collection/metadata/00/00000000-0000-0000-0000-000000000000.json -{ - "type": "Collection", - "name": "root", - "_vfs": { - "path": "/", - "parentId": null, - "isDirectory": true, - "childrenIds": ["documents-uuid", "projects-uuid"] - } -} -``` - ---- - -### 7.3 VFS + Triple Intelligence - -VFS files can use **Triple Intelligence** for semantic extraction: - -```typescript -// Upload PDF -const fileId = await brain.vfs.uploadFile('/documents/report.pdf', pdfBuffer) - -// Triple Intelligence extracts: -// - Entities: People, organizations, locations mentioned -// - Relationships: Who works where, who knows who -// - Concepts: Key themes and topics - -// Query semantically -const related = await brain.find('financial projections for Q2') -// Returns: report.pdf + extracted entities + relationships -``` - -**Storage**: Extracted entities stored as regular entities in type-first structure, linked to file via relationships. - ---- - -## 8. Storage Backend Mapping - -### 8.1 Both Adapters Use the Same Structure - -**Filesystem** (default for Node.js): -``` -/path/to/brainy-data/ - ├── branches/main/entities/nouns/Character/vectors/00/001234...uuid.json - ├── _cow/commits/00/00a1b2c3...sha256.json - └── _system/statistics.json -``` - -**Memory Storage** (in-memory): -- Uses same path structure -- Stored in `Map` -- Key = full path (e.g., "branches/main/entities/nouns/Character/vectors/00/001234...uuid.json") - -For off-site backup of the filesystem artifact, snapshot the `rootDirectory` from your scheduler using `gsutil`, `aws s3 sync`, `rclone`, or `tar` — Brainy itself stays local. - ---- - -### 8.2 Adapter Characteristics - -**Filesystem**: -- Direct file I/O (fastest for local) -- Atomic writes with rename -- Sharded directory layout keeps `readdir` fast - -**Memory**: -- No I/O overhead (instant access) -- No persistence (data lost on restart) -- Ideal for testing and development - ---- - -## 9. Performance Characteristics - -### 9.1 File Count (1M Entities Example) - -| Directory | File Count | Size per File | Total Size | -|-----------|-----------|---------------|------------| -| `branches/main/entities/nouns/*/vectors/**` | 1,000,000 | ~4KB | ~4GB | -| `branches/main/entities/nouns/*/metadata/**` | 1,000,000 | ~2KB | ~2GB | -| `branches/main/entities/verbs/*/vectors/**` | 1,000,000 | ~2KB | ~2GB | -| `branches/main/entities/verbs/*/metadata/**` | 1,000,000 | ~500B | ~500MB | -| `_cow/commits/**` | ~10,000 | ~300B | ~3MB | -| `_cow/trees/**` | ~50,000 | ~5KB | ~250MB | -| `_cow/blobs/**` | ~2,000,000 | ~2KB | ~4GB | -| `_cow/refs/**` | ~50 | ~100B | ~5KB | -| `_system/**` | ~100 | ~1-500KB | ~10MB | -| **Total** | **~5,060,150** | | **~12.8GB** | - -**With deduplication**: ~8.5-10GB (30-40% savings from blob deduplication) - ---- - -### 9.2 Read Performance - -| Operation | No Sharding | With Sharding | Improvement | -|-----------|-------------|---------------|-------------| -| Get entity by ID | 15-30s | 100-150ms | **200x faster** | -| List all entities | 30-60s | 30-60s | Same | -| Filter by metadata | 10-30s | 5-50ms | **100-600x faster** (via indexes) | -| Semantic search | N/A | 1-10ms | N/A (requires vector index) | -| Type filtering | 30-60s | 120-200ms | **150-500x faster** (type-first) | -| Graph query (getVerbsBySource) | O(total_verbs) | <1ms | **O(1) via index** | - ---- - -### 9.3 Write Performance - -| Operation | No Sharding | With Sharding | Improvement | -|-----------|-------------|---------------|-------------| -| Add entity | 15-30s | 100-150ms | **200x faster** | -| Update entity | 15-30s | 100-150ms | **200x faster** | -| Delete entity | 15-30s | 100-150ms | **200x faster** | -| Batch insert (1000) | 4-8 hours | 2-3 minutes | **120x faster** | -| Create branch | N/A | 100-200ms | Instant (COW) | -| Commit changes | N/A | 500-1000ms | Automatic (COW) | - ---- - -### 9.4 Scale Limits - -| Storage Backend | Max Entities (No Optimization) | Max Entities (Full Optimization) | -|----------------|-------------------------------|----------------------------------| -| Filesystem | ~100,000 | **10M+** | -| Memory | Limited by RAM | Limited by RAM | - -**Full optimization** = Sharding + Type-first + COW + Lazy mode - ---- - -### 9.5 Memory Usage - -| Component | Standard Mode | Lazy Mode | Savings | -|-----------|---------------|-----------|---------| -| **Vector Index (100K entities)** | 149MB | 15-33MB | 5-10x | -| **Graph Index (100K verbs)** | 100MB | 100MB | N/A | -| **Metadata Indexes** | 10-50MB | 10-50MB | N/A | -| **UnifiedCache** | 2GB | 2GB | N/A | -| **Total (100K entities)** | ~2.3GB | ~2.2GB | Minimal | -| **Total (1M entities)** | ~3.5GB | ~2.3GB | **34% less** | -| **Total (10M entities)** | ~15GB | ~3.0GB | **80% less** | - -**Lazy mode activates automatically** when vectors exceed available cache. - ---- - -## 10. Best Practices - -### 10.1 Data Organization - -✅ **Do**: -- Use UUIDs for all entities and relationships -- Let Brainy handle sharding automatically (type-first + UUID sharding) -- Use metadata indexes for filtering -- Use batch operations for bulk deletions -- Snapshot `rootDirectory` to off-site storage from your scheduler (`gsutil` / `aws s3 sync` / `rclone`) -- Create branches for experimentation (instant, zero-cost) - -❌ **Don't**: -- Try to organize files manually -- Assume file paths are predictable (use IDs, not paths) -- Store large binary data in metadata (use VFS for blobs) -- Disable COW (always enabled) - ---- - -### 10.2 clear() Operation - -**What clear() deletes**: - -✅ Deletes: -- `branches/` → ALL entity data (all types, all shards, all branches, all forks) -- `_cow/` → ALL version control (commits, trees, blobs, refs) -- `_system/` → ALL indexes (statistics, vector index, metadata) - -✅ Resets: -- COW managers (refManager, blobStorage, commitLog) → `undefined` -- Entity counts → 0 -- Statistics cache → `null` - -✅ Behavior: -- COW **auto-reinitializes** on next operation (can't be disabled) -- Branches recreated automatically when new data added -- clean slate for fresh start - -**Example**: -```typescript -await brain.storage.clear() // ✅ Deletes ALL data correctly -await brain.add({ data: 'Alice', type: 'person' }) // ✅ COW reinitializes automatically -``` - ---- - -### 10.3 Querying - -✅ **Do**: -- Use type filtering for known types: `brain.getNouns({ type: 'Character' })` -- Use metadata filters when possible: `brain.getNouns({ filter: { metadata: { status: 'active' } } })` -- Limit result sets with pagination: `brain.getNouns({ limit: 100, offset: 0 })` -- Use semantic search for similarity queries: `brain.find('concept similar to...')` -- Use graph queries for relationships: `brain.getVerbsBySource(userId)` - -❌ **Don't**: -- Load all entities into memory: `const all = await brain.getNouns()` (use pagination!) -- Filter in application code (use metadata indexes instead) -- Scan all entities for simple queries (use indexes) - ---- - -## 10.4 Common Storage Scenarios - -Understanding how Brainy's storage architecture handles common scenarios. - -### Scenario 1: Adding an Entity - -**User code**: -```typescript -await brain.add({ data: 'Alice', type: 'person' }) -``` - -**What happens in storage**: -``` -1. Generate UUID: "3fa85f64-5717-4562-b3fc-2c963f66afa6" -2. Compute vector embedding: [0.1, 0.2, 0.3, ...] -3. Extract shard: "3f" -4. Write vector file: - → branches/main/entities/nouns/person/vectors/3f/3fa85f64-5717-4562-b3fc-2c963f66afa6.json -5. Write metadata file: - → branches/main/entities/nouns/person/metadata/3f/3fa85f64-5717-4562-b3fc-2c963f66afa6.json -6. Create COW commit: - → _cow/commits/00/00a1b2c3...sha256.json -7. Create COW tree (directory snapshot): - → _cow/trees/ab/abcdef12...sha256.json -8. Create COW blobs (content-addressable): - → _cow/blobs/3f/3fa85f64...sha256.bin (vector data) - → _cow/blobs/7b/7b2f5e3c...sha256.bin (metadata) -9. Update branch ref: - → _cow/refs/heads/main.json (points to new commit) -10. Update statistics: - → _system/statistics.json (increment person count) -11. Update vector index (in-memory): - → Connect to nearest neighbors -12. Update graph index (in-memory): - → Add to adjacency maps -``` - -**Files created**: 5-7 files (2 entity files + 1 commit + 1 tree + 2-3 blobs + 1 ref update) - ---- - -### Scenario 2: Querying by Type - -**User code**: -```typescript -const characters = await brain.getNouns({ type: 'Character', limit: 100 }) -``` - -**What happens in storage**: -``` -1. Type-first optimization: - → Scan only: branches/main/entities/nouns/Character/** - → Skip all other types (41 other type directories) -2. List all shards in parallel: - → branches/main/entities/nouns/Character/metadata/00/ - → branches/main/entities/nouns/Character/metadata/01/ - → ... (256 parallel operations) -3. Read first 100 metadata files -4. Return results (no vector load needed for listing) -``` - -**Performance**: 120-200ms for 100K entities (vs 30-60s without type-first) - ---- - -### Scenario 3: Semantic Search - -**User code**: -```typescript -const results = await brain.find('medieval castle', { k: 10 }) -``` - -**What happens in storage**: -``` -1. Compute query vector: [0.1, 0.2, 0.3, ...] -2. Use vector index (in-memory): - → Navigate graph from entry point - → Find 10 nearest neighbors (1-10ms) -3. Load vectors from cache or storage: - → Standard mode: All vectors already in memory - → Lazy mode: Load missing vectors from storage - branches/main/entities/nouns/Place/vectors/3f/3fa85f64...uuid.json -4. Return results with metadata -``` - -**Performance**: 1-10ms (standard mode), 2-15ms (lazy mode with cache) - ---- - -### Scenario 4: Creating a Branch - -**User code**: -```typescript -await brain.branch.create('feature-experiment') -``` - -**What happens in storage**: -``` -1. Copy ref (instant): - _cow/refs/heads/main.json → _cow/refs/heads/feature-experiment.json -2. Create branch directory (empty initially): - branches/feature-experiment/ -3. NO data copying (lazy COW): - → All data shared with main branch - → Only modified entities copied on write -4. Result: Branch created in 100-200ms -``` - -**Storage overhead**: ~100 bytes (just the ref file) -**Data duplication**: 0% (shared with main until modified) - ---- - -### Scenario 5: Time-Travel Query - -**User code**: -```typescript -const yesterday = await brain.asOf(Date.now() - 86400000) -const historicalData = await yesterday.getNouns({ type: 'Character' }) -``` - -**What happens in storage**: -``` -1. Find commit at timestamp: - → Search _cow/commits/** for timestamp match -2. Load commit object: - → _cow/commits/00/00a1b2c3...sha256.json -3. Load tree from commit: - → _cow/trees/ab/abcdef12...sha256.json -4. Lazy-load entities from historical tree: - → Read blob hashes from tree - → Load blobs: _cow/blobs/3f/3fa85f64...sha256.bin - → Reconstruct entities from historical state -5. Return read-only view (writes blocked) -``` - -**Performance**: 500-1000ms for first query (loads commit tree), 100-200ms for subsequent queries (cached) - ---- - -### Scenario 6: Clearing Storage - -**User code**: -```typescript -await brain.storage.clear() -``` - -**What happens in storage**: -``` -1. Delete all entity data: - → Remove: branches/ (entire directory) - → Result: ALL types, ALL shards, ALL branches deleted -2. Delete all version control: - → Remove: _cow/ (entire directory) - → Result: ALL commits, trees, blobs, refs deleted -3. Delete all indexes: - → Remove: _system/ (entire directory) - → Result: Statistics, vector index, metadata indexes deleted -4. Reset COW managers in memory: - → refManager = undefined - → blobStorage = undefined - → commitLog = undefined -5. Reset counters: - → totalNounCount = 0 - → totalVerbCount = 0 -6. Next operation auto-reinitializes COW: - → COW managers recreate automatically - → Fresh branches/main/ created - → New _cow/ initialized -``` - -**Storage after clear()**: Empty (all data deleted) -**COW status**: Always enabled (auto-reinitializes) - ---- - -### Scenario 7: Cold Start (Index Rebuild) - -**User code**: -```typescript -const brain = new Brainy({ storage: existingStorage }) await brain.init() +1. Acquire locks/_writer.lock (or open read-only) +2. Crash recovery: drop _generations/{N} newer than manifest.json +3. Load _system singletons (counts, statistics, field registry, id mapper) +4. Vector index: hnsw-system.json + entity vectors.json (lazy mode if large) +5. Graph adjacency: load LSM SSTables from _system/idx/ +6. Metadata index: column-store manifests + bitmap chunks on demand ``` -**What happens in storage**: -``` -1. Check for persisted indexes: - → Load: _system/vector/system.json (entry point, max level) - → Load: _system/vector/nodes/** (graph connections) - → Load: _system/statistics.json (entity counts) -2. Decide standard vs lazy mode: - → Check: entityCount × vectorSize vs. available cache - → Auto-enable lazy mode if needed -3. Rebuild vector index: - → Standard mode: Load all vectors into memory - → Lazy mode: Load only graph structure (~24 bytes/node) -4. Rebuild Graph Adjacency index: - → Load: branches/main/entities/verbs/*/metadata/** (all verbs) - → Build: sourceToTargets and targetToSources maps -5. Load Metadata indexes: - → Read: _system/metadata_indexes/** (on-demand) -6. Ready for queries (1-5 seconds for 100K entities) -``` +### Snapshot and restore -**Performance**: -- 100K entities: 1-5 seconds -- 1M entities: 10-30 seconds -- 10M entities: 1-3 minutes +``` +const db = brain.now(); await db.persist('/backups/today'); await db.release() +→ hard-links everything except locks/ into a self-contained directory + +await Brainy.load('/backups/today') // open snapshot read-only as a Db +await brain.restore('/backups/today', { confirm: true }) // replace store state +``` --- -### Scenario 8: Bulk Import +## 13. Summary -**User code**: -```typescript -await brain.addBatch([ - { data: 'Alice', type: 'person' }, - { data: 'Bob', type: 'person' }, - // ... 10,000 more -]) -``` - -**What happens in storage**: -``` -1. Batch vector computation (parallel) -2. Batch shard distribution: - → 10,000 entities → ~39 entities per shard (256 shards) -3. Parallel writes to storage: - → 256 shards written in parallel - → Each shard: ~39 files written -4. Single COW commit for entire batch: - → 1 commit object - → 1 tree object (or tree fan-out for large trees) - → 10,000+ blobs (deduplicated) -5. Update indexes in batch: - → Vector index: Batch insert (optimized) - → Graph: Batch update - → Metadata: Batch index update -``` - -**Performance**: 2-3 minutes for 10,000 entities (vs 4-8 hours without batching) - ---- - -## 11. Summary - -**Complete Storage Structure**: -- **3 storage layers**: branches/ (data), _cow/ (versions), _system/ (indexes) -- **2 files per entity**: metadata.json + vector.json (optimized I/O) -- **4 indexes**: Vector (semantic), Type Index (metadata-based), Graph (relationships), Metadata (fields) -- **256 shards**: UUID-based (uniform distribution) -- **42 noun types + 127 verb types**: Type is metadata, not storage structure -- **Git-like COW**: Branches, commits, trees, blobs, refs -- **VFS support**: Traditional file/folder hierarchies - -**Scalability**: -- **ID-First Storage**: 40x faster directory lookups (eliminates 42-type search) -- Sharding: 200x faster file lookups -- Type filtering: Still O(type_count) via metadata index -- Lazy mode: 5-10x less memory for large datasets -- COW: Instant branches, efficient forks -- Deduplication: 30-90% storage savings - -**Production Features**: -- Batch operations (efficient I/O) -- Operator-layer backup via `gsutil` / `aws s3 sync` / `rclone` against `rootDirectory` -- Auto-reinitialization (COW always-on, can't be broken) -- **Clean architecture**: Removed 500+ lines of type cache complexity +- **Two backends** (filesystem, memory), one path vocabulary. +- **Two files per entity** under ID-first `entities/{kind}/{shard}/{id}/`. +- **Type and subtype are metadata**, not directory structure; type queries go + through the metadata index, not the filesystem. +- **`_system/`** holds singletons plus 256 hash buckets of index state. +- **`_generations/` + `manifest.json` + `tx-log.jsonl`** implement + generational MVCC; `transact()` is the unit of history. +- **`_column_index/` + `_blobs/`** hold the columnar metadata runs and binary + blobs (VFS content included). +- **`locks/`** coordinates the single writer and reader flush requests, and + never travels with snapshots. --- ## Next Steps -- [Storage Architecture](./storage-architecture.md) - Adapter and backup details -- [VFS Guide](../vfs/README.md) - Use Virtual File System features -- [Triple Intelligence](../vfs/TRIPLE_INTELLIGENCE.md) - Semantic file extraction -- [Scaling Guide](../SCALING.md) - Handle 10M+ entities -- [Performance Tuning](../PERFORMANCE.md) - Optimize for your use case - ---- - -**Last Updated**: 2026 -**Key Features**: ID-first storage, COW always-on, metadata-based type index, 4-index architecture, VFS support, billion-scale optimization +- [ADR-001 — Generational MVCC](../ADR-001-generational-mvcc.md) +- [Index Architecture](./index-architecture.md) +- [Consistency Model](../concepts/consistency-model.md) +- [VFS Guide](../vfs/README.md) diff --git a/docs/architecture/multiprocess-storage-mixin.md b/docs/architecture/multiprocess-storage-mixin.md index cf0b4691..7b53fa33 100644 --- a/docs/architecture/multiprocess-storage-mixin.md +++ b/docs/architecture/multiprocess-storage-mixin.md @@ -86,9 +86,8 @@ Benefits: `FileSystemStorage` is the only one in-tree, but Cortex's `MmapFileSystemStorage` inherits from it — interface relocation would ripple through the plugin ecosystem. -- The current state works. Real failure modes - (BR-CX-INTERFACE-GAP) were build/install artifacts, not type-system - failures. +- The current state works. The real failure modes seen in the field + were build/install artifacts, not type-system failures. - 7.22.0 just shipped a clean fix. Stacking another refactor before consumers absorb it adds churn without urgency. - The `hasStorageMethod()` guard accomplishes the same runtime safety the diff --git a/docs/eli5.md b/docs/eli5.md index c5ace717..303de040 100644 --- a/docs/eli5.md +++ b/docs/eli5.md @@ -133,7 +133,7 @@ Brainy is the only row with every box checked. And it runs all of them in a sing ### One library, any scale -Brainy scales from a single laptop to billions of entities without changing a line of code. Small datasets live in memory. Larger ones spill to disk. At cloud scale, Brainy uses S3-compatible storage and automatically shards across nodes — the same API the whole way. Up to ten billion entities is fully implemented today. +Brainy scales from a quick experiment to serious production datasets without changing a line of code. Small datasets live entirely in memory. Larger ones spill to disk, where Brainy shards and compresses everything automatically. Need a backup or a copy? Snapshots are instant — the same API the whole way. Add Cortex and you also unlock memory-mapped storage — aggregate state lives directly in the operating system's memory with zero serialization overhead, as fast as the hardware allows. diff --git a/docs/vfs/SEMANTIC_VFS.md b/docs/vfs/SEMANTIC_VFS.md index 03639cc8..797f0078 100644 --- a/docs/vfs/SEMANTIC_VFS.md +++ b/docs/vfs/SEMANTIC_VFS.md @@ -47,16 +47,18 @@ const authFiles = await vfs.readdir('/by-concept/authentication') const aliceFiles = await vfs.readdir('/by-author/alice') ``` -### 2. **Time Travel** -See your codebase as it existed at any point: +### 2. **Change Tracking by Date** +List the files that were modified on any given day: ```typescript -// Code from March 15th -const snapshot = await vfs.readdir('/as-of/2024-03-15') +// Files that changed on March 15th +const changed = await vfs.readdir('/as-of/2024-03-15') -// Compare with today +// Everything under /src right now const current = await vfs.readdir('/src') ``` +`/as-of/` selects by *modification date* — it reads the files' current content, not historical versions. For true point-in-time queries over entity state, use the Db API (`brain.asOf(generation)`). + ### 3. **Knowledge Graph Navigation** Navigate by semantic relationships: ```typescript @@ -118,13 +120,14 @@ await vfs.stat('/by-author/alice/config.ts') ### 4. By Time (Temporal) ✅ **Production** ```typescript await vfs.readdir('/as-of/2024-03-15') -// Files modified on March 15, 2024 +// Files modified on March 15, 2024 (24-hour window) -await vfs.readFile('/as-of/2024-03-15/src/auth.ts') -// Read auth.ts as it existed that day +await vfs.readFile('/as-of/2024-03-15/auth.ts') +// Current content of auth.ts, addressed by modification date — +// the path only resolves if auth.ts was modified that day ``` -**How it works:** Tracks `modified` timestamp. Uses B-tree range queries (`greaterEqual`/`lessEqual`) for O(log n) performance. +**How it works:** Tracks the `modified` timestamp on every file and runs a range query (`greaterEqual`/`lessEqual`) over one 24-hour window for O(log n) performance. The VFS does not store historical file contents — `/as-of/` filters by *when a file last changed*; reads return the current bytes. For point-in-time state, use the Db API (`brain.asOf(generation)`). **Status:** ✅ Fully implemented and tested at 10K file scale @@ -222,7 +225,7 @@ console.log(authFiles) // ['login.ts', 'signup.ts', 'oauth.ts'] ``` -### Example 2: Time Travel +### Example 2: Changes by Day ```typescript // See what changed today const today = new Date().toISOString().split('T')[0] @@ -232,8 +235,8 @@ const todaysFiles = await vfs.readdir(`/as-of/${today}`) const yesterday = new Date(Date.now() - 86400000).toISOString().split('T')[0] const yesterdaysFiles = await vfs.readdir(`/as-of/${yesterday}`) -const newFiles = todaysFiles.filter(f => !yesterdaysFiles.includes(f)) -console.log('New files today:', newFiles) +const onlyToday = todaysFiles.filter(f => !yesterdaysFiles.includes(f)) +console.log('Changed today (untouched yesterday):', onlyToday) ``` ### Example 3: Graph Navigation @@ -432,25 +435,17 @@ await vfs.writeFile(path, code, { }) ``` -### 3. Optimize for Your Scale +### 3. Combine Dimensions ```typescript -// For < 100K files: Post-filtering is fine -// For > 100K files: Use flattened indexes - -// Force index refresh after bulk operations -await brain.storage.rebuildIndexes() -``` - -### 4. Combine Dimensions -```typescript -// Find security files Alice worked on this week +// Find security files Alice changed on a given day +// (each /as-of/ path covers exactly that one day) const aliceFiles = await vfs.readdir('/by-author/alice') const securityFiles = await vfs.readdir('/by-tag/security') -const thisWeek = await vfs.readdir(`/as-of/${weekAgo}`) +const changedThatDay = await vfs.readdir('/as-of/2024-03-15') const intersection = aliceFiles .filter(f => securityFiles.includes(f)) - .filter(f => thisWeek.includes(f)) + .filter(f => changedThatDay.includes(f)) ``` --- @@ -469,14 +464,14 @@ console.log(entity.metadata.concepts) ### Slow Queries on Large Datasets ```typescript -// Check if indexes are built -const stats = await brain.storage.getIndexStats() +// Check if indexes are built and populated +const stats = await brain.getIndexStats() console.log(stats) - -// Rebuild if needed -await brain.storage.rebuildIndexes() ``` +If an index looks empty or inconsistent, rebuild from raw storage with the CLI +(stop the live writer first): `brainy inspect repair `. + ### Semantic Path Returns Empty ```typescript // Check if metadata exists diff --git a/src/brainy.ts b/src/brainy.ts index c80f7d36..8710f79e 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -94,7 +94,10 @@ import { BatchResult, BrainyConfig, BrainyStats, - ScoreExplanation + ScoreExplanation, + FillSubtypeRule, + FillSubtypeRules, + FillSubtypesResult } from './types/brainy.types.js' import { NounType, VerbType, TypeUtils } from './types/graphTypes.js' import { BrainyInterface } from './types/brainyInterface.js' @@ -2885,7 +2888,7 @@ export class Brainy implements BrainyInterface { * }) * * @example Lock down management relationships - * brain.requireSubtype(VerbType.Manages, { + * brain.requireSubtype(VerbType.ReportsTo, { * values: ['direct', 'dotted-line'], * required: true * }) @@ -7330,10 +7333,10 @@ export class Brainy implements BrainyInterface { * @param subtype - Optional specific subtype string for O(1) point count * * @example - * brain.counts.byRelationshipSubtype(VerbType.Manages) + * brain.counts.byRelationshipSubtype(VerbType.ReportsTo) * // → { direct: 12, 'dotted-line': 3 } * - * brain.counts.byRelationshipSubtype(VerbType.Manages, 'direct') + * brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct') * // → 12 */ byRelationshipSubtype: (verb: VerbType, subtype?: string): number | Record => { @@ -7361,7 +7364,7 @@ export class Brainy implements BrainyInterface { * `topSubtypes` for verbs. * * @example - * brain.counts.topRelationshipSubtypes(VerbType.Manages, 5) + * brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 5) * // → [['direct', 12], ['dotted-line', 3]] */ topRelationshipSubtypes: (verb: VerbType, n: number = 10): Array<[string, number]> => { @@ -7533,7 +7536,7 @@ export class Brainy implements BrainyInterface { * @returns Sorted list of distinct subtype strings (empty if none) * * @example - * const variants = brain.relationshipSubtypesOf(VerbType.Manages) + * const variants = brain.relationshipSubtypesOf(VerbType.ReportsTo) * // → ['direct', 'dotted-line'] */ relationshipSubtypesOf(verb: VerbType): string[] { @@ -7550,17 +7553,17 @@ export class Brainy implements BrainyInterface { /** * Find entities and relationships missing a `subtype` value, grouped by type. * - * The diagnostic pair to `migrateField()` / `fillSubtypes()` — answers the - * question "what would break if I enabled strict subtype enforcement?". Run - * this before adopting an SDK that registers `requireSubtype()` rules on - * common NounTypes, or before upgrading to Brainy 8.0 (which makes - * `requireSubtype: true` the default). + * The diagnostic pair to `fillSubtypes()` / `migrateField()` — answers the + * question "what would strict subtype enforcement reject?". 8.0 makes + * `requireSubtype: true` the default, so run this when opening a pre-8.0 + * brain (with `requireSubtype: false` as the temporary escape hatch), then + * back-fill the reported gaps with `fillSubtypes(rules)`. * * Streams the brain via the same paginated `storage.getNouns()` / - * `storage.getVerbs()` pattern `migrateField()` uses — safe for large brains - * but linear in `O(N)`. Cortex 3.0+ may proxy this through the native - * column-store null-subtype bitmap for sub-linear performance on billion-scale - * brains (tracked in `CTX-SUBTYPE-8.0-CONTRACT`). + * `storage.getVerbs()` pattern `fillSubtypes()` uses — safe for large brains + * but linear in `O(N)`. A native index provider may serve this from a + * column-store null-subtype bitmap in the future for sub-linear performance + * on billion-scale brains. * * @param options.includeVFS - When `false` (default), entities marked with * `metadata.isVFSEntity` or `metadata.isVFS` are excluded from the report — @@ -7570,7 +7573,7 @@ export class Brainy implements BrainyInterface { * @param options.onProgress - Optional progress callback invoked after each batch. * @returns Report with per-type counts of entities/relationships without a * subtype, plus the overall total and a one-line recommendation pointing at - * `migrateField()` (7.x) or `fillSubtypes()` (8.0). + * `fillSubtypes()`. * * @example Find pre-existing gaps before turning on strict mode * const report = await brain.audit() @@ -7655,7 +7658,7 @@ export class Brainy implements BrainyInterface { const recommendation = missingSubtype === 0 ? 'No subtype gaps detected — this brain is strict-mode-ready.' - : 'Found ' + missingSubtype + ' entries without subtype. Migrate via `brain.migrateField()` (7.x) — or wait for `brain.fillSubtypes()` (8.0) which closes the same gap with caller-supplied rules.' + : 'Found ' + missingSubtype + ' entries without subtype. Back-fill with `brain.fillSubtypes(rules)` — one rule per NounType/VerbType, literal default or per-entry function. To lift an existing field into `subtype` instead, use `brain.migrateField({ from, to: \'subtype\' })`.' return { entitiesWithoutSubtype, @@ -7666,6 +7669,262 @@ export class Brainy implements BrainyInterface { } } + /** + * Back-fill missing `subtype` values across the whole brain — the 8.0 + * migration helper for data written before subtype became required. + * + * 8.0 enforces a non-empty `subtype` on every write by default + * (`requireSubtype: true`). A brain created on 7.x typically carries entities + * and relationships without one; this method clears that debt in a single + * idempotent pass so the opt-out (`requireSubtype: false`) can be removed. + * The intended upgrade flow: + * + * 1. Open the brain with `requireSubtype: false` (temporary escape hatch). + * 2. `await brain.audit()` — see what's missing, grouped by type. + * 3. `await brain.fillSubtypes(rules)` — back-fill with one rule per type. + * 4. Re-run `audit()` until `total === 0`, then drop the opt-out. + * + * **Rules.** One rule per NounType (entities) and/or VerbType + * (relationships) — the two vocabularies don't overlap, so a single map + * covers both sides. A rule is either a literal subtype string (blanket + * default) or a function deriving the subtype from the entry; functions + * subsume `where`-style filtering by returning `undefined` for entries they + * decline (those stay untouched and count as `skipped`, so a later run with + * a stricter rule can pick them up). + * + * **What is never touched:** entries that already carry a non-empty + * `subtype` (re-running is a no-op on them), and — unless + * `includeVFS: true` — Brainy's own VFS infrastructure entries + * (`metadata.isVFSEntity` / `metadata.isVFS`), which bypass enforcement + * anyway and are not migration debt. + * + * **Write strategy (deliberate):** each fill goes through the public + * `update()` / `updateRelation()` paths, so every write is individually + * atomic (storage record + indexes + subtype rollups commit together) and + * bumps the entry's `_rev` like any other update. The pass is *not* one + * whole-brain transaction: a migration over millions of entries inside a + * single transaction would hold an unbounded working set and turn one bad + * entry into an all-or-nothing failure. Idempotence is the recovery model — + * a crashed or partially-failed run is resumed safely by re-running, because + * only entries still missing a subtype are written. + * + * Streams via the same paginated `storage.getNouns()` / `storage.getVerbs()` + * walk `audit()` uses — `O(N)` but constant memory. The noun pass is skipped + * entirely when the map has no NounType rules, and vice versa. + * + * @param rules - Map of NounType/VerbType → literal subtype or rule function. + * Must contain at least one valid type key; invalid keys, empty-string + * literals, and non-string/non-function values throw before any data is + * touched. + * @param options.includeVFS - Also fill VFS infrastructure entries (default + * `false` — they bypass enforcement and don't need a subtype). + * @param options.batchSize - Pagination batch size (default `200`). + * @param options.onProgress - Optional callback invoked after each batch. + * @returns `{ scanned, filled, skipped, errors, byType }` — see + * {@link FillSubtypesResult}. After a clean run, `skipped` equals the + * remaining `audit().total`. + * @throws If the brain is read-only, the rule map is empty/malformed, or a + * key is not a valid NounType/VerbType. Per-entry write failures do NOT + * throw — they are collected in `errors` and the pass continues. + * + * @example Back-fill entities and relationships in one pass + * const report = await brain.fillSubtypes({ + * [NounType.Person]: (e) => e.metadata?.kind ?? 'unspecified', + * [NounType.Document]: 'general', + * [VerbType.RelatedTo]: 'unspecified' + * }) + * // → { scanned: 5200, filled: 1429, skipped: 0, errors: [], byType: { person: 800, document: 600, relatedTo: 29 } } + * + * @example Selective fill — decline entries a rule can't classify + * await brain.fillSubtypes({ + * [NounType.Person]: (e) => e.metadata?.department ? 'employee' : undefined + * }) + * // Persons without a department stay untouched (counted as skipped). + * + * @since 8.0.0 + */ + async fillSubtypes( + rules: FillSubtypeRules, + options: { + includeVFS?: boolean + batchSize?: number + onProgress?: (progress: { scanned: number; filled: number; skipped: number }) => void + } = {} + ): Promise { + this.assertWritable('fillSubtypes') + await this.ensureInitialized() + + // Validate the rule map up front — fail fast on shape errors before any + // data is touched. + if (!rules || typeof rules !== 'object' || Array.isArray(rules)) { + throw new Error( + 'fillSubtypes: rules must be a map of NounType/VerbType → subtype string or rule function' + ) + } + const nounTypeValues = new Set(Object.values(NounType)) + const verbTypeValues = new Set(Object.values(VerbType)) + const nounRules = new Map>>() + const verbRules = new Map>>() + for (const [key, rule] of Object.entries(rules)) { + if (rule === undefined) continue + if (typeof rule !== 'string' && typeof rule !== 'function') { + throw new Error( + `fillSubtypes: rule for '${key}' must be a subtype string or a function (got ${typeof rule})` + ) + } + if (typeof rule === 'string' && rule.length === 0) { + throw new Error( + `fillSubtypes: rule for '${key}' is an empty string — a subtype must be non-empty` + ) + } + if (nounTypeValues.has(key)) { + nounRules.set(key, rule as FillSubtypeRule>) + } else if (verbTypeValues.has(key)) { + verbRules.set(key, rule as FillSubtypeRule>) + } else { + throw new Error(`fillSubtypes: '${key}' is not a valid NounType or VerbType`) + } + } + if (nounRules.size === 0 && verbRules.size === 0) { + throw new Error( + 'fillSubtypes: rules map is empty — provide at least one NounType or VerbType rule' + ) + } + + const includeVFS = options.includeVFS === true + const batchSize = Math.max(1, options.batchSize ?? 200) + + let scanned = 0 + let filled = 0 + let skipped = 0 + const errors: Array<{ id: string; error: string }> = [] + const byType: Record = {} + + const reportProgress = (): void => { + if (options.onProgress) options.onProgress({ scanned, filled, skipped }) + } + + // Normalize a rule's output: only a non-empty string is a fill; anything + // else (undefined, empty string) is a decline. An empty subtype would fail + // the very strict-mode check this helper exists to satisfy. + const normalize = (value: string | undefined): string | undefined => + typeof value === 'string' && value.length > 0 ? value : undefined + + // Pass 1: entities (skipped entirely when the map has no NounType rules). + if (nounRules.size > 0) { + let offset = 0 + while (true) { + const page = await this.storage.getNouns({ pagination: { offset, limit: batchSize } }) + if (page.items.length === 0) break + for (const noun of page.items) { + scanned++ + // VFS infrastructure entries bypass enforcement via their marker, so + // they're not migration debt — leave them alone unless asked. + if ( + !includeVFS && + (noun.metadata?.isVFSEntity === true || noun.metadata?.isVFS === true) + ) { + continue + } + if (typeof noun.subtype === 'string' && noun.subtype.length > 0) continue // already filled — never overwrite + const rule = nounRules.get(noun.type) + if (rule === undefined) { + skipped++ + continue + } + try { + let subtype: string | undefined + if (typeof rule === 'function') { + const entity = await this.convertNounToEntity(noun) + subtype = normalize(rule(entity)) + } else { + subtype = rule + } + if (subtype === undefined) { + skipped++ + continue + } + await this.update({ id: noun.id, subtype }) + filled++ + byType[noun.type] = (byType[noun.type] || 0) + 1 + } catch (err) { + errors.push({ + id: noun.id, + error: err instanceof Error ? err.message : String(err) + }) + } + } + reportProgress() + if (!page.hasMore) break + offset += page.items.length + } + } + + // Pass 2: relationships (skipped entirely when the map has no VerbType rules). + if (verbRules.size > 0) { + let offset = 0 + while (true) { + const page = await this.storage.getVerbs({ pagination: { offset, limit: batchSize } }) + if (page.items.length === 0) break + for (const verb of page.items) { + scanned++ + if ( + !includeVFS && + (verb.metadata?.isVFSEntity === true || verb.metadata?.isVFS === true) + ) { + continue + } + if (typeof verb.subtype === 'string' && verb.subtype.length > 0) continue // already filled — never overwrite + const rule = verbRules.get(verb.verb) + if (rule === undefined) { + skipped++ + continue + } + try { + let subtype: string | undefined + if (typeof rule === 'function') { + // Project the stored verb onto the public Relation shape the + // rule function is typed against. + const relation: Relation = { + id: verb.id, + from: verb.sourceId, + to: verb.targetId, + type: verb.verb, + weight: verb.weight, + data: verb.data, + metadata: verb.metadata as T, + service: verb.service, + createdAt: verb.createdAt, + updatedAt: verb.updatedAt, + confidence: verb.confidence + } + subtype = normalize(rule(relation)) + } else { + subtype = rule + } + if (subtype === undefined) { + skipped++ + continue + } + await this.updateRelation({ id: verb.id, subtype }) + filled++ + byType[verb.verb] = (byType[verb.verb] || 0) + 1 + } catch (err) { + errors.push({ + id: verb.id, + error: err instanceof Error ? err.message : String(err) + }) + } + } + reportProgress() + if (!page.hasMore) break + offset += page.items.length + } + } + + return { scanned, filled, skipped, errors, byType } + } + /** * Stream-and-rewrite a field across every entity in the brain. * @@ -8845,6 +9104,16 @@ export class Brainy implements BrainyInterface { private async executeProximitySearch(params: FindParams): Promise[]> { if (!params.near) return [] + // Teaching error: without an anchor id the constraint is meaningless, and + // letting it fall through produces an opaque storage-layer sharding error. + if (!params.near.id) { + throw new Error( + "find({ near }): 'near.id' is required — pass the entity to search around, " + + 'e.g. near: { id, threshold }. To impose a minimum score on plain semantic ' + + 'results, filter on result.score instead.' + ) + } + const nearEntity = await this.get(params.near.id) if (!nearEntity) return [] @@ -9680,7 +9949,7 @@ export class Brainy implements BrainyInterface { */ private normalizeConfig(config?: BrainyConfig): Required { // Validate storage configuration. Brainy 8.0 ships two adapters only — - // FileSystemStorage and MemoryStorage — per BR-BRAINY-80-STORAGE-SIMPLIFY. + // FileSystemStorage and MemoryStorage (cloud + OPFS adapters were removed). // Cloud backup remains supported via operator tooling (db.persist() + // gsutil / aws s3 cp / rclone / azcopy). Pre-constructed adapter // instances bypass the type check (they ARE the storage). diff --git a/src/cli/commands/core.ts b/src/cli/commands/core.ts index 11407a11..398c301c 100644 --- a/src/cli/commands/core.ts +++ b/src/cli/commands/core.ts @@ -132,6 +132,7 @@ export const coreCommands = { const spinner = ora('Adding to neural database...').start() const brain = getBrainy() + await brain.init() let metadata: any = {} if (options.metadata) { @@ -204,6 +205,12 @@ export const coreCommands = { } else { formatOutput({ id: result, metadata, confidence: addParams.confidence, weight: addParams.weight }, options) } + + // close() releases the writer lock and indexes, but global timers + // (UnifiedCache bookkeeping, PathResolver stats) keep the event loop + // alive. CLI commands are one-shot — exit explicitly. + await brain.close() + process.exit(0) } catch (error: any) { if (spinner) spinner.fail('Failed to add data') console.error(chalk.red('Failed to add data:', error.message)) @@ -280,6 +287,7 @@ export const coreCommands = { const spinner = ora('Searching with Triple Intelligence™...').start() const brain = getBrainy() + await brain.init() // Build comprehensive search params const searchParams: any = { @@ -292,11 +300,6 @@ export const coreCommands = { searchParams.offset = parseInt(options.offset) } - // Vector Intelligence - similarity threshold - if (options.threshold) { - searchParams.near = { threshold: parseFloat(options.threshold) } - } - // Metadata Intelligence - type filtering if (options.type) { const types = options.type.split(',').map(t => t.trim()) @@ -314,7 +317,9 @@ export const coreCommands = { } } - // Vector Intelligence - proximity search + // Vector Intelligence - proximity search around an anchor entity. + // `near` requires an id; a bare --threshold (no --near) is applied as a + // plain score floor on the fused results after find() returns. if (options.near) { searchParams.near = { id: options.near, @@ -371,7 +376,14 @@ export const coreCommands = { } } - const results = await brain.find(searchParams) + let results = await brain.find(searchParams) + + // Without --near there is no proximity anchor; apply --threshold as a + // minimum-score filter on the fused results instead. + if (!options.near && options.threshold) { + const minScore = parseFloat(options.threshold) + results = results.filter((r) => r.score === undefined || r.score >= minScore) + } spinner.succeed(`Found ${results.length} results`) @@ -457,6 +469,10 @@ export const coreCommands = { } else { formatOutput(results, options) } + + // One-shot command — see add() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { if (spinner) spinner.fail('Search failed') console.error(chalk.red('Search failed:', error.message)) @@ -496,6 +512,7 @@ export const coreCommands = { const spinner = ora('Fetching item...').start() const brain = getBrainy() + await brain.init() // Try to get the item const item = await brain.get(id) @@ -529,6 +546,10 @@ export const coreCommands = { } else { formatOutput(item, options) } + + // One-shot command — see add() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { if (spinner) spinner.fail('Failed to get item') console.error(chalk.red('Failed to get item:', error.message)) @@ -589,6 +610,7 @@ export const coreCommands = { const spinner = ora('Creating relationship...').start() const brain = getBrainy() + await brain.init() let metadata: any = {} if (options.metadata) { @@ -627,6 +649,10 @@ export const coreCommands = { } else { formatOutput({ id: result, source, verb, target, metadata }, options) } + + // One-shot command — see add() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { if (spinner) spinner.fail('Failed to create relationship') console.error(chalk.red('Failed to create relationship:', error.message)) @@ -683,6 +709,7 @@ export const coreCommands = { spinner = ora('Updating entity...').start() const brain = getBrainy() + await brain.init() // Get existing entity first const existing = await brain.get(id) @@ -731,6 +758,10 @@ export const coreCommands = { } else { formatOutput({ id, updated: true }, options) } + + // One-shot command — see add() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { if (spinner) spinner.fail('Failed to update entity') console.error(chalk.red('Update failed:', error.message)) @@ -784,6 +815,7 @@ export const coreCommands = { spinner = ora('Deleting entity...').start() const brain = getBrainy() + await brain.init() await brain.delete(id) @@ -794,6 +826,10 @@ export const coreCommands = { } else { formatOutput({ id, deleted: true }, options) } + + // One-shot command — see add() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { if (spinner) spinner.fail('Failed to delete entity') console.error(chalk.red('Delete failed:', error.message)) @@ -846,6 +882,7 @@ export const coreCommands = { spinner = ora('Removing relationship...').start() const brain = getBrainy() + await brain.init() await brain.unrelate(id) @@ -856,6 +893,10 @@ export const coreCommands = { } else { formatOutput({ id, removed: true }, options) } + + // One-shot command — see add() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { if (spinner) spinner.fail('Failed to remove relationship') console.error(chalk.red('Unrelate failed:', error.message)) @@ -875,7 +916,9 @@ export const coreCommands = { if (options.json) { formatOutput(diag, options) - return + // One-shot command — see add() for why the explicit close + exit. + await brain.close() + process.exit(0) } console.log(chalk.bold('\nBrainy Diagnostics')) @@ -907,6 +950,10 @@ export const coreCommands = { console.log(` Metadata: ${diag.indexes.metadata.type} (initialized: ${diag.indexes.metadata.initialized})`) console.log(` Graph: ${diag.indexes.graph.type} (initialized: ${diag.indexes.graph.initialized}, wired: ${diag.indexes.graph.wiredToStorage})`) console.log() + + // One-shot command — see add() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { console.error(chalk.red('Diagnostics failed: ' + error.message)) process.exit(1) diff --git a/src/cli/commands/import.ts b/src/cli/commands/import.ts index 2a107f90..d4db46fe 100644 --- a/src/cli/commands/import.ts +++ b/src/cli/commands/import.ts @@ -152,6 +152,7 @@ export const importCommands = { spinner = ora('Initializing import...').start() const brain = getBrainy() + await brain.init() // Handle different source types let result: any @@ -401,6 +402,12 @@ export const importCommands = { } else if (options.json) { formatOutput(result, options) } + + // close() releases the writer lock and indexes, but global timers + // (UnifiedCache bookkeeping, PathResolver stats) keep the event loop + // alive. CLI commands are one-shot — exit explicitly. + await brain.close() + process.exit(0) } catch (error: any) { if (spinner) spinner.fail('Import failed') console.error(chalk.red('Import failed:', error.message)) @@ -472,9 +479,10 @@ export const importCommands = { spinner = ora('Initializing VFS import...').start() const brain = getBrainy() + await brain.init() // Get VFS - const vfs = await brain.vfs + const vfs = brain.vfs // Load DirectoryImporter const { DirectoryImporter } = await import('../../vfs/importers/DirectoryImporter.js') @@ -536,6 +544,10 @@ export const importCommands = { } else if (options.json) { formatOutput(result, options) } + + // One-shot command — see import() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { if (spinner) spinner.fail('VFS import failed') console.error(chalk.red('VFS import failed:', error.message)) diff --git a/src/cli/commands/insights.ts b/src/cli/commands/insights.ts index 907d2802..218ab4c2 100644 --- a/src/cli/commands/insights.ts +++ b/src/cli/commands/insights.ts @@ -41,6 +41,7 @@ export const insightsCommands = { try { const brain = getBrainy() + await brain.init() // Get insights from Brainy const insights = await brain.insights() @@ -102,6 +103,12 @@ export const insightsCommands = { } else { formatOutput(insights, options) } + + // close() releases the writer lock and indexes, but global timers + // (UnifiedCache bookkeeping, PathResolver stats) keep the event loop + // alive. CLI commands are one-shot — exit explicitly. + await brain.close() + process.exit(0) } catch (error: any) { spinner.fail('Failed to get insights') console.error(chalk.red('Insights failed:', error.message)) @@ -120,6 +127,7 @@ export const insightsCommands = { try { const brain = getBrainy() + await brain.init() // Get available fields from metadata index const fields = await brain.getAvailableFields() @@ -166,6 +174,10 @@ export const insightsCommands = { })) formatOutput(fieldsWithStats, options) } + + // One-shot command — see insights() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { spinner.fail('Failed to get fields') console.error(chalk.red('Fields analysis failed:', error.message)) @@ -186,6 +198,7 @@ export const insightsCommands = { if (!field) { spinner = ora('Getting available fields...').start() const brain = getBrainy() + await brain.init() const availableFields = await brain.getAvailableFields() spinner.stop() @@ -202,6 +215,7 @@ export const insightsCommands = { spinner = ora(`Getting values for field: ${field}...`).start() const brain = getBrainy() + await brain.init() const values = await brain.getFieldValues(field) const limit = options.limit ? parseInt(options.limit) : 100 @@ -242,6 +256,10 @@ export const insightsCommands = { } else { formatOutput({ field, values, count: values.length }, options) } + + // One-shot command — see insights() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { if (spinner) spinner.fail('Failed to get field values') console.error(chalk.red('Field values failed:', error.message)) @@ -289,6 +307,7 @@ export const insightsCommands = { spinner = ora('Analyzing optimal query plan...').start() const brain = getBrainy() + await brain.init() const plan = await brain.getOptimalQueryPlan(filters) @@ -328,6 +347,10 @@ export const insightsCommands = { } else { formatOutput({ filters, plan }, options) } + + // One-shot command — see insights() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { if (spinner) spinner.fail('Failed to generate query plan') console.error(chalk.red('Query plan failed:', error.message)) diff --git a/src/cli/commands/neural.ts b/src/cli/commands/neural.ts index 2c878071..be334a4c 100644 --- a/src/cli/commands/neural.ts +++ b/src/cli/commands/neural.ts @@ -1,7 +1,11 @@ /** - * 🧠 Neural Similarity API Commands - * - * CLI interface for semantic similarity, clustering, and neural operations + * @module cli/commands/neural + * @description Neural CLI commands: semantic similarity, clustering, + * hierarchy, neighbors, outlier detection, and visualization data export. + * Registered in `src/cli/index.ts` as `similar` / `cluster` / `related` / + * `hierarchy` / `outliers` / `visualize`. Each command is one-shot: it + * initializes the shared Brainy instance, runs the neural operation, then + * closes the store and exits explicitly. */ import inquirer from 'inquirer' @@ -25,136 +29,13 @@ interface CommandArguments { _: string[]; } -export const neuralCommand = { - command: 'neural [action]', - describe: '🧠 Neural similarity and clustering operations', - - builder: (yargs: any) => { - return yargs - .positional('action', { - describe: 'Neural operation to perform', - type: 'string', - choices: ['similar', 'clusters', 'hierarchy', 'neighbors', 'path', 'outliers', 'visualize'] - }) - .option('id', { - describe: 'Item ID for similarity operations', - type: 'string', - alias: 'i' - }) - .option('query', { - describe: 'Query text for similarity search', - type: 'string', - alias: 'q' - }) - .option('threshold', { - describe: 'Similarity threshold (0-1)', - type: 'number', - default: 0.7, - alias: 't' - }) - .option('format', { - describe: 'Output format', - type: 'string', - choices: ['json', 'table', 'tree', 'graph'], - default: 'table', - alias: 'f' - }) - .option('output', { - describe: 'Output file path', - type: 'string', - alias: 'o' - }) - .option('limit', { - describe: 'Maximum number of results', - type: 'number', - default: 10, - alias: 'l' - }) - .option('algorithm', { - describe: 'Clustering algorithm', - type: 'string', - choices: ['hierarchical', 'kmeans', 'dbscan', 'auto'], - default: 'auto', - alias: 'a' - }) - .option('dimensions', { - describe: 'Visualization dimensions (2 or 3)', - type: 'number', - choices: [2, 3], - default: 2, - alias: 'd' - }) - .option('explain', { - describe: 'Include detailed explanations', - type: 'boolean', - default: false, - alias: 'e' - }) - }, - - handler: async (argv: CommandArguments) => { - console.log(chalk.cyan('\n🧠 NEURAL SIMILARITY API')) - console.log(chalk.gray('━'.repeat(50))) +let brainyInstance: Brainy | null = null - // Initialize Brainy and Neural API - const brain = new Brainy() - const neural = brain.neural() - - try { - const action = argv.action || await promptForAction() - - switch (action) { - case 'similar': - await handleSimilarCommand(neural, argv) - break - case 'clusters': - await handleClustersCommand(neural, argv) - break - case 'hierarchy': - await handleHierarchyCommand(neural, argv) - break - case 'neighbors': - await handleNeighborsCommand(neural, argv) - break - case 'path': - console.log(chalk.yellow('\n⚠️ Semantic path finding coming soon')) - console.log(chalk.dim('This feature requires implementing graph traversal algorithms')) - console.log(chalk.dim('Use "neighbors" and "hierarchy" commands to explore connections')) - break - case 'outliers': - await handleOutliersCommand(neural, argv) - break - case 'visualize': - await handleVisualizeCommand(neural, argv) - break - default: - console.log(chalk.red(`❌ Unknown action: ${action}`)) - showHelp() - } - } catch (error) { - console.error(chalk.red('💥 Error:'), error instanceof Error ? error.message : error) - process.exit(1) - } +const getBrainy = (): Brainy => { + if (!brainyInstance) { + brainyInstance = new Brainy() } -} - -async function promptForAction(): Promise { - const answer = await inquirer.prompt([{ - type: 'list', - name: 'action', - message: 'Choose a neural operation:', - choices: [ - { name: '🔗 Calculate similarity between items', value: 'similar' }, - { name: '🎯 Find semantic clusters', value: 'clusters' }, - { name: '🌳 Show item hierarchy', value: 'hierarchy' }, - { name: '🕸️ Find semantic neighbors', value: 'neighbors' }, - { name: '🛣️ Find semantic path between items (coming soon)', value: 'path', disabled: true }, - { name: '🚨 Detect outliers', value: 'outliers' }, - { name: '📊 Generate visualization data', value: 'visualize' } - ] - }]) - - return answer.action + return brainyInstance } async function handleSimilarCommand(neural: any, argv: CommandArguments): Promise { @@ -436,63 +317,6 @@ async function handleNeighborsCommand(neural: any, argv: CommandArguments): Prom } } -async function handlePathCommand(neural: any, argv: CommandArguments): Promise { - const spinner = ora('🛣️ Finding semantic path...').start() - - try { - let fromId: string, toId: string - - if (argv._ && argv._.length >= 3) { - fromId = argv._[1] - toId = argv._[2] - } else { - spinner.stop() - const answers = await inquirer.prompt([ - { - type: 'input', - name: 'from', - message: 'From item ID:', - validate: (input: string) => input.length > 0 - }, - { - type: 'input', - name: 'to', - message: 'To item ID:', - validate: (input: string) => input.length > 0 - } - ]) - fromId = answers.from - toId = answers.to - spinner.start() - } - - const path = await neural.semanticPath(fromId, toId) - - if (path.length === 0) { - spinner.warn('🚫 No semantic path found') - console.log(`No path found between ${chalk.cyan(fromId)} and ${chalk.cyan(toId)}`) - } else { - spinner.succeed(`✅ Found path with ${path.length} hops`) - - console.log(`\n🛣️ Semantic Path from ${chalk.cyan(fromId)} to ${chalk.cyan(toId)}:`) - console.log(`${chalk.cyan(fromId)} (start)`) - - path.forEach((hop, index) => { - console.log(`${' '.repeat(index + 1)}↓ ${(hop.similarity * 100).toFixed(1)}%`) - console.log(`${' '.repeat(index + 1)}${hop.id} (hop ${hop.hop})`) - }) - } - - if (argv.output) { - await saveToFile(argv.output, path, argv.format!) - } - - } catch (error) { - spinner.fail('💥 Failed to find path') - throw error - } -} - async function handleOutliersCommand(neural: any, argv: CommandArguments): Promise { const spinner = ora('🚨 Detecting semantic outliers...').start() @@ -592,32 +416,32 @@ function formatAsTable(data: any): string { return JSON.stringify(data, null, 2) } -function showHelp(): void { - console.log('\n🧠 Neural Similarity API Commands:') - console.log('') - console.log(' brainy neural similar Calculate similarity') - console.log(' brainy neural clusters Find semantic clusters') - console.log(' brainy neural hierarchy Show item hierarchy') - console.log(' brainy neural neighbors Find semantic neighbors') - console.log(' brainy neural path Find semantic path') - console.log(' brainy neural outliers Detect outliers') - console.log(' brainy neural visualize Generate visualization data') - console.log('') - console.log('Options:') - console.log(' --threshold, -t Similarity threshold (0-1)') - console.log(' --format, -f Output format (json|table|tree|graph)') - console.log(' --output, -o Save to file') - console.log(' --limit, -l Maximum results') - console.log(' --explain, -e Include explanations') - console.log('') +/** + * @description Run a neural CLI handler with the one-shot lifecycle every + * Brainy command follows: init → work → `close()` → explicit `process.exit`. + * close() releases the writer lock and indexes, but global timers + * (UnifiedCache bookkeeping, PathResolver stats) keep the event loop alive, + * so one-shot commands must exit explicitly. + * @param work - The handler body, given the initialized neural API. + */ +async function runNeuralCommand(work: (neural: ReturnType) => Promise): Promise { + try { + const brain = getBrainy() + await brain.init() + + await work(brain.neural()) + + await brain.close() + process.exit(0) + } catch (error) { + console.error(chalk.red('💥 Error:'), error instanceof Error ? error.message : error) + process.exit(1) + } } // Commander-compatible wrappers export const neuralCommands = { async similar(a?: string, b?: string, options?: any) { - const brain = new Brainy() - const neural = brain.neural() - // Build argv-style object for handler const argv: CommandArguments = { _: ['neural', 'similar', a || '', b || ''].filter(x => x), @@ -627,13 +451,10 @@ export const neuralCommands = { ...options } - await handleSimilarCommand(neural, argv) + await runNeuralCommand((neural) => handleSimilarCommand(neural, argv)) }, async cluster(options?: any) { - const brain = new Brainy() - const neural = brain.neural() - const argv: CommandArguments = { _: ['neural', 'cluster'], algorithm: options?.algorithm || 'hierarchical', @@ -643,26 +464,20 @@ export const neuralCommands = { ...options } - await handleClustersCommand(neural, argv) + await runNeuralCommand((neural) => handleClustersCommand(neural, argv)) }, async hierarchy(id?: string, options?: any) { - const brain = new Brainy() - const neural = brain.neural() - const argv: CommandArguments = { _: ['neural', 'hierarchy', id || ''].filter(x => x), id, ...options } - await handleHierarchyCommand(neural, argv) + await runNeuralCommand((neural) => handleHierarchyCommand(neural, argv)) }, async related(id?: string, options?: any) { - const brain = new Brainy() - const neural = brain.neural() - const argv: CommandArguments = { _: ['neural', 'related', id || ''].filter(x => x), id, @@ -671,13 +486,10 @@ export const neuralCommands = { ...options } - await handleNeighborsCommand(neural, argv) + await runNeuralCommand((neural) => handleNeighborsCommand(neural, argv)) }, async outliers(options?: any) { - const brain = new Brainy() - const neural = brain.neural() - const argv: CommandArguments = { _: ['neural', 'outliers'], threshold: options?.threshold ? parseFloat(options.threshold) : 0.3, @@ -685,13 +497,10 @@ export const neuralCommands = { ...options } - await handleOutliersCommand(neural, argv) + await runNeuralCommand((neural) => handleOutliersCommand(neural, argv)) }, async visualize(options?: any) { - const brain = new Brainy() - const neural = brain.neural() - const argv: CommandArguments = { _: ['neural', 'visualize'], format: options?.format || 'json', @@ -701,8 +510,6 @@ export const neuralCommands = { ...options } - await handleVisualizeCommand(neural, argv) + await runNeuralCommand((neural) => handleVisualizeCommand(neural, argv)) } -} - -export default neuralCommand \ No newline at end of file +} \ No newline at end of file diff --git a/src/cli/commands/nlp.ts b/src/cli/commands/nlp.ts index 652a3699..8b7547da 100644 --- a/src/cli/commands/nlp.ts +++ b/src/cli/commands/nlp.ts @@ -62,6 +62,7 @@ export const nlpCommands = { spinner = ora('Extracting entities with neural NLP...').start() const brain = getBrainy() + await brain.init() // Extract entities using Brainy's neural entity extractor const entities = await brain.extract(text) @@ -105,6 +106,12 @@ export const nlpCommands = { } else { formatOutput(entities, options) } + + // close() releases the writer lock and indexes, but global timers + // (UnifiedCache bookkeeping, PathResolver stats) keep the event loop + // alive. CLI commands are one-shot — exit explicitly. + await brain.close() + process.exit(0) } catch (error: any) { if (spinner) spinner.fail('Entity extraction failed') console.error(chalk.red('Extraction failed:', error.message)) @@ -147,6 +154,7 @@ export const nlpCommands = { spinner = ora('Extracting concepts with neural analysis...').start() const brain = getBrainy() + await brain.init() const confidence = options.threshold ? parseFloat(options.threshold) : 0.5 const concepts = await brain.extractConcepts(text, { confidence }) @@ -171,6 +179,10 @@ export const nlpCommands = { } else { formatOutput(concepts, options) } + + // One-shot command — see extract() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { if (spinner) spinner.fail('Concept extraction failed') console.error(chalk.red('Extraction failed:', error.message)) @@ -201,6 +213,7 @@ export const nlpCommands = { spinner = ora('Analyzing text with neural NLP...').start() const brain = getBrainy() + await brain.init() // Run both entity extraction and concept extraction const [entities, concepts] = await Promise.all([ @@ -265,6 +278,10 @@ export const nlpCommands = { concepts }, options) } + + // One-shot command — see extract() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { if (spinner) spinner.fail('Analysis failed') console.error(chalk.red('Analysis failed:', error.message)) diff --git a/src/cli/commands/storage.ts b/src/cli/commands/storage.ts index 54a19044..8d7889bd 100644 --- a/src/cli/commands/storage.ts +++ b/src/cli/commands/storage.ts @@ -1,16 +1,33 @@ /** - * 💾 Storage Management Commands + * @module cli/commands/storage + * @description Storage management CLI commands for the 8.0 storage surface + * (filesystem + memory adapters). * - * Modern interactive CLI for storage lifecycle, cost optimization, and management + * - `storage status` — backend, root directory, entity/relationship counts, + * writer-lock holder, and operational mode (rendered from `brain.stats()`). + * - `storage batch-delete ` — delete a newline-separated list of entity + * IDs through the public `brain.delete()` path (vectors, metadata, indexes, + * and statistics all stay consistent), with per-ID retry and an optional + * continue-on-error mode. + * + * The 7.x cloud-era subcommands (lifecycle policies, tiering, cost + * estimation, runtime compression toggles) were removed with the cloud + * storage adapters in 8.0. Filesystem gzip compression is a constructor + * option (`storage.options.compression`, on by default), not a runtime + * toggle. */ import chalk from 'chalk' import ora from 'ora' import inquirer from 'inquirer' import Table from 'cli-table3' -import { readFileSync, writeFileSync } from 'node:fs' +import { readFileSync } from 'node:fs' import { Brainy } from '../../brainy.js' +/** + * @description Shared CLI flags every storage command accepts (`--verbose`, + * `--json`, `--pretty`), mirroring the other command modules. + */ interface StorageOptions { verbose?: boolean json?: boolean @@ -19,6 +36,11 @@ interface StorageOptions { let brainyInstance: Brainy | null = null +/** + * @description Lazily construct the module's shared `Brainy` instance so + * repeated handler invocations (and tests) reuse one store handle. + * @returns The shared `Brainy` instance. + */ const getBrainy = (): Brainy => { if (!brainyInstance) { brainyInstance = new Brainy() @@ -26,111 +48,93 @@ const getBrainy = (): Brainy => { return brainyInstance } -const formatBytes = (bytes: number): string => { - if (bytes === 0) return '0 B' - const k = 1024 - const sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB'] - const i = Math.floor(Math.log(bytes) / Math.log(k)) - return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i] -} - -const formatCurrency = (amount: number): string => { - return new Intl.NumberFormat('en-US', { - style: 'currency', - currency: 'USD', - minimumFractionDigits: 0, - maximumFractionDigits: 0 - }).format(amount) -} - -const formatOutput = (data: any, options: StorageOptions): void => { +/** + * @description Emit machine-readable output when `--json` is set; honors + * `--pretty` for indented JSON. + * @param data - The value to serialize. + * @param options - Shared CLI flags. + */ +const formatOutput = (data: unknown, options: StorageOptions): void => { if (options.json) { console.log(options.pretty ? JSON.stringify(data, null, 2) : JSON.stringify(data)) } } +/** + * @description The `storage status` / `storage batch-delete` command handlers + * registered in `src/cli/index.ts`. + */ export const storageCommands = { /** - * Show storage status and health + * @description Show storage status: backend adapter, root directory, + * entity/relationship counts, writer-lock holder, and operational mode. + * @param options - Shared CLI flags. + * @example + * ```bash + * brainy storage status # human-readable report + * brainy storage status --json # machine-readable + * ``` */ - async status(options: StorageOptions & { detailed?: boolean; quota?: boolean }) { + async status(options: StorageOptions) { const spinner = ora('Checking storage status...').start() try { const brain = getBrainy() - const storage = (brain as any).storage + await brain.init() - const status = await storage.getStorageStatus() + const stats = await brain.stats() spinner.succeed('Storage status retrieved') if (options.json) { - formatOutput(status, options) - return + formatOutput( + { + backend: stats.storage.backend, + rootDir: stats.storage.rootDir, + entityCount: stats.entityCount, + relationCount: stats.relationCount, + mode: stats.mode, + writerLock: stats.writerLock ?? null + }, + options + ) + // close() releases the writer lock and indexes, but global timers + // (UnifiedCache bookkeeping, PathResolver stats) keep the event loop + // alive. CLI commands are one-shot — exit explicitly. + await brain.close() + process.exit(0) } console.log(chalk.cyan('\n💾 Storage Status\n')) - // Basic info table const infoTable = new Table({ head: [chalk.cyan('Property'), chalk.cyan('Value')], style: { head: [], border: [] } }) infoTable.push( - ['Type', chalk.green(status.type || 'Unknown')], - ['Status', status.healthy ? chalk.green('✓ Healthy') : chalk.red('✗ Unhealthy')] + ['Backend', chalk.green(stats.storage.backend)], + ['Entities', String(stats.entityCount)], + ['Relationships', String(stats.relationCount)], + ['Mode', stats.mode] ) - - if (status.details) { - if (status.details.bucket) { - infoTable.push(['Bucket', status.details.bucket]) - } - if (status.details.region) { - infoTable.push(['Region', status.details.region]) - } - if (status.details.path) { - infoTable.push(['Path', status.details.path]) - } - if (status.details.compression !== undefined) { - infoTable.push(['Compression', status.details.compression ? chalk.green('Enabled') : chalk.dim('Disabled')]) - } + if (stats.storage.rootDir) { + infoTable.push(['Root', chalk.dim(stats.storage.rootDir)]) } console.log(infoTable.toString()) - // Quota info (for OPFS) - if (options.quota && status.details?.quota) { - console.log(chalk.cyan('\n📊 Quota Information\n')) - - const quotaTable = new Table({ - head: [chalk.cyan('Metric'), chalk.cyan('Value')], - style: { head: [], border: [] } - }) - - const usagePercent = status.details.usagePercent || 0 - const usageColor = usagePercent > 80 ? chalk.red : usagePercent > 60 ? chalk.yellow : chalk.green - - quotaTable.push( - ['Usage', formatBytes(status.details.usage)], - ['Quota', formatBytes(status.details.quota)], - ['Used', usageColor(`${usagePercent.toFixed(1)}%`)] + if (stats.writerLock) { + console.log(chalk.bold('\nWriter Lock:')) + console.log( + ` PID ${stats.writerLock.pid} on ${stats.writerLock.hostname} ` + + chalk.dim(`(since ${stats.writerLock.startedAt})`) ) - - console.log(quotaTable.toString()) - - if (usagePercent > 80) { - console.log(chalk.yellow('\n⚠️ Warning: Approaching quota limit!')) - console.log(chalk.dim(' Consider cleaning up old data or requesting more quota')) - } - } - - // Detailed info - if (options.detailed && status.details) { - console.log(chalk.cyan('\n🔍 Detailed Information\n')) - console.log(chalk.dim(JSON.stringify(status.details, null, 2))) } + // One-shot command — see the --json branch for why the explicit exit. + await brain.close() + process.exit(0) } catch (error: any) { spinner.fail('Failed to get storage status') console.error(chalk.red(error.message)) @@ -139,474 +143,14 @@ export const storageCommands = { }, /** - * Lifecycle policy management - */ - lifecycle: { - /** - * Set lifecycle policy (interactive or from file) - */ - async set(configFile?: string, options: StorageOptions & { validate?: boolean } = {}) { - const brain = getBrainy() - const storage = (brain as any).storage - - let policy: any - - if (configFile) { - // Load from file - const spinner = ora('Loading policy from file...').start() - try { - const content = readFileSync(configFile, 'utf-8') - policy = JSON.parse(content) - spinner.succeed('Policy loaded') - } catch (error: any) { - spinner.fail('Failed to load policy file') - console.error(chalk.red(error.message)) - process.exit(1) - } - } else { - // Interactive mode - console.log(chalk.cyan('\n📋 Lifecycle Policy Builder\n')) - - const storageStatus = await storage.getStorageStatus() - const storageType = storageStatus.type - - // Detect storage provider - let provider: 'aws' | 'gcs' | 'azure' | 'r2' | 'unknown' = 'unknown' - if (storageType === 's3-compatible') { - const endpoint = storageStatus.details?.endpoint || '' - if (endpoint.includes('r2.cloudflarestorage.com')) { - provider = 'r2' - } else if (endpoint.includes('amazonaws.com')) { - provider = 'aws' - } - } else if (storageType === 'gcs') { - provider = 'gcs' - } else if (storageType === 'azure') { - provider = 'azure' - } - - if (provider === 'unknown') { - console.log(chalk.yellow('⚠️ Could not detect storage provider')) - console.log(chalk.dim('Lifecycle policies require: AWS S3, GCS, or Azure Blob Storage')) - process.exit(1) - } - - console.log(chalk.green(`✓ Detected: ${provider.toUpperCase()}\n`)) - - // Provider-specific interactive prompts - if (provider === 'aws' || provider === 'r2') { - const answers = await inquirer.prompt([ - { - type: 'input', - name: 'prefix', - message: 'Path prefix to apply policy to:', - default: 'entities/', - validate: (input: string) => input.length > 0 - }, - { - type: 'list', - name: 'strategy', - message: 'Choose optimization strategy:', - choices: [ - { name: '🎯 Intelligent-Tiering (Recommended - Automatic)', value: 'intelligent' }, - { name: '📅 Lifecycle Policies (Manual tier transitions)', value: 'lifecycle' }, - { name: '🚀 Aggressive Archival (Maximum savings)', value: 'aggressive' } - ] - } - ]) - - if (answers.strategy === 'intelligent') { - // Intelligent-Tiering - const tierAnswers = await inquirer.prompt([ - { - type: 'input', - name: 'configName', - message: 'Configuration name:', - default: 'brainy-auto-tier' - } - ]) - - const spinner = ora('Enabling Intelligent-Tiering...').start() - try { - await storage.enableIntelligentTiering(answers.prefix, tierAnswers.configName) - spinner.succeed('Intelligent-Tiering enabled!') - - console.log(chalk.cyan('\n💰 Cost Impact:\n')) - console.log(chalk.green('✓ Automatic optimization based on access patterns')) - console.log(chalk.green('✓ No retrieval fees')) - console.log(chalk.green('✓ Expected savings: 50-70%')) - console.log(chalk.dim('\nObjects automatically move between tiers:')) - console.log(chalk.dim(' • Frequent Access Tier (accessed within 30 days)')) - console.log(chalk.dim(' • Infrequent Access Tier (not accessed for 30+ days)')) - console.log(chalk.dim(' • Archive Instant Access Tier (not accessed for 90+ days)')) - - return - } catch (error: any) { - spinner.fail('Failed to enable Intelligent-Tiering') - console.error(chalk.red(error.message)) - process.exit(1) - } - } else if (answers.strategy === 'lifecycle') { - // Custom lifecycle policy - const lifecycleAnswers = await inquirer.prompt([ - { - type: 'number', - name: 'standardIA', - message: 'Move to Standard-IA after (days):', - default: 30, - validate: (input: number) => input > 0 - }, - { - type: 'number', - name: 'glacier', - message: 'Move to Glacier after (days):', - default: 90, - validate: (input: number) => input > 0 - }, - { - type: 'number', - name: 'deepArchive', - message: 'Move to Deep Archive after (days):', - default: 365, - validate: (input: number) => input > 0 - } - ]) - - policy = { - rules: [{ - id: 'brainy-lifecycle', - prefix: answers.prefix, - status: 'Enabled', - transitions: [ - { days: lifecycleAnswers.standardIA, storageClass: 'STANDARD_IA' }, - { days: lifecycleAnswers.glacier, storageClass: 'GLACIER' }, - { days: lifecycleAnswers.deepArchive, storageClass: 'DEEP_ARCHIVE' } - ] - }] - } - } else { - // Aggressive archival - policy = { - rules: [{ - id: 'brainy-aggressive', - prefix: answers.prefix, - status: 'Enabled', - transitions: [ - { days: 7, storageClass: 'STANDARD_IA' }, - { days: 30, storageClass: 'GLACIER' }, - { days: 90, storageClass: 'DEEP_ARCHIVE' } - ] - }] - } - } - } else if (provider === 'gcs') { - // GCS Autoclass - const answers = await inquirer.prompt([ - { - type: 'confirm', - name: 'useAutoclass', - message: 'Enable Autoclass (automatic tier management)?', - default: true - } - ]) - - if (answers.useAutoclass) { - const autoclassAnswers = await inquirer.prompt([ - { - type: 'list', - name: 'terminalClass', - message: 'Terminal storage class:', - choices: [ - { name: 'Archive (Lowest cost)', value: 'ARCHIVE' }, - { name: 'Nearline (Balance)', value: 'NEARLINE' } - ], - default: 'ARCHIVE' - } - ]) - - const spinner = ora('Enabling Autoclass...').start() - try { - await storage.enableAutoclass({ terminalStorageClass: autoclassAnswers.terminalClass }) - spinner.succeed('Autoclass enabled!') - - console.log(chalk.cyan('\n💰 Cost Impact:\n')) - console.log(chalk.green('✓ Automatic optimization (no manual policies needed)')) - console.log(chalk.green('✓ Expected savings: 60-94%')) - console.log(chalk.dim('\nObjects automatically move:')) - console.log(chalk.dim(' • Standard → Nearline → Coldline → Archive')) - console.log(chalk.dim(' • Based on access patterns')) - - return - } catch (error: any) { - spinner.fail('Failed to enable Autoclass') - console.error(chalk.red(error.message)) - process.exit(1) - } - } - } else if (provider === 'azure') { - // Azure lifecycle - const answers = await inquirer.prompt([ - { - type: 'number', - name: 'coolAfter', - message: 'Move to Cool tier after (days):', - default: 30 - }, - { - type: 'number', - name: 'archiveAfter', - message: 'Move to Archive tier after (days):', - default: 90 - } - ]) - - policy = { - rules: [{ - name: 'brainy-lifecycle', - enabled: true, - type: 'Lifecycle', - definition: { - filters: { blobTypes: ['blockBlob'] }, - actions: { - baseBlob: { - tierToCool: { daysAfterModificationGreaterThan: answers.coolAfter }, - tierToArchive: { daysAfterModificationGreaterThan: answers.archiveAfter } - } - } - } - }] - } - } - } - - // Validate policy - if (options.validate && policy) { - console.log(chalk.cyan('\n📋 Policy Preview:\n')) - console.log(chalk.dim(JSON.stringify(policy, null, 2))) - - const { confirm } = await inquirer.prompt([{ - type: 'confirm', - name: 'confirm', - message: 'Apply this policy?', - default: true - }]) - - if (!confirm) { - console.log(chalk.yellow('Policy not applied')) - return - } - } - - // Apply policy - const spinner = ora('Applying lifecycle policy...').start() - try { - await storage.setLifecyclePolicy(policy) - spinner.succeed('Lifecycle policy applied!') - - // Calculate estimated savings - if (!options.json) { - console.log(chalk.cyan('\n💰 Estimated Annual Savings:\n')) - - const savingsTable = new Table({ - head: [chalk.cyan('Scale'), chalk.cyan('Before'), chalk.cyan('After'), chalk.cyan('Savings')], - style: { head: [], border: [] } - }) - - const scenarios = [ - { size: 5, before: 1380, after: 59, savings: 1321, percent: 96 }, - { size: 50, before: 13800, after: 594, savings: 13206, percent: 96 }, - { size: 500, before: 138000, after: 5940, savings: 132060, percent: 96 } - ] - - scenarios.forEach(s => { - savingsTable.push([ - `${s.size}TB`, - formatCurrency(s.before), - chalk.green(formatCurrency(s.after)), - chalk.green(`${formatCurrency(s.savings)} (${s.percent}%)`) - ]) - }) - - console.log(savingsTable.toString()) - console.log(chalk.dim('\n💡 Tip: Monitor costs with: brainy monitor cost --breakdown')) - } - - if (options.json) { - formatOutput({ success: true, policy }, options) - } - } catch (error: any) { - spinner.fail('Failed to apply lifecycle policy') - console.error(chalk.red(error.message)) - process.exit(1) - } - }, - - /** - * Get current lifecycle policy - */ - async get(options: StorageOptions & { format?: 'json' | 'yaml' } = {}) { - const spinner = ora('Retrieving lifecycle policy...').start() - - try { - const brain = getBrainy() - const storage = (brain as any).storage - - const policy = await storage.getLifecyclePolicy() - - spinner.succeed('Policy retrieved') - - if (options.json || options.format === 'json') { - console.log(JSON.stringify(policy, null, 2)) - } else { - console.log(chalk.cyan('\n📋 Current Lifecycle Policy:\n')) - console.log(chalk.dim(JSON.stringify(policy, null, 2))) - } - } catch (error: any) { - spinner.fail('Failed to get lifecycle policy') - console.error(chalk.red(error.message)) - process.exit(1) - } - }, - - /** - * Remove lifecycle policy - */ - async remove(options: StorageOptions) { - const { confirm } = await inquirer.prompt([{ - type: 'confirm', - name: 'confirm', - message: chalk.yellow('⚠️ Remove lifecycle policy? (This will stop cost optimization)'), - default: false - }]) - - if (!confirm) { - console.log(chalk.yellow('Policy not removed')) - return - } - - const spinner = ora('Removing lifecycle policy...').start() - - try { - const brain = getBrainy() - const storage = (brain as any).storage - - await storage.removeLifecyclePolicy() - - spinner.succeed('Lifecycle policy removed') - - if (!options.json) { - console.log(chalk.yellow('\n⚠️ Cost optimization disabled')) - console.log(chalk.dim(' Storage costs will increase to standard rates')) - console.log(chalk.dim(' Run "brainy storage lifecycle set" to re-enable')) - } - } catch (error: any) { - spinner.fail('Failed to remove lifecycle policy') - console.error(chalk.red(error.message)) - process.exit(1) - } - } - }, - - /** - * Compression management (FileSystem storage) - */ - compression: { - async enable(options: StorageOptions) { - const spinner = ora('Enabling compression...').start() - - try { - const brain = getBrainy() - const storage = (brain as any).storage - - const status = await storage.getStorageStatus() - - if (status.type !== 'filesystem') { - spinner.fail('Compression is only available for FileSystem storage') - console.log(chalk.yellow('\n⚠️ Current storage type: ' + status.type)) - console.log(chalk.dim(' Compression works with: filesystem')) - process.exit(1) - } - - // Enable compression (would need to update storage config) - spinner.succeed('Compression enabled!') - - if (!options.json) { - console.log(chalk.cyan('\n📦 Compression Settings:\n')) - console.log(chalk.green('✓ Gzip compression enabled')) - console.log(chalk.dim(' Expected space savings: 60-80%')) - console.log(chalk.dim(' All new files will be compressed')) - console.log(chalk.dim('\n💡 Tip: Existing files will be compressed during next write')) - } - } catch (error: any) { - spinner.fail('Failed to enable compression') - console.error(chalk.red(error.message)) - process.exit(1) - } - }, - - async disable(options: StorageOptions) { - const spinner = ora('Disabling compression...').start() - - try { - spinner.succeed('Compression disabled') - - if (!options.json) { - console.log(chalk.yellow('\n⚠️ Compression disabled')) - console.log(chalk.dim(' Files will no longer be compressed')) - console.log(chalk.dim(' Existing compressed files will still be readable')) - } - } catch (error: any) { - spinner.fail('Failed to disable compression') - console.error(chalk.red(error.message)) - process.exit(1) - } - }, - - async status(options: StorageOptions) { - const spinner = ora('Checking compression status...').start() - - try { - const brain = getBrainy() - const storage = (brain as any).storage - - const status = await storage.getStorageStatus() - - spinner.succeed('Status retrieved') - - const compressionEnabled = status.details?.compression || false - - if (!options.json) { - console.log(chalk.cyan('\n📦 Compression Status:\n')) - - const table = new Table({ - head: [chalk.cyan('Property'), chalk.cyan('Value')], - style: { head: [], border: [] } - }) - - table.push( - ['Status', compressionEnabled ? chalk.green('✓ Enabled') : chalk.dim('Disabled')], - ['Algorithm', compressionEnabled ? 'gzip' : 'None'], - ['Space Savings', compressionEnabled ? chalk.green('60-80%') : chalk.dim('0%')] - ) - - console.log(table.toString()) - - if (!compressionEnabled) { - console.log(chalk.dim('\n💡 Enable compression: brainy storage compression enable')) - } - } else { - formatOutput({ enabled: compressionEnabled }, options) - } - } catch (error: any) { - spinner.fail('Failed to check compression status') - console.error(chalk.red(error.message)) - process.exit(1) - } - } - }, - - /** - * Batch delete with retry logic + * @description Delete a list of entity IDs (one per line in a file) through + * the public `brain.delete()` path so vectors, metadata, graph edges, + * indexes, and statistics all stay consistent. Asks for confirmation before + * deleting; failed IDs are retried up to `--max-retries` times, and + * `--continue-on-error` keeps going past IDs that still fail. + * @param file - Path to a newline-separated file of entity IDs. + * @param options - Shared CLI flags plus `--max-retries` and + * `--continue-on-error`. */ async batchDelete( file: string, @@ -615,227 +159,107 @@ export const storageCommands = { const spinner = ora('Loading entity IDs...').start() try { - const brain = getBrainy() - const storage = (brain as any).storage - - // Read IDs from file + // Read IDs from file before touching the store. const content = readFileSync(file, 'utf-8') - const ids = content.split('\n').filter(line => line.trim()) + const ids = content + .split('\n') + .map((line) => line.trim()) + .filter((line) => line.length > 0) spinner.succeed(`Loaded ${ids.length} entity IDs`) - // Confirm - const { confirm } = await inquirer.prompt([{ - type: 'confirm', - name: 'confirm', - message: chalk.yellow(`⚠️ Delete ${ids.length} entities? This cannot be undone.`), - default: false - }]) - - if (!confirm) { - console.log(chalk.yellow('Deletion cancelled')) - return + if (ids.length === 0) { + console.log(chalk.yellow('No entity IDs found in file — nothing to delete')) + process.exit(0) } - // Generate paths for all entities (vectors + metadata) - const paths: string[] = [] - for (const id of ids) { - const shard = id.substring(0, 2) - paths.push(`entities/nouns/vectors/${shard}/${id}.json`) - paths.push(`entities/nouns/metadata/${shard}/${id}.json`) - } - - // Batch delete with progress - const deleteSpinner = ora('Deleting entities...').start() - const startTime = Date.now() - - try { - await storage.batchDelete(paths, { - maxRetries: options.maxRetries ? parseInt(options.maxRetries) : 3, - continueOnError: options.continueOnError || false - }) - - const duration = ((Date.now() - startTime) / 1000).toFixed(1) - const rate = (ids.length / parseFloat(duration)).toFixed(0) - - deleteSpinner.succeed(`Deleted ${ids.length} entities in ${duration}s (${rate}/sec)`) - - if (!options.json) { - console.log(chalk.green(`\n✓ Batch delete complete`)) - console.log(chalk.dim(` Entities: ${ids.length}`)) - console.log(chalk.dim(` Duration: ${duration}s`)) - console.log(chalk.dim(` Rate: ${rate} entities/sec`)) - } else { - formatOutput({ - deleted: ids.length, - duration: parseFloat(duration), - rate: parseFloat(rate) - }, options) - } - } catch (error: any) { - deleteSpinner.fail('Batch delete failed') - console.error(chalk.red(error.message)) - process.exit(1) - } - } catch (error: any) { - spinner.fail('Failed to load entity IDs') - console.error(chalk.red(error.message)) - process.exit(1) - } - }, - - /** - * Cost estimation tool - */ - async costEstimate( - options: StorageOptions & { - provider?: 'aws' | 'gcs' | 'azure' | 'r2' - size?: string - operations?: string - } = {} - ) { - console.log(chalk.cyan('\n💰 Cloud Storage Cost Estimator\n')) - - let provider: string - let sizeGB: number - let operations: number - - if (!options.provider || !options.size || !options.operations) { - // Interactive mode - const answers = await inquirer.prompt([ + // Confirm — destructive and irreversible. + const { confirm } = await inquirer.prompt([ { - type: 'list', - name: 'provider', - message: 'Cloud provider:', - choices: [ - { name: 'AWS S3', value: 'aws' }, - { name: 'Google Cloud Storage', value: 'gcs' }, - { name: 'Azure Blob Storage', value: 'azure' }, - { name: 'Cloudflare R2', value: 'r2' } - ], - when: !options.provider - }, - { - type: 'number', - name: 'sizeGB', - message: 'Total data size (GB):', - default: 1000, - validate: (input: number) => input > 0, - when: !options.size - }, - { - type: 'number', - name: 'operations', - message: 'Monthly operations (reads + writes):', - default: 1000000, - validate: (input: number) => input >= 0, - when: !options.operations + type: 'confirm', + name: 'confirm', + message: chalk.yellow(`⚠️ Delete ${ids.length} entities? This cannot be undone.`), + default: false } ]) - provider = options.provider || answers.provider - sizeGB = options.size ? parseFloat(options.size) : answers.sizeGB - operations = options.operations ? parseInt(options.operations) : answers.operations - } else { - provider = options.provider - sizeGB = parseFloat(options.size) - operations = parseInt(options.operations) - } - - // Calculate costs - const spinner = ora('Calculating costs...').start() - - // Pricing (2025 estimates) - const pricing: Record = { - aws: { - standard: { storage: 0.023, operations: 0.005 }, - ia: { storage: 0.0125, operations: 0.01 }, - glacier: { storage: 0.004, operations: 0.05 }, - deepArchive: { storage: 0.00099, operations: 0.10 } - }, - gcs: { - standard: { storage: 0.020, operations: 0.005 }, - nearline: { storage: 0.010, operations: 0.010 }, - coldline: { storage: 0.004, operations: 0.050 }, - archive: { storage: 0.0012, operations: 0.050 } - }, - azure: { - hot: { storage: 0.0184, operations: 0.005 }, - cool: { storage: 0.010, operations: 0.010 }, - archive: { storage: 0.00099, operations: 0.050 } - }, - r2: { - standard: { storage: 0.015, operations: 0.0045 } - } - } - - const providerPricing = pricing[provider] - const results: any = {} - - for (const [tier, prices] of Object.entries(providerPricing)) { - const storageCost = sizeGB * (prices as any).storage - const opsCost = (operations / 1000000) * (prices as any).operations - const monthly = storageCost + opsCost - const annual = monthly * 12 - - results[tier] = { - storage: storageCost, - operations: opsCost, - monthly, - annual - } - } - - spinner.succeed('Cost estimation complete') - - if (!options.json) { - console.log(chalk.cyan(`\n💰 Cost Estimate for ${provider.toUpperCase()}\n`)) - console.log(chalk.dim(`Data Size: ${sizeGB} GB (${formatBytes(sizeGB * 1024 * 1024 * 1024)})`)) - console.log(chalk.dim(`Operations: ${operations.toLocaleString()}/month\n`)) - - const table = new Table({ - head: [ - chalk.cyan('Tier'), - chalk.cyan('Storage/mo'), - chalk.cyan('Ops/mo'), - chalk.cyan('Total/mo'), - chalk.cyan('Annual') - ], - style: { head: [], border: [] } - }) - - for (const [tier, costs] of Object.entries(results)) { - table.push([ - tier.toUpperCase(), - formatCurrency((costs as any).storage), - formatCurrency((costs as any).operations), - formatCurrency((costs as any).monthly), - chalk.green(formatCurrency((costs as any).annual)) - ]) + if (!confirm) { + console.log(chalk.yellow('Deletion cancelled')) + process.exit(0) } - console.log(table.toString()) + const brain = getBrainy() + await brain.init() - // Show savings - const tiers = Object.keys(results) - if (tiers.length > 1) { - const highest = results[tiers[0]] - const lowest = results[tiers[tiers.length - 1]] - const savings = highest.annual - lowest.annual - const savingsPercent = ((savings / highest.annual) * 100).toFixed(0) + const maxRetries = options.maxRetries ? parseInt(options.maxRetries, 10) : 3 - console.log(chalk.cyan('\n💡 Potential Savings:\n')) - console.log(chalk.green(` ${formatCurrency(savings)}/year (${savingsPercent}%) by using lifecycle policies`)) - console.log(chalk.dim(` ${tiers[0].toUpperCase()} → ${tiers[tiers.length - 1].toUpperCase()}`)) + const deleteSpinner = ora(`Deleting ${ids.length} entities...`).start() + const startTime = Date.now() + let deleted = 0 + const failed: Array<{ id: string; error: string }> = [] + + for (const id of ids) { + let lastError: unknown = null + let succeeded = false + + for (let attempt = 0; attempt <= maxRetries; attempt++) { + try { + await brain.delete(id) + succeeded = true + break + } catch (error) { + lastError = error + } + } + + if (succeeded) { + deleted++ + deleteSpinner.text = `Deleting entities... ${deleted}/${ids.length}` + } else { + failed.push({ + id, + error: lastError instanceof Error ? lastError.message : String(lastError) + }) + if (!options.continueOnError) { + break + } + } } - if (provider === 'r2') { - console.log(chalk.cyan('\n✨ R2 Advantage:\n')) - console.log(chalk.green(' $0 egress fees (unlimited data transfer out)')) - console.log(chalk.dim(' Perfect for high-traffic applications')) + const duration = ((Date.now() - startTime) / 1000).toFixed(1) + + if (failed.length === 0) { + deleteSpinner.succeed(`Deleted ${deleted} entities in ${duration}s`) + } else { + deleteSpinner.warn(`Deleted ${deleted}/${ids.length} entities in ${duration}s (${failed.length} failed)`) } - } else { - formatOutput(results, options) + + if (!options.json) { + console.log(chalk.green(`\n✓ Batch delete complete`)) + console.log(chalk.dim(` Deleted: ${deleted}`)) + console.log(chalk.dim(` Failed: ${failed.length}`)) + console.log(chalk.dim(` Duration: ${duration}s`)) + + if (failed.length > 0 && options.verbose) { + console.log(chalk.bold('\n⚠️ Failed IDs:')) + for (const f of failed.slice(0, 10)) { + console.log(` ${chalk.dim(f.id)}: ${chalk.red(f.error)}`) + } + if (failed.length > 10) { + console.log(chalk.dim(` ... and ${failed.length - 10} more`)) + } + } + } else { + formatOutput({ deleted, failed, duration: parseFloat(duration) }, options) + } + + // One-shot command — see status() for why the explicit close + exit. + await brain.close() + process.exit(failed.length > 0 && !options.continueOnError ? 1 : 0) + } catch (error: any) { + spinner.fail('Batch delete failed') + console.error(chalk.red(error.message)) + process.exit(1) } } } diff --git a/src/cli/commands/utility.ts b/src/cli/commands/utility.ts index e934bebd..41fd5e18 100644 --- a/src/cli/commands/utility.ts +++ b/src/cli/commands/utility.ts @@ -22,8 +22,7 @@ interface StatsOptions extends UtilityOptions { } interface CleanOptions extends UtilityOptions { - removeOrphans?: boolean - rebuildIndex?: boolean + force?: boolean } interface BenchmarkOptions extends UtilityOptions { @@ -63,6 +62,7 @@ export const utilityCommands = { try { const brain = getBrainy() + await brain.init() const nounCount = await brain.getNounCount() const verbCount = await brain.getVerbCount() const memUsage = process.memoryUsage() @@ -77,7 +77,11 @@ export const utilityCommands = { if (options.json) { formatOutput(stats, options) - return + // close() releases the writer lock and indexes, but global timers + // (UnifiedCache bookkeeping, PathResolver stats) keep the event loop + // alive. CLI commands are one-shot — exit explicitly. + await brain.close() + process.exit(0) } console.log(chalk.cyan('\n📊 Database Statistics\n')) @@ -112,7 +116,10 @@ export const utilityCommands = { ) console.log(memTable.toString()) - + + // One-shot command — see the --json branch for why the explicit exit. + await brain.close() + process.exit(0) } catch (error: any) { spinner.fail('Failed to gather statistics') console.error(chalk.red(error.message)) @@ -121,30 +128,33 @@ export const utilityCommands = { }, /** - * Clean and optimize database + * Clear the database (all entities, relationships, and indexes). + * Destructive — asks for confirmation unless --force is passed. */ async clean(options: CleanOptions) { - const spinner = ora('Cleaning database...').start() + let spinner: ReturnType | null = null try { - const brain = getBrainy() + // Destructive operation — confirm first (skipped with --force). + if (!options.force) { + const inquirer = (await import('inquirer')).default + const { confirm } = await inquirer.prompt([{ + type: 'confirm', + name: 'confirm', + message: chalk.yellow('⚠️ Permanently delete ALL data (entities, relationships, indexes)?'), + default: false + }]) - // For now, only support full clear - // removeOrphans and rebuildIndex would require new Brainy APIs - if (options.removeOrphans || options.rebuildIndex) { - spinner.warn('Advanced cleanup options not yet implemented') - console.log(chalk.yellow('\n⚠️ Advanced cleanup features coming soon:')) - console.log(chalk.dim(' • --remove-orphans: Remove disconnected items')) - console.log(chalk.dim(' • --rebuild-index: Rebuild vector index')) - console.log(chalk.dim('\nUse "brainy clean" without options to clear the database')) - return + if (!confirm) { + console.log(chalk.yellow('Clean cancelled')) + process.exit(0) + } } - // Show warning before clearing - console.log(chalk.yellow('\n⚠️ WARNING: This will permanently delete ALL data!')) + const brain = getBrainy() // Clear all data (entities, relationships, and every index) - spinner.text = 'Clearing all data...' + spinner = ora('Clearing all data...').start() await brain.init() await brain.clear() @@ -163,7 +173,7 @@ export const utilityCommands = { await brain.close() process.exit(0) } catch (error: any) { - spinner.fail('Cleanup failed') + if (spinner) spinner.fail('Cleanup failed') console.error(chalk.red(error.message)) process.exit(1) } @@ -185,7 +195,8 @@ export const utilityCommands = { try { const brain = getBrainy() - + await brain.init() + // Benchmark different operations const benchmarks = [ { name: 'add', enabled: operations === 'all' || operations.includes('add') }, @@ -205,7 +216,8 @@ export const utilityCommands = { switch (bench.name) { case 'add': - await brain.add({ data: `Test item ${i}`, type: NounType.Thing, metadata: { benchmark: true } }) + // 8.0 requires a subtype on every write by default. + await brain.add({ data: `Test item ${i}`, type: NounType.Thing, subtype: 'benchmark', metadata: { benchmark: true } }) break case 'search': await brain.find({ query: 'test', limit: 10 }) @@ -284,7 +296,10 @@ export const utilityCommands = { } else { formatOutput(results, options) } - + + // One-shot command — see stats() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { console.error(chalk.red('Benchmark failed:'), error.message) process.exit(1) diff --git a/src/cli/commands/vfs.ts b/src/cli/commands/vfs.ts index 32132670..e07f1198 100644 --- a/src/cli/commands/vfs.ts +++ b/src/cli/commands/vfs.ts @@ -70,6 +70,12 @@ export const vfsCommands = { } else { formatOutput({ path, content: buffer.toString(), size: buffer.length }, options) } + + // close() releases the writer lock and indexes, but global timers + // (UnifiedCache bookkeeping, PathResolver stats) keep the event loop + // alive. CLI commands are one-shot — exit explicitly. + await brain.close() + process.exit(0) } catch (error: any) { spinner.fail('Failed to read file') console.error(chalk.red(error.message)) @@ -109,6 +115,10 @@ export const vfsCommands = { } else { formatOutput({ path, size: Buffer.byteLength(data) }, options) } + + // One-shot command — see read() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { spinner.fail('Failed to write file') console.error(chalk.red(error.message)) @@ -132,7 +142,9 @@ export const vfsCommands = { if (!options.json) { if (!Array.isArray(entries) || entries.length === 0) { console.log(chalk.yellow('Directory is empty')) - return + // One-shot command — see read() for why the explicit close + exit. + await brain.close() + process.exit(0) } if (options.long) { @@ -171,6 +183,10 @@ export const vfsCommands = { } else { formatOutput(entries, options) } + + // One-shot command — see read() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { spinner.fail('Failed to list directory') console.error(chalk.red(error.message)) @@ -202,6 +218,10 @@ export const vfsCommands = { } else { formatOutput(stats, options) } + + // One-shot command — see read() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { spinner.fail('Failed to get stats') console.error(chalk.red(error.message)) @@ -227,6 +247,10 @@ export const vfsCommands = { } else { formatOutput({ path, created: true }, options) } + + // One-shot command — see read() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { spinner.fail('Failed to create directory') console.error(chalk.red(error.message)) @@ -258,12 +282,16 @@ export const vfsCommands = { } else { formatOutput({ path, removed: true }, options) } + + // One-shot command — see read() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { spinner.fail('Failed to remove') console.error(chalk.red(error.message)) - if (!options.force) { - process.exit(1) - } + // --force tolerates the failure but the process is still one-shot: + // exit cleanly (0) instead of falling off the event loop. + process.exit(options.force ? 0 : 1) } }, @@ -303,6 +331,10 @@ export const vfsCommands = { } else { formatOutput(results, options) } + + // One-shot command — see read() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { spinner.fail('Search failed') console.error(chalk.red(error.message)) @@ -343,6 +375,10 @@ export const vfsCommands = { } else { formatOutput(results, options) } + + // One-shot command — see read() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { spinner.fail('Failed to find similar files') console.error(chalk.red(error.message)) @@ -371,6 +407,10 @@ export const vfsCommands = { } else { formatOutput(tree, options) } + + // One-shot command — see read() for why the explicit close + exit. + await brain.close() + process.exit(0) } catch (error: any) { spinner.fail('Failed to build tree') console.error(chalk.red(error.message)) diff --git a/src/cli/index.ts b/src/cli/index.ts index 7fa4c1b8..6c371bff 100644 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -19,6 +19,7 @@ import { insightsCommands } from './commands/insights.js' import { importCommands } from './commands/import.js' import { snapshotCommands } from './commands/snapshot.js' import { inspectCommands } from './commands/inspect.js' +import { types as typesCommand, validate as validateCommand } from './commands/types.js' import { readFileSync } from 'fs' import { fileURLToPath } from 'url' import { dirname, join } from 'path' @@ -70,14 +71,9 @@ ${chalk.cyan('Examples:')} $ brainy vfs similar /code/Button.tsx ${chalk.dim('# Storage management')} - $ brainy storage status --quota - $ brainy storage lifecycle set ${chalk.dim('# Interactive mode')} - $ brainy storage cost-estimate + $ brainy storage status $ brainy storage batch-delete old-ids.txt - ${chalk.dim('# Interactive mode')} - $ brainy interactive - ${chalk.cyan('Documentation:')} ${chalk.dim('Full docs:')} https://github.com/soulcraftlabs/brainy ${chalk.dim('Report issues:')} https://github.com/soulcraftlabs/brainy/issues @@ -109,7 +105,7 @@ program .description('Advanced search with Triple Intelligence™ (interactive if no query)') .option('-k, --limit ', 'Number of results', '10') .option('--offset ', 'Skip N results (pagination)') - .option('-t, --threshold ', 'Similarity threshold (0-1)', '0.7') + .option('-t, --threshold ', 'Minimum similarity score (0-1); with --near, the proximity threshold') .option('--type ', 'Filter by type(s) - comma separated') .option('--where ', 'Metadata filters (JSON)') .option('--near ', 'Find items near this ID') @@ -181,6 +177,23 @@ program .option('--pretty', 'Pretty print JSON') .action(coreCommands.diagnostics) +// ===== Type Commands ===== + +program + .command('types') + .description('List all NounType and VerbType values') + .option('--noun', 'Show noun types only') + .option('--verb', 'Show verb types only') + .option('--json', 'Output as JSON') + .action(typesCommand) + +program + .command('validate [type]') + .description('Check whether a type string is a valid NounType or VerbType (interactive if no type)') + .option('--verb', 'Validate as a verb type (default: noun type)') + .option('--json', 'Output as JSON') + .action(validateCommand) + // ===== Neural Commands ===== program @@ -222,17 +235,6 @@ program .option('--children-only', 'Show only child hierarchy') .action(neuralCommands.hierarchy) -program - .command('path ') - .description('Find semantic path between items') - .option('--steps', 'Show step-by-step path') - .option('--max-hops ', 'Maximum path length', '5') - .action(() => { - console.log(chalk.yellow('\n⚠️ Semantic path finding coming soon')) - console.log(chalk.dim('This feature requires implementing graph traversal algorithms')) - console.log(chalk.dim('Use "brainy neighbors" and "brainy hierarchy" to explore connections')) - }) - program .command('outliers') .alias('anomalies') @@ -457,89 +459,26 @@ program program .command('storage') - .description('💾 Storage management and cost optimization') + .description('💾 Storage management (filesystem + memory backends)') .addCommand( new Command('status') - .description('Show storage status and health') - .option('--detailed', 'Show detailed information') - .option('--quota', 'Show quota information (OPFS)') + .description('Show storage backend, counts, root directory, and writer lock') + .option('--json', 'Output as JSON') + .option('--pretty', 'Pretty-print JSON') .action((options) => { storageCommands.status(options) }) ) - .addCommand( - new Command('lifecycle') - .description('Lifecycle policy management') - .addCommand( - new Command('set') - .argument('[config-file]', 'Policy configuration file (JSON)') - .description('Set lifecycle policy (interactive if no file)') - .option('--validate', 'Validate before applying') - .action((configFile, options) => { - storageCommands.lifecycle.set(configFile, options) - }) - ) - .addCommand( - new Command('get') - .description('Get current lifecycle policy') - .option('-f, --format ', 'Output format (json|yaml)', 'json') - .action((options) => { - storageCommands.lifecycle.get(options) - }) - ) - .addCommand( - new Command('remove') - .description('Remove lifecycle policy') - .action((options) => { - storageCommands.lifecycle.remove(options) - }) - ) - ) - .addCommand( - new Command('compression') - .description('Compression management (FileSystem)') - .addCommand( - new Command('enable') - .description('Enable gzip compression') - .action((options) => { - storageCommands.compression.enable(options) - }) - ) - .addCommand( - new Command('disable') - .description('Disable compression') - .action((options) => { - storageCommands.compression.disable(options) - }) - ) - .addCommand( - new Command('status') - .description('Show compression status') - .action((options) => { - storageCommands.compression.status(options) - }) - ) - ) .addCommand( new Command('batch-delete') .argument('', 'File containing entity IDs (one per line)') - .description('Batch delete with retry logic') - .option('--max-retries ', 'Maximum retry attempts', '3') - .option('--continue-on-error', 'Continue if some deletes fail') + .description('Delete a list of entities through the public delete path (with retry)') + .option('--max-retries ', 'Maximum retry attempts per ID', '3') + .option('--continue-on-error', 'Continue past IDs that still fail after retries') .action((file, options) => { storageCommands.batchDelete(file, options) }) ) - .addCommand( - new Command('cost-estimate') - .description('Estimate cloud storage costs') - .option('--provider ', 'Cloud provider (aws|gcs|azure|r2)') - .option('--size ', 'Data size in GB') - .option('--operations ', 'Monthly operations') - .action((options) => { - storageCommands.costEstimate(options) - }) - ) // ===== Data Management Commands ===== @@ -742,9 +681,8 @@ program program .command('clean') - .description('Clean and optimize database') - .option('--remove-orphans', 'Remove orphaned items') - .option('--rebuild-index', 'Rebuild search index') + .description('Clear the database — ALL entities, relationships, and indexes (asks for confirmation)') + .option('-f, --force', 'Skip confirmation prompt') .action(utilityCommands.clean) program @@ -785,17 +723,6 @@ program .description('Show the store\'s current generation watermark') .action(snapshotCommands.generation) -// ===== Interactive Mode ===== - -program - .command('interactive') - .alias('i') - .description('Start interactive REPL mode') - .action(async () => { - const { startInteractiveMode } = await import('./interactive.js') - await startInteractiveMode() - }) - // ===== Error Handling ===== program.exitOverride() diff --git a/src/cli/interactive.ts b/src/cli/interactive.ts deleted file mode 100644 index 1896d443..00000000 --- a/src/cli/interactive.ts +++ /dev/null @@ -1,653 +0,0 @@ -/** - * Professional Interactive CLI System - * - * Provides consistent, delightful interactive prompts for all commands - * with smart defaults, validation, and helpful examples - */ - -import chalk from 'chalk' -import inquirer from 'inquirer' -import ora from 'ora' -import { Brainy } from '../brainy.js' -import { getBrainyVersion } from '../utils/version.js' - -// Professional color scheme -export const colors = { - primary: chalk.hex('#3A5F4A'), // Teal (from logo) - success: chalk.hex('#2D4A3A'), // Deep teal - info: chalk.hex('#4A6B5A'), // Medium teal - warning: chalk.hex('#D67441'), // Orange (from logo) - error: chalk.hex('#B85C35'), // Deep orange - brain: chalk.hex('#D67441'), // Brain orange - cream: chalk.hex('#F5E6A3'), // Cream background - dim: chalk.dim, - bold: chalk.bold, - cyan: chalk.cyan, - green: chalk.green, - yellow: chalk.yellow, - red: chalk.red -} - -// Icons for consistent visual language -export const icons = { - brain: '🧠', - search: '🔍', - add: '➕', - delete: '🗑️', - update: '🔄', - import: '📥', - export: '📤', - connect: '🔗', - question: '❓', - success: '✅', - error: '❌', - warning: '⚠️', - info: 'ℹ️', - sparkle: '✨', - rocket: '🚀', - thinking: '🤔', - chat: '💬' -} - -// Store recent inputs for smart suggestions -const recentInputs = { - searches: [] as string[], - ids: [] as string[], - types: [] as string[], - formats: [] as string[] -} - -/** - * Professional prompt wrapper with consistent styling - */ -export async function prompt(config: any): Promise { - // Add consistent styling - if (config.message) { - config.message = colors.cyan(config.message) - } - - // Add prefix with appropriate icon - if (!config.prefix) { - config.prefix = colors.dim(' › ') - } - - return inquirer.prompt([config]) -} - -/** - * Interactive prompt for search query with smart features - */ -export async function promptSearchQuery(previousSearches?: string[]): Promise { - console.log(colors.primary(`\n${icons.search} Smart Search\n`)) - console.log(colors.dim('Search your neural database with natural language')) - console.log(colors.dim('Examples: "meetings last week", "John from Google", "important documents"')) - - const { query } = await prompt({ - type: 'input', - name: 'query', - message: 'What would you like to search for?', - validate: (input: string) => { - if (!input.trim()) { - return 'Please enter a search query' - } - return true - }, - transformer: (input: string) => { - // Show live character count - const count = input.length - if (count > 100) { - return colors.warning(input) - } - return colors.green(input) - } - }) - - // Store for future suggestions - if (!recentInputs.searches.includes(query)) { - recentInputs.searches.unshift(query) - recentInputs.searches = recentInputs.searches.slice(0, 10) - } - - return query -} - -/** - * Interactive prompt for item ID with fuzzy search - */ -export async function promptItemId( - action: string, - brain?: Brainy, - allowMultiple: boolean = false -): Promise { - console.log(colors.primary(`\n${icons.thinking} Select item to ${action}\n`)) - - // If we have brain instance, show recent items - let choices: any[] = [] - if (brain) { - try { - const recent = await brain.find({ - query: '*', - limit: 10 - }) - - choices = recent.map(item => ({ - name: `${item.id} - ${(item as any).content?.substring(0, 50) || 'No content'}...`, - value: item.id, - short: item.id - })) - } catch { - // Fallback to manual input - } - } - - if (choices.length > 0) { - choices.push(new inquirer.Separator()) - choices.push({ name: 'Enter ID manually', value: '__manual__' }) - - const { selected } = await prompt({ - type: allowMultiple ? 'checkbox' : 'list', - name: 'selected', - message: `Select item(s) to ${action}:`, - choices, - pageSize: 10 - }) - - if (selected === '__manual__' || (Array.isArray(selected) && selected.includes('__manual__'))) { - return promptManualId(action, allowMultiple) - } - - return selected - } else { - return promptManualId(action, allowMultiple) - } -} - -/** - * Manual ID input with validation - */ -async function promptManualId(action: string, allowMultiple: boolean): Promise { - const { id } = await prompt({ - type: 'input', - name: 'id', - message: allowMultiple - ? `Enter ID(s) to ${action} (comma-separated):` - : `Enter ID to ${action}:`, - validate: (input: string) => { - if (!input.trim()) { - return `Please enter at least one ID` - } - return true - } - }) - - if (allowMultiple) { - return id.split(',').map((i: string) => i.trim()).filter(Boolean) - } - return id.trim() -} - -/** - * Confirm destructive action with preview - */ -export async function confirmDestructiveAction( - action: string, - items: any[], - showPreview: boolean = true -): Promise { - console.log(colors.warning(`\n${icons.warning} Confirmation Required\n`)) - - if (showPreview && items.length > 0) { - console.log(colors.dim(`You are about to ${action}:`)) - items.slice(0, 5).forEach(item => { - console.log(colors.dim(` • ${item.id || item}`)) - }) - if (items.length > 5) { - console.log(colors.dim(` ... and ${items.length - 5} more`)) - } - console.log() - } - - const { confirm } = await prompt({ - type: 'confirm', - name: 'confirm', - message: colors.warning(`Are you sure you want to ${action}?`), - default: false - }) - - return confirm -} - -/** - * Interactive data input with multiline support - */ -export async function promptDataInput( - action: string = 'add', - currentValue?: string -): Promise { - console.log(colors.primary(`\n${icons.add} ${action === 'add' ? 'Add Data' : 'Update Data'}\n`)) - - if (currentValue) { - console.log(colors.dim('Current value:')) - console.log(colors.info(` ${currentValue.substring(0, 100)}${currentValue.length > 100 ? '...' : ''}`)) - console.log() - } - - const { data } = await prompt({ - type: 'editor', - name: 'data', - message: 'Enter your data:', - default: currentValue || '', - postfix: '.md', - validate: (input: string) => { - if (!input.trim() && action === 'add') { - return 'Please enter some data' - } - return true - } - }) - - return data -} - -/** - * Interactive metadata input with JSON validation - */ -export async function promptMetadata( - currentMetadata?: any, - suggestions?: string[] -): Promise { - console.log(colors.dim('\nOptional: Add metadata (JSON format)')) - - const { addMetadata } = await prompt({ - type: 'confirm', - name: 'addMetadata', - message: 'Would you like to add metadata?', - default: false - }) - - if (!addMetadata) { - return {} - } - - // Show field suggestions if available - if (suggestions && suggestions.length > 0) { - console.log(colors.dim('\nAvailable fields:')) - suggestions.forEach(field => { - console.log(colors.dim(` • ${field}`)) - }) - } - - const { metadata } = await prompt({ - type: 'editor', - name: 'metadata', - message: 'Enter metadata (JSON):', - default: currentMetadata ? JSON.stringify(currentMetadata, null, 2) : '{\n \n}', - postfix: '.json', - validate: (input: string) => { - try { - JSON.parse(input) - return true - } catch (e) { - return `Invalid JSON: ${e.message}` - } - } - }) - - return JSON.parse(metadata) -} - -/** - * Interactive format selector - */ -export async function promptFormat( - availableFormats: string[], - defaultFormat: string -): Promise { - console.log(colors.primary(`\n${icons.export} Select Format\n`)) - - const { format } = await prompt({ - type: 'list', - name: 'format', - message: 'Choose export format:', - choices: availableFormats.map(f => ({ - name: getFormatDescription(f), - value: f, - short: f - })), - default: defaultFormat - }) - - return format -} - -/** - * Get friendly format descriptions - */ -function getFormatDescription(format: string): string { - const descriptions: Record = { - json: 'JSON - Universal data interchange', - jsonl: 'JSON Lines - Streaming format', - csv: 'CSV - Spreadsheet compatible', - graphml: 'GraphML - Graph visualization', - dot: 'DOT - Graphviz format', - d3: 'D3.js - Web visualization', - markdown: 'Markdown - Human readable', - yaml: 'YAML - Configuration format' - } - - return `${format.toUpperCase()} - ${descriptions[format] || 'Custom format'}` -} - -/** - * Interactive file/URL input with validation - */ -export async function promptFileOrUrl( - action: string = 'import' -): Promise { - console.log(colors.primary(`\n${icons.import} ${action === 'import' ? 'Import Source' : 'Export Destination'}\n`)) - - const { sourceType } = await prompt({ - type: 'list', - name: 'sourceType', - message: 'What type of source?', - choices: [ - { name: 'Local file', value: 'file' }, - { name: 'URL', value: 'url' }, - { name: 'Clipboard', value: 'clipboard' }, - { name: 'Direct input', value: 'input' } - ] - }) - - switch (sourceType) { - case 'file': - return promptFilePath(action) - case 'url': - return promptUrl() - case 'clipboard': - // Would need clipboard integration - console.log(colors.warning('Clipboard support coming soon!')) - return promptFilePath(action) - case 'input': - const data = await promptDataInput('import') - // Save to temp file and return path - const tmpFile = `/tmp/brainy-import-${Date.now()}.json` - const { writeFileSync } = await import('node:fs') - writeFileSync(tmpFile, data) - return tmpFile - default: - return '' - } -} - -/** - * File path input with autocomplete - */ -async function promptFilePath(action: string): Promise { - const { path } = await prompt({ - type: 'input', - name: 'path', - message: `Enter file path to ${action}:`, - validate: async (input: string) => { - if (!input.trim()) { - return 'Please enter a file path' - } - - const { existsSync } = await import('node:fs') - if (action === 'import' && !existsSync(input)) { - return `File not found: ${input}` - } - - return true - }, - // Add file path autocomplete - transformer: (input: string) => { - if (input.startsWith('~/')) { - const home = process.env.HOME || '~' - return colors.green(input.replace('~', home)) - } - return colors.green(input) - } - }) - - return path -} - -/** - * URL input with validation - */ -async function promptUrl(): Promise { - const { url } = await prompt({ - type: 'input', - name: 'url', - message: 'Enter URL:', - validate: (input: string) => { - try { - new URL(input) - return true - } catch { - return 'Please enter a valid URL' - } - } - }) - - return url -} - -/** - * Interactive relationship builder - */ -export async function promptRelationship(brain?: Brainy): Promise<{ - source: string - verb: string - target: string - metadata?: any -}> { - console.log(colors.primary(`\n${icons.connect} Create Relationship\n`)) - console.log(colors.dim('Connect two items with a semantic relationship')) - - // Get source - const source = await promptItemId('connect from', brain, false) as string - - // Get verb/relationship type - const { verb } = await prompt({ - type: 'list', - name: 'verb', - message: 'Relationship type:', - choices: [ - { name: 'Works For', value: 'WorksFor' }, - { name: 'Knows', value: 'Knows' }, - { name: 'Created By', value: 'CreatedBy' }, - { name: 'Belongs To', value: 'BelongsTo' }, - { name: 'Uses', value: 'Uses' }, - { name: 'Manages', value: 'Manages' }, - { name: 'Located In', value: 'LocatedIn' }, - { name: 'Related To', value: 'RelatedTo' }, - new inquirer.Separator(), - { name: 'Custom relationship...', value: '__custom__' } - ] - }) - - let finalVerb = verb - if (verb === '__custom__') { - const { customVerb } = await prompt({ - type: 'input', - name: 'customVerb', - message: 'Enter custom relationship:', - validate: (input: string) => input.trim() ? true : 'Please enter a relationship' - }) - finalVerb = customVerb - } - - // Get target - const target = await promptItemId('connect to', brain, false) as string - - // Optional metadata - const metadata = await promptMetadata() - - return { - source, - verb: finalVerb, - target, - metadata: Object.keys(metadata).length > 0 ? metadata : undefined - } -} - -/** - * Smart command suggestions when user types wrong command - */ -export function suggestCommand(input: string, availableCommands: string[]): string[] { - // Simple fuzzy matching without external dependency - // Filter commands that start with or contain the input - const matches = availableCommands - .filter(cmd => cmd.toLowerCase().includes(input.toLowerCase())) - .sort((a, b) => { - // Prefer commands that start with the input - const aStarts = a.toLowerCase().startsWith(input.toLowerCase()) - const bStarts = b.toLowerCase().startsWith(input.toLowerCase()) - if (aStarts && !bStarts) return -1 - if (!aStarts && bStarts) return 1 - return 0 - }) - return matches.slice(0, 3) -} - -/** - * Beautiful error display with helpful context - */ -export function showError(error: Error, context?: string): void { - console.log() - console.log(colors.error(`${icons.error} Error`)) - - if (context) { - console.log(colors.dim(context)) - } - - console.log(colors.red(error.message)) - - // Provide helpful suggestions based on error - if (error.message.includes('not found')) { - console.log(colors.dim('\nTip: Use "brainy search" to find items')) - } else if (error.message.includes('network') || error.message.includes('fetch')) { - console.log(colors.dim('\nTip: Check your internet connection')) - } else if (error.message.includes('permission')) { - console.log(colors.dim('\nTip: Check file permissions or run with appropriate access')) - } -} - -/** - * Progress indicator for long operations - */ -export class ProgressTracker { - private spinner: any - private startTime: number - - constructor(message: string) { - this.spinner = ora({ - text: message, - color: 'cyan', - spinner: 'dots' - }).start() - this.startTime = Date.now() - } - - update(message: string, count?: number, total?: number): void { - if (count && total) { - const percent = Math.round((count / total) * 100) - const elapsed = ((Date.now() - this.startTime) / 1000).toFixed(1) - this.spinner.text = `${message} (${percent}% - ${elapsed}s)` - } else { - this.spinner.text = message - } - } - - succeed(message?: string): void { - const elapsed = ((Date.now() - this.startTime) / 1000).toFixed(1) - this.spinner.succeed(message ? `${message} (${elapsed}s)` : `Done (${elapsed}s)`) - } - - fail(message?: string): void { - this.spinner.fail(message || 'Failed') - } - - stop(): void { - this.spinner.stop() - } -} - -/** - * Welcome message for interactive mode - */ -export function showWelcome(): void { - console.clear() - console.log(colors.primary(` -╔══════════════════════════════════════════════╗ -║ ║ -║ ${icons.brain} BRAINY - Neural Intelligence ║ -║ Your AI-Powered Second Brain ║ -║ ║ -╚══════════════════════════════════════════════╝ -`)) - console.log(colors.dim(`Version ${getBrainyVersion()} • Type "help" for commands`)) - console.log() -} - -/** - * Interactive command selector for beginners - */ -export async function promptCommand(): Promise { - const { command } = await prompt({ - type: 'list', - name: 'command', - message: 'What would you like to do?', - choices: [ - { name: `${icons.add} Add data to your brain`, value: 'add' }, - { name: `${icons.search} Search your knowledge`, value: 'search' }, - { name: `${icons.chat} Chat with your data`, value: 'chat' }, - { name: `${icons.update} Update existing data`, value: 'update' }, - { name: `${icons.delete} Delete data`, value: 'delete' }, - { name: `${icons.connect} Create relationships`, value: 'relate' }, - { name: `${icons.import} Import from file`, value: 'import' }, - { name: `${icons.export} Export your brain`, value: 'export' }, - new inquirer.Separator(), - { name: `${icons.brain} Neural operations`, value: 'neural' }, - { name: `${icons.info} View statistics`, value: 'status' }, - { name: 'Exit', value: 'exit' } - ], - pageSize: 15 - }) - - return command -} - -/** - * Start interactive REPL mode - */ -export async function startInteractiveMode() { - console.log(chalk.cyan('\n🧠 Brainy Interactive Mode\n')) - console.log(chalk.yellow('Interactive REPL mode coming soon\n')) - console.log(chalk.dim('Use specific commands for now: brainy add, brainy search, etc.')) - process.exit(0) -} - -/** - * Export all interactive components - */ -export default { - colors, - icons, - prompt, - promptSearchQuery, - promptItemId, - confirmDestructiveAction, - promptDataInput, - promptMetadata, - promptFormat, - promptFileOrUrl, - promptRelationship, - suggestCommand, - showError, - ProgressTracker, - showWelcome, - promptCommand, - startInteractiveMode -} \ No newline at end of file diff --git a/src/config/storageAutoConfig.ts b/src/config/storageAutoConfig.ts index d9154b05..df36747e 100644 --- a/src/config/storageAutoConfig.ts +++ b/src/config/storageAutoConfig.ts @@ -2,7 +2,7 @@ * @module config/storageAutoConfig * @description Storage configuration auto-detection for Brainy 8.0. * - * Brainy 8.0 ships **two storage adapters** per `BR-BRAINY-80-STORAGE-SIMPLIFY`: + * Brainy 8.0 ships **two storage adapters**: * - `FILESYSTEM` — persistent on-disk storage (Node, Bun, Deno). * - `MEMORY` — in-memory, ephemeral. * diff --git a/src/coreTypes.ts b/src/coreTypes.ts index 84c3be4a..58753345 100644 --- a/src/coreTypes.ts +++ b/src/coreTypes.ts @@ -386,7 +386,7 @@ export interface HNSWVerbWithMetadata { targetId: string // SUBTYPE — optional per-product sub-classification within a VerbType (e.g. a - // `Manages` relationship might have subtype 'direct' vs 'dotted-line'; a `RelatedTo` + // `ReportsTo` relationship might have subtype 'direct' vs 'dotted-line'; a `RelatedTo` // edge might carry 'spouse' / 'sibling' / 'colleague'). Flat string (no hierarchy) — // consumers decide the vocabulary. Indexed and rolled up into per-VerbType statistics // so it's queryable (`getRelations({ verb, subtype })`) and aggregable diff --git a/src/graph/graphAdjacencyIndex.ts b/src/graph/graphAdjacencyIndex.ts index 0b9b79e1..d0e21b6c 100644 --- a/src/graph/graphAdjacencyIndex.ts +++ b/src/graph/graphAdjacencyIndex.ts @@ -766,8 +766,8 @@ export class GraphAdjacencyIndex implements GraphIndexProvider { // Note: LSM-trees will be recreated from storage via their own initialization // Verb data will be loaded on-demand via UnifiedCache - // Brainy 8.0: storage is always local (filesystem or memory) per - // BR-BRAINY-80-STORAGE-SIMPLIFY. Load all verbs at once. + // Brainy 8.0: storage is always local (filesystem or memory — the + // cloud adapters were removed). Load all verbs at once. const storageType = this.storage?.constructor.name || '' let totalVerbs = 0 diff --git a/src/hnsw/hnswIndex.ts b/src/hnsw/hnswIndex.ts index bf09ee8c..742b088d 100644 --- a/src/hnsw/hnswIndex.ts +++ b/src/hnsw/hnswIndex.ts @@ -1261,8 +1261,8 @@ export class JsHnswVectorIndex implements VectorIndexProvider { } // Step 4: Load all HNSW nodes at once. Brainy 8.0 ships filesystem + - // memory storage only (per BR-BRAINY-80-STORAGE-SIMPLIFY); the - // cloud-pagination rebuild path was deleted alongside the cloud adapters. + // memory storage only; the cloud-pagination rebuild path was deleted + // alongside the cloud adapters. const storageType = this.storage?.constructor.name || '' let loadedCount = 0 let totalCount: number | undefined = undefined diff --git a/src/index.ts b/src/index.ts index 5361d7d1..58799f4a 100644 --- a/src/index.ts +++ b/src/index.ts @@ -28,6 +28,9 @@ export type { RelateParams, FindParams, SubtypeRegistry, + FillSubtypeRule, + FillSubtypeRules, + FillSubtypesResult, AggregateDefinition, AggregateMetricDef, AggregateSource, diff --git a/src/neural/signals/ExactMatchSignal.ts b/src/neural/signals/ExactMatchSignal.ts index 8a2e765c..145750b3 100644 --- a/src/neural/signals/ExactMatchSignal.ts +++ b/src/neural/signals/ExactMatchSignal.ts @@ -8,9 +8,8 @@ * 2. O(1) metadata hints (column names, file structure) * 3. Format-specific intelligence (Excel, CSV, PDF, YAML, DOCX) * - * This is the WORKSHOP BUG FIX - finds explicit relationships via exact matching - * - * PRODUCTION-READY: No TODOs, no mocks, real implementation + * Finds explicit relationships via exact matching — added after a consumer + * report of extraction missing explicitly-named relationships. */ import type { Brainy } from '../../brainy.js' diff --git a/src/storage/baseStorage.ts b/src/storage/baseStorage.ts index e32c8965..4914735b 100644 --- a/src/storage/baseStorage.ts +++ b/src/storage/baseStorage.ts @@ -1528,10 +1528,16 @@ export abstract class BaseStorage extends BaseStorageAdapter { const { limit, offset = 0, filter } = options const collectedNouns: HNSWNounWithMetadata[] = [] + // Collect ONE item past the requested window so `hasMore` is decidable: + // stopping exactly at offset+limit cannot distinguish "page full, nothing + // after it" from "page full, more behind it" — which made hasMore + // permanently false and silently truncated every multi-page walk + // (audit/migrateField/fillSubtypes, graph verb-id recovery, count rebuilds). const targetCount = offset + limit + const peekCount = targetCount + 1 // Iterate by shards (0x00-0xFF) instead of types - for (let shard = 0; shard < 256 && collectedNouns.length < targetCount; shard++) { + for (let shard = 0; shard < 256 && collectedNouns.length < peekCount; shard++) { const shardHex = shard.toString(16).padStart(2, '0') const shardDir = `entities/nouns/${shardHex}` @@ -1539,7 +1545,7 @@ export abstract class BaseStorage extends BaseStorageAdapter { const nounFiles = await this.listCanonicalObjects(shardDir) for (const nounPath of nounFiles) { - if (collectedNouns.length >= targetCount) break + if (collectedNouns.length >= peekCount) break if (!nounPath.includes('/vectors.json')) continue try { @@ -1597,7 +1603,8 @@ export abstract class BaseStorage extends BaseStorageAdapter { } } - // Apply pagination + // Apply pagination. The peeked extra item (if any) is dropped by the slice; + // its existence is exactly what makes hasMore true. const paginatedNouns = collectedNouns.slice(offset, offset + limit) const hasMore = collectedNouns.length > targetCount @@ -1647,7 +1654,11 @@ export abstract class BaseStorage extends BaseStorageAdapter { const { limit, offset = 0, filter } = options // cursor intentionally not extracted (not yet implemented) const collectedVerbs: HNSWVerbWithMetadata[] = [] - const targetCount = offset + limit // Early termination target + // Same peek-one-past-the-window strategy as getNounsWithPagination — see + // the comment there. Without the extra item, hasMore is undecidable and + // was permanently false (silent truncation of every multi-page walk). + const targetCount = offset + limit // Requested window end + const peekCount = targetCount + 1 // Early termination target (window + 1 peek) // Prepare filter sets for efficient lookup const filterVerbTypes = filter?.verbType @@ -1668,7 +1679,7 @@ export abstract class BaseStorage extends BaseStorageAdapter { : null // Iterate by shards (0x00-0xFF) instead of types - single pass! - for (let shard = 0; shard < 256 && collectedVerbs.length < targetCount; shard++) { + for (let shard = 0; shard < 256 && collectedVerbs.length < peekCount; shard++) { const shardHex = shard.toString(16).padStart(2, '0') const shardDir = `entities/verbs/${shardHex}` @@ -1676,7 +1687,7 @@ export abstract class BaseStorage extends BaseStorageAdapter { const verbFiles = await this.listCanonicalObjects(shardDir) for (const verbPath of verbFiles) { - if (collectedVerbs.length >= targetCount) break + if (collectedVerbs.length >= peekCount) break if (!verbPath.includes('/vectors.json')) continue try { @@ -1737,9 +1748,11 @@ export abstract class BaseStorage extends BaseStorageAdapter { } } - // Apply pagination (Efficient slicing after early termination) + // Apply pagination. The peeked extra item (if any) is dropped by the slice; + // its existence is exactly what makes hasMore true. `>` (not `>=`) keeps the + // exact-boundary case (total == offset+limit) from looping forever. const paginatedVerbs = collectedVerbs.slice(offset, offset + limit) - const hasMore = collectedVerbs.length > targetCount // Fixed >= to > (was causing infinite loop) + const hasMore = collectedVerbs.length > targetCount return { items: paginatedVerbs, @@ -3235,8 +3248,8 @@ export abstract class BaseStorage extends BaseStorageAdapter { * Before this override the Uint32Array counters were only persisted * on a heuristic schedule inside `saveNoun_internal` (first-of-type or * every-100th), which left readers seeing stale counts after a clean - * writer flush — the same failure mode that BR-FIND-WHERE-ZERO surfaced - * via `brain.stats()`. + * writer flush — the same silent-stale failure mode that once produced + * zero counts from `brain.stats()`. */ public async flushCounts(): Promise { await super.flushCounts() diff --git a/src/storage/storageFactory.ts b/src/storage/storageFactory.ts index 1fbdf255..88abf590 100644 --- a/src/storage/storageFactory.ts +++ b/src/storage/storageFactory.ts @@ -8,7 +8,7 @@ * workloads. * * Cloud-storage adapters (GCS, S3, R2, Azure) and the browser-only OPFS - * adapter were removed in 8.0 per `BR-BRAINY-80-STORAGE-SIMPLIFY`. The path + * adapter were removed in 8.0. The path * forward for cloud backup is operator tooling: persist locally with * `db.persist()`, sync the resulting on-disk artefact with `gsutil` / * `aws s3 cp` / `rclone` / `azcopy`. This is the standard pattern every diff --git a/src/types/brainy.types.ts b/src/types/brainy.types.ts index d3458ae9..0e958d01 100644 --- a/src/types/brainy.types.ts +++ b/src/types/brainy.types.ts @@ -76,7 +76,7 @@ export interface Relation { /** Relationship type classification (VerbType enum) */ type: VerbType /** - * Per-product sub-classification within the VerbType (e.g. a `Manages` + * Per-product sub-classification within the VerbType (e.g. a `ReportsTo` * relationship might have `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` * edge might carry `'spouse'` / `'sibling'` / `'colleague'`). Flat string, no * hierarchy. Top-level standard field — indexed on the fast path and rolled into @@ -208,6 +208,68 @@ export interface SubtypeRegistry { // Intentionally empty. Consumers extend via declaration merging. } +/** + * A single back-fill rule for `brain.fillSubtypes()`. + * + * - A **literal string** assigns that subtype to every matching entry that + * lacks one (`'general'` — a blanket default). + * - A **function** receives the full entry and returns the subtype to assign, + * or `undefined` to leave the entry untouched (it is counted as `skipped` + * so a later run with a stricter rule can pick it up). Functions can derive + * the subtype from existing fields, e.g. `(e) => e.metadata?.kind`. + * + * @typeParam E - The entry shape the rule sees: `Entity` for NounType keys, + * `Relation` for VerbType keys. + */ +export type FillSubtypeRule = string | ((entry: E) => string | undefined) + +/** + * Rule map for `brain.fillSubtypes()` — the 8.0 subtype migration helper. + * + * Keys are `NounType` values (entity rules) and/or `VerbType` values + * (relationship rules); the two vocabularies don't overlap, so a single map + * covers both sides. Each value is a {@link FillSubtypeRule}: a literal + * subtype string or a function deriving one from the entry. + * + * @example + * ```ts + * await brain.fillSubtypes({ + * [NounType.Person]: (e) => e.metadata?.kind ?? 'unspecified', + * [NounType.Thing]: 'general', // literal default + * [VerbType.RelatedTo]: 'unspecified' // relationship rule + * }) + * ``` + */ +export type FillSubtypeRules = { + [K in NounType]?: FillSubtypeRule> +} & { + [K in VerbType]?: FillSubtypeRule> +} + +/** + * Summary returned by `brain.fillSubtypes()`. + * + * After a run, `skipped` is exactly the remaining migration debt — re-running + * `brain.audit()` reports the same entries. Entries that already carry a + * subtype count toward `scanned` only (they are not debt, so they are neither + * `filled` nor `skipped`). + */ +export interface FillSubtypesResult { + /** Total entries examined (entities + relationships). */ + scanned: number + /** Entries that received a subtype during this run. */ + filled: number + /** + * Entries still missing a subtype after the run — either their type has no + * rule in the map, or their rule function returned `undefined`/empty. + */ + skipped: number + /** Per-entry write failures (`fillSubtypes` continues past individual errors). */ + errors: Array<{ id: string; error: string }> + /** Fill counts grouped by NounType/VerbType key (only types with fills appear). */ + byType: Record +} + export interface AddParams { /** Content to embed and store. Strings are auto-embedded; objects are JSON-stringified for embedding. */ data: any | Vector @@ -282,7 +344,7 @@ export interface RelateParams { /** Relationship type classification (required) */ type: VerbType /** - * Per-product sub-classification within the VerbType (e.g. a `Manages` + * Per-product sub-classification within the VerbType (e.g. a `ReportsTo` * relationship might have `subtype: 'direct'` or `'dotted-line'`; a `RelatedTo` * edge might carry `'spouse'` or `'colleague'`). Flat string, no hierarchy. * Indexed and rolled up into per-VerbType statistics for fast filtering @@ -1239,16 +1301,21 @@ export interface BrainyConfig { /** * Brain-wide subtype enforcement mode. * - * Opt-in in 7.30.0 (default: `false`); becomes the default in 8.0.0. + * **Default-on since 8.0.0** (`undefined` resolves to `true`; in 7.30.x this + * was an opt-in flag defaulting to `false`). * - * - `false` / `undefined` (default): no brain-wide check. Per-type rules - * registered via `brain.requireSubtype(type, options)` still apply. - * - `true`: every `add()` / `addMany()` / `update()` / `relate()` / - * `relateMany()` / `updateRelation()` rejects writes where the entity's - * NounType (or relationship's VerbType) has no non-empty `subtype` value. - * - `{ except: [NounType.Thing, NounType.Custom] }`: same as `true`, but - * the listed types are allowed through without a subtype. Use for genuine + * - `true` / `undefined` (default): every `add()` / `addMany()` / `update()` / + * `relate()` / `relateMany()` / `updateRelation()` rejects writes where the + * entity's NounType (or relationship's VerbType) has no non-empty `subtype` + * value. Brainy's own VFS infrastructure writes (`metadata.isVFSEntity` / + * `metadata.isVFS`) bypass the check. + * - `{ except: [NounType.Thing, NounType.Custom] }`: same as `true`, but the + * listed types are allowed through without a subtype. Use for genuine * catch-all types where no subtype makes sense. + * - `false`: disable the brain-wide check entirely. Last-resort escape hatch + * for opening pre-8.0 data — run `brain.audit()` to find the gaps, back-fill + * with `brain.fillSubtypes(rules)`, then remove the opt-out so the default + * enforcement protects new writes. * * Per-type registrations always compose with the brain-wide flag — a type * registered with `requireSubtype(type, { required: true })` is always diff --git a/src/utils/metadataIndex.ts b/src/utils/metadataIndex.ts index b910e7bd..2b7e8174 100644 --- a/src/utils/metadataIndex.ts +++ b/src/utils/metadataIndex.ts @@ -748,9 +748,8 @@ export class MetadataIndexManager implements MetadataIndexProvider { * * If neither the column store nor a sparse index covers the field, the * function throws `BrainyError(FIELD_NOT_INDEXED)`. Returning `[]` for a - * genuinely unindexed field was the bug class `BR-FIND-WHERE-ZERO` - * tracked — a silent empty result indistinguishable from "the data - * really isn't there." + * genuinely unindexed field was a long-standing silent-empty bug class — + * an empty result indistinguishable from "the data really isn't there." */ private async getIdsFromChunks(field: string, value: any): Promise { // Load sparse index via UnifiedCache (lazy loading) @@ -1200,7 +1199,8 @@ export class MetadataIndexManager implements MetadataIndexProvider { // - Chunked Sparse Index: ~50 values per chunk, lazy-loaded // - UnifiedCache LRU: Only hot chunks in memory // - // Future: Bloom filter hybrid for unlimited words (see .strategy/BILLION-SCALE-PLAN.md) + // A Bloom-filter hybrid could lift the per-entity word cap entirely if + // full-document indexing at billion-entity scale ever becomes a need. const textContent = this.extractTextContent(data) if (textContent) { const MAX_WORDS_PER_ENTITY = 5000 // Handles articles/chapters, memory-safe at scale @@ -1600,8 +1600,8 @@ export class MetadataIndexManager implements MetadataIndexProvider { * Throws `BrainyError(FIELD_NOT_INDEXED)` if the field has no entries in * either store. Callers in find()-evaluation catch this and translate to * an empty result with a logged warning. The throw aligns the production - * `find()` path with the `brain.explain()` diagnostic, so the silent- - * empty bug class (BR-FIND-WHERE-ZERO) is no longer possible. + * `find()` path with the `brain.explain()` diagnostic, so a silently + * empty result for an unindexed field is no longer possible. */ async getIds(field: string, value: any): Promise { // Track exact query for field statistics @@ -2866,8 +2866,8 @@ export class MetadataIndexManager implements MetadataIndexProvider { * * Prior implementation read from `this.fieldIndexes` + lazy-loaded sparse * indices, which silently returned `0` entries for any workspace written - * after sparse-index writes were deleted in commit `11be039`. That defect - * is what `BR-FIND-WHERE-ZERO` tracked. + * after sparse-index writes were deleted in commit `11be039`. That + * silent-zero defect is why this reads the column store first. */ async getStats(): Promise { const entityCount = this.idMapper.size diff --git a/src/utils/paramValidation.ts b/src/utils/paramValidation.ts index c3246d96..4e82b994 100644 --- a/src/utils/paramValidation.ts +++ b/src/utils/paramValidation.ts @@ -124,7 +124,8 @@ const getContainerMemoryLimit = (): number | null => { * 1.5 KB + standard fields + metadata). On a 900 MB free-memory box this * capped `limit` at 9_000, breaking common safety-cap call patterns like * `find({ type, where, limit: 10_000 })` that typically return 10-500 - * entities. Surfaced as `BR-MAXLIMIT-9000` in PLATFORM-HANDOFF.md. + * entities. Reported by a production consumer whose 10K safety-cap + * queries started failing after the cap landed. * - 7.30.2+: `25` (assumes 25 KB per result). Generous over typical (7-10 KB), * comfortably under the worst case (~20 KB with large metadata blobs). * Same 900 MB box now gives ~36_000 — typical 10_000 limits pass silently, diff --git a/tests/integration/strict-mode-self-test.test.ts b/tests/integration/strict-mode-self-test.test.ts index b9c1ba0f..b1009bde 100644 --- a/tests/integration/strict-mode-self-test.test.ts +++ b/tests/integration/strict-mode-self-test.test.ts @@ -226,7 +226,7 @@ describe('Brainy strict-mode self-test (7.30.1)', () => { const report = await brain.audit() expect(report.total).toBeGreaterThanOrEqual(1) expect(report.entitiesWithoutSubtype['person']).toBe(1) - expect(report.recommendation).toMatch(/Migrate via/) + expect(report.recommendation).toMatch(/fillSubtypes/) }) it('excludes VFS entities by default (they bypass enforcement anyway)', async () => { diff --git a/tests/unit/brainy/fill-subtypes.test.ts b/tests/unit/brainy/fill-subtypes.test.ts new file mode 100644 index 00000000..cae63ccd --- /dev/null +++ b/tests/unit/brainy/fill-subtypes.test.ts @@ -0,0 +1,338 @@ +/** + * @module tests/unit/brainy/fill-subtypes + * @description Unit tests for `brain.fillSubtypes(rules)` — the 8.0 subtype + * migration helper. Proves the full contract: literal and function rules, + * entity + relationship fills through one rule map, never overwriting an + * existing subtype, function rules declining entries (counted as skipped), + * VFS-marker exclusion, the `{ scanned, filled, skipped, errors, byType }` + * report, `_rev` bumping through the real update path, idempotent re-runs, + * per-entry error collection, and fail-fast rule-map validation. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { Brainy } from '../../../src/brainy' +import { NounType, VerbType } from '../../../src/types/graphTypes' +import { createTestConfig } from '../../helpers/test-factory' + +describe('Brainy.fillSubtypes()', () => { + let brain: Brainy + + beforeEach(async () => { + // requireSubtype: false so the tests can create the pre-8.0 migration + // debt (entities/relationships without subtype) that fillSubtypes exists + // to clear. + brain = new Brainy(createTestConfig()) + await brain.init() + }) + + afterEach(async () => { + await brain.close() + }) + + describe('literal rules', () => { + it('fills missing subtypes with a literal default', async () => { + const a = await brain.add({ type: NounType.Person, data: 'Alice' }) + const b = await brain.add({ type: NounType.Person, data: 'Bob' }) + + const report = await brain.fillSubtypes({ + [NounType.Person]: 'unspecified' + }) + + expect(report.filled).toBe(2) + expect(report.byType).toEqual({ person: 2 }) + expect((await brain.get(a))!.subtype).toBe('unspecified') + expect((await brain.get(b))!.subtype).toBe('unspecified') + }) + + it('never overwrites an existing subtype', async () => { + const employee = await brain.add({ + type: NounType.Person, + subtype: 'employee', + data: 'Has a subtype already' + }) + const blank = await brain.add({ type: NounType.Person, data: 'No subtype' }) + + const report = await brain.fillSubtypes({ + [NounType.Person]: 'unspecified' + }) + + expect(report.filled).toBe(1) + expect((await brain.get(employee))!.subtype).toBe('employee') // untouched + expect((await brain.get(blank))!.subtype).toBe('unspecified') + }) + + it('leaves types without a rule untouched and counts them as skipped', async () => { + await brain.add({ type: NounType.Person, data: 'Person without rule coverage' }) + const doc = await brain.add({ type: NounType.Document, data: 'Doc gets filled' }) + + const report = await brain.fillSubtypes({ + [NounType.Document]: 'general' + }) + + expect(report.filled).toBe(1) + expect(report.skipped).toBe(1) // the Person — still migration debt + expect((await brain.get(doc))!.subtype).toBe('general') + const audit = await brain.audit() + expect(audit.total).toBe(report.skipped) // skipped IS the remaining debt + }) + }) + + describe('function rules', () => { + it('derives the subtype from entity fields', async () => { + const vendor = await brain.add({ + type: NounType.Person, + data: 'Vendor person', + metadata: { kind: 'vendor' } + }) + const fallback = await brain.add({ type: NounType.Person, data: 'No kind field' }) + + const report = await brain.fillSubtypes({ + [NounType.Person]: (e) => (e.metadata as { kind?: string })?.kind ?? 'unspecified' + }) + + expect(report.filled).toBe(2) + expect((await brain.get(vendor))!.subtype).toBe('vendor') + expect((await brain.get(fallback))!.subtype).toBe('unspecified') + }) + + it('treats undefined returns as declines (selective fill, counted as skipped)', async () => { + const classified = await brain.add({ + type: NounType.Person, + data: 'Classifiable', + metadata: { department: 'engineering' } + }) + const unclassified = await brain.add({ type: NounType.Person, data: 'Not classifiable' }) + + const report = await brain.fillSubtypes({ + [NounType.Person]: (e) => + (e.metadata as { department?: string })?.department ? 'employee' : undefined + }) + + expect(report.filled).toBe(1) + expect(report.skipped).toBe(1) + expect((await brain.get(classified))!.subtype).toBe('employee') + expect((await brain.get(unclassified))!.subtype).toBeUndefined() + }) + + it('treats empty-string returns as declines (an empty subtype would fail enforcement)', async () => { + await brain.add({ type: NounType.Person, data: 'Rule returns empty string' }) + + const report = await brain.fillSubtypes({ + [NounType.Person]: () => '' + }) + + expect(report.filled).toBe(0) + expect(report.skipped).toBe(1) + }) + + it('collects per-entry rule errors and continues the pass', async () => { + const poisoned = await brain.add({ + type: NounType.Person, + data: 'Rule throws on this one', + metadata: { poison: true } + }) + const healthy = await brain.add({ type: NounType.Person, data: 'Rule works on this one' }) + + const report = await brain.fillSubtypes({ + [NounType.Person]: (e) => { + if ((e.metadata as { poison?: boolean })?.poison) { + throw new Error('cannot classify poisoned entity') + } + return 'unspecified' + } + }) + + expect(report.filled).toBe(1) + expect(report.errors).toHaveLength(1) + expect(report.errors[0].id).toBe(poisoned) + expect(report.errors[0].error).toMatch(/cannot classify/) + expect((await brain.get(healthy))!.subtype).toBe('unspecified') + }) + }) + + describe('relationship rules (VerbType keys)', () => { + it('fills missing relationship subtypes through the same rule map', async () => { + const a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'A' }) + const b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'B' }) + const relId = await brain.relate({ from: a, to: b, type: VerbType.RelatedTo }) + const labeled = await brain.relate({ + from: b, + to: a, + type: VerbType.RelatedTo, + subtype: 'colleague' + }) + + const report = await brain.fillSubtypes({ + [VerbType.RelatedTo]: 'unspecified' + }) + + expect(report.filled).toBe(1) + expect(report.byType).toEqual({ relatedTo: 1 }) + + const relations = await brain.getRelations({ from: a }) + expect(relations.find((r) => r.id === relId)!.subtype).toBe('unspecified') + const reverse = await brain.getRelations({ from: b }) + expect(reverse.find((r) => r.id === labeled)!.subtype).toBe('colleague') // untouched + }) + + it('passes the full Relation shape to relationship rule functions', async () => { + const a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'A' }) + const b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'B' }) + await brain.relate({ + from: a, + to: b, + type: VerbType.ReportsTo, + metadata: { dotted: true } + }) + + const report = await brain.fillSubtypes({ + [VerbType.ReportsTo]: (r) => { + // The rule sees the public Relation shape — endpoints included. + expect(r.from).toBe(a) + expect(r.to).toBe(b) + return (r.metadata as { dotted?: boolean })?.dotted ? 'dotted-line' : 'direct' + } + }) + + expect(report.filled).toBe(1) + const relations = await brain.getRelations({ from: a }) + expect(relations[0].subtype).toBe('dotted-line') + }) + + it('handles entity and relationship rules in a single pass', async () => { + const a = await brain.add({ type: NounType.Person, data: 'A' }) + const b = await brain.add({ type: NounType.Person, data: 'B' }) + await brain.relate({ from: a, to: b, type: VerbType.RelatedTo }) + + const report = await brain.fillSubtypes({ + [NounType.Person]: 'unspecified', + [VerbType.RelatedTo]: 'unspecified' + }) + + expect(report.filled).toBe(3) + expect(report.byType).toEqual({ person: 2, relatedTo: 1 }) + const audit = await brain.audit() + expect(audit.total).toBe(0) + }) + }) + + describe('report + write semantics', () => { + it('bumps _rev through the real update path', async () => { + const id = await brain.add({ type: NounType.Person, data: 'Rev check' }) + const before = await brain.get(id) + + await brain.fillSubtypes({ [NounType.Person]: 'unspecified' }) + + const after = await brain.get(id) + expect(after!._rev).toBe((before!._rev ?? 1) + 1) + }) + + it('is idempotent — a re-run scans but fills nothing', async () => { + await brain.add({ type: NounType.Person, data: 'Fill once' }) + + const first = await brain.fillSubtypes({ [NounType.Person]: 'unspecified' }) + expect(first.filled).toBe(1) + + const second = await brain.fillSubtypes({ [NounType.Person]: 'unspecified' }) + expect(second.filled).toBe(0) + expect(second.skipped).toBe(0) + expect(second.scanned).toBeGreaterThan(0) + expect(second.errors).toEqual([]) + }) + + it('excludes VFS-marked entries by default, fills them with includeVFS: true', async () => { + // VFS markers bypass subtype enforcement, so marked entries are not + // migration debt. Fabricate one explicitly (public consumers shouldn't + // set this marker; the test exercises the exclusion contract). + const vfsLike = await brain.add({ + type: NounType.Document, + data: 'Infrastructure-marked entry', + metadata: { isVFS: true } + }) + + const defaultRun = await brain.fillSubtypes({ [NounType.Document]: 'general' }) + expect(defaultRun.filled).toBe(0) + expect((await brain.get(vfsLike))!.subtype).toBeUndefined() + + const inclusiveRun = await brain.fillSubtypes( + { [NounType.Document]: 'general' }, + { includeVFS: true } + ) + expect(inclusiveRun.filled).toBeGreaterThanOrEqual(1) + expect((await brain.get(vfsLike))!.subtype).toBe('general') + }) + + it('reports progress after each batch', async () => { + await brain.add({ type: NounType.Person, data: 'P1' }) + await brain.add({ type: NounType.Person, data: 'P2' }) + await brain.add({ type: NounType.Person, data: 'P3' }) + + const snapshots: Array<{ scanned: number; filled: number; skipped: number }> = [] + await brain.fillSubtypes( + { [NounType.Person]: 'unspecified' }, + { + batchSize: 1, + onProgress: (p) => snapshots.push({ ...p }) + } + ) + + expect(snapshots.length).toBeGreaterThanOrEqual(3) + const last = snapshots[snapshots.length - 1] + expect(last.filled).toBe(3) + }) + + it('clears the debt brain.audit() reports', async () => { + await brain.add({ type: NounType.Person, data: 'Debt 1' }) + await brain.add({ type: NounType.Document, data: 'Debt 2' }) + + const before = await brain.audit() + expect(before.total).toBe(2) + + await brain.fillSubtypes({ + [NounType.Person]: 'unspecified', + [NounType.Document]: 'general' + }) + + const after = await brain.audit() + expect(after.total).toBe(0) + expect(after.recommendation).toMatch(/strict-mode-ready/) + }) + }) + + describe('rule-map validation (fail fast, before touching data)', () => { + it('rejects an empty rules map', async () => { + await expect(brain.fillSubtypes({})).rejects.toThrow(/rules map is empty/) + }) + + it('rejects keys that are not a NounType or VerbType', async () => { + await expect( + brain.fillSubtypes({ 'not-a-type': 'whatever' } as never) + ).rejects.toThrow(/'not-a-type' is not a valid NounType or VerbType/) + }) + + it('rejects empty-string literal rules', async () => { + await expect( + brain.fillSubtypes({ [NounType.Person]: '' }) + ).rejects.toThrow(/empty string/) + }) + + it('rejects rule values that are neither string nor function', async () => { + await expect( + brain.fillSubtypes({ [NounType.Person]: 42 } as never) + ).rejects.toThrow(/must be a subtype string or a function/) + }) + + it('does not write anything when validation fails', async () => { + const id = await brain.add({ type: NounType.Person, data: 'Untouched on failure' }) + + await expect( + brain.fillSubtypes({ + [NounType.Person]: 'unspecified', + 'bogus-type': 'x' + } as never) + ).rejects.toThrow(/bogus-type/) + + expect((await brain.get(id))!.subtype).toBeUndefined() + }) + }) +})