diff --git a/README.md b/README.md index 19b69f25..40f82a37 100644 --- a/README.md +++ b/README.md @@ -247,6 +247,27 @@ Plugins are opt-in. Brainy never auto-imports packages unless listed in `plugins Model any domain — healthcare (`Patient → diagnoses → Condition`), finance (`Account → transfers → Transaction`), education (`Student → completes → Course`), or your own. +### Subtypes — sub-classification within a NounType + +The 42 NounTypes are intentionally coarse. Use the top-level `subtype` field to sub-classify entities within a type — flat string, no hierarchy, your choice of vocabulary: + +```javascript +await brain.add({ + data: 'Avery Brooks — runs the AI lab', + type: NounType.Person, + subtype: 'employee' // 'customer', 'vendor', 'contractor', … +}) + +// Filter on the fast path — column-store hit, not metadata fallback: +const employees = await brain.find({ type: NounType.Person, subtype: 'employee' }) + +// O(1) counts via the persisted rollup: +brain.counts.bySubtype(NounType.Person) +// → { employee: 12, customer: 847, vendor: 34 } +``` + +For other facets you want counted (`status`, `source`, `role`), register them with `brain.trackField(name)`. Renaming an existing convention to `subtype`? Use `brain.migrateField({from, to})`. Full guide: **[Subtypes & Facets](docs/guides/subtypes-and-facets.md)**. + **[Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)** | **[Stage 3 Canonical Reference](docs/STAGE3-CANONICAL-TAXONOMY.md)** --- diff --git a/RELEASES.md b/RELEASES.md index 875e69bd..3913618e 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -11,6 +11,139 @@ Collective. The SDK wraps it — most products never call Brainy directly. Read --- +## v7.29.0 — 2026-06-04 + +**Affected products:** anyone modeling entities with per-product sub-classification — every +consumer that's been reaching for `metadata.kind`, a custom `metadata.subtype` field, a +`data.kind` shape inside the payload, or similar ad-hoc conventions. Additive; drop-in from +7.28.x. No deprecations. + +### New — `subtype` promoted to a top-level standard field + +`subtype?: string` is now a first-class standard field on every entity, alongside `type` / +`confidence` / `weight`. Use it as the platform-standard primitive for sub-classifying entities +within a NounType (`Person` → `'employee'` / `'customer'`; `Document` → `'invoice'` / `'contract'`; +`Event` → `'milestone'` / `'meeting'`): + +```typescript +await brain.add({ + data: 'Avery Brooks — runs the AI lab', + type: NounType.Person, + subtype: 'employee', // top-level write param + metadata: { department: 'ai-lab' } +}) + +// Fast-path filter — column-store hit, not metadata fallback: +const employees = await brain.find({ type: NounType.Person, subtype: 'employee' }) + +// Set membership: +const internal = await brain.find({ + type: NounType.Person, + subtype: ['employee', 'contractor'] +}) +``` + +Flat string, no hierarchy — your vocabulary, your choice. Brainy stores and counts, never validates. + +### New — O(1) subtype counts via the persisted rollup + +A new `_system/subtype-statistics.json` rollup is maintained incrementally as entities are added, +updated, and deleted — mirroring the existing `nounCountsByType` machinery with the same self-heal +behavior on poison detection. Counts are O(1) at billion scale: + +```typescript +brain.counts.bySubtype(NounType.Person) +// → { employee: 12, customer: 847, vendor: 34 } + +brain.counts.bySubtype(NounType.Person, 'employee') // O(1) point count +// → 12 + +brain.counts.topSubtypes(NounType.Person, 3) +// → [['customer', 847], ['employee', 12], ['vendor', 34]] + +brain.subtypesOf(NounType.Person) +// → ['customer', 'employee', 'vendor'] +``` + +### New — `brain.trackField()` for other metadata facets + +For facets that aren't the *primary* sub-classification (`status`, `source`, `role`, `paradigm`), +`trackField` registers a field for cardinality + per-NounType breakdown stats without promoting +each one to a top-level slot. Piggybacks on the existing aggregation engine, so backfill-on-define +applies — registering on a populated brain scans existing entities on the first query: + +```typescript +brain.trackField('status', { perType: true }) + +await brain.counts.byField('status') +// → { todo: 12, doing: 3, done: 47 } + +await brain.counts.byField('status', { type: NounType.Task }) +// → { todo: 8, doing: 2, done: 30 } + +// Opt-in vocabulary validation: +brain.trackField('priority', { values: ['low', 'medium', 'high'] }) +// brain.add({ ..., metadata: { priority: 'urgent' } }) → throws +``` + +### New — generic `brain.migrateField()` for one-shot rewrites + +Streams every entity and copies a value from one field path to another. Supports top-level +standard fields, `metadata.*`, and `data.*` paths; idempotent; with optional `readBoth: true` +deprecation window that preserves the source field alongside the new one: + +```typescript +// Phase 1: dual-populate (legacy readers still work) +await brain.migrateField({ + from: 'metadata.kind', + to: 'subtype', + readBoth: true +}) + +// ...readers migrate to subtype at their own pace... + +// Phase 2: clear the source field +await brain.migrateField({ from: 'metadata.kind', to: 'subtype' }) +// → { scanned: 1500, migrated: 1500, skipped: 0, errors: [] } +``` + +Supports `batchSize` and `onProgress` for large brains. + +### Aggregation composition + +`subtype` is a standard-field group-by dimension out of the box — no `where:` wrapper, no +metadata-fallback overhead: + +```typescript +await brain.find({ + aggregate: { + groupBy: ['type', 'subtype'], + metrics: { count: { op: 'COUNT' } } + } +}) +// [ +// { groupKey: { type: 'person', subtype: 'customer' }, count: 847 }, +// { groupKey: { type: 'person', subtype: 'employee' }, count: 12 }, +// { groupKey: { type: 'document', subtype: 'invoice' }, count: 2103 }, +// ... +// ] +``` + +### Cortex compatibility + +Subtype works under Cortex out of the box via auto-field-indexing. Cortex will land its own +native-side query-planner recognition + `subtypeCountsByType` native rollup in the next Cortex +release for full parity with `type`'s fast path. Not a Brainy blocker — subtype reads/writes +function correctly with current Cortex. + +### Docs + +Full guide: `docs/guides/subtypes-and-facets.md`. Subtype is documented as a core primitive +throughout `README.md`, `docs/DATA_MODEL.md`, `docs/architecture/finite-type-system.md`, +`docs/api/README.md`, and `docs/QUERY_OPERATORS.md`. + +--- + ## v7.24.0 — 2026-05-26 **Affected products:** aggregation/reporting users + anyone calling `extractEntities` on diff --git a/docs/DATA_MODEL.md b/docs/DATA_MODEL.md index d85e7c8f..ec28ecb8 100644 --- a/docs/DATA_MODEL.md +++ b/docs/DATA_MODEL.md @@ -186,6 +186,7 @@ When you add an entity, Brainy stores these standard fields in the metadata obje | Field | Set By | Description | |-------|--------|-------------| | `noun` | System | Entity type (NounType enum value) | +| `subtype` | User | Per-NounType sub-classification (e.g. `'employee'`, `'invoice'`, `'milestone'`). Flat string, no hierarchy. Indexed on the fast path and rolled into per-NounType statistics. | | `data` | System | The raw `data` value (stored opaquely) | | `createdAt` | System | Creation timestamp | | `updatedAt` | System | Last update timestamp | @@ -196,6 +197,30 @@ When you add an entity, Brainy stores these standard fields in the metadata obje On read, these standard fields are extracted to top-level Entity properties. The `metadata` field on the returned Entity contains **only your custom fields**. +### Subtype — sub-classification within a NounType + +`type` (NounType) is a stable 42-value enum. `subtype` is the consumer-chosen string vocabulary *within* a type: + +```typescript +// A Person who is an employee: +await brain.add({ + data: 'Avery Brooks — runs the AI lab', + type: NounType.Person, + subtype: 'employee', + metadata: { department: 'ai-lab' } +}) + +// A Document that is an invoice: +await brain.add({ + data: 'INV-2026-001', + type: NounType.Document, + subtype: 'invoice', + metadata: { amount: 1500 } +}) +``` + +`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's how `find({ type, subtype })` routes through the standard-field fast path (column-store hit) instead of the metadata fallback. See **[Subtypes & Facets](./guides/subtypes-and-facets.md)** for the full guide including `trackField()` and `migrateField()`. + --- ## See Also diff --git a/docs/QUERY_OPERATORS.md b/docs/QUERY_OPERATORS.md index 93f9cd2a..5cfc5410 100644 --- a/docs/QUERY_OPERATORS.md +++ b/docs/QUERY_OPERATORS.md @@ -194,6 +194,26 @@ brain.find({ where: { noun: NounType.Person } }) brain.find({ type: [NounType.Person, NounType.Agent] }) ``` +### Filter by subtype + +`subtype` is a top-level standard field — takes the column-store fast path, not the metadata fallback. Pair with `type` for the typical "Person who is an employee" query: + +```typescript +// Equality on subtype: +brain.find({ type: NounType.Person, subtype: 'employee' }) + +// Set membership: +brain.find({ type: NounType.Person, subtype: ['employee', 'contractor'] }) + +// Operator-form predicates use `where`: +brain.find({ + type: NounType.Person, + where: { subtype: { exists: true } } +}) +``` + +See the **[Subtypes & Facets guide](./guides/subtypes-and-facets.md)** for the full surface. + ### Combine semantic search with filters ```typescript diff --git a/docs/api/README.md b/docs/api/README.md index 5035cba8..22516a83 100644 --- a/docs/api/README.md +++ b/docs/api/README.md @@ -109,11 +109,12 @@ Add a single entity to the database. ```typescript const id = await brain.add({ - data: 'JavaScript is a programming language', // Text or pre-computed vector - type: NounType.Concept, // Required: Entity type - metadata: { // Optional metadata - category: 'programming', - year: 1995 + data: 'JavaScript is a programming language', // Text or pre-computed vector + type: NounType.Concept, // Required: Entity type + subtype: 'language', // Optional: sub-classification + metadata: { // Optional: queryable fields + category: 'programming', + year: 1995 } }) ``` @@ -121,6 +122,7 @@ const id = await brain.add({ **Parameters:** - `data`: `string | number[]` - Content to embed (text auto-embeds) or pre-computed vector - `type`: `NounType` - Entity type (required) +- `subtype?`: `string` - Per-product sub-classification within the NounType (top-level standard field, indexed on the fast path). See [Subtypes & Facets](../guides/subtypes-and-facets.md). - `metadata?`: `object` - Structured queryable fields (indexed by MetadataIndex, used in `where` filters) - `id?`: `string` - Custom ID (auto-generated UUID if not provided) - `vector?`: `number[]` - Pre-computed vector (skips auto-embedding) @@ -158,15 +160,20 @@ Update an existing entity. ```typescript await brain.update({ id: entityId, - data: 'Updated content', // Optional: new data - metadata: { updated: true } // Optional: new metadata (merges) + data: 'Updated content', // Optional: new data + subtype: 'archived', // Optional: change sub-classification + metadata: { updated: true } // Optional: new metadata (merges) }) ``` **Parameters:** - `id`: `string` - Entity ID - `data?`: `string | number[]` - New data/vector -- `metadata?`: `object` - Metadata to merge +- `type?`: `NounType` - Change entity type +- `subtype?`: `string` - Change subtype (omit to preserve existing) +- `metadata?`: `object` - Metadata to merge (or replace with `merge: false`) +- `confidence?`: `number` - Update classification confidence +- `weight?`: `number` - Update entity importance **Returns:** `Promise` @@ -221,6 +228,7 @@ const results = await brain.find({ **FindParams:** - `query?`: `string` - Text for semantic + hybrid search (searches `data` via HNSW + text index) - `type?`: `NounType | NounType[]` - Filter by entity type(s). Alias for `where.noun`. +- `subtype?`: `string | string[]` - Filter by sub-classification (top-level standard field, fast path). Single string for equality, array for set membership. - `where?`: `object` - Metadata filters. See **[Query Operators](../QUERY_OPERATORS.md)** for all operators. - `connected?`: `object` - Graph traversal options - `to?`: `string` - Target entity ID @@ -1928,6 +1936,86 @@ const count = await brain.getVerbCount() --- +### Subtype & facet APIs + +Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**. + +#### `counts.bySubtype(type, subtype?)` → `Record | number` + +O(1) subtype counts for a NounType (backed by the persisted rollup). + +```typescript +brain.counts.bySubtype(NounType.Person) +// → { employee: 12, customer: 847, vendor: 34 } + +brain.counts.bySubtype(NounType.Person, 'employee') +// → 12 +``` + +#### `counts.topSubtypes(type, n=10)` → `Array<[subtype, count]>` + +Top N subtypes ranked by count. + +```typescript +brain.counts.topSubtypes(NounType.Person, 3) +// → [['customer', 847], ['employee', 12], ['vendor', 34]] +``` + +#### `subtypesOf(type)` → `string[]` + +Sorted distinct subtypes seen for a NounType. + +```typescript +brain.subtypesOf(NounType.Person) +// → ['customer', 'employee', 'vendor'] +``` + +#### `trackField(name, options?)` → `void` + +Register a metadata field for cardinality + per-NounType breakdown stats. With `values: [...]`, validates against the whitelist on `add()`/`update()`. + +```typescript +brain.trackField('status') // basic +brain.trackField('status', { perType: true }) // with per-NounType breakdown +brain.trackField('priority', { values: ['low', 'med', 'high'] }) // strict vocabulary +``` + +#### `counts.byField(name, options?)` → `Promise>` + +Counts by value for a tracked field. Requires `perType: true` registration if filtering by NounType. + +```typescript +await brain.counts.byField('status') +// → { todo: 12, doing: 3, done: 47 } + +await brain.counts.byField('status', { type: NounType.Task }) +// → { todo: 8, doing: 2, done: 30 } +``` + +#### `migrateField(options)` → `Promise` + +Stream-and-rewrite a field across the brain. Supports `metadata.X`, `data.X`, and top-level paths. Idempotent. + +```typescript +// One-shot rewrite +await brain.migrateField({ from: 'metadata.kind', to: 'subtype' }) + +// Deprecation window — keep source field readable +await brain.migrateField({ from: 'data.kind', to: 'subtype', readBoth: true }) + +// With progress reporting +await brain.migrateField({ + from: 'metadata.kind', + to: 'subtype', + batchSize: 500, + onProgress: ({ scanned, migrated }) => console.log(`${scanned} / ${migrated}`) +}) +``` + +Returns `{ scanned: number, migrated: number, skipped: number, errors: Array<{id, error}> }`. + +--- + ### `embed(data)` → `Promise` ✨ Generate embedding vector from text or data. diff --git a/docs/architecture/finite-type-system.md b/docs/architecture/finite-type-system.md index 51ba515c..c8f70c75 100644 --- a/docs/architecture/finite-type-system.md +++ b/docs/architecture/finite-type-system.md @@ -449,6 +449,26 @@ brain.registerNounType('chemical_compound', { }) ``` +### 1a. Subtypes — sub-classification without hierarchy + +The 42-type taxonomy is intentionally coarse. Per-product vocabulary fits on the **`subtype`** axis — a top-level standard string field on every entity. Flat by design — no hierarchy, no parent chain, no recursive resolution. That preserves the Uint32Array-backed O(1) type stats while giving consumers a place to put `'employee'` / `'customer'` / `'invoice'` / `'milestone'` without burning a slot in the global enum. + +```typescript +// Same NounType, different subtypes: +await brain.add({ type: NounType.Person, subtype: 'employee' }) +await brain.add({ type: NounType.Person, subtype: 'customer' }) +await brain.add({ type: NounType.Document, subtype: 'invoice' }) + +// Fast path — column-store hit, not metadata fallback: +await brain.find({ type: NounType.Person, subtype: 'employee' }) + +// Per-NounType-per-subtype counts maintained incrementally: +brain.counts.bySubtype(NounType.Person) +// → { employee: 12, customer: 847 } +``` + +Subtype has its own statistics rollup (`_system/subtype-statistics.json`) maintained alongside `nounCountsByType`, so per-subtype counts stay O(1) at billion scale. Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**. + ### 2. Semantic not Structural ```typescript diff --git a/docs/guides/quick-start.md b/docs/guides/quick-start.md index 61c1a2b1..a4e287f1 100644 --- a/docs/guides/quick-start.md +++ b/docs/guides/quick-start.md @@ -39,16 +39,20 @@ That's it. Brainy auto-configures storage, loads the embedding model, and builds const reactId: string = await brain.add({ data: 'React is a JavaScript library for building user interfaces', type: NounType.Concept, + subtype: 'library', // Sub-classification within Concept metadata: { category: 'frontend', year: 2013 } }) const nextId: string = await brain.add({ data: 'Next.js framework for React with server-side rendering', type: NounType.Concept, + subtype: 'framework', metadata: { category: 'framework', year: 2016 } }) ``` +`type` is one of Brainy's 42 stable NounTypes. `subtype` is your free-form sub-classification within that type — flat string, no hierarchy, indexed on the fast path. See **[Subtypes & Facets](./subtypes-and-facets.md)** for the full guide. + ## 4. Create Relationships ```typescript diff --git a/docs/guides/subtypes-and-facets.md b/docs/guides/subtypes-and-facets.md new file mode 100644 index 00000000..0ba1fcde --- /dev/null +++ b/docs/guides/subtypes-and-facets.md @@ -0,0 +1,285 @@ +--- +title: Subtypes & Facets +slug: guides/subtypes-and-facets +public: true +category: guides +template: guide +order: 7 +description: Use the top-level `subtype` field to sub-classify entities within a NounType, track other metadata facets like status or role, and migrate field names without downtime. +next: + - guides/aggregation + - api/reference +--- + +# Subtypes & Facets + +> Sub-classify entities within a NounType, track arbitrary metadata facets, and migrate field names without downtime. + +## Why this exists + +Brainy's `NounType` (Person, Document, Event, Concept, Task, …) is a stable, flat 42-type taxonomy. It's deliberately coarse — every product needs a way to further classify entities *within* a type. Is this `Person` an employee or a customer? Is this `Document` an invoice or a contract? Is this `Event` a meeting or a milestone? + +Three layers solve this: + +| Layer | Use it for | Shape | +|---|---|---| +| **`subtype`** | The primary sub-classification of an entity, one value per entity | Top-level standard field | +| **`trackField()`** | Other facets you want to count or filter on (`status`, `source`, `role`, `paradigm`) | Registered metadata field | +| **`migrateField()`** | Renaming or restructuring fields across an existing dataset | One-shot stream-and-rewrite | + +## Layer 1 — `subtype` + +`subtype` is a top-level standard field on every entity, alongside `type` / `confidence` / `weight`. Flat string, no hierarchy. The vocabulary is your choice — Brainy stores and counts, never validates. + +### Write + +```typescript +import { Brainy, NounType } from '@soulcraft/brainy' + +const brain = new Brainy() +await brain.init() + +await brain.add({ + data: 'Avery Brooks — runs the AI lab', + type: NounType.Person, + subtype: 'employee', // top-level write param + metadata: { department: 'ai-lab' } +}) +``` + +### Read + +```typescript +// Top-level filter — standard-field fast path, no `where` wrapper: +const employees = await brain.find({ type: NounType.Person, subtype: 'employee' }) + +// Set membership: +const internal = await brain.find({ + type: NounType.Person, + subtype: ['employee', 'contractor'] +}) + +// Operator-form predicates use `where`: +const typed = await brain.find({ + type: NounType.Person, + where: { subtype: { exists: true } } +}) +``` + +### What it looks like + +```json +{ + "id": "01HZK3M7TW...", + "type": "person", + "subtype": "employee", + "data": "Avery Brooks — runs the AI lab", + "metadata": { "department": "ai-lab" } +} +``` + +`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's what makes the fast path possible: queries on `subtype` hit the column-store index directly, never the metadata fallback. + +### Counts (O(1)) + +```typescript +// All subtypes for a NounType +brain.counts.bySubtype(NounType.Person) +// → { employee: 12, customer: 847, vendor: 34 } + +// Point count +brain.counts.bySubtype(NounType.Person, 'employee') +// → 12 + +// Top N +brain.counts.topSubtypes(NounType.Person, 3) +// → [['customer', 847], ['employee', 12], ['vendor', 34]] + +// Distinct subtypes for a NounType +brain.subtypesOf(NounType.Person) +// → ['customer', 'employee', 'vendor'] +``` + +These are O(1) lookups backed by `_system/subtype-statistics.json` — no scan, no storage round-trip. The rollup is incrementally maintained as entities are added, updated, and deleted. + +### Aggregation + +`subtype` is a first-class group-by dimension: + +```typescript +const rows = await brain.find({ + aggregate: { + groupBy: ['type', 'subtype'], + metrics: { count: { op: 'COUNT' } } + } +}) +// [ +// { groupKey: { type: 'person', subtype: 'customer' }, count: 847 }, +// { groupKey: { type: 'person', subtype: 'employee' }, count: 12 }, +// { groupKey: { type: 'document', subtype: 'invoice' }, count: 2103 }, +// ... +// ] +``` + +## Layer 2 — `trackField()` + +Some metadata fields aren't *the* sub-classification of an entity but are still worth counting: `status`, `source`, `role`, `paradigm`. Promoting each one to top-level would clutter the contract. `trackField()` registers a metadata field for cardinality + per-NounType breakdown stats without the contract growth. + +### Register and query + +```typescript +// Track a single facet +brain.trackField('status') + +await brain.add({ data: 'Ship subtype', type: NounType.Task, metadata: { status: 'todo' } }) +await brain.add({ data: 'Write docs', type: NounType.Task, metadata: { status: 'done' } }) + +await brain.counts.byField('status') +// → { todo: 1, done: 1 } +``` + +### Per-NounType breakdown + +```typescript +brain.trackField('status', { perType: true }) + +await brain.counts.byField('status', { type: NounType.Task }) +// → { todo: 1, done: 1 } +``` + +### Vocabulary whitelist (opt-in validation) + +```typescript +brain.trackField('priority', { values: ['low', 'medium', 'high'] }) + +// Throws — 'urgent' isn't in the vocabulary: +await brain.add({ + data: 'Fix bug', + type: NounType.Task, + metadata: { priority: 'urgent' } +}) +``` + +`trackField` piggybacks on the existing aggregation engine (see the [Aggregation guide](./aggregation.md) for the underlying mechanism). Backfill-on-define means the first call to `counts.byField()` scans existing entities; subsequent calls are O(groups). + +### subtype vs trackField — when to use which + +- **`subtype`** when there's one primary sub-classification per entity. Limit yourself to one per NounType. Examples: Person→employee/customer/vendor; Document→invoice/contract/policy. +- **`trackField`** for anything else you want to count or filter on. No limit on how many you register. Examples: status, source, role, paradigm. + +## Layer 3 — `migrateField()` + +Use this when you need to rename or restructure a field across an entire dataset — for example, moving a `metadata.kind` convention up to the top-level `subtype` standard field. + +### One-shot rewrite + +```typescript +// Starting state: every entity has metadata.kind +const result = await brain.migrateField({ + from: 'metadata.kind', + to: 'subtype' +}) + +console.log(result) +// { +// scanned: 1500, +// migrated: 1500, +// skipped: 0, +// errors: [] +// } +``` + +After this returns, every entity has `subtype` populated from the old `metadata.kind` value, and `metadata.kind` is cleared. + +### Deprecation window — keep both fields readable + +When you can't coordinate all readers and the migration in a single deploy, use `readBoth: true` to preserve the source field alongside the new one: + +```typescript +// Phase 1: dual-populate (existing readers still work against metadata.kind): +await brain.migrateField({ + from: 'metadata.kind', + to: 'subtype', + readBoth: true +}) + +// ... readers migrate to query subtype at their own pace ... + +// Phase 2: clear the source field when ready: +await brain.migrateField({ from: 'metadata.kind', to: 'subtype' }) +``` + +### Supported paths + +| Path form | Refers to | +|---|---| +| `'subtype'`, `'type'`, `'confidence'` | Top-level standard fields | +| `'metadata.X'` | A key under `entity.metadata` | +| `'data.X'` | A key under `entity.data` (when `data` is an object) | +| `'X'` (bare, non-standard) | Shorthand for `metadata.X` | + +### Idempotent + +`migrateField` is safe to re-run. Entities where the source is absent, or where the destination already holds the same value, are skipped. This makes it safe to use in a deploy-once-then-cleanup workflow. + +### Progress reporting + +```typescript +await brain.migrateField({ + from: 'metadata.kind', + to: 'subtype', + batchSize: 500, + onProgress: ({ scanned, migrated }) => { + console.log(`${scanned} scanned, ${migrated} migrated`) + } +}) +``` + +## Putting it together + +A realistic adoption sequence for a brain that started without these primitives: + +```typescript +import { Brainy, NounType } from '@soulcraft/brainy' + +const brain = new Brainy({ storage: { type: 'filesystem', options: { path: './brain-data' } } }) +await brain.init() + +// 1. Migrate any existing metadata.kind convention to the new top-level subtype +await brain.migrateField({ from: 'metadata.kind', to: 'subtype', readBoth: true }) + +// 2. Register the other facets you want counted +brain.trackField('status', { perType: true }) +brain.trackField('source') + +// 3. Use subtype on every new write +await brain.add({ + data: 'Quarterly review', + type: NounType.Event, + subtype: 'milestone', + metadata: { status: 'todo', source: 'planning-session' } +}) + +// 4. Query the breakdowns +brain.counts.bySubtype(NounType.Event) +// → { milestone: 14, meeting: 203, deadline: 7 } + +await brain.counts.byField('status', { type: NounType.Event }) +// → { todo: 12, done: 212 } + +// 5. Once all readers are on the new field, drop the source: +await brain.migrateField({ from: 'metadata.kind', to: 'subtype' }) +``` + +## Reference + +- `brain.add({ ..., subtype: 'value' })` — write the field +- `brain.update({ id, subtype: 'value' })` — change the field +- `brain.find({ type, subtype })` — filter (fast path) +- `brain.find({ subtype: ['a', 'b'] })` — set membership +- `brain.counts.bySubtype(type, subtype?)` — O(1) counts +- `brain.counts.topSubtypes(type, n?)` — top N by count +- `brain.subtypesOf(type)` — distinct subtype list +- `brain.trackField(name, { perType?, values? })` — register a facet +- `brain.counts.byField(name, { type? })` — facet counts +- `brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })` — rewrite a field diff --git a/src/brainy.ts b/src/brainy.ts index 6bbf284a..2ecbc28a 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -11,7 +11,7 @@ import { HNSWIndex } from './hnsw/hnswIndex.js' // for the 99% of queries that don't filter by type (avoids searching 42 separate graphs) import { createStorage } from './storage/storageFactory.js' import { BaseStorage } from './storage/baseStorage.js' -import { StorageAdapter, Vector, DistanceFunction, EmbeddingFunction, GraphVerb } from './coreTypes.js' +import { StorageAdapter, Vector, DistanceFunction, EmbeddingFunction, GraphVerb, STANDARD_ENTITY_FIELDS } from './coreTypes.js' import { defaultEmbeddingFunction, cosineDistance, @@ -97,7 +97,7 @@ import { BrainyStats, ScoreExplanation } from './types/brainy.types.js' -import { NounType, VerbType } from './types/graphTypes.js' +import { NounType, VerbType, TypeUtils } from './types/graphTypes.js' import { BrainyInterface } from './types/brainyInterface.js' import type { IntegrationHub } from './integrations/core/IntegrationHub.js' import { MigrationRunner } from './migration/MigrationRunner.js' @@ -189,6 +189,14 @@ export class Brainy implements BrainyInterface { private _pendingMigrationRunner?: MigrationRunner // Deferred migration runner for large datasets private _aggregationIndex?: AggregationIndex // Incremental aggregation engine private _materializer?: AggregateMaterializer // Debounced materialization of aggregate results + /** + * Fields registered via `brain.trackField()` — drives optional value validation on + * `add()`/`update()` and powers `brain.counts.byField()`. Outer key is the field + * name (`'status'`, `'paradigm'`, `'role'`, ...); inner record carries the per-NounType + * flag and an optional value whitelist (when provided, writes with off-vocabulary + * values are rejected). + */ + private _trackedFields: Map }> = new Map() // State private initialized = false @@ -1016,6 +1024,15 @@ export class Brainy implements BrainyInterface { // Zero-config validation (static import for performance) validateAddParams(params) + // Tracked-field vocabulary enforcement (Layer 2). Walks both bags so a + // tracked field declared at top level (e.g. 'subtype') and one declared in + // metadata (e.g. 'status') both validate. + this.enforceTrackedFieldValues(params.metadata as Record | undefined, 'metadata') + this.enforceTrackedFieldValues( + { subtype: params.subtype } as Record, + 'top-level' + ) + // Generate ID if not provided const id = params.id || uuidv4() @@ -1038,6 +1055,7 @@ export class Brainy implements BrainyInterface { ...params.metadata, data: params.data, noun: params.type, + ...(params.subtype !== undefined && { subtype: params.subtype }), service: params.service, createdAt: Date.now(), updatedAt: Date.now(), @@ -1057,6 +1075,7 @@ export class Brainy implements BrainyInterface { connections: new Map(), level: 0, type: params.type, + ...(params.subtype !== undefined && { subtype: params.subtype }), ...(params.confidence !== undefined && { confidence: params.confidence }), ...(params.weight !== undefined && { weight: params.weight }), createdAt: Date.now(), @@ -1329,6 +1348,7 @@ export class Brainy implements BrainyInterface { score, // Flatten common entity fields to top level type: entity.type, + subtype: entity.subtype, metadata: entity.metadata, data: entity.data, confidence: entity.confidence, @@ -1357,6 +1377,7 @@ export class Brainy implements BrainyInterface { id: noun.id, vector: noun.vector, type: noun.type || NounType.Thing, + subtype: noun.subtype, // Standard fields at top-level confidence: noun.confidence, @@ -1398,12 +1419,13 @@ export class Brainy implements BrainyInterface { // Extract standard fields, rest are custom metadata // Same destructuring as baseStorage.getNoun() to ensure consistency - const { noun, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata + const { noun, subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata const entity: Entity = { id, vector: [], // Stub vector (empty array - vectors not loaded for metadata-only) type: noun as NounType || NounType.Thing, + subtype, // Standard fields from metadata confidence, @@ -1478,6 +1500,17 @@ export class Brainy implements BrainyInterface { // Zero-config validation (static import for performance) validateUpdateParams(params) + // Tracked-field vocabulary enforcement (Layer 2). Same as add() — the + // metadata bag carries fields registered via trackField(), and subtype is + // a tracked top-level candidate. + this.enforceTrackedFieldValues(params.metadata as Record | undefined, 'metadata') + if (params.subtype !== undefined) { + this.enforceTrackedFieldValues( + { subtype: params.subtype } as Record, + 'top-level' + ) + } + // Get existing entity with vectors (fix for regression) // We need includeVectors: true because: // 1. SaveNounOperation requires the vector @@ -1513,7 +1546,10 @@ export class Brainy implements BrainyInterface { ...(params.confidence !== undefined && { confidence: params.confidence }), ...(params.weight !== undefined && { weight: params.weight }), ...(params.confidence === undefined && existing.confidence !== undefined && { confidence: existing.confidence }), - ...(params.weight === undefined && existing.weight !== undefined && { weight: existing.weight }) + ...(params.weight === undefined && existing.weight !== undefined && { weight: existing.weight }), + // Update subtype if provided, otherwise preserve existing + ...(params.subtype !== undefined && { subtype: params.subtype }), + ...(params.subtype === undefined && existing.subtype !== undefined && { subtype: existing.subtype }) } // Build entity structure for metadata index (with top-level fields) @@ -1523,6 +1559,7 @@ export class Brainy implements BrainyInterface { connections: new Map(), level: 0, type: params.type || existing.type, + subtype: params.subtype !== undefined ? params.subtype : existing.subtype, confidence: params.confidence !== undefined ? params.confidence : existing.confidence, weight: params.weight !== undefined ? params.weight : existing.weight, createdAt: existing.createdAt, @@ -2291,6 +2328,104 @@ export class Brainy implements BrainyInterface { this._aggregationIndex!.defineAggregate(def) } + /** + * Register a field for cardinality + per-NounType breakdown stats. + * + * Layer 2 of the subtype-and-facets primitive: a lightweight wrapper over the + * aggregation engine that auto-defines an internal `__fieldCounts__` + * aggregate so consumers can query value frequencies without writing the + * aggregate definition themselves. Backfill-on-define (shipped 7.23.0) means + * existing entities are scanned on the first query, not at registration time. + * + * Use this for facets that don't warrant top-level promotion (e.g. `status`, + * `source`, `role`). For sub-classification within a NounType, prefer the + * top-level `subtype` field — it takes the standard-field fast path and has + * its own statistics rollup. + * + * @param name - Field name to track. Resolves via the standard-fields-first / + * metadata-fallback path, so both `'subtype'` (top-level) and `'status'` + * (metadata) work the same way. + * @param options - `perType` adds `noun` to the groupBy so counts are split + * by NounType; `values` registers a whitelist that rejects writes containing + * off-vocabulary values (validated in `add()`/`update()`). + * + * @example Track status without per-type breakdown + * brain.trackField('status') + * const counts = await brain.counts.byField('status') + * // → { todo: 12, doing: 3, done: 47 } + * + * @example Track status with per-NounType breakdown + * brain.trackField('status', { perType: true }) + * const taskCounts = await brain.counts.byField('status', { type: NounType.Task }) + * // → { todo: 8, doing: 2, done: 30 } + * + * @example Strict vocabulary + * brain.trackField('priority', { values: ['low', 'medium', 'high'] }) + * // brain.add({ ..., metadata: { priority: 'urgent' } }) throws + */ + trackField( + name: string, + options: { perType?: boolean; values?: string[] } = {} + ): void { + if (!name || typeof name !== 'string') { + throw new Error('trackField: name must be a non-empty string') + } + const perType = options.perType === true + const valuesSet = options.values && options.values.length > 0 + ? new Set(options.values) + : undefined + this._trackedFields.set(name, { perType, values: valuesSet }) + + // Auto-define the backing aggregate. groupBy uses 'noun' (storage field name + // for type) when perType is on, so the column-store key matches the persisted + // shape and resolveEntityField doesn't need to rewrite the dimension. + this.ensureAggregationIndex() + const aggregateName = this.fieldCountsAggregateName(name) + if (!this._aggregationIndex!.hasAggregate(aggregateName)) { + this._aggregationIndex!.defineAggregate({ + name: aggregateName, + source: {}, + groupBy: perType ? [name, 'noun'] : [name], + metrics: { count: { op: 'count' } } + }) + } + } + + /** + * Internal aggregate name for a tracked field. Centralized so `trackField()` + * and `counts.byField()` agree on the convention. + */ + private fieldCountsAggregateName(name: string): string { + return `__fieldCounts__${name}` + } + + /** + * Validate a metadata bag (or top-level field assignment) against any registered + * value whitelists. Called from `add()`/`update()` after the standard zero-config + * validation. Throws on the first off-vocabulary value to fail fast. + * + * Tracked fields with no `values` whitelist are skipped — registration alone + * does not imply validation. + */ + private enforceTrackedFieldValues( + bag: Record | undefined, + bagLabel: 'metadata' | 'top-level' + ): void { + if (!bag || this._trackedFields.size === 0) return + for (const [field, def] of this._trackedFields.entries()) { + if (!def.values) continue + if (!(field in bag)) continue + const value = bag[field] + if (value === undefined || value === null) continue + const asString = typeof value === 'string' ? value : String(value) + if (!def.values.has(asString)) { + throw new Error( + `trackField('${field}') rejected ${bagLabel} value '${asString}': not in registered vocabulary [${Array.from(def.values).join(', ')}]` + ) + } + } + } + /** * Remove a named aggregate and clean up its state. * @@ -2373,7 +2508,7 @@ export class Brainy implements BrainyInterface { // Distinguish between search criteria (need vector search) and filter criteria (metadata only) // Treat empty string query as no query const hasVectorSearchCriteria = (params.query && params.query.trim() !== '') || params.vector || params.near - const hasFilterCriteria = params.where || params.type || params.service + const hasFilterCriteria = params.where || params.type || params.subtype || params.service const hasGraphCriteria = params.connected // Handle metadata-only queries (no vector search needed) @@ -2390,6 +2525,15 @@ export class Brainy implements BrainyInterface { } if (params.service) filter.service = params.service + // Subtype (top-level standard field — fast path, not metadata fallback). + // Must be assigned BEFORE the type-array expansion below so the spread + // into each anyOf branch carries it through. + if (params.subtype !== undefined) { + filter.subtype = Array.isArray(params.subtype) + ? { oneOf: params.subtype } + : params.subtype + } + if (params.type) { const types = Array.isArray(params.type) ? params.type : [params.type] if (types.length === 1) { @@ -2528,7 +2672,7 @@ export class Brainy implements BrainyInterface { let preResolvedMetadataIds: string[] | null = null let preResolvedFilter: any = null - if (params.where || params.type || params.service || params.excludeVFS) { + if (params.where || params.type || params.subtype || params.service || params.excludeVFS) { preResolvedFilter = {} if (params.where) { Object.assign(preResolvedFilter, params.where) @@ -2543,6 +2687,13 @@ export class Brainy implements BrainyInterface { preResolvedFilter.vfsType = { exists: false } preResolvedFilter.isVFSEntity = { ne: true } } + // Subtype (top-level standard field — fast path). + // Must be assigned BEFORE the type-array expansion below. + if (params.subtype !== undefined) { + preResolvedFilter.subtype = Array.isArray(params.subtype) + ? { oneOf: params.subtype } + : params.subtype + } if (params.type) { const types = Array.isArray(params.type) ? params.type : [params.type] if (types.length === 1) { @@ -5776,13 +5927,13 @@ export class Brainy implements BrainyInterface { // Get total count for pagination UI (O(1) when possible) count: async (params: Omit, 'limit' | 'offset'>) => { // For simple type queries, use O(1) index counting - if (params.type && !params.query && !params.where && !params.connected) { + if (params.type && !params.subtype && !params.query && !params.where && !params.connected) { const types = Array.isArray(params.type) ? params.type : [params.type] return types.reduce((sum, type) => sum + this.metadataIndex.getEntityCountByType(type), 0) } // For complex queries, use metadata index for efficient counting - if (params.where || params.service) { + if (params.where || params.subtype || params.service) { let filter: any = {} if (params.where) { Object.assign(filter, params.where) @@ -5793,6 +5944,11 @@ export class Brainy implements BrainyInterface { } } if (params.service) filter.service = params.service + if (params.subtype !== undefined) { + filter.subtype = Array.isArray(params.subtype) + ? { oneOf: params.subtype } + : params.subtype + } if (params.type) { const types = Array.isArray(params.type) ? params.type : [params.type] if (types.length === 1) { @@ -5846,7 +6002,7 @@ export class Brainy implements BrainyInterface { return { // Stream all entities with optional filtering entities: async function* (this: Brainy, filter?: Partial>) { - if (filter?.type || filter?.where || filter?.service) { + if (filter?.type || filter?.subtype || filter?.where || filter?.service) { // Use MetadataIndexManager for efficient filtered streaming let filterObj: any = {} if (filter.where) { @@ -5858,6 +6014,11 @@ export class Brainy implements BrainyInterface { } } if (filter.service) filterObj.service = filter.service + if (filter.subtype !== undefined) { + filterObj.subtype = Array.isArray(filter.subtype) + ? { oneOf: filter.subtype } + : filter.subtype + } if (filter.type) { const types = Array.isArray(filter.type) ? filter.type : [filter.type] if (types.length === 1) { @@ -6042,6 +6203,70 @@ export class Brainy implements BrainyInterface { return this.metadataIndex.getTopNounTypes(n) }, + /** + * O(1) subtype counts for a given NounType. + * + * Returns the count for a single (type, subtype) pair when `subtype` is + * passed; returns the full subtype → count map for that NounType when omitted. + * Backed by the persisted `_system/subtype-statistics.json` rollup — no + * scan, no storage round-trip. + * + * @param type - The NounType to count subtypes within + * @param subtype - Optional specific subtype string for O(1) point count + * @returns A number when `subtype` is given, otherwise a `Record` (empty `{}` if none) + * + * @example Get all subtypes of Person + * const counts = brain.counts.bySubtype(NounType.Person) + * // → { employee: 56, customer: 847, vendor: 34 } + * + * @example O(1) point count + * const employees = brain.counts.bySubtype(NounType.Person, 'employee') + * // → 56 + */ + bySubtype: (type: NounType, subtype?: string): number | Record => { + const subtypeMap = typeof (this.storage as any).getSubtypeCountsByType === 'function' + ? (this.storage as any).getSubtypeCountsByType() as Map> + : null + if (!subtypeMap) { + return subtype !== undefined ? 0 : {} + } + const typeIdx = TypeUtils.getNounIndex(type) + const inner = subtypeMap.get(typeIdx) + if (!inner) { + return subtype !== undefined ? 0 : {} + } + if (subtype !== undefined) { + return inner.get(subtype) || 0 + } + const result: Record = {} + for (const [k, v] of inner.entries()) result[k] = v + return result + }, + + /** + * Top N subtypes for a NounType, sorted by count (descending). + * + * @param type - The NounType to rank subtypes within + * @param n - Maximum number of (subtype, count) pairs to return (default: 10) + * @returns Array of `[subtype, count]` tuples, highest count first + * + * @example + * const top3 = brain.counts.topSubtypes(NounType.Person, 3) + * // → [['customer', 847], ['employee', 56], ['vendor', 34]] + */ + topSubtypes: (type: NounType, n: number = 10): Array<[string, number]> => { + const subtypeMap = typeof (this.storage as any).getSubtypeCountsByType === 'function' + ? (this.storage as any).getSubtypeCountsByType() as Map> + : null + if (!subtypeMap) return [] + const typeIdx = TypeUtils.getNounIndex(type) + const inner = subtypeMap.get(typeIdx) + if (!inner) return [] + return Array.from(inner.entries()) + .sort((a, b) => b[1] - a[1]) + .slice(0, n) + }, + // Phase 1b: Get top N verb types by count topVerbTypes: (n: number = 10) => { return this.metadataIndex.getTopVerbTypes(n) @@ -6071,6 +6296,60 @@ export class Brainy implements BrainyInterface { return this.metadataIndex.getCountForCriteria(field, value) }, + /** + * Counts by value for a field registered via `brain.trackField()`. Reads + * from the backing `__fieldCounts__` aggregate (Layer 2 of the + * subtype-and-facets primitive). Backfill-on-define means the first call + * scans existing entities, subsequent calls are O(groups). + * + * Without `options.type`: returns the cross-NounType total per value. + * With `options.type`: returns per-value counts for that NounType only — + * requires the field to have been registered with `perType: true`. + * + * @param name - The tracked field name + * @param options.type - Optional NounType filter (requires perType registration) + * @returns `{ value: count }` map. Empty when the field wasn't tracked + * (no aggregate exists) or no entities have set it yet. + * @throws If `options.type` is passed but the field was not registered with `perType: true` + * + * @example + * brain.trackField('status', { perType: true }) + * await brain.add({ data: 'Ship it', type: NounType.Task, metadata: { status: 'todo' } }) + * await brain.counts.byField('status') + * // → { todo: 1 } + * await brain.counts.byField('status', { type: NounType.Task }) + * // → { todo: 1 } + */ + byField: async ( + name: string, + options?: { type?: NounType } + ): Promise> => { + const tracked = this._trackedFields.get(name) + if (!tracked) return {} + if (options?.type !== undefined && !tracked.perType) { + throw new Error( + `counts.byField('${name}'): per-type breakdown requested but the field was registered without perType:true. Re-call trackField('${name}', { perType: true }).` + ) + } + const aggregateName = this.fieldCountsAggregateName(name) + if (!this._aggregationIndex || !this._aggregationIndex.hasAggregate(aggregateName)) { + return {} + } + const rows = await this.queryAggregate(aggregateName) + const result: Record = {} + for (const row of rows) { + const value = row.groupKey?.[name] + // Skip the aggregation engine's "missing-value" sentinel: entities that + // don't have the tracked field at all (e.g. the VFS root) bucket under + // '__null__' and would otherwise pollute the count map. + if (value === undefined || value === null || value === '__null__') continue + if (options?.type !== undefined && row.groupKey?.['noun'] !== options.type) continue + const key = String(value) + result[key] = (result[key] || 0) + (typeof row.metrics?.count === 'number' ? row.metrics.count : row.count) + } + return result + }, + // Get all type counts as Map for performance-critical operations getAllTypeCounts: () => this.metadataIndex.getAllEntityCounts(), @@ -6132,6 +6411,255 @@ export class Brainy implements BrainyInterface { return this.counts.getStats(options) } + /** + * Distinct subtypes seen for a given NounType. + * + * Reads from the subtype-statistics rollup — no scan, no storage round-trip. + * The returned list is the vocabulary actually observed in the data, not a + * registered schema (Brainy doesn't validate subtype vocabulary; that's a + * consumer concern). + * + * @param type - The NounType to enumerate subtypes for + * @returns Sorted list of distinct subtype strings (empty if none) + * + * @example + * const personSubtypes = brain.subtypesOf(NounType.Person) + * // → ['customer', 'employee', 'vendor'] + */ + subtypesOf(type: NounType): string[] { + const subtypeMap = typeof (this.storage as any).getSubtypeCountsByType === 'function' + ? (this.storage as any).getSubtypeCountsByType() as Map> + : null + if (!subtypeMap) return [] + const typeIdx = TypeUtils.getNounIndex(type) + const inner = subtypeMap.get(typeIdx) + if (!inner) return [] + return Array.from(inner.keys()).sort() + } + + /** + * Stream-and-rewrite a field across every entity in the brain. + * + * Layer 3 of the subtype-and-facets primitive. Reads the value at `from`, + * writes it to `to`, and (unless `readBoth: true`) clears the source. Use this + * when migrating between field-name conventions or moving a value from + * `metadata.*` / `data.*` up to a top-level standard field like `subtype`. + * + * **Path forms:** + * - `'subtype'`, `'type'`, `'confidence'`, etc. — top-level standard fields + * (whatever appears in `STANDARD_ENTITY_FIELDS`) + * - `'metadata.X'` — a key under `entity.metadata` + * - `'data.X'` — a key under `entity.data` (when `data` is an object) + * - `'X'` (bare, not standard) — shorthand for `metadata.X` + * + * **Behavior:** + * - Streams entities in batches and rewrites in-place via `brain.update()`. + * Aggregations and indexes refresh automatically. + * - Entities where the source path is absent or where the target already + * holds the same value are skipped (idempotent — safe to re-run). + * - With `readBoth: true`, the source value is preserved alongside the new + * target so legacy consumers can keep querying the old path during a + * deprecation window. Re-run with `readBoth: false` when ready to clear. + * + * @param options.from - Source path + * @param options.to - Destination path + * @param options.readBoth - If true, leave the source value in place (default false → source cleared) + * @param options.batchSize - Entities per batch (default 100) + * @param options.onProgress - Optional callback invoked after each batch + * @returns Migration summary — counts and per-entity errors + * + * @example One-shot migration + * await brain.migrateField({ from: 'metadata.kind', to: 'subtype' }) + * // → { scanned: 1500, migrated: 1500, skipped: 0, errors: [] } + * + * @example Deprecation window — keep both fields readable + * await brain.migrateField({ from: 'data.kind', to: 'subtype', readBoth: true }) + * // ...consumers switch reads to subtype over time... + * await brain.migrateField({ from: 'data.kind', to: 'subtype' }) + * // ...source cleared in the final sweep. + */ + async migrateField(options: { + from: string + to: string + readBoth?: boolean + batchSize?: number + onProgress?: (progress: { scanned: number; migrated: number }) => void + }): Promise<{ + scanned: number + migrated: number + skipped: number + errors: Array<{ id: string; error: string }> + }> { + this.assertWritable('migrateField') + await this.ensureInitialized() + + if (!options.from || typeof options.from !== 'string') { + throw new Error('migrateField: `from` must be a non-empty string path') + } + if (!options.to || typeof options.to !== 'string') { + throw new Error('migrateField: `to` must be a non-empty string path') + } + if (options.from === options.to) { + throw new Error('migrateField: `from` and `to` are identical — nothing to do') + } + + const fromPath = this.parseMigrationPath(options.from) + const toPath = this.parseMigrationPath(options.to) + const batchSize = Math.max(1, options.batchSize ?? 100) + const readBoth = options.readBoth === true + + let scanned = 0 + let migrated = 0 + let skipped = 0 + const errors: Array<{ id: string; error: string }> = [] + let offset = 0 + + // Stream via storage pagination — same shape used by streaming.entities() + // when no filter is set. Avoids a full in-memory load on large brains. + while (true) { + const page = await this.storage.getNouns({ pagination: { offset, limit: batchSize } }) + if (page.items.length === 0) break + + for (const noun of page.items) { + scanned++ + try { + const entity = await this.convertNounToEntity(noun as any) + const sourceValue = this.readPath(entity, fromPath) + if (sourceValue === undefined || sourceValue === null) { + skipped++ + continue + } + const targetValue = this.readPath(entity, toPath) + const sourceCleared = readBoth || fromPath.kind === toPath.kind && fromPath.field === toPath.field + if (targetValue === sourceValue && (readBoth || !this.pathExists(entity, fromPath))) { + // Already migrated and (readBoth || source already gone) → no-op + skipped++ + continue + } + + const update = this.buildMigrationUpdate(entity, fromPath, toPath, sourceValue, readBoth) + await this.update(update) + migrated++ + void sourceCleared + } catch (err) { + errors.push({ + id: (noun as any).id ?? '', + error: err instanceof Error ? err.message : String(err) + }) + } + } + + if (options.onProgress) { + options.onProgress({ scanned, migrated }) + } + if (!page.hasMore) break + offset += page.items.length + } + + return { scanned, migrated, skipped, errors } + } + + /** + * Parse a dotted path used by `migrateField` into its routing kind + field name. + * `'subtype'` → top-level standard; `'metadata.X'` → metadata; `'data.X'` → data; + * bare non-standard names → metadata (matching `resolveEntityField`'s fallback). + */ + private parseMigrationPath(path: string): { kind: 'top' | 'metadata' | 'data'; field: string } { + const dotIdx = path.indexOf('.') + if (dotIdx === -1) { + if (STANDARD_ENTITY_FIELDS.has(path)) return { kind: 'top', field: path } + return { kind: 'metadata', field: path } + } + const head = path.slice(0, dotIdx) + const tail = path.slice(dotIdx + 1) + if (!tail) throw new Error(`migrateField: invalid path '${path}' (trailing dot)`) + if (head === 'metadata') return { kind: 'metadata', field: tail } + if (head === 'data') return { kind: 'data', field: tail } + throw new Error( + `migrateField: unsupported path prefix '${head}'. Supported: 'metadata.X', 'data.X', or a bare field name.` + ) + } + + /** Read the value at a parsed migration path. Returns `undefined` if absent. */ + private readPath(entity: Entity, path: { kind: 'top' | 'metadata' | 'data'; field: string }): unknown { + if (path.kind === 'top') return (entity as any)[path.field] + if (path.kind === 'metadata') { + const bag = entity.metadata as unknown as Record | undefined + return bag?.[path.field] + } + const data = entity.data + if (data && typeof data === 'object') { + return (data as Record)[path.field] + } + return undefined + } + + /** Whether a parsed path resolves to a defined (non-undefined) value. */ + private pathExists(entity: Entity, path: { kind: 'top' | 'metadata' | 'data'; field: string }): boolean { + return this.readPath(entity, path) !== undefined + } + + /** + * Build an `UpdateParams` payload that copies the value from the source path + * to the destination, and (unless `readBoth`) clears the source. Uses + * `merge: false` on the metadata bag when clearing so we can omit the source + * key authoritatively instead of relying on `undefined` round-trip behavior. + */ + private buildMigrationUpdate( + entity: Entity, + from: { kind: 'top' | 'metadata' | 'data'; field: string }, + to: { kind: 'top' | 'metadata' | 'data'; field: string }, + value: unknown, + readBoth: boolean + ): UpdateParams { + const id = entity.id + const update: UpdateParams = { id } + + // Apply destination write. + if (to.kind === 'top') { + ;(update as any)[to.field] = value + } else if (to.kind === 'metadata') { + const base = (entity.metadata as unknown as Record) ?? {} + const nextMeta: Record = { ...base, [to.field]: value } + if (!readBoth && from.kind === 'metadata') { + delete nextMeta[from.field] + } + update.metadata = nextMeta as T + update.merge = false + } else { + // data path — only meaningful when data is an object + const base = (entity.data && typeof entity.data === 'object') ? { ...(entity.data as Record) } : {} + base[to.field] = value + if (!readBoth && from.kind === 'data') { + delete base[from.field] + } + update.data = base + } + + // Apply source clear if not already handled by the destination bag write. + if (!readBoth) { + if (from.kind === 'metadata' && to.kind !== 'metadata') { + const base = (entity.metadata as unknown as Record) ?? {} + const nextMeta: Record = { ...base } + delete nextMeta[from.field] + update.metadata = nextMeta as T + update.merge = false + } else if (from.kind === 'data' && to.kind !== 'data') { + const base = (entity.data && typeof entity.data === 'object') ? { ...(entity.data as Record) } : null + if (base) { + delete base[from.field] + update.data = base + } + } else if (from.kind === 'top' && (from.field === 'subtype')) { + // Only subtype currently supports clearing at the top level via UpdateParams. + // Other standard fields aren't user-mutable through this path. + ;(update as any).subtype = undefined + } + } + + return update + } + // ============= NEW EMBEDDING & ANALYSIS APIs ============= /** diff --git a/src/coreTypes.ts b/src/coreTypes.ts index c5389c6d..a4203107 100644 --- a/src/coreTypes.ts +++ b/src/coreTypes.ts @@ -214,6 +214,14 @@ export interface HNSWNounWithMetadata { // TYPE (required, explicit) type: NounType + // SUBTYPE — optional per-product sub-classification within a NounType (e.g. a Person + // entity might have subtype 'employee' / 'customer' / 'vendor'; an Event might have + // subtype 'meeting' / 'milestone'). Flat string (no hierarchy) — consumers decide the + // vocabulary. Indexed and rolled up into per-NounType statistics so it's queryable + // (`find({ type, subtype })`) and aggregable (`groupBy:['subtype']`) on the standard-field + // fast path, never falling through to the metadata fallback. + subtype?: string + // QUALITY METRICS (top-level, explicit) confidence?: number weight?: number @@ -250,6 +258,7 @@ export const STANDARD_ENTITY_FIELDS: ReadonlySet = new Set([ 'connections', 'level', 'type', + 'subtype', 'confidence', 'weight', 'createdAt', diff --git a/src/storage/baseStorage.ts b/src/storage/baseStorage.ts index 7b33c807..60846572 100644 --- a/src/storage/baseStorage.ts +++ b/src/storage/baseStorage.ts @@ -224,6 +224,18 @@ export abstract class BaseStorage extends BaseStorageAdapter { protected nounCountsByType = new Uint32Array(NOUN_TYPE_COUNT) // 168 bytes (Stage 3: 42 types) protected verbCountsByType = new Uint32Array(VERB_TYPE_COUNT) // 508 bytes (Stage 3: 127 types) + /** + * Per-NounType-per-subtype counts. Outer key is the NounType index (matching + * `nounCountsByType` indexing); inner key is the subtype string. Populated + * incrementally as entities are saved and decremented on delete; persisted + * to `_system/subtype-statistics.json` alongside type-statistics. + * + * Memory: one Map entry per (type, subtype) pair actually used — typically + * tens of inner entries per NounType in production. Sparse by design — types + * with no subtype-bearing entities have no outer entry. + */ + protected subtypeCountsByType = new Map>() + /** * In-memory map from noun id → its `noun` (NounType) value, populated when * `saveNounMetadata_internal()` runs. `saveNoun_internal()` consults this @@ -238,6 +250,14 @@ export abstract class BaseStorage extends BaseStorageAdapter { * its lifetime and prunes on delete. */ protected nounTypeByIdCache = new Map() + + /** + * In-memory map from noun id → its `subtype` string (when set). Parallel to + * `nounTypeByIdCache`. Lets `deleteNounMetadata()` decrement the correct + * subtype bucket without re-reading metadata. Sparse — only ids with a + * non-empty subtype get an entry. + */ + protected nounSubtypeByIdCache = new Map() // Total: 676 bytes (99.2% reduction vs Map-based tracking) // Type caches REMOVED - ID-first paths eliminate need for type lookups! @@ -364,6 +384,7 @@ export abstract class BaseStorage extends BaseStorageAdapter { try { // Load type statistics from storage (if they exist) await this.loadTypeStatistics() + await this.loadSubtypeStatistics() // GraphAdjacencyIndex is now SINGLETON via getGraphIndex() // - Removed direct creation here to fix dual-ownership bug @@ -864,7 +885,7 @@ export abstract class BaseStorage extends BaseStorageAdapter { } // Combine into HNSWNounWithMetadata - Extract standard fields to top-level - const { noun, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata + const { noun, subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata return { id: vector.id, @@ -873,6 +894,7 @@ export abstract class BaseStorage extends BaseStorageAdapter { level: vector.level, // Standard fields at top-level type: (noun as NounType) || NounType.Thing, + subtype: subtype as string | undefined, createdAt: (createdAt as number) || Date.now(), updatedAt: (updatedAt as number) || Date.now(), confidence: confidence as number | undefined, @@ -901,12 +923,13 @@ export abstract class BaseStorage extends BaseStorageAdapter { for (const noun of nouns) { const metadata = await this.getNounMetadata(noun.id) if (metadata) { - const { noun: nounType, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata + const { noun: nounType, subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata nounsWithMetadata.push({ ...noun, // Standard fields at top-level type: (nounType as NounType) || NounType.Thing, + subtype: subtype as string | undefined, createdAt: (createdAt as number) || Date.now(), updatedAt: (updatedAt as number) || Date.now(), confidence: confidence as number | undefined, @@ -2202,6 +2225,27 @@ export abstract class BaseStorage extends BaseStorageAdapter { this.nounTypeByIdCache.set(id, metadata.noun as NounType) } + // Track subtype changes: on type or subtype change via update(), decrement + // the prior bucket before incrementing the new one. Symmetric with the + // delete-path decrement in `deleteNounMetadata()`. + const priorSubtype = this.nounSubtypeByIdCache.get(id) + const priorTypeForSubtype = isNew ? undefined : (existingMetadata?.noun as NounType | undefined) + const newSubtype = typeof metadata.subtype === 'string' && metadata.subtype.length > 0 + ? metadata.subtype as string + : undefined + const newType = metadata.noun as NounType | undefined + + if (priorSubtype && priorTypeForSubtype && (priorSubtype !== newSubtype || priorTypeForSubtype !== newType)) { + this.decrementSubtypeCount(priorTypeForSubtype, priorSubtype) + } + if (newSubtype && newType && (isNew || priorSubtype !== newSubtype || priorTypeForSubtype !== newType)) { + this.incrementSubtypeCount(newType, newSubtype) + this.nounSubtypeByIdCache.set(id, newSubtype) + } else if (!newSubtype && priorSubtype) { + // Subtype cleared by an update — drop the cache entry (decrement already done above) + this.nounSubtypeByIdCache.delete(id) + } + // CRITICAL FIX: Increment count for new entities // This runs AFTER metadata is saved, guaranteeing type information is available // Uses synchronous increment since storage operations are already serialized @@ -2385,7 +2429,7 @@ export abstract class BaseStorage extends BaseStorageAdapter { const noun = this.deserializeNoun(vectorData) // Extract standard fields to top-level - const { noun: nounType, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadataData + const { noun: nounType, subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadataData results.set(id, { id: noun.id, @@ -2394,6 +2438,7 @@ export abstract class BaseStorage extends BaseStorageAdapter { level: noun.level, // Standard fields at top-level type: (nounType as NounType) || NounType.Thing, + subtype: subtype as string | undefined, createdAt: (createdAt as number) || Date.now(), updatedAt: (updatedAt as number) || Date.now(), confidence: confidence as number | undefined, @@ -2602,6 +2647,13 @@ export abstract class BaseStorage extends BaseStorageAdapter { if (this.nounCountsByType[idx] > 0) { this.nounCountsByType[idx]-- } + + // Symmetric subtype decrement + const priorSubtype = this.nounSubtypeByIdCache.get(id) + if (priorSubtype) { + this.nounSubtypeByIdCache.delete(id) + this.decrementSubtypeCount(priorType, priorSubtype) + } } } @@ -2798,6 +2850,121 @@ export abstract class BaseStorage extends BaseStorageAdapter { await this.writeObjectToPath(`${SYSTEM_DIR}/type-statistics.json`, stats) } + /** + * Increment the (type, subtype) count, creating the inner map on first use. + * Indexed by NounType index so it lines up with `nounCountsByType` and can + * be reduced into per-type totals without re-keying. + */ + protected incrementSubtypeCount(type: NounType, subtype: string): void { + const typeIdx = TypeUtils.getNounIndex(type) + if (typeIdx < 0) return + let inner = this.subtypeCountsByType.get(typeIdx) + if (!inner) { + inner = new Map() + this.subtypeCountsByType.set(typeIdx, inner) + } + inner.set(subtype, (inner.get(subtype) || 0) + 1) + } + + /** + * Decrement the (type, subtype) count. Deletes the inner key when it reaches + * 0 and the outer entry when its inner map is empty, so the persisted shape + * stays compact across heavy churn. + */ + protected decrementSubtypeCount(type: NounType, subtype: string): void { + const typeIdx = TypeUtils.getNounIndex(type) + if (typeIdx < 0) return + const inner = this.subtypeCountsByType.get(typeIdx) + if (!inner) return + const next = (inner.get(subtype) || 0) - 1 + if (next <= 0) { + inner.delete(subtype) + if (inner.size === 0) this.subtypeCountsByType.delete(typeIdx) + } else { + inner.set(subtype, next) + } + } + + /** + * Load `_system/subtype-statistics.json` into `subtypeCountsByType`. + * Persisted shape: `{ counts: { [typeIdx]: { [subtype]: count } }, updatedAt }`. + * Missing file or parse error → start from empty (matches loadTypeStatistics). + */ + protected async loadSubtypeStatistics(): Promise { + try { + const stats = await this.readObjectFromPath(`${SYSTEM_DIR}/subtype-statistics.json`) + if (stats && stats.counts && typeof stats.counts === 'object') { + this.subtypeCountsByType.clear() + for (const [typeKey, subtypeMap] of Object.entries(stats.counts as Record>)) { + const typeIdx = Number(typeKey) + if (!Number.isInteger(typeIdx) || typeIdx < 0 || typeIdx >= NOUN_TYPE_COUNT) continue + const inner = new Map() + for (const [subtype, count] of Object.entries(subtypeMap)) { + if (typeof count === 'number' && count > 0) inner.set(subtype, count) + } + if (inner.size > 0) this.subtypeCountsByType.set(typeIdx, inner) + } + } + } catch { + // No existing subtype statistics, starting fresh. + } + } + + /** + * Save subtype statistics to storage. Mirrors the type-statistics persistence + * cadence (called from `flushCounts()` and the periodic save in + * `saveNoun_internal`). + */ + protected async saveSubtypeStatistics(): Promise { + const counts: Record> = {} + for (const [typeIdx, inner] of this.subtypeCountsByType.entries()) { + const innerObj: Record = {} + for (const [subtype, count] of inner.entries()) innerObj[subtype] = count + counts[String(typeIdx)] = innerObj + } + await this.writeObjectToPath(`${SYSTEM_DIR}/subtype-statistics.json`, { + counts, + updatedAt: Date.now() + }) + } + + /** + * Rebuild subtype counts from on-disk metadata. Companion to `rebuildTypeCounts()` + * — used for poison recovery and explicit repair via `brainy inspect repair`. + * O(N) over all nouns. + */ + public async rebuildSubtypeCounts(): Promise { + prodLog.info('[BaseStorage] Rebuilding subtype counts from storage...') + this.subtypeCountsByType.clear() + this.nounSubtypeByIdCache.clear() + + for (let shard = 0; shard < 256; shard++) { + const shardHex = shard.toString(16).padStart(2, '0') + const shardDir = `entities/nouns/${shardHex}` + try { + const paths = await this.listObjectsInBranch(shardDir) + for (const path of paths) { + if (!path.includes('/metadata.json')) continue + try { + const metadata = await this.readWithInheritance(path) + if (metadata && metadata.noun && typeof metadata.subtype === 'string' && metadata.subtype.length > 0) { + this.incrementSubtypeCount(metadata.noun as NounType, metadata.subtype) + // Path is `entities/nouns///metadata.json` — extract id segment. + const segments = path.split('/') + const idSeg = segments[segments.length - 2] + if (idSeg) this.nounSubtypeByIdCache.set(idSeg, metadata.subtype) + } + } catch { /* skip unreadable entities */ } + } + } catch { /* skip missing shards */ } + } + + await this.saveSubtypeStatistics() + const totals = Array.from(this.subtypeCountsByType.values()) + .reduce((sum, inner) => sum + Array.from(inner.values()).reduce((s, n) => s + n, 0), 0) + prodLog.info(`[BaseStorage] Rebuilt subtype counts: ${totals} entities across ${this.subtypeCountsByType.size} NounTypes`) + } + /** * Persist both counter systems atomically when an explicit flush is * requested. `super.flushCounts()` writes `entityCounts` (the Map in @@ -2814,6 +2981,7 @@ export abstract class BaseStorage extends BaseStorageAdapter { public async flushCounts(): Promise { await super.flushCounts() await this.saveTypeStatistics() + await this.saveSubtypeStatistics() } /** @@ -2825,6 +2993,17 @@ export abstract class BaseStorage extends BaseStorageAdapter { return this.nounCountsByType } + /** + * Get subtype counts by NounType (O(1) access to subtype statistics). + * Returned map is the live in-memory view — callers must treat it as + * read-only. Outer key: NounType index. Inner map: subtype → count. + * + * @returns Map keyed by NounType index → Map of subtype → count + */ + public getSubtypeCountsByType(): Map> { + return this.subtypeCountsByType + } + /** * Get verb counts by type (O(1) access to type statistics) * Exposed for MetadataIndexManager to use as single source of truth diff --git a/src/types/brainy.types.ts b/src/types/brainy.types.ts index 11755c0e..5005f792 100644 --- a/src/types/brainy.types.ts +++ b/src/types/brainy.types.ts @@ -26,6 +26,13 @@ export interface Entity { vector: Vector /** Entity type classification (NounType enum) */ type: NounType + /** + * Per-product sub-classification within the NounType (e.g. a `Person` entity might + * have `subtype: 'employee'` or `'customer'`; a `Document` might have `subtype: 'invoice'`). + * Flat string, no hierarchy. Top-level standard field — indexed on the fast path and + * rolled into per-NounType statistics. + */ + subtype?: string /** Opaque content — used for embeddings and semantic search. Not indexed by MetadataIndex. */ data?: any /** User-defined structured fields — indexed and queryable via `where` filters. */ @@ -105,6 +112,7 @@ export interface Result { // Convenience: Common entity fields flattened to top level type?: NounType // Entity type (from entity.type) + subtype?: string // Per-product sub-classification (from entity.subtype) metadata?: T // Entity metadata (from entity.metadata) data?: any // Entity data (from entity.data) confidence?: number // Type classification confidence (from entity.confidence) @@ -158,6 +166,14 @@ export interface AddParams { data: any | Vector /** Entity type classification (required) */ type: NounType + /** + * Per-product sub-classification within the NounType (e.g. a `Person` entity might have + * `subtype: 'employee'` or `'customer'`; a `Document` might have `subtype: 'invoice'`). + * Flat string, no hierarchy — consumers choose the vocabulary. Indexed and rolled up into + * per-NounType statistics for fast `find({ type, subtype })` filtering and `groupBy:['subtype']` + * aggregation on the standard-field fast path. + */ + subtype?: string /** Structured queryable fields — indexed by MetadataIndex, used in `where` filters */ metadata?: T /** Custom entity ID (auto-generated UUID v4 if not provided) */ @@ -181,6 +197,7 @@ export interface UpdateParams { id: string // Entity to update data?: any // New content to re-embed type?: NounType // Change type + subtype?: string // Change subtype (set to '' or null-equivalent via dedicated unset is future work) metadata?: Partial // Metadata to update merge?: boolean // Merge or replace metadata (default: true) vector?: Vector // New pre-computed vector @@ -252,6 +269,13 @@ export interface FindParams { // Metadata Intelligence /** Filter by entity type(s). Alias for `where.noun`. */ type?: NounType | NounType[] + /** + * Filter by per-product subtype (top-level standard field — uses the fast path, not the + * metadata fallback). Pass a single string for equality, or an array for set membership + * (e.g. `subtype: ['employee', 'contractor']`). For operator-form predicates (e.g. + * `{ exists: true }`, `{ missing: true }`) use `where: { subtype: { …operators… } }`. + */ + subtype?: string | string[] /** Metadata filters using BFO operators (e.g., `{ year: { greaterThan: 2020 } }`) */ where?: Partial diff --git a/src/utils/paramValidation.ts b/src/utils/paramValidation.ts index ee7e704c..ffc2bd91 100644 --- a/src/utils/paramValidation.ts +++ b/src/utils/paramValidation.ts @@ -373,7 +373,15 @@ export function validateUpdateParams(params: UpdateParams): void { } // Universal truth: must update something - if (!params.data && !params.metadata && !params.type && !params.vector) { + if ( + !params.data && + !params.metadata && + !params.type && + !params.vector && + params.subtype === undefined && + params.confidence === undefined && + params.weight === undefined + ) { throw new Error('must specify at least one field to update') } diff --git a/tests/integration/subtype-and-facets.test.ts b/tests/integration/subtype-and-facets.test.ts new file mode 100644 index 00000000..d17c681a --- /dev/null +++ b/tests/integration/subtype-and-facets.test.ts @@ -0,0 +1,333 @@ +/** + * @module tests/integration/subtype-and-facets + * @description Integration coverage for the 7.29.0 subtype-and-facets primitive + * (Layer 1 top-level `subtype` field + rollup + counts API, Layer 2 + * `brain.trackField()`, Layer 3 `brain.migrateField()`). + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { Brainy } from '../../src/brainy' +import { NounType } from '../../src/types/graphTypes' + +describe('Subtype + Facets (7.29.0)', () => { + let brain: Brainy + + beforeEach(async () => { + brain = new Brainy({ storage: { type: 'memory' }, silent: true }) + await brain.init() + }) + + afterEach(async () => { + await brain.close() + }) + + describe('Layer 1: subtype as top-level standard field', () => { + it('persists subtype on add() and round-trips through get()', async () => { + const id = await brain.add({ + data: 'Avery operates the AI lab', + type: NounType.Person, + subtype: 'operator', + metadata: { department: 'ai-lab' } + }) + + const entity = await brain.get(id) + expect(entity).not.toBeNull() + expect(entity!.subtype).toBe('operator') + expect(entity!.type).toBe(NounType.Person) + // subtype lives at top-level, NOT in metadata + expect((entity!.metadata as any).subtype).toBeUndefined() + expect((entity!.metadata as any).department).toBe('ai-lab') + }) + + it('allows omitting subtype (optional field)', async () => { + const id = await brain.add({ data: 'plain entity', type: NounType.Thing }) + const entity = await brain.get(id) + expect(entity!.subtype).toBeUndefined() + }) + + it('updates subtype via update() without disturbing other fields', async () => { + const id = await brain.add({ + data: 'switched roles', + type: NounType.Person, + subtype: 'contractor', + metadata: { department: 'eng' } + }) + + await brain.update({ id, subtype: 'employee' }) + const entity = await brain.get(id) + expect(entity!.subtype).toBe('employee') + expect((entity!.metadata as any).department).toBe('eng') + }) + + it('preserves existing subtype when update() omits it', async () => { + const id = await brain.add({ + data: 'unchanged subtype', + type: NounType.Person, + subtype: 'employee' + }) + await brain.update({ id, metadata: { tag: 'updated' } }) + const entity = await brain.get(id) + expect(entity!.subtype).toBe('employee') + }) + + it('find({ type, subtype }) hits the fast path and matches', async () => { + await brain.add({ data: 'a', type: NounType.Person, subtype: 'employee' }) + await brain.add({ data: 'b', type: NounType.Person, subtype: 'employee' }) + await brain.add({ data: 'c', type: NounType.Person, subtype: 'customer' }) + await brain.add({ data: 'd', type: NounType.Document, subtype: 'employee' }) + + const employees = await brain.find({ type: NounType.Person, subtype: 'employee' }) + expect(employees).toHaveLength(2) + for (const r of employees) { + expect(r.subtype).toBe('employee') + expect(r.type).toBe(NounType.Person) + } + }) + + it('find({ subtype: [...] }) matches set membership', async () => { + await brain.add({ data: 'a', type: NounType.Person, subtype: 'employee' }) + await brain.add({ data: 'b', type: NounType.Person, subtype: 'contractor' }) + await brain.add({ data: 'c', type: NounType.Person, subtype: 'customer' }) + + const both = await brain.find({ + type: NounType.Person, + subtype: ['employee', 'contractor'] + }) + expect(both).toHaveLength(2) + const subtypes = both.map(r => r.subtype).sort() + expect(subtypes).toEqual(['contractor', 'employee']) + }) + + it('Result objects expose subtype at the top level', async () => { + await brain.add({ data: 'a', type: NounType.Person, subtype: 'customer' }) + const results = await brain.find({ subtype: 'customer' }) + expect(results[0].subtype).toBe('customer') + expect(results[0].entity?.subtype).toBe('customer') + }) + }) + + describe('Layer 1: counts API', () => { + beforeEach(async () => { + await brain.add({ data: 'a', type: NounType.Person, subtype: 'employee' }) + await brain.add({ data: 'b', type: NounType.Person, subtype: 'employee' }) + await brain.add({ data: 'c', type: NounType.Person, subtype: 'customer' }) + await brain.add({ data: 'd', type: NounType.Person, subtype: 'customer' }) + await brain.add({ data: 'e', type: NounType.Person, subtype: 'customer' }) + await brain.add({ data: 'f', type: NounType.Document, subtype: 'invoice' }) + }) + + it('counts.bySubtype(type) returns the full breakdown', async () => { + const counts = brain.counts.bySubtype(NounType.Person) + expect(counts).toEqual({ employee: 2, customer: 3 }) + }) + + it('counts.bySubtype(type, subtype) returns the O(1) point count', async () => { + expect(brain.counts.bySubtype(NounType.Person, 'employee')).toBe(2) + expect(brain.counts.bySubtype(NounType.Person, 'customer')).toBe(3) + expect(brain.counts.bySubtype(NounType.Person, 'never-existed')).toBe(0) + }) + + it('counts.topSubtypes ranks by count, respects N', async () => { + const top1 = brain.counts.topSubtypes(NounType.Person, 1) + expect(top1).toEqual([['customer', 3]]) + + const all = brain.counts.topSubtypes(NounType.Person, 10) + expect(all).toEqual([ + ['customer', 3], + ['employee', 2] + ]) + }) + + it('brain.subtypesOf(type) returns sorted distinct subtypes', async () => { + expect(brain.subtypesOf(NounType.Person)).toEqual(['customer', 'employee']) + expect(brain.subtypesOf(NounType.Document)).toEqual(['invoice']) + }) + + it('rollup decrements on delete', async () => { + const id = await brain.add({ data: 'g', type: NounType.Person, subtype: 'employee' }) + expect(brain.counts.bySubtype(NounType.Person, 'employee')).toBe(3) + + await brain.delete(id) + expect(brain.counts.bySubtype(NounType.Person, 'employee')).toBe(2) + }) + + it('rollup re-routes on subtype change via update', async () => { + const id = await brain.add({ data: 'h', type: NounType.Person, subtype: 'employee' }) + expect(brain.counts.bySubtype(NounType.Person, 'employee')).toBe(3) + expect(brain.counts.bySubtype(NounType.Person, 'customer')).toBe(3) + + await brain.update({ id, subtype: 'customer' }) + expect(brain.counts.bySubtype(NounType.Person, 'employee')).toBe(2) + expect(brain.counts.bySubtype(NounType.Person, 'customer')).toBe(4) + }) + }) + + describe('Layer 2: trackField() + counts.byField()', () => { + it('tracks a metadata field and returns value frequencies', async () => { + brain.trackField('status') + await brain.add({ data: 'a', type: NounType.Task, metadata: { status: 'todo' } }) + await brain.add({ data: 'b', type: NounType.Task, metadata: { status: 'todo' } }) + await brain.add({ data: 'c', type: NounType.Task, metadata: { status: 'done' } }) + await brain.add({ data: 'd', type: NounType.Task, metadata: { status: 'done' } }) + await brain.add({ data: 'e', type: NounType.Task, metadata: { status: 'done' } }) + + const byStatus = await brain.counts.byField('status') + expect(byStatus).toEqual({ todo: 2, done: 3 }) + }) + + it('perType breakdown returns per-NounType counts when filtered', async () => { + brain.trackField('status', { perType: true }) + await brain.add({ data: 'a', type: NounType.Task, metadata: { status: 'todo' } }) + await brain.add({ data: 'b', type: NounType.Task, metadata: { status: 'done' } }) + await brain.add({ data: 'c', type: NounType.Event, metadata: { status: 'todo' } }) + + const taskStatuses = await brain.counts.byField('status', { type: NounType.Task }) + expect(taskStatuses).toEqual({ todo: 1, done: 1 }) + }) + + it('throws when per-type breakdown requested but field tracked without perType', async () => { + brain.trackField('status') + await expect( + brain.counts.byField('status', { type: NounType.Task }) + ).rejects.toThrow(/perType/) + }) + + it('returns {} for fields never registered', async () => { + const result = await brain.counts.byField('unregistered') + expect(result).toEqual({}) + }) + + it('values whitelist rejects off-vocabulary writes', async () => { + brain.trackField('priority', { values: ['low', 'medium', 'high'] }) + await expect( + brain.add({ data: 'bad', type: NounType.Task, metadata: { priority: 'urgent' } }) + ).rejects.toThrow(/priority.*urgent.*vocabulary/i) + }) + + it('values whitelist accepts on-vocabulary writes', async () => { + brain.trackField('priority', { values: ['low', 'medium', 'high'] }) + const id = await brain.add({ + data: 'ok', + type: NounType.Task, + metadata: { priority: 'high' } + }) + const entity = await brain.get(id) + expect((entity!.metadata as any).priority).toBe('high') + }) + }) + + describe('Layer 3: migrateField()', () => { + it('migrates metadata.X → top-level subtype and clears the source', async () => { + // Set up the legacy shape: store kind inside metadata. + const id1 = await brain.add({ + data: 'a', + type: NounType.Person, + metadata: { kind: 'employee', department: 'eng' } + }) + const id2 = await brain.add({ + data: 'b', + type: NounType.Person, + metadata: { kind: 'customer' } + }) + + const result = await brain.migrateField({ + from: 'metadata.kind', + to: 'subtype' + }) + expect(result.errors).toEqual([]) + expect(result.migrated).toBe(2) + expect(result.scanned).toBeGreaterThanOrEqual(2) + + const e1 = await brain.get(id1) + expect(e1!.subtype).toBe('employee') + expect((e1!.metadata as any).kind).toBeUndefined() + expect((e1!.metadata as any).department).toBe('eng') + + const e2 = await brain.get(id2) + expect(e2!.subtype).toBe('customer') + expect((e2!.metadata as any).kind).toBeUndefined() + }) + + it('readBoth: true preserves the source field for the deprecation window', async () => { + const id = await brain.add({ + data: 'a', + type: NounType.Person, + metadata: { kind: 'employee' } + }) + + const result = await brain.migrateField({ + from: 'metadata.kind', + to: 'subtype', + readBoth: true + }) + expect(result.migrated).toBe(1) + + const entity = await brain.get(id) + expect(entity!.subtype).toBe('employee') + expect((entity!.metadata as any).kind).toBe('employee') + }) + + it('is idempotent — re-running on already-migrated data is a no-op', async () => { + await brain.add({ + data: 'a', + type: NounType.Person, + metadata: { kind: 'employee' } + }) + + await brain.migrateField({ from: 'metadata.kind', to: 'subtype' }) + const second = await brain.migrateField({ from: 'metadata.kind', to: 'subtype' }) + // All entities have already migrated → every entity is skipped on the second pass + expect(second.migrated).toBe(0) + expect(second.skipped).toBeGreaterThan(0) + }) + + it('migrates data.X → subtype (when data is an object)', async () => { + const id = await brain.add({ + data: { description: 'thing', kind: 'character' } as any, + type: NounType.Person + }) + + const result = await brain.migrateField({ from: 'data.kind', to: 'subtype' }) + expect(result.migrated).toBe(1) + expect(result.errors).toEqual([]) + + const entity = await brain.get(id) + expect(entity!.subtype).toBe('character') + expect((entity!.data as any).description).toBe('thing') + expect((entity!.data as any).kind).toBeUndefined() + }) + + it('throws on identical from/to', async () => { + await expect( + brain.migrateField({ from: 'subtype', to: 'subtype' }) + ).rejects.toThrow(/identical/) + }) + + it('rejects unsupported path prefixes', async () => { + await expect( + brain.migrateField({ from: 'metadata.kind', to: 'connections.foo' }) + ).rejects.toThrow(/unsupported path prefix/) + }) + + it('migrates with the rollup tracking the new subtype counts', async () => { + await brain.add({ + data: 'a', + type: NounType.Person, + metadata: { kind: 'employee' } + }) + await brain.add({ + data: 'b', + type: NounType.Person, + metadata: { kind: 'customer' } + }) + + await brain.migrateField({ from: 'metadata.kind', to: 'subtype' }) + + // Rollup should now reflect the new subtype values + expect(brain.counts.bySubtype(NounType.Person)).toEqual({ + employee: 1, + customer: 1 + }) + }) + }) +})