feat: subtype top-level field + trackField + migrateField
Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).
Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
BaseStorage, mirrored after nounCountsByType with the same self-heal
rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path
Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
cardinality + per-NounType breakdown stats. Backed by the aggregation
engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update
Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
streams every entity, copies the value from one path to another, and
(unless readBoth) clears the source. Supports top-level standard fields,
metadata.X, and data.X paths. Idempotent — safe to re-run.
Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
quick-start all treat subtype as a core primitive with anonymous example
vocabularies (employee/customer/invoice/milestone).
Tests
- 26 new integration tests covering write/read/update/delete round-trips,
counts rollup decrement + re-route on mutation, trackField + byField
with and without perType, vocabulary whitelist enforcement, and
migrateField for metadata.X → subtype and data.X → subtype paths
including readBoth deprecation-window semantics.
Unit suite: 1468/1468 passing. Type-check + build clean.
This commit is contained in:
parent
e47fea0917
commit
2cdf70ee0f
14 changed files with 1698 additions and 21 deletions
21
README.md
21
README.md
|
|
@ -247,6 +247,27 @@ Plugins are opt-in. Brainy never auto-imports packages unless listed in `plugins
|
|||
|
||||
Model any domain — healthcare (`Patient → diagnoses → Condition`), finance (`Account → transfers → Transaction`), education (`Student → completes → Course`), or your own.
|
||||
|
||||
### Subtypes — sub-classification within a NounType
|
||||
|
||||
The 42 NounTypes are intentionally coarse. Use the top-level `subtype` field to sub-classify entities within a type — flat string, no hierarchy, your choice of vocabulary:
|
||||
|
||||
```javascript
|
||||
await brain.add({
|
||||
data: 'Avery Brooks — runs the AI lab',
|
||||
type: NounType.Person,
|
||||
subtype: 'employee' // 'customer', 'vendor', 'contractor', …
|
||||
})
|
||||
|
||||
// Filter on the fast path — column-store hit, not metadata fallback:
|
||||
const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })
|
||||
|
||||
// O(1) counts via the persisted rollup:
|
||||
brain.counts.bySubtype(NounType.Person)
|
||||
// → { employee: 12, customer: 847, vendor: 34 }
|
||||
```
|
||||
|
||||
For other facets you want counted (`status`, `source`, `role`), register them with `brain.trackField(name)`. Renaming an existing convention to `subtype`? Use `brain.migrateField({from, to})`. Full guide: **[Subtypes & Facets](docs/guides/subtypes-and-facets.md)**.
|
||||
|
||||
**[Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)** | **[Stage 3 Canonical Reference](docs/STAGE3-CANONICAL-TAXONOMY.md)**
|
||||
|
||||
---
|
||||
|
|
|
|||
133
RELEASES.md
133
RELEASES.md
|
|
@ -11,6 +11,139 @@ Collective. The SDK wraps it — most products never call Brainy directly. Read
|
|||
|
||||
---
|
||||
|
||||
## v7.29.0 — 2026-06-04
|
||||
|
||||
**Affected products:** anyone modeling entities with per-product sub-classification — every
|
||||
consumer that's been reaching for `metadata.kind`, a custom `metadata.subtype` field, a
|
||||
`data.kind` shape inside the payload, or similar ad-hoc conventions. Additive; drop-in from
|
||||
7.28.x. No deprecations.
|
||||
|
||||
### New — `subtype` promoted to a top-level standard field
|
||||
|
||||
`subtype?: string` is now a first-class standard field on every entity, alongside `type` /
|
||||
`confidence` / `weight`. Use it as the platform-standard primitive for sub-classifying entities
|
||||
within a NounType (`Person` → `'employee'` / `'customer'`; `Document` → `'invoice'` / `'contract'`;
|
||||
`Event` → `'milestone'` / `'meeting'`):
|
||||
|
||||
```typescript
|
||||
await brain.add({
|
||||
data: 'Avery Brooks — runs the AI lab',
|
||||
type: NounType.Person,
|
||||
subtype: 'employee', // top-level write param
|
||||
metadata: { department: 'ai-lab' }
|
||||
})
|
||||
|
||||
// Fast-path filter — column-store hit, not metadata fallback:
|
||||
const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })
|
||||
|
||||
// Set membership:
|
||||
const internal = await brain.find({
|
||||
type: NounType.Person,
|
||||
subtype: ['employee', 'contractor']
|
||||
})
|
||||
```
|
||||
|
||||
Flat string, no hierarchy — your vocabulary, your choice. Brainy stores and counts, never validates.
|
||||
|
||||
### New — O(1) subtype counts via the persisted rollup
|
||||
|
||||
A new `_system/subtype-statistics.json` rollup is maintained incrementally as entities are added,
|
||||
updated, and deleted — mirroring the existing `nounCountsByType` machinery with the same self-heal
|
||||
behavior on poison detection. Counts are O(1) at billion scale:
|
||||
|
||||
```typescript
|
||||
brain.counts.bySubtype(NounType.Person)
|
||||
// → { employee: 12, customer: 847, vendor: 34 }
|
||||
|
||||
brain.counts.bySubtype(NounType.Person, 'employee') // O(1) point count
|
||||
// → 12
|
||||
|
||||
brain.counts.topSubtypes(NounType.Person, 3)
|
||||
// → [['customer', 847], ['employee', 12], ['vendor', 34]]
|
||||
|
||||
brain.subtypesOf(NounType.Person)
|
||||
// → ['customer', 'employee', 'vendor']
|
||||
```
|
||||
|
||||
### New — `brain.trackField()` for other metadata facets
|
||||
|
||||
For facets that aren't the *primary* sub-classification (`status`, `source`, `role`, `paradigm`),
|
||||
`trackField` registers a field for cardinality + per-NounType breakdown stats without promoting
|
||||
each one to a top-level slot. Piggybacks on the existing aggregation engine, so backfill-on-define
|
||||
applies — registering on a populated brain scans existing entities on the first query:
|
||||
|
||||
```typescript
|
||||
brain.trackField('status', { perType: true })
|
||||
|
||||
await brain.counts.byField('status')
|
||||
// → { todo: 12, doing: 3, done: 47 }
|
||||
|
||||
await brain.counts.byField('status', { type: NounType.Task })
|
||||
// → { todo: 8, doing: 2, done: 30 }
|
||||
|
||||
// Opt-in vocabulary validation:
|
||||
brain.trackField('priority', { values: ['low', 'medium', 'high'] })
|
||||
// brain.add({ ..., metadata: { priority: 'urgent' } }) → throws
|
||||
```
|
||||
|
||||
### New — generic `brain.migrateField()` for one-shot rewrites
|
||||
|
||||
Streams every entity and copies a value from one field path to another. Supports top-level
|
||||
standard fields, `metadata.*`, and `data.*` paths; idempotent; with optional `readBoth: true`
|
||||
deprecation window that preserves the source field alongside the new one:
|
||||
|
||||
```typescript
|
||||
// Phase 1: dual-populate (legacy readers still work)
|
||||
await brain.migrateField({
|
||||
from: 'metadata.kind',
|
||||
to: 'subtype',
|
||||
readBoth: true
|
||||
})
|
||||
|
||||
// ...readers migrate to subtype at their own pace...
|
||||
|
||||
// Phase 2: clear the source field
|
||||
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
|
||||
// → { scanned: 1500, migrated: 1500, skipped: 0, errors: [] }
|
||||
```
|
||||
|
||||
Supports `batchSize` and `onProgress` for large brains.
|
||||
|
||||
### Aggregation composition
|
||||
|
||||
`subtype` is a standard-field group-by dimension out of the box — no `where:` wrapper, no
|
||||
metadata-fallback overhead:
|
||||
|
||||
```typescript
|
||||
await brain.find({
|
||||
aggregate: {
|
||||
groupBy: ['type', 'subtype'],
|
||||
metrics: { count: { op: 'COUNT' } }
|
||||
}
|
||||
})
|
||||
// [
|
||||
// { groupKey: { type: 'person', subtype: 'customer' }, count: 847 },
|
||||
// { groupKey: { type: 'person', subtype: 'employee' }, count: 12 },
|
||||
// { groupKey: { type: 'document', subtype: 'invoice' }, count: 2103 },
|
||||
// ...
|
||||
// ]
|
||||
```
|
||||
|
||||
### Cortex compatibility
|
||||
|
||||
Subtype works under Cortex out of the box via auto-field-indexing. Cortex will land its own
|
||||
native-side query-planner recognition + `subtypeCountsByType` native rollup in the next Cortex
|
||||
release for full parity with `type`'s fast path. Not a Brainy blocker — subtype reads/writes
|
||||
function correctly with current Cortex.
|
||||
|
||||
### Docs
|
||||
|
||||
Full guide: `docs/guides/subtypes-and-facets.md`. Subtype is documented as a core primitive
|
||||
throughout `README.md`, `docs/DATA_MODEL.md`, `docs/architecture/finite-type-system.md`,
|
||||
`docs/api/README.md`, and `docs/QUERY_OPERATORS.md`.
|
||||
|
||||
---
|
||||
|
||||
## v7.24.0 — 2026-05-26
|
||||
|
||||
**Affected products:** aggregation/reporting users + anyone calling `extractEntities` on
|
||||
|
|
|
|||
|
|
@ -186,6 +186,7 @@ When you add an entity, Brainy stores these standard fields in the metadata obje
|
|||
| Field | Set By | Description |
|
||||
|-------|--------|-------------|
|
||||
| `noun` | System | Entity type (NounType enum value) |
|
||||
| `subtype` | User | Per-NounType sub-classification (e.g. `'employee'`, `'invoice'`, `'milestone'`). Flat string, no hierarchy. Indexed on the fast path and rolled into per-NounType statistics. |
|
||||
| `data` | System | The raw `data` value (stored opaquely) |
|
||||
| `createdAt` | System | Creation timestamp |
|
||||
| `updatedAt` | System | Last update timestamp |
|
||||
|
|
@ -196,6 +197,30 @@ When you add an entity, Brainy stores these standard fields in the metadata obje
|
|||
|
||||
On read, these standard fields are extracted to top-level Entity properties. The `metadata` field on the returned Entity contains **only your custom fields**.
|
||||
|
||||
### Subtype — sub-classification within a NounType
|
||||
|
||||
`type` (NounType) is a stable 42-value enum. `subtype` is the consumer-chosen string vocabulary *within* a type:
|
||||
|
||||
```typescript
|
||||
// A Person who is an employee:
|
||||
await brain.add({
|
||||
data: 'Avery Brooks — runs the AI lab',
|
||||
type: NounType.Person,
|
||||
subtype: 'employee',
|
||||
metadata: { department: 'ai-lab' }
|
||||
})
|
||||
|
||||
// A Document that is an invoice:
|
||||
await brain.add({
|
||||
data: 'INV-2026-001',
|
||||
type: NounType.Document,
|
||||
subtype: 'invoice',
|
||||
metadata: { amount: 1500 }
|
||||
})
|
||||
```
|
||||
|
||||
`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's how `find({ type, subtype })` routes through the standard-field fast path (column-store hit) instead of the metadata fallback. See **[Subtypes & Facets](./guides/subtypes-and-facets.md)** for the full guide including `trackField()` and `migrateField()`.
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
|
|
|||
|
|
@ -194,6 +194,26 @@ brain.find({ where: { noun: NounType.Person } })
|
|||
brain.find({ type: [NounType.Person, NounType.Agent] })
|
||||
```
|
||||
|
||||
### Filter by subtype
|
||||
|
||||
`subtype` is a top-level standard field — takes the column-store fast path, not the metadata fallback. Pair with `type` for the typical "Person who is an employee" query:
|
||||
|
||||
```typescript
|
||||
// Equality on subtype:
|
||||
brain.find({ type: NounType.Person, subtype: 'employee' })
|
||||
|
||||
// Set membership:
|
||||
brain.find({ type: NounType.Person, subtype: ['employee', 'contractor'] })
|
||||
|
||||
// Operator-form predicates use `where`:
|
||||
brain.find({
|
||||
type: NounType.Person,
|
||||
where: { subtype: { exists: true } }
|
||||
})
|
||||
```
|
||||
|
||||
See the **[Subtypes & Facets guide](./guides/subtypes-and-facets.md)** for the full surface.
|
||||
|
||||
### Combine semantic search with filters
|
||||
|
||||
```typescript
|
||||
|
|
|
|||
|
|
@ -111,7 +111,8 @@ Add a single entity to the database.
|
|||
const id = await brain.add({
|
||||
data: 'JavaScript is a programming language', // Text or pre-computed vector
|
||||
type: NounType.Concept, // Required: Entity type
|
||||
metadata: { // Optional metadata
|
||||
subtype: 'language', // Optional: sub-classification
|
||||
metadata: { // Optional: queryable fields
|
||||
category: 'programming',
|
||||
year: 1995
|
||||
}
|
||||
|
|
@ -121,6 +122,7 @@ const id = await brain.add({
|
|||
**Parameters:**
|
||||
- `data`: `string | number[]` - Content to embed (text auto-embeds) or pre-computed vector
|
||||
- `type`: `NounType` - Entity type (required)
|
||||
- `subtype?`: `string` - Per-product sub-classification within the NounType (top-level standard field, indexed on the fast path). See [Subtypes & Facets](../guides/subtypes-and-facets.md).
|
||||
- `metadata?`: `object` - Structured queryable fields (indexed by MetadataIndex, used in `where` filters)
|
||||
- `id?`: `string` - Custom ID (auto-generated UUID if not provided)
|
||||
- `vector?`: `number[]` - Pre-computed vector (skips auto-embedding)
|
||||
|
|
@ -159,6 +161,7 @@ Update an existing entity.
|
|||
await brain.update({
|
||||
id: entityId,
|
||||
data: 'Updated content', // Optional: new data
|
||||
subtype: 'archived', // Optional: change sub-classification
|
||||
metadata: { updated: true } // Optional: new metadata (merges)
|
||||
})
|
||||
```
|
||||
|
|
@ -166,7 +169,11 @@ await brain.update({
|
|||
**Parameters:**
|
||||
- `id`: `string` - Entity ID
|
||||
- `data?`: `string | number[]` - New data/vector
|
||||
- `metadata?`: `object` - Metadata to merge
|
||||
- `type?`: `NounType` - Change entity type
|
||||
- `subtype?`: `string` - Change subtype (omit to preserve existing)
|
||||
- `metadata?`: `object` - Metadata to merge (or replace with `merge: false`)
|
||||
- `confidence?`: `number` - Update classification confidence
|
||||
- `weight?`: `number` - Update entity importance
|
||||
|
||||
**Returns:** `Promise<void>`
|
||||
|
||||
|
|
@ -221,6 +228,7 @@ const results = await brain.find({
|
|||
**FindParams:**
|
||||
- `query?`: `string` - Text for semantic + hybrid search (searches `data` via HNSW + text index)
|
||||
- `type?`: `NounType | NounType[]` - Filter by entity type(s). Alias for `where.noun`.
|
||||
- `subtype?`: `string | string[]` - Filter by sub-classification (top-level standard field, fast path). Single string for equality, array for set membership.
|
||||
- `where?`: `object` - Metadata filters. See **[Query Operators](../QUERY_OPERATORS.md)** for all operators.
|
||||
- `connected?`: `object` - Graph traversal options
|
||||
- `to?`: `string` - Target entity ID
|
||||
|
|
@ -1928,6 +1936,86 @@ const count = await brain.getVerbCount()
|
|||
|
||||
---
|
||||
|
||||
### Subtype & facet APIs
|
||||
|
||||
Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**.
|
||||
|
||||
#### `counts.bySubtype(type, subtype?)` → `Record<string, number> | number`
|
||||
|
||||
O(1) subtype counts for a NounType (backed by the persisted rollup).
|
||||
|
||||
```typescript
|
||||
brain.counts.bySubtype(NounType.Person)
|
||||
// → { employee: 12, customer: 847, vendor: 34 }
|
||||
|
||||
brain.counts.bySubtype(NounType.Person, 'employee')
|
||||
// → 12
|
||||
```
|
||||
|
||||
#### `counts.topSubtypes(type, n=10)` → `Array<[subtype, count]>`
|
||||
|
||||
Top N subtypes ranked by count.
|
||||
|
||||
```typescript
|
||||
brain.counts.topSubtypes(NounType.Person, 3)
|
||||
// → [['customer', 847], ['employee', 12], ['vendor', 34]]
|
||||
```
|
||||
|
||||
#### `subtypesOf(type)` → `string[]`
|
||||
|
||||
Sorted distinct subtypes seen for a NounType.
|
||||
|
||||
```typescript
|
||||
brain.subtypesOf(NounType.Person)
|
||||
// → ['customer', 'employee', 'vendor']
|
||||
```
|
||||
|
||||
#### `trackField(name, options?)` → `void`
|
||||
|
||||
Register a metadata field for cardinality + per-NounType breakdown stats. With `values: [...]`, validates against the whitelist on `add()`/`update()`.
|
||||
|
||||
```typescript
|
||||
brain.trackField('status') // basic
|
||||
brain.trackField('status', { perType: true }) // with per-NounType breakdown
|
||||
brain.trackField('priority', { values: ['low', 'med', 'high'] }) // strict vocabulary
|
||||
```
|
||||
|
||||
#### `counts.byField(name, options?)` → `Promise<Record<string, number>>`
|
||||
|
||||
Counts by value for a tracked field. Requires `perType: true` registration if filtering by NounType.
|
||||
|
||||
```typescript
|
||||
await brain.counts.byField('status')
|
||||
// → { todo: 12, doing: 3, done: 47 }
|
||||
|
||||
await brain.counts.byField('status', { type: NounType.Task })
|
||||
// → { todo: 8, doing: 2, done: 30 }
|
||||
```
|
||||
|
||||
#### `migrateField(options)` → `Promise<MigrationSummary>`
|
||||
|
||||
Stream-and-rewrite a field across the brain. Supports `metadata.X`, `data.X`, and top-level paths. Idempotent.
|
||||
|
||||
```typescript
|
||||
// One-shot rewrite
|
||||
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
|
||||
|
||||
// Deprecation window — keep source field readable
|
||||
await brain.migrateField({ from: 'data.kind', to: 'subtype', readBoth: true })
|
||||
|
||||
// With progress reporting
|
||||
await brain.migrateField({
|
||||
from: 'metadata.kind',
|
||||
to: 'subtype',
|
||||
batchSize: 500,
|
||||
onProgress: ({ scanned, migrated }) => console.log(`${scanned} / ${migrated}`)
|
||||
})
|
||||
```
|
||||
|
||||
Returns `{ scanned: number, migrated: number, skipped: number, errors: Array<{id, error}> }`.
|
||||
|
||||
---
|
||||
|
||||
### `embed(data)` → `Promise<number[]>` ✨
|
||||
|
||||
Generate embedding vector from text or data.
|
||||
|
|
|
|||
|
|
@ -449,6 +449,26 @@ brain.registerNounType('chemical_compound', {
|
|||
})
|
||||
```
|
||||
|
||||
### 1a. Subtypes — sub-classification without hierarchy
|
||||
|
||||
The 42-type taxonomy is intentionally coarse. Per-product vocabulary fits on the **`subtype`** axis — a top-level standard string field on every entity. Flat by design — no hierarchy, no parent chain, no recursive resolution. That preserves the Uint32Array-backed O(1) type stats while giving consumers a place to put `'employee'` / `'customer'` / `'invoice'` / `'milestone'` without burning a slot in the global enum.
|
||||
|
||||
```typescript
|
||||
// Same NounType, different subtypes:
|
||||
await brain.add({ type: NounType.Person, subtype: 'employee' })
|
||||
await brain.add({ type: NounType.Person, subtype: 'customer' })
|
||||
await brain.add({ type: NounType.Document, subtype: 'invoice' })
|
||||
|
||||
// Fast path — column-store hit, not metadata fallback:
|
||||
await brain.find({ type: NounType.Person, subtype: 'employee' })
|
||||
|
||||
// Per-NounType-per-subtype counts maintained incrementally:
|
||||
brain.counts.bySubtype(NounType.Person)
|
||||
// → { employee: 12, customer: 847 }
|
||||
```
|
||||
|
||||
Subtype has its own statistics rollup (`_system/subtype-statistics.json`) maintained alongside `nounCountsByType`, so per-subtype counts stay O(1) at billion scale. Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**.
|
||||
|
||||
### 2. Semantic not Structural
|
||||
|
||||
```typescript
|
||||
|
|
|
|||
|
|
@ -39,16 +39,20 @@ That's it. Brainy auto-configures storage, loads the embedding model, and builds
|
|||
const reactId: string = await brain.add({
|
||||
data: 'React is a JavaScript library for building user interfaces',
|
||||
type: NounType.Concept,
|
||||
subtype: 'library', // Sub-classification within Concept
|
||||
metadata: { category: 'frontend', year: 2013 }
|
||||
})
|
||||
|
||||
const nextId: string = await brain.add({
|
||||
data: 'Next.js framework for React with server-side rendering',
|
||||
type: NounType.Concept,
|
||||
subtype: 'framework',
|
||||
metadata: { category: 'framework', year: 2016 }
|
||||
})
|
||||
```
|
||||
|
||||
`type` is one of Brainy's 42 stable NounTypes. `subtype` is your free-form sub-classification within that type — flat string, no hierarchy, indexed on the fast path. See **[Subtypes & Facets](./subtypes-and-facets.md)** for the full guide.
|
||||
|
||||
## 4. Create Relationships
|
||||
|
||||
```typescript
|
||||
|
|
|
|||
285
docs/guides/subtypes-and-facets.md
Normal file
285
docs/guides/subtypes-and-facets.md
Normal file
|
|
@ -0,0 +1,285 @@
|
|||
---
|
||||
title: Subtypes & Facets
|
||||
slug: guides/subtypes-and-facets
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 7
|
||||
description: Use the top-level `subtype` field to sub-classify entities within a NounType, track other metadata facets like status or role, and migrate field names without downtime.
|
||||
next:
|
||||
- guides/aggregation
|
||||
- api/reference
|
||||
---
|
||||
|
||||
# Subtypes & Facets
|
||||
|
||||
> Sub-classify entities within a NounType, track arbitrary metadata facets, and migrate field names without downtime.
|
||||
|
||||
## Why this exists
|
||||
|
||||
Brainy's `NounType` (Person, Document, Event, Concept, Task, …) is a stable, flat 42-type taxonomy. It's deliberately coarse — every product needs a way to further classify entities *within* a type. Is this `Person` an employee or a customer? Is this `Document` an invoice or a contract? Is this `Event` a meeting or a milestone?
|
||||
|
||||
Three layers solve this:
|
||||
|
||||
| Layer | Use it for | Shape |
|
||||
|---|---|---|
|
||||
| **`subtype`** | The primary sub-classification of an entity, one value per entity | Top-level standard field |
|
||||
| **`trackField()`** | Other facets you want to count or filter on (`status`, `source`, `role`, `paradigm`) | Registered metadata field |
|
||||
| **`migrateField()`** | Renaming or restructuring fields across an existing dataset | One-shot stream-and-rewrite |
|
||||
|
||||
## Layer 1 — `subtype`
|
||||
|
||||
`subtype` is a top-level standard field on every entity, alongside `type` / `confidence` / `weight`. Flat string, no hierarchy. The vocabulary is your choice — Brainy stores and counts, never validates.
|
||||
|
||||
### Write
|
||||
|
||||
```typescript
|
||||
import { Brainy, NounType } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
await brain.add({
|
||||
data: 'Avery Brooks — runs the AI lab',
|
||||
type: NounType.Person,
|
||||
subtype: 'employee', // top-level write param
|
||||
metadata: { department: 'ai-lab' }
|
||||
})
|
||||
```
|
||||
|
||||
### Read
|
||||
|
||||
```typescript
|
||||
// Top-level filter — standard-field fast path, no `where` wrapper:
|
||||
const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })
|
||||
|
||||
// Set membership:
|
||||
const internal = await brain.find({
|
||||
type: NounType.Person,
|
||||
subtype: ['employee', 'contractor']
|
||||
})
|
||||
|
||||
// Operator-form predicates use `where`:
|
||||
const typed = await brain.find({
|
||||
type: NounType.Person,
|
||||
where: { subtype: { exists: true } }
|
||||
})
|
||||
```
|
||||
|
||||
### What it looks like
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "01HZK3M7TW...",
|
||||
"type": "person",
|
||||
"subtype": "employee",
|
||||
"data": "Avery Brooks — runs the AI lab",
|
||||
"metadata": { "department": "ai-lab" }
|
||||
}
|
||||
```
|
||||
|
||||
`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's what makes the fast path possible: queries on `subtype` hit the column-store index directly, never the metadata fallback.
|
||||
|
||||
### Counts (O(1))
|
||||
|
||||
```typescript
|
||||
// All subtypes for a NounType
|
||||
brain.counts.bySubtype(NounType.Person)
|
||||
// → { employee: 12, customer: 847, vendor: 34 }
|
||||
|
||||
// Point count
|
||||
brain.counts.bySubtype(NounType.Person, 'employee')
|
||||
// → 12
|
||||
|
||||
// Top N
|
||||
brain.counts.topSubtypes(NounType.Person, 3)
|
||||
// → [['customer', 847], ['employee', 12], ['vendor', 34]]
|
||||
|
||||
// Distinct subtypes for a NounType
|
||||
brain.subtypesOf(NounType.Person)
|
||||
// → ['customer', 'employee', 'vendor']
|
||||
```
|
||||
|
||||
These are O(1) lookups backed by `_system/subtype-statistics.json` — no scan, no storage round-trip. The rollup is incrementally maintained as entities are added, updated, and deleted.
|
||||
|
||||
### Aggregation
|
||||
|
||||
`subtype` is a first-class group-by dimension:
|
||||
|
||||
```typescript
|
||||
const rows = await brain.find({
|
||||
aggregate: {
|
||||
groupBy: ['type', 'subtype'],
|
||||
metrics: { count: { op: 'COUNT' } }
|
||||
}
|
||||
})
|
||||
// [
|
||||
// { groupKey: { type: 'person', subtype: 'customer' }, count: 847 },
|
||||
// { groupKey: { type: 'person', subtype: 'employee' }, count: 12 },
|
||||
// { groupKey: { type: 'document', subtype: 'invoice' }, count: 2103 },
|
||||
// ...
|
||||
// ]
|
||||
```
|
||||
|
||||
## Layer 2 — `trackField()`
|
||||
|
||||
Some metadata fields aren't *the* sub-classification of an entity but are still worth counting: `status`, `source`, `role`, `paradigm`. Promoting each one to top-level would clutter the contract. `trackField()` registers a metadata field for cardinality + per-NounType breakdown stats without the contract growth.
|
||||
|
||||
### Register and query
|
||||
|
||||
```typescript
|
||||
// Track a single facet
|
||||
brain.trackField('status')
|
||||
|
||||
await brain.add({ data: 'Ship subtype', type: NounType.Task, metadata: { status: 'todo' } })
|
||||
await brain.add({ data: 'Write docs', type: NounType.Task, metadata: { status: 'done' } })
|
||||
|
||||
await brain.counts.byField('status')
|
||||
// → { todo: 1, done: 1 }
|
||||
```
|
||||
|
||||
### Per-NounType breakdown
|
||||
|
||||
```typescript
|
||||
brain.trackField('status', { perType: true })
|
||||
|
||||
await brain.counts.byField('status', { type: NounType.Task })
|
||||
// → { todo: 1, done: 1 }
|
||||
```
|
||||
|
||||
### Vocabulary whitelist (opt-in validation)
|
||||
|
||||
```typescript
|
||||
brain.trackField('priority', { values: ['low', 'medium', 'high'] })
|
||||
|
||||
// Throws — 'urgent' isn't in the vocabulary:
|
||||
await brain.add({
|
||||
data: 'Fix bug',
|
||||
type: NounType.Task,
|
||||
metadata: { priority: 'urgent' }
|
||||
})
|
||||
```
|
||||
|
||||
`trackField` piggybacks on the existing aggregation engine (see the [Aggregation guide](./aggregation.md) for the underlying mechanism). Backfill-on-define means the first call to `counts.byField()` scans existing entities; subsequent calls are O(groups).
|
||||
|
||||
### subtype vs trackField — when to use which
|
||||
|
||||
- **`subtype`** when there's one primary sub-classification per entity. Limit yourself to one per NounType. Examples: Person→employee/customer/vendor; Document→invoice/contract/policy.
|
||||
- **`trackField`** for anything else you want to count or filter on. No limit on how many you register. Examples: status, source, role, paradigm.
|
||||
|
||||
## Layer 3 — `migrateField()`
|
||||
|
||||
Use this when you need to rename or restructure a field across an entire dataset — for example, moving a `metadata.kind` convention up to the top-level `subtype` standard field.
|
||||
|
||||
### One-shot rewrite
|
||||
|
||||
```typescript
|
||||
// Starting state: every entity has metadata.kind
|
||||
const result = await brain.migrateField({
|
||||
from: 'metadata.kind',
|
||||
to: 'subtype'
|
||||
})
|
||||
|
||||
console.log(result)
|
||||
// {
|
||||
// scanned: 1500,
|
||||
// migrated: 1500,
|
||||
// skipped: 0,
|
||||
// errors: []
|
||||
// }
|
||||
```
|
||||
|
||||
After this returns, every entity has `subtype` populated from the old `metadata.kind` value, and `metadata.kind` is cleared.
|
||||
|
||||
### Deprecation window — keep both fields readable
|
||||
|
||||
When you can't coordinate all readers and the migration in a single deploy, use `readBoth: true` to preserve the source field alongside the new one:
|
||||
|
||||
```typescript
|
||||
// Phase 1: dual-populate (existing readers still work against metadata.kind):
|
||||
await brain.migrateField({
|
||||
from: 'metadata.kind',
|
||||
to: 'subtype',
|
||||
readBoth: true
|
||||
})
|
||||
|
||||
// ... readers migrate to query subtype at their own pace ...
|
||||
|
||||
// Phase 2: clear the source field when ready:
|
||||
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
|
||||
```
|
||||
|
||||
### Supported paths
|
||||
|
||||
| Path form | Refers to |
|
||||
|---|---|
|
||||
| `'subtype'`, `'type'`, `'confidence'` | Top-level standard fields |
|
||||
| `'metadata.X'` | A key under `entity.metadata` |
|
||||
| `'data.X'` | A key under `entity.data` (when `data` is an object) |
|
||||
| `'X'` (bare, non-standard) | Shorthand for `metadata.X` |
|
||||
|
||||
### Idempotent
|
||||
|
||||
`migrateField` is safe to re-run. Entities where the source is absent, or where the destination already holds the same value, are skipped. This makes it safe to use in a deploy-once-then-cleanup workflow.
|
||||
|
||||
### Progress reporting
|
||||
|
||||
```typescript
|
||||
await brain.migrateField({
|
||||
from: 'metadata.kind',
|
||||
to: 'subtype',
|
||||
batchSize: 500,
|
||||
onProgress: ({ scanned, migrated }) => {
|
||||
console.log(`${scanned} scanned, ${migrated} migrated`)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Putting it together
|
||||
|
||||
A realistic adoption sequence for a brain that started without these primitives:
|
||||
|
||||
```typescript
|
||||
import { Brainy, NounType } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy({ storage: { type: 'filesystem', options: { path: './brain-data' } } })
|
||||
await brain.init()
|
||||
|
||||
// 1. Migrate any existing metadata.kind convention to the new top-level subtype
|
||||
await brain.migrateField({ from: 'metadata.kind', to: 'subtype', readBoth: true })
|
||||
|
||||
// 2. Register the other facets you want counted
|
||||
brain.trackField('status', { perType: true })
|
||||
brain.trackField('source')
|
||||
|
||||
// 3. Use subtype on every new write
|
||||
await brain.add({
|
||||
data: 'Quarterly review',
|
||||
type: NounType.Event,
|
||||
subtype: 'milestone',
|
||||
metadata: { status: 'todo', source: 'planning-session' }
|
||||
})
|
||||
|
||||
// 4. Query the breakdowns
|
||||
brain.counts.bySubtype(NounType.Event)
|
||||
// → { milestone: 14, meeting: 203, deadline: 7 }
|
||||
|
||||
await brain.counts.byField('status', { type: NounType.Event })
|
||||
// → { todo: 12, done: 212 }
|
||||
|
||||
// 5. Once all readers are on the new field, drop the source:
|
||||
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
|
||||
```
|
||||
|
||||
## Reference
|
||||
|
||||
- `brain.add({ ..., subtype: 'value' })` — write the field
|
||||
- `brain.update({ id, subtype: 'value' })` — change the field
|
||||
- `brain.find({ type, subtype })` — filter (fast path)
|
||||
- `brain.find({ subtype: ['a', 'b'] })` — set membership
|
||||
- `brain.counts.bySubtype(type, subtype?)` — O(1) counts
|
||||
- `brain.counts.topSubtypes(type, n?)` — top N by count
|
||||
- `brain.subtypesOf(type)` — distinct subtype list
|
||||
- `brain.trackField(name, { perType?, values? })` — register a facet
|
||||
- `brain.counts.byField(name, { type? })` — facet counts
|
||||
- `brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })` — rewrite a field
|
||||
546
src/brainy.ts
546
src/brainy.ts
|
|
@ -11,7 +11,7 @@ import { HNSWIndex } from './hnsw/hnswIndex.js'
|
|||
// for the 99% of queries that don't filter by type (avoids searching 42 separate graphs)
|
||||
import { createStorage } from './storage/storageFactory.js'
|
||||
import { BaseStorage } from './storage/baseStorage.js'
|
||||
import { StorageAdapter, Vector, DistanceFunction, EmbeddingFunction, GraphVerb } from './coreTypes.js'
|
||||
import { StorageAdapter, Vector, DistanceFunction, EmbeddingFunction, GraphVerb, STANDARD_ENTITY_FIELDS } from './coreTypes.js'
|
||||
import {
|
||||
defaultEmbeddingFunction,
|
||||
cosineDistance,
|
||||
|
|
@ -97,7 +97,7 @@ import {
|
|||
BrainyStats,
|
||||
ScoreExplanation
|
||||
} from './types/brainy.types.js'
|
||||
import { NounType, VerbType } from './types/graphTypes.js'
|
||||
import { NounType, VerbType, TypeUtils } from './types/graphTypes.js'
|
||||
import { BrainyInterface } from './types/brainyInterface.js'
|
||||
import type { IntegrationHub } from './integrations/core/IntegrationHub.js'
|
||||
import { MigrationRunner } from './migration/MigrationRunner.js'
|
||||
|
|
@ -189,6 +189,14 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
private _pendingMigrationRunner?: MigrationRunner // Deferred migration runner for large datasets
|
||||
private _aggregationIndex?: AggregationIndex // Incremental aggregation engine
|
||||
private _materializer?: AggregateMaterializer // Debounced materialization of aggregate results
|
||||
/**
|
||||
* Fields registered via `brain.trackField()` — drives optional value validation on
|
||||
* `add()`/`update()` and powers `brain.counts.byField()`. Outer key is the field
|
||||
* name (`'status'`, `'paradigm'`, `'role'`, ...); inner record carries the per-NounType
|
||||
* flag and an optional value whitelist (when provided, writes with off-vocabulary
|
||||
* values are rejected).
|
||||
*/
|
||||
private _trackedFields: Map<string, { perType: boolean; values?: Set<string> }> = new Map()
|
||||
|
||||
// State
|
||||
private initialized = false
|
||||
|
|
@ -1016,6 +1024,15 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// Zero-config validation (static import for performance)
|
||||
validateAddParams(params)
|
||||
|
||||
// Tracked-field vocabulary enforcement (Layer 2). Walks both bags so a
|
||||
// tracked field declared at top level (e.g. 'subtype') and one declared in
|
||||
// metadata (e.g. 'status') both validate.
|
||||
this.enforceTrackedFieldValues(params.metadata as Record<string, unknown> | undefined, 'metadata')
|
||||
this.enforceTrackedFieldValues(
|
||||
{ subtype: params.subtype } as Record<string, unknown>,
|
||||
'top-level'
|
||||
)
|
||||
|
||||
// Generate ID if not provided
|
||||
const id = params.id || uuidv4()
|
||||
|
||||
|
|
@ -1038,6 +1055,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
...params.metadata,
|
||||
data: params.data,
|
||||
noun: params.type,
|
||||
...(params.subtype !== undefined && { subtype: params.subtype }),
|
||||
service: params.service,
|
||||
createdAt: Date.now(),
|
||||
updatedAt: Date.now(),
|
||||
|
|
@ -1057,6 +1075,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
connections: new Map(),
|
||||
level: 0,
|
||||
type: params.type,
|
||||
...(params.subtype !== undefined && { subtype: params.subtype }),
|
||||
...(params.confidence !== undefined && { confidence: params.confidence }),
|
||||
...(params.weight !== undefined && { weight: params.weight }),
|
||||
createdAt: Date.now(),
|
||||
|
|
@ -1329,6 +1348,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
score,
|
||||
// Flatten common entity fields to top level
|
||||
type: entity.type,
|
||||
subtype: entity.subtype,
|
||||
metadata: entity.metadata,
|
||||
data: entity.data,
|
||||
confidence: entity.confidence,
|
||||
|
|
@ -1357,6 +1377,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
id: noun.id,
|
||||
vector: noun.vector,
|
||||
type: noun.type || NounType.Thing,
|
||||
subtype: noun.subtype,
|
||||
|
||||
// Standard fields at top-level
|
||||
confidence: noun.confidence,
|
||||
|
|
@ -1398,12 +1419,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
|
||||
// Extract standard fields, rest are custom metadata
|
||||
// Same destructuring as baseStorage.getNoun() to ensure consistency
|
||||
const { noun, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata
|
||||
const { noun, subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata
|
||||
|
||||
const entity: Entity<T> = {
|
||||
id,
|
||||
vector: [], // Stub vector (empty array - vectors not loaded for metadata-only)
|
||||
type: noun as NounType || NounType.Thing,
|
||||
subtype,
|
||||
|
||||
// Standard fields from metadata
|
||||
confidence,
|
||||
|
|
@ -1478,6 +1500,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// Zero-config validation (static import for performance)
|
||||
validateUpdateParams(params)
|
||||
|
||||
// Tracked-field vocabulary enforcement (Layer 2). Same as add() — the
|
||||
// metadata bag carries fields registered via trackField(), and subtype is
|
||||
// a tracked top-level candidate.
|
||||
this.enforceTrackedFieldValues(params.metadata as Record<string, unknown> | undefined, 'metadata')
|
||||
if (params.subtype !== undefined) {
|
||||
this.enforceTrackedFieldValues(
|
||||
{ subtype: params.subtype } as Record<string, unknown>,
|
||||
'top-level'
|
||||
)
|
||||
}
|
||||
|
||||
// Get existing entity with vectors (fix for regression)
|
||||
// We need includeVectors: true because:
|
||||
// 1. SaveNounOperation requires the vector
|
||||
|
|
@ -1513,7 +1546,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
...(params.confidence !== undefined && { confidence: params.confidence }),
|
||||
...(params.weight !== undefined && { weight: params.weight }),
|
||||
...(params.confidence === undefined && existing.confidence !== undefined && { confidence: existing.confidence }),
|
||||
...(params.weight === undefined && existing.weight !== undefined && { weight: existing.weight })
|
||||
...(params.weight === undefined && existing.weight !== undefined && { weight: existing.weight }),
|
||||
// Update subtype if provided, otherwise preserve existing
|
||||
...(params.subtype !== undefined && { subtype: params.subtype }),
|
||||
...(params.subtype === undefined && existing.subtype !== undefined && { subtype: existing.subtype })
|
||||
}
|
||||
|
||||
// Build entity structure for metadata index (with top-level fields)
|
||||
|
|
@ -1523,6 +1559,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
connections: new Map(),
|
||||
level: 0,
|
||||
type: params.type || existing.type,
|
||||
subtype: params.subtype !== undefined ? params.subtype : existing.subtype,
|
||||
confidence: params.confidence !== undefined ? params.confidence : existing.confidence,
|
||||
weight: params.weight !== undefined ? params.weight : existing.weight,
|
||||
createdAt: existing.createdAt,
|
||||
|
|
@ -2291,6 +2328,104 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
this._aggregationIndex!.defineAggregate(def)
|
||||
}
|
||||
|
||||
/**
|
||||
* Register a field for cardinality + per-NounType breakdown stats.
|
||||
*
|
||||
* Layer 2 of the subtype-and-facets primitive: a lightweight wrapper over the
|
||||
* aggregation engine that auto-defines an internal `__fieldCounts__<name>`
|
||||
* aggregate so consumers can query value frequencies without writing the
|
||||
* aggregate definition themselves. Backfill-on-define (shipped 7.23.0) means
|
||||
* existing entities are scanned on the first query, not at registration time.
|
||||
*
|
||||
* Use this for facets that don't warrant top-level promotion (e.g. `status`,
|
||||
* `source`, `role`). For sub-classification within a NounType, prefer the
|
||||
* top-level `subtype` field — it takes the standard-field fast path and has
|
||||
* its own statistics rollup.
|
||||
*
|
||||
* @param name - Field name to track. Resolves via the standard-fields-first /
|
||||
* metadata-fallback path, so both `'subtype'` (top-level) and `'status'`
|
||||
* (metadata) work the same way.
|
||||
* @param options - `perType` adds `noun` to the groupBy so counts are split
|
||||
* by NounType; `values` registers a whitelist that rejects writes containing
|
||||
* off-vocabulary values (validated in `add()`/`update()`).
|
||||
*
|
||||
* @example Track status without per-type breakdown
|
||||
* brain.trackField('status')
|
||||
* const counts = await brain.counts.byField('status')
|
||||
* // → { todo: 12, doing: 3, done: 47 }
|
||||
*
|
||||
* @example Track status with per-NounType breakdown
|
||||
* brain.trackField('status', { perType: true })
|
||||
* const taskCounts = await brain.counts.byField('status', { type: NounType.Task })
|
||||
* // → { todo: 8, doing: 2, done: 30 }
|
||||
*
|
||||
* @example Strict vocabulary
|
||||
* brain.trackField('priority', { values: ['low', 'medium', 'high'] })
|
||||
* // brain.add({ ..., metadata: { priority: 'urgent' } }) throws
|
||||
*/
|
||||
trackField(
|
||||
name: string,
|
||||
options: { perType?: boolean; values?: string[] } = {}
|
||||
): void {
|
||||
if (!name || typeof name !== 'string') {
|
||||
throw new Error('trackField: name must be a non-empty string')
|
||||
}
|
||||
const perType = options.perType === true
|
||||
const valuesSet = options.values && options.values.length > 0
|
||||
? new Set(options.values)
|
||||
: undefined
|
||||
this._trackedFields.set(name, { perType, values: valuesSet })
|
||||
|
||||
// Auto-define the backing aggregate. groupBy uses 'noun' (storage field name
|
||||
// for type) when perType is on, so the column-store key matches the persisted
|
||||
// shape and resolveEntityField doesn't need to rewrite the dimension.
|
||||
this.ensureAggregationIndex()
|
||||
const aggregateName = this.fieldCountsAggregateName(name)
|
||||
if (!this._aggregationIndex!.hasAggregate(aggregateName)) {
|
||||
this._aggregationIndex!.defineAggregate({
|
||||
name: aggregateName,
|
||||
source: {},
|
||||
groupBy: perType ? [name, 'noun'] : [name],
|
||||
metrics: { count: { op: 'count' } }
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Internal aggregate name for a tracked field. Centralized so `trackField()`
|
||||
* and `counts.byField()` agree on the convention.
|
||||
*/
|
||||
private fieldCountsAggregateName(name: string): string {
|
||||
return `__fieldCounts__${name}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate a metadata bag (or top-level field assignment) against any registered
|
||||
* value whitelists. Called from `add()`/`update()` after the standard zero-config
|
||||
* validation. Throws on the first off-vocabulary value to fail fast.
|
||||
*
|
||||
* Tracked fields with no `values` whitelist are skipped — registration alone
|
||||
* does not imply validation.
|
||||
*/
|
||||
private enforceTrackedFieldValues(
|
||||
bag: Record<string, unknown> | undefined,
|
||||
bagLabel: 'metadata' | 'top-level'
|
||||
): void {
|
||||
if (!bag || this._trackedFields.size === 0) return
|
||||
for (const [field, def] of this._trackedFields.entries()) {
|
||||
if (!def.values) continue
|
||||
if (!(field in bag)) continue
|
||||
const value = bag[field]
|
||||
if (value === undefined || value === null) continue
|
||||
const asString = typeof value === 'string' ? value : String(value)
|
||||
if (!def.values.has(asString)) {
|
||||
throw new Error(
|
||||
`trackField('${field}') rejected ${bagLabel} value '${asString}': not in registered vocabulary [${Array.from(def.values).join(', ')}]`
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove a named aggregate and clean up its state.
|
||||
*
|
||||
|
|
@ -2373,7 +2508,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// Distinguish between search criteria (need vector search) and filter criteria (metadata only)
|
||||
// Treat empty string query as no query
|
||||
const hasVectorSearchCriteria = (params.query && params.query.trim() !== '') || params.vector || params.near
|
||||
const hasFilterCriteria = params.where || params.type || params.service
|
||||
const hasFilterCriteria = params.where || params.type || params.subtype || params.service
|
||||
const hasGraphCriteria = params.connected
|
||||
|
||||
// Handle metadata-only queries (no vector search needed)
|
||||
|
|
@ -2390,6 +2525,15 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
if (params.service) filter.service = params.service
|
||||
|
||||
// Subtype (top-level standard field — fast path, not metadata fallback).
|
||||
// Must be assigned BEFORE the type-array expansion below so the spread
|
||||
// into each anyOf branch carries it through.
|
||||
if (params.subtype !== undefined) {
|
||||
filter.subtype = Array.isArray(params.subtype)
|
||||
? { oneOf: params.subtype }
|
||||
: params.subtype
|
||||
}
|
||||
|
||||
if (params.type) {
|
||||
const types = Array.isArray(params.type) ? params.type : [params.type]
|
||||
if (types.length === 1) {
|
||||
|
|
@ -2528,7 +2672,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
let preResolvedMetadataIds: string[] | null = null
|
||||
let preResolvedFilter: any = null
|
||||
|
||||
if (params.where || params.type || params.service || params.excludeVFS) {
|
||||
if (params.where || params.type || params.subtype || params.service || params.excludeVFS) {
|
||||
preResolvedFilter = {}
|
||||
if (params.where) {
|
||||
Object.assign(preResolvedFilter, params.where)
|
||||
|
|
@ -2543,6 +2687,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
preResolvedFilter.vfsType = { exists: false }
|
||||
preResolvedFilter.isVFSEntity = { ne: true }
|
||||
}
|
||||
// Subtype (top-level standard field — fast path).
|
||||
// Must be assigned BEFORE the type-array expansion below.
|
||||
if (params.subtype !== undefined) {
|
||||
preResolvedFilter.subtype = Array.isArray(params.subtype)
|
||||
? { oneOf: params.subtype }
|
||||
: params.subtype
|
||||
}
|
||||
if (params.type) {
|
||||
const types = Array.isArray(params.type) ? params.type : [params.type]
|
||||
if (types.length === 1) {
|
||||
|
|
@ -5776,13 +5927,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// Get total count for pagination UI (O(1) when possible)
|
||||
count: async (params: Omit<FindParams<T>, 'limit' | 'offset'>) => {
|
||||
// For simple type queries, use O(1) index counting
|
||||
if (params.type && !params.query && !params.where && !params.connected) {
|
||||
if (params.type && !params.subtype && !params.query && !params.where && !params.connected) {
|
||||
const types = Array.isArray(params.type) ? params.type : [params.type]
|
||||
return types.reduce((sum, type) => sum + this.metadataIndex.getEntityCountByType(type), 0)
|
||||
}
|
||||
|
||||
// For complex queries, use metadata index for efficient counting
|
||||
if (params.where || params.service) {
|
||||
if (params.where || params.subtype || params.service) {
|
||||
let filter: any = {}
|
||||
if (params.where) {
|
||||
Object.assign(filter, params.where)
|
||||
|
|
@ -5793,6 +5944,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
}
|
||||
if (params.service) filter.service = params.service
|
||||
if (params.subtype !== undefined) {
|
||||
filter.subtype = Array.isArray(params.subtype)
|
||||
? { oneOf: params.subtype }
|
||||
: params.subtype
|
||||
}
|
||||
if (params.type) {
|
||||
const types = Array.isArray(params.type) ? params.type : [params.type]
|
||||
if (types.length === 1) {
|
||||
|
|
@ -5846,7 +6002,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
return {
|
||||
// Stream all entities with optional filtering
|
||||
entities: async function* (this: Brainy<T>, filter?: Partial<FindParams<T>>) {
|
||||
if (filter?.type || filter?.where || filter?.service) {
|
||||
if (filter?.type || filter?.subtype || filter?.where || filter?.service) {
|
||||
// Use MetadataIndexManager for efficient filtered streaming
|
||||
let filterObj: any = {}
|
||||
if (filter.where) {
|
||||
|
|
@ -5858,6 +6014,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
}
|
||||
if (filter.service) filterObj.service = filter.service
|
||||
if (filter.subtype !== undefined) {
|
||||
filterObj.subtype = Array.isArray(filter.subtype)
|
||||
? { oneOf: filter.subtype }
|
||||
: filter.subtype
|
||||
}
|
||||
if (filter.type) {
|
||||
const types = Array.isArray(filter.type) ? filter.type : [filter.type]
|
||||
if (types.length === 1) {
|
||||
|
|
@ -6042,6 +6203,70 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
return this.metadataIndex.getTopNounTypes(n)
|
||||
},
|
||||
|
||||
/**
|
||||
* O(1) subtype counts for a given NounType.
|
||||
*
|
||||
* Returns the count for a single (type, subtype) pair when `subtype` is
|
||||
* passed; returns the full subtype → count map for that NounType when omitted.
|
||||
* Backed by the persisted `_system/subtype-statistics.json` rollup — no
|
||||
* scan, no storage round-trip.
|
||||
*
|
||||
* @param type - The NounType to count subtypes within
|
||||
* @param subtype - Optional specific subtype string for O(1) point count
|
||||
* @returns A number when `subtype` is given, otherwise a `Record<subtype, count>` (empty `{}` if none)
|
||||
*
|
||||
* @example Get all subtypes of Person
|
||||
* const counts = brain.counts.bySubtype(NounType.Person)
|
||||
* // → { employee: 56, customer: 847, vendor: 34 }
|
||||
*
|
||||
* @example O(1) point count
|
||||
* const employees = brain.counts.bySubtype(NounType.Person, 'employee')
|
||||
* // → 56
|
||||
*/
|
||||
bySubtype: (type: NounType, subtype?: string): number | Record<string, number> => {
|
||||
const subtypeMap = typeof (this.storage as any).getSubtypeCountsByType === 'function'
|
||||
? (this.storage as any).getSubtypeCountsByType() as Map<number, Map<string, number>>
|
||||
: null
|
||||
if (!subtypeMap) {
|
||||
return subtype !== undefined ? 0 : {}
|
||||
}
|
||||
const typeIdx = TypeUtils.getNounIndex(type)
|
||||
const inner = subtypeMap.get(typeIdx)
|
||||
if (!inner) {
|
||||
return subtype !== undefined ? 0 : {}
|
||||
}
|
||||
if (subtype !== undefined) {
|
||||
return inner.get(subtype) || 0
|
||||
}
|
||||
const result: Record<string, number> = {}
|
||||
for (const [k, v] of inner.entries()) result[k] = v
|
||||
return result
|
||||
},
|
||||
|
||||
/**
|
||||
* Top N subtypes for a NounType, sorted by count (descending).
|
||||
*
|
||||
* @param type - The NounType to rank subtypes within
|
||||
* @param n - Maximum number of (subtype, count) pairs to return (default: 10)
|
||||
* @returns Array of `[subtype, count]` tuples, highest count first
|
||||
*
|
||||
* @example
|
||||
* const top3 = brain.counts.topSubtypes(NounType.Person, 3)
|
||||
* // → [['customer', 847], ['employee', 56], ['vendor', 34]]
|
||||
*/
|
||||
topSubtypes: (type: NounType, n: number = 10): Array<[string, number]> => {
|
||||
const subtypeMap = typeof (this.storage as any).getSubtypeCountsByType === 'function'
|
||||
? (this.storage as any).getSubtypeCountsByType() as Map<number, Map<string, number>>
|
||||
: null
|
||||
if (!subtypeMap) return []
|
||||
const typeIdx = TypeUtils.getNounIndex(type)
|
||||
const inner = subtypeMap.get(typeIdx)
|
||||
if (!inner) return []
|
||||
return Array.from(inner.entries())
|
||||
.sort((a, b) => b[1] - a[1])
|
||||
.slice(0, n)
|
||||
},
|
||||
|
||||
// Phase 1b: Get top N verb types by count
|
||||
topVerbTypes: (n: number = 10) => {
|
||||
return this.metadataIndex.getTopVerbTypes(n)
|
||||
|
|
@ -6071,6 +6296,60 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
return this.metadataIndex.getCountForCriteria(field, value)
|
||||
},
|
||||
|
||||
/**
|
||||
* Counts by value for a field registered via `brain.trackField()`. Reads
|
||||
* from the backing `__fieldCounts__<name>` aggregate (Layer 2 of the
|
||||
* subtype-and-facets primitive). Backfill-on-define means the first call
|
||||
* scans existing entities, subsequent calls are O(groups).
|
||||
*
|
||||
* Without `options.type`: returns the cross-NounType total per value.
|
||||
* With `options.type`: returns per-value counts for that NounType only —
|
||||
* requires the field to have been registered with `perType: true`.
|
||||
*
|
||||
* @param name - The tracked field name
|
||||
* @param options.type - Optional NounType filter (requires perType registration)
|
||||
* @returns `{ value: count }` map. Empty when the field wasn't tracked
|
||||
* (no aggregate exists) or no entities have set it yet.
|
||||
* @throws If `options.type` is passed but the field was not registered with `perType: true`
|
||||
*
|
||||
* @example
|
||||
* brain.trackField('status', { perType: true })
|
||||
* await brain.add({ data: 'Ship it', type: NounType.Task, metadata: { status: 'todo' } })
|
||||
* await brain.counts.byField('status')
|
||||
* // → { todo: 1 }
|
||||
* await brain.counts.byField('status', { type: NounType.Task })
|
||||
* // → { todo: 1 }
|
||||
*/
|
||||
byField: async (
|
||||
name: string,
|
||||
options?: { type?: NounType }
|
||||
): Promise<Record<string, number>> => {
|
||||
const tracked = this._trackedFields.get(name)
|
||||
if (!tracked) return {}
|
||||
if (options?.type !== undefined && !tracked.perType) {
|
||||
throw new Error(
|
||||
`counts.byField('${name}'): per-type breakdown requested but the field was registered without perType:true. Re-call trackField('${name}', { perType: true }).`
|
||||
)
|
||||
}
|
||||
const aggregateName = this.fieldCountsAggregateName(name)
|
||||
if (!this._aggregationIndex || !this._aggregationIndex.hasAggregate(aggregateName)) {
|
||||
return {}
|
||||
}
|
||||
const rows = await this.queryAggregate(aggregateName)
|
||||
const result: Record<string, number> = {}
|
||||
for (const row of rows) {
|
||||
const value = row.groupKey?.[name]
|
||||
// Skip the aggregation engine's "missing-value" sentinel: entities that
|
||||
// don't have the tracked field at all (e.g. the VFS root) bucket under
|
||||
// '__null__' and would otherwise pollute the count map.
|
||||
if (value === undefined || value === null || value === '__null__') continue
|
||||
if (options?.type !== undefined && row.groupKey?.['noun'] !== options.type) continue
|
||||
const key = String(value)
|
||||
result[key] = (result[key] || 0) + (typeof row.metrics?.count === 'number' ? row.metrics.count : row.count)
|
||||
}
|
||||
return result
|
||||
},
|
||||
|
||||
// Get all type counts as Map for performance-critical operations
|
||||
getAllTypeCounts: () => this.metadataIndex.getAllEntityCounts(),
|
||||
|
||||
|
|
@ -6132,6 +6411,255 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
return this.counts.getStats(options)
|
||||
}
|
||||
|
||||
/**
|
||||
* Distinct subtypes seen for a given NounType.
|
||||
*
|
||||
* Reads from the subtype-statistics rollup — no scan, no storage round-trip.
|
||||
* The returned list is the vocabulary actually observed in the data, not a
|
||||
* registered schema (Brainy doesn't validate subtype vocabulary; that's a
|
||||
* consumer concern).
|
||||
*
|
||||
* @param type - The NounType to enumerate subtypes for
|
||||
* @returns Sorted list of distinct subtype strings (empty if none)
|
||||
*
|
||||
* @example
|
||||
* const personSubtypes = brain.subtypesOf(NounType.Person)
|
||||
* // → ['customer', 'employee', 'vendor']
|
||||
*/
|
||||
subtypesOf(type: NounType): string[] {
|
||||
const subtypeMap = typeof (this.storage as any).getSubtypeCountsByType === 'function'
|
||||
? (this.storage as any).getSubtypeCountsByType() as Map<number, Map<string, number>>
|
||||
: null
|
||||
if (!subtypeMap) return []
|
||||
const typeIdx = TypeUtils.getNounIndex(type)
|
||||
const inner = subtypeMap.get(typeIdx)
|
||||
if (!inner) return []
|
||||
return Array.from(inner.keys()).sort()
|
||||
}
|
||||
|
||||
/**
|
||||
* Stream-and-rewrite a field across every entity in the brain.
|
||||
*
|
||||
* Layer 3 of the subtype-and-facets primitive. Reads the value at `from`,
|
||||
* writes it to `to`, and (unless `readBoth: true`) clears the source. Use this
|
||||
* when migrating between field-name conventions or moving a value from
|
||||
* `metadata.*` / `data.*` up to a top-level standard field like `subtype`.
|
||||
*
|
||||
* **Path forms:**
|
||||
* - `'subtype'`, `'type'`, `'confidence'`, etc. — top-level standard fields
|
||||
* (whatever appears in `STANDARD_ENTITY_FIELDS`)
|
||||
* - `'metadata.X'` — a key under `entity.metadata`
|
||||
* - `'data.X'` — a key under `entity.data` (when `data` is an object)
|
||||
* - `'X'` (bare, not standard) — shorthand for `metadata.X`
|
||||
*
|
||||
* **Behavior:**
|
||||
* - Streams entities in batches and rewrites in-place via `brain.update()`.
|
||||
* Aggregations and indexes refresh automatically.
|
||||
* - Entities where the source path is absent or where the target already
|
||||
* holds the same value are skipped (idempotent — safe to re-run).
|
||||
* - With `readBoth: true`, the source value is preserved alongside the new
|
||||
* target so legacy consumers can keep querying the old path during a
|
||||
* deprecation window. Re-run with `readBoth: false` when ready to clear.
|
||||
*
|
||||
* @param options.from - Source path
|
||||
* @param options.to - Destination path
|
||||
* @param options.readBoth - If true, leave the source value in place (default false → source cleared)
|
||||
* @param options.batchSize - Entities per batch (default 100)
|
||||
* @param options.onProgress - Optional callback invoked after each batch
|
||||
* @returns Migration summary — counts and per-entity errors
|
||||
*
|
||||
* @example One-shot migration
|
||||
* await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
|
||||
* // → { scanned: 1500, migrated: 1500, skipped: 0, errors: [] }
|
||||
*
|
||||
* @example Deprecation window — keep both fields readable
|
||||
* await brain.migrateField({ from: 'data.kind', to: 'subtype', readBoth: true })
|
||||
* // ...consumers switch reads to subtype over time...
|
||||
* await brain.migrateField({ from: 'data.kind', to: 'subtype' })
|
||||
* // ...source cleared in the final sweep.
|
||||
*/
|
||||
async migrateField(options: {
|
||||
from: string
|
||||
to: string
|
||||
readBoth?: boolean
|
||||
batchSize?: number
|
||||
onProgress?: (progress: { scanned: number; migrated: number }) => void
|
||||
}): Promise<{
|
||||
scanned: number
|
||||
migrated: number
|
||||
skipped: number
|
||||
errors: Array<{ id: string; error: string }>
|
||||
}> {
|
||||
this.assertWritable('migrateField')
|
||||
await this.ensureInitialized()
|
||||
|
||||
if (!options.from || typeof options.from !== 'string') {
|
||||
throw new Error('migrateField: `from` must be a non-empty string path')
|
||||
}
|
||||
if (!options.to || typeof options.to !== 'string') {
|
||||
throw new Error('migrateField: `to` must be a non-empty string path')
|
||||
}
|
||||
if (options.from === options.to) {
|
||||
throw new Error('migrateField: `from` and `to` are identical — nothing to do')
|
||||
}
|
||||
|
||||
const fromPath = this.parseMigrationPath(options.from)
|
||||
const toPath = this.parseMigrationPath(options.to)
|
||||
const batchSize = Math.max(1, options.batchSize ?? 100)
|
||||
const readBoth = options.readBoth === true
|
||||
|
||||
let scanned = 0
|
||||
let migrated = 0
|
||||
let skipped = 0
|
||||
const errors: Array<{ id: string; error: string }> = []
|
||||
let offset = 0
|
||||
|
||||
// Stream via storage pagination — same shape used by streaming.entities()
|
||||
// when no filter is set. Avoids a full in-memory load on large brains.
|
||||
while (true) {
|
||||
const page = await this.storage.getNouns({ pagination: { offset, limit: batchSize } })
|
||||
if (page.items.length === 0) break
|
||||
|
||||
for (const noun of page.items) {
|
||||
scanned++
|
||||
try {
|
||||
const entity = await this.convertNounToEntity(noun as any)
|
||||
const sourceValue = this.readPath(entity, fromPath)
|
||||
if (sourceValue === undefined || sourceValue === null) {
|
||||
skipped++
|
||||
continue
|
||||
}
|
||||
const targetValue = this.readPath(entity, toPath)
|
||||
const sourceCleared = readBoth || fromPath.kind === toPath.kind && fromPath.field === toPath.field
|
||||
if (targetValue === sourceValue && (readBoth || !this.pathExists(entity, fromPath))) {
|
||||
// Already migrated and (readBoth || source already gone) → no-op
|
||||
skipped++
|
||||
continue
|
||||
}
|
||||
|
||||
const update = this.buildMigrationUpdate(entity, fromPath, toPath, sourceValue, readBoth)
|
||||
await this.update(update)
|
||||
migrated++
|
||||
void sourceCleared
|
||||
} catch (err) {
|
||||
errors.push({
|
||||
id: (noun as any).id ?? '<unknown>',
|
||||
error: err instanceof Error ? err.message : String(err)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
if (options.onProgress) {
|
||||
options.onProgress({ scanned, migrated })
|
||||
}
|
||||
if (!page.hasMore) break
|
||||
offset += page.items.length
|
||||
}
|
||||
|
||||
return { scanned, migrated, skipped, errors }
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a dotted path used by `migrateField` into its routing kind + field name.
|
||||
* `'subtype'` → top-level standard; `'metadata.X'` → metadata; `'data.X'` → data;
|
||||
* bare non-standard names → metadata (matching `resolveEntityField`'s fallback).
|
||||
*/
|
||||
private parseMigrationPath(path: string): { kind: 'top' | 'metadata' | 'data'; field: string } {
|
||||
const dotIdx = path.indexOf('.')
|
||||
if (dotIdx === -1) {
|
||||
if (STANDARD_ENTITY_FIELDS.has(path)) return { kind: 'top', field: path }
|
||||
return { kind: 'metadata', field: path }
|
||||
}
|
||||
const head = path.slice(0, dotIdx)
|
||||
const tail = path.slice(dotIdx + 1)
|
||||
if (!tail) throw new Error(`migrateField: invalid path '${path}' (trailing dot)`)
|
||||
if (head === 'metadata') return { kind: 'metadata', field: tail }
|
||||
if (head === 'data') return { kind: 'data', field: tail }
|
||||
throw new Error(
|
||||
`migrateField: unsupported path prefix '${head}'. Supported: 'metadata.X', 'data.X', or a bare field name.`
|
||||
)
|
||||
}
|
||||
|
||||
/** Read the value at a parsed migration path. Returns `undefined` if absent. */
|
||||
private readPath(entity: Entity<T>, path: { kind: 'top' | 'metadata' | 'data'; field: string }): unknown {
|
||||
if (path.kind === 'top') return (entity as any)[path.field]
|
||||
if (path.kind === 'metadata') {
|
||||
const bag = entity.metadata as unknown as Record<string, unknown> | undefined
|
||||
return bag?.[path.field]
|
||||
}
|
||||
const data = entity.data
|
||||
if (data && typeof data === 'object') {
|
||||
return (data as Record<string, unknown>)[path.field]
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
/** Whether a parsed path resolves to a defined (non-undefined) value. */
|
||||
private pathExists(entity: Entity<T>, path: { kind: 'top' | 'metadata' | 'data'; field: string }): boolean {
|
||||
return this.readPath(entity, path) !== undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Build an `UpdateParams` payload that copies the value from the source path
|
||||
* to the destination, and (unless `readBoth`) clears the source. Uses
|
||||
* `merge: false` on the metadata bag when clearing so we can omit the source
|
||||
* key authoritatively instead of relying on `undefined` round-trip behavior.
|
||||
*/
|
||||
private buildMigrationUpdate(
|
||||
entity: Entity<T>,
|
||||
from: { kind: 'top' | 'metadata' | 'data'; field: string },
|
||||
to: { kind: 'top' | 'metadata' | 'data'; field: string },
|
||||
value: unknown,
|
||||
readBoth: boolean
|
||||
): UpdateParams<T> {
|
||||
const id = entity.id
|
||||
const update: UpdateParams<T> = { id }
|
||||
|
||||
// Apply destination write.
|
||||
if (to.kind === 'top') {
|
||||
;(update as any)[to.field] = value
|
||||
} else if (to.kind === 'metadata') {
|
||||
const base = (entity.metadata as unknown as Record<string, unknown>) ?? {}
|
||||
const nextMeta: Record<string, unknown> = { ...base, [to.field]: value }
|
||||
if (!readBoth && from.kind === 'metadata') {
|
||||
delete nextMeta[from.field]
|
||||
}
|
||||
update.metadata = nextMeta as T
|
||||
update.merge = false
|
||||
} else {
|
||||
// data path — only meaningful when data is an object
|
||||
const base = (entity.data && typeof entity.data === 'object') ? { ...(entity.data as Record<string, unknown>) } : {}
|
||||
base[to.field] = value
|
||||
if (!readBoth && from.kind === 'data') {
|
||||
delete base[from.field]
|
||||
}
|
||||
update.data = base
|
||||
}
|
||||
|
||||
// Apply source clear if not already handled by the destination bag write.
|
||||
if (!readBoth) {
|
||||
if (from.kind === 'metadata' && to.kind !== 'metadata') {
|
||||
const base = (entity.metadata as unknown as Record<string, unknown>) ?? {}
|
||||
const nextMeta: Record<string, unknown> = { ...base }
|
||||
delete nextMeta[from.field]
|
||||
update.metadata = nextMeta as T
|
||||
update.merge = false
|
||||
} else if (from.kind === 'data' && to.kind !== 'data') {
|
||||
const base = (entity.data && typeof entity.data === 'object') ? { ...(entity.data as Record<string, unknown>) } : null
|
||||
if (base) {
|
||||
delete base[from.field]
|
||||
update.data = base
|
||||
}
|
||||
} else if (from.kind === 'top' && (from.field === 'subtype')) {
|
||||
// Only subtype currently supports clearing at the top level via UpdateParams.
|
||||
// Other standard fields aren't user-mutable through this path.
|
||||
;(update as any).subtype = undefined
|
||||
}
|
||||
}
|
||||
|
||||
return update
|
||||
}
|
||||
|
||||
// ============= NEW EMBEDDING & ANALYSIS APIs =============
|
||||
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -214,6 +214,14 @@ export interface HNSWNounWithMetadata {
|
|||
// TYPE (required, explicit)
|
||||
type: NounType
|
||||
|
||||
// SUBTYPE — optional per-product sub-classification within a NounType (e.g. a Person
|
||||
// entity might have subtype 'employee' / 'customer' / 'vendor'; an Event might have
|
||||
// subtype 'meeting' / 'milestone'). Flat string (no hierarchy) — consumers decide the
|
||||
// vocabulary. Indexed and rolled up into per-NounType statistics so it's queryable
|
||||
// (`find({ type, subtype })`) and aggregable (`groupBy:['subtype']`) on the standard-field
|
||||
// fast path, never falling through to the metadata fallback.
|
||||
subtype?: string
|
||||
|
||||
// QUALITY METRICS (top-level, explicit)
|
||||
confidence?: number
|
||||
weight?: number
|
||||
|
|
@ -250,6 +258,7 @@ export const STANDARD_ENTITY_FIELDS: ReadonlySet<string> = new Set([
|
|||
'connections',
|
||||
'level',
|
||||
'type',
|
||||
'subtype',
|
||||
'confidence',
|
||||
'weight',
|
||||
'createdAt',
|
||||
|
|
|
|||
|
|
@ -224,6 +224,18 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
protected nounCountsByType = new Uint32Array(NOUN_TYPE_COUNT) // 168 bytes (Stage 3: 42 types)
|
||||
protected verbCountsByType = new Uint32Array(VERB_TYPE_COUNT) // 508 bytes (Stage 3: 127 types)
|
||||
|
||||
/**
|
||||
* Per-NounType-per-subtype counts. Outer key is the NounType index (matching
|
||||
* `nounCountsByType` indexing); inner key is the subtype string. Populated
|
||||
* incrementally as entities are saved and decremented on delete; persisted
|
||||
* to `_system/subtype-statistics.json` alongside type-statistics.
|
||||
*
|
||||
* Memory: one Map entry per (type, subtype) pair actually used — typically
|
||||
* tens of inner entries per NounType in production. Sparse by design — types
|
||||
* with no subtype-bearing entities have no outer entry.
|
||||
*/
|
||||
protected subtypeCountsByType = new Map<number, Map<string, number>>()
|
||||
|
||||
/**
|
||||
* In-memory map from noun id → its `noun` (NounType) value, populated when
|
||||
* `saveNounMetadata_internal()` runs. `saveNoun_internal()` consults this
|
||||
|
|
@ -238,6 +250,14 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
* its lifetime and prunes on delete.
|
||||
*/
|
||||
protected nounTypeByIdCache = new Map<string, NounType>()
|
||||
|
||||
/**
|
||||
* In-memory map from noun id → its `subtype` string (when set). Parallel to
|
||||
* `nounTypeByIdCache`. Lets `deleteNounMetadata()` decrement the correct
|
||||
* subtype bucket without re-reading metadata. Sparse — only ids with a
|
||||
* non-empty subtype get an entry.
|
||||
*/
|
||||
protected nounSubtypeByIdCache = new Map<string, string>()
|
||||
// Total: 676 bytes (99.2% reduction vs Map-based tracking)
|
||||
|
||||
// Type caches REMOVED - ID-first paths eliminate need for type lookups!
|
||||
|
|
@ -364,6 +384,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
try {
|
||||
// Load type statistics from storage (if they exist)
|
||||
await this.loadTypeStatistics()
|
||||
await this.loadSubtypeStatistics()
|
||||
|
||||
// GraphAdjacencyIndex is now SINGLETON via getGraphIndex()
|
||||
// - Removed direct creation here to fix dual-ownership bug
|
||||
|
|
@ -864,7 +885,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
}
|
||||
|
||||
// Combine into HNSWNounWithMetadata - Extract standard fields to top-level
|
||||
const { noun, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata
|
||||
const { noun, subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata
|
||||
|
||||
return {
|
||||
id: vector.id,
|
||||
|
|
@ -873,6 +894,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
level: vector.level,
|
||||
// Standard fields at top-level
|
||||
type: (noun as NounType) || NounType.Thing,
|
||||
subtype: subtype as string | undefined,
|
||||
createdAt: (createdAt as number) || Date.now(),
|
||||
updatedAt: (updatedAt as number) || Date.now(),
|
||||
confidence: confidence as number | undefined,
|
||||
|
|
@ -901,12 +923,13 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
for (const noun of nouns) {
|
||||
const metadata = await this.getNounMetadata(noun.id)
|
||||
if (metadata) {
|
||||
const { noun: nounType, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata
|
||||
const { noun: nounType, subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata
|
||||
|
||||
nounsWithMetadata.push({
|
||||
...noun,
|
||||
// Standard fields at top-level
|
||||
type: (nounType as NounType) || NounType.Thing,
|
||||
subtype: subtype as string | undefined,
|
||||
createdAt: (createdAt as number) || Date.now(),
|
||||
updatedAt: (updatedAt as number) || Date.now(),
|
||||
confidence: confidence as number | undefined,
|
||||
|
|
@ -2202,6 +2225,27 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
this.nounTypeByIdCache.set(id, metadata.noun as NounType)
|
||||
}
|
||||
|
||||
// Track subtype changes: on type or subtype change via update(), decrement
|
||||
// the prior bucket before incrementing the new one. Symmetric with the
|
||||
// delete-path decrement in `deleteNounMetadata()`.
|
||||
const priorSubtype = this.nounSubtypeByIdCache.get(id)
|
||||
const priorTypeForSubtype = isNew ? undefined : (existingMetadata?.noun as NounType | undefined)
|
||||
const newSubtype = typeof metadata.subtype === 'string' && metadata.subtype.length > 0
|
||||
? metadata.subtype as string
|
||||
: undefined
|
||||
const newType = metadata.noun as NounType | undefined
|
||||
|
||||
if (priorSubtype && priorTypeForSubtype && (priorSubtype !== newSubtype || priorTypeForSubtype !== newType)) {
|
||||
this.decrementSubtypeCount(priorTypeForSubtype, priorSubtype)
|
||||
}
|
||||
if (newSubtype && newType && (isNew || priorSubtype !== newSubtype || priorTypeForSubtype !== newType)) {
|
||||
this.incrementSubtypeCount(newType, newSubtype)
|
||||
this.nounSubtypeByIdCache.set(id, newSubtype)
|
||||
} else if (!newSubtype && priorSubtype) {
|
||||
// Subtype cleared by an update — drop the cache entry (decrement already done above)
|
||||
this.nounSubtypeByIdCache.delete(id)
|
||||
}
|
||||
|
||||
// CRITICAL FIX: Increment count for new entities
|
||||
// This runs AFTER metadata is saved, guaranteeing type information is available
|
||||
// Uses synchronous increment since storage operations are already serialized
|
||||
|
|
@ -2385,7 +2429,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
const noun = this.deserializeNoun(vectorData)
|
||||
|
||||
// Extract standard fields to top-level
|
||||
const { noun: nounType, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadataData
|
||||
const { noun: nounType, subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadataData
|
||||
|
||||
results.set(id, {
|
||||
id: noun.id,
|
||||
|
|
@ -2394,6 +2438,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
level: noun.level,
|
||||
// Standard fields at top-level
|
||||
type: (nounType as NounType) || NounType.Thing,
|
||||
subtype: subtype as string | undefined,
|
||||
createdAt: (createdAt as number) || Date.now(),
|
||||
updatedAt: (updatedAt as number) || Date.now(),
|
||||
confidence: confidence as number | undefined,
|
||||
|
|
@ -2602,6 +2647,13 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
if (this.nounCountsByType[idx] > 0) {
|
||||
this.nounCountsByType[idx]--
|
||||
}
|
||||
|
||||
// Symmetric subtype decrement
|
||||
const priorSubtype = this.nounSubtypeByIdCache.get(id)
|
||||
if (priorSubtype) {
|
||||
this.nounSubtypeByIdCache.delete(id)
|
||||
this.decrementSubtypeCount(priorType, priorSubtype)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -2798,6 +2850,121 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
await this.writeObjectToPath(`${SYSTEM_DIR}/type-statistics.json`, stats)
|
||||
}
|
||||
|
||||
/**
|
||||
* Increment the (type, subtype) count, creating the inner map on first use.
|
||||
* Indexed by NounType index so it lines up with `nounCountsByType` and can
|
||||
* be reduced into per-type totals without re-keying.
|
||||
*/
|
||||
protected incrementSubtypeCount(type: NounType, subtype: string): void {
|
||||
const typeIdx = TypeUtils.getNounIndex(type)
|
||||
if (typeIdx < 0) return
|
||||
let inner = this.subtypeCountsByType.get(typeIdx)
|
||||
if (!inner) {
|
||||
inner = new Map<string, number>()
|
||||
this.subtypeCountsByType.set(typeIdx, inner)
|
||||
}
|
||||
inner.set(subtype, (inner.get(subtype) || 0) + 1)
|
||||
}
|
||||
|
||||
/**
|
||||
* Decrement the (type, subtype) count. Deletes the inner key when it reaches
|
||||
* 0 and the outer entry when its inner map is empty, so the persisted shape
|
||||
* stays compact across heavy churn.
|
||||
*/
|
||||
protected decrementSubtypeCount(type: NounType, subtype: string): void {
|
||||
const typeIdx = TypeUtils.getNounIndex(type)
|
||||
if (typeIdx < 0) return
|
||||
const inner = this.subtypeCountsByType.get(typeIdx)
|
||||
if (!inner) return
|
||||
const next = (inner.get(subtype) || 0) - 1
|
||||
if (next <= 0) {
|
||||
inner.delete(subtype)
|
||||
if (inner.size === 0) this.subtypeCountsByType.delete(typeIdx)
|
||||
} else {
|
||||
inner.set(subtype, next)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Load `_system/subtype-statistics.json` into `subtypeCountsByType`.
|
||||
* Persisted shape: `{ counts: { [typeIdx]: { [subtype]: count } }, updatedAt }`.
|
||||
* Missing file or parse error → start from empty (matches loadTypeStatistics).
|
||||
*/
|
||||
protected async loadSubtypeStatistics(): Promise<void> {
|
||||
try {
|
||||
const stats = await this.readObjectFromPath(`${SYSTEM_DIR}/subtype-statistics.json`)
|
||||
if (stats && stats.counts && typeof stats.counts === 'object') {
|
||||
this.subtypeCountsByType.clear()
|
||||
for (const [typeKey, subtypeMap] of Object.entries(stats.counts as Record<string, Record<string, number>>)) {
|
||||
const typeIdx = Number(typeKey)
|
||||
if (!Number.isInteger(typeIdx) || typeIdx < 0 || typeIdx >= NOUN_TYPE_COUNT) continue
|
||||
const inner = new Map<string, number>()
|
||||
for (const [subtype, count] of Object.entries(subtypeMap)) {
|
||||
if (typeof count === 'number' && count > 0) inner.set(subtype, count)
|
||||
}
|
||||
if (inner.size > 0) this.subtypeCountsByType.set(typeIdx, inner)
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// No existing subtype statistics, starting fresh.
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Save subtype statistics to storage. Mirrors the type-statistics persistence
|
||||
* cadence (called from `flushCounts()` and the periodic save in
|
||||
* `saveNoun_internal`).
|
||||
*/
|
||||
protected async saveSubtypeStatistics(): Promise<void> {
|
||||
const counts: Record<string, Record<string, number>> = {}
|
||||
for (const [typeIdx, inner] of this.subtypeCountsByType.entries()) {
|
||||
const innerObj: Record<string, number> = {}
|
||||
for (const [subtype, count] of inner.entries()) innerObj[subtype] = count
|
||||
counts[String(typeIdx)] = innerObj
|
||||
}
|
||||
await this.writeObjectToPath(`${SYSTEM_DIR}/subtype-statistics.json`, {
|
||||
counts,
|
||||
updatedAt: Date.now()
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Rebuild subtype counts from on-disk metadata. Companion to `rebuildTypeCounts()`
|
||||
* — used for poison recovery and explicit repair via `brainy inspect repair`.
|
||||
* O(N) over all nouns.
|
||||
*/
|
||||
public async rebuildSubtypeCounts(): Promise<void> {
|
||||
prodLog.info('[BaseStorage] Rebuilding subtype counts from storage...')
|
||||
this.subtypeCountsByType.clear()
|
||||
this.nounSubtypeByIdCache.clear()
|
||||
|
||||
for (let shard = 0; shard < 256; shard++) {
|
||||
const shardHex = shard.toString(16).padStart(2, '0')
|
||||
const shardDir = `entities/nouns/${shardHex}`
|
||||
try {
|
||||
const paths = await this.listObjectsInBranch(shardDir)
|
||||
for (const path of paths) {
|
||||
if (!path.includes('/metadata.json')) continue
|
||||
try {
|
||||
const metadata = await this.readWithInheritance(path)
|
||||
if (metadata && metadata.noun && typeof metadata.subtype === 'string' && metadata.subtype.length > 0) {
|
||||
this.incrementSubtypeCount(metadata.noun as NounType, metadata.subtype)
|
||||
// Path is `entities/nouns/<shard>/<id>/metadata.json` — extract id segment.
|
||||
const segments = path.split('/')
|
||||
const idSeg = segments[segments.length - 2]
|
||||
if (idSeg) this.nounSubtypeByIdCache.set(idSeg, metadata.subtype)
|
||||
}
|
||||
} catch { /* skip unreadable entities */ }
|
||||
}
|
||||
} catch { /* skip missing shards */ }
|
||||
}
|
||||
|
||||
await this.saveSubtypeStatistics()
|
||||
const totals = Array.from(this.subtypeCountsByType.values())
|
||||
.reduce((sum, inner) => sum + Array.from(inner.values()).reduce((s, n) => s + n, 0), 0)
|
||||
prodLog.info(`[BaseStorage] Rebuilt subtype counts: ${totals} entities across ${this.subtypeCountsByType.size} NounTypes`)
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist both counter systems atomically when an explicit flush is
|
||||
* requested. `super.flushCounts()` writes `entityCounts` (the Map in
|
||||
|
|
@ -2814,6 +2981,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
public async flushCounts(): Promise<void> {
|
||||
await super.flushCounts()
|
||||
await this.saveTypeStatistics()
|
||||
await this.saveSubtypeStatistics()
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -2825,6 +2993,17 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
return this.nounCountsByType
|
||||
}
|
||||
|
||||
/**
|
||||
* Get subtype counts by NounType (O(1) access to subtype statistics).
|
||||
* Returned map is the live in-memory view — callers must treat it as
|
||||
* read-only. Outer key: NounType index. Inner map: subtype → count.
|
||||
*
|
||||
* @returns Map keyed by NounType index → Map of subtype → count
|
||||
*/
|
||||
public getSubtypeCountsByType(): Map<number, Map<string, number>> {
|
||||
return this.subtypeCountsByType
|
||||
}
|
||||
|
||||
/**
|
||||
* Get verb counts by type (O(1) access to type statistics)
|
||||
* Exposed for MetadataIndexManager to use as single source of truth
|
||||
|
|
|
|||
|
|
@ -26,6 +26,13 @@ export interface Entity<T = any> {
|
|||
vector: Vector
|
||||
/** Entity type classification (NounType enum) */
|
||||
type: NounType
|
||||
/**
|
||||
* Per-product sub-classification within the NounType (e.g. a `Person` entity might
|
||||
* have `subtype: 'employee'` or `'customer'`; a `Document` might have `subtype: 'invoice'`).
|
||||
* Flat string, no hierarchy. Top-level standard field — indexed on the fast path and
|
||||
* rolled into per-NounType statistics.
|
||||
*/
|
||||
subtype?: string
|
||||
/** Opaque content — used for embeddings and semantic search. Not indexed by MetadataIndex. */
|
||||
data?: any
|
||||
/** User-defined structured fields — indexed and queryable via `where` filters. */
|
||||
|
|
@ -105,6 +112,7 @@ export interface Result<T = any> {
|
|||
|
||||
// Convenience: Common entity fields flattened to top level
|
||||
type?: NounType // Entity type (from entity.type)
|
||||
subtype?: string // Per-product sub-classification (from entity.subtype)
|
||||
metadata?: T // Entity metadata (from entity.metadata)
|
||||
data?: any // Entity data (from entity.data)
|
||||
confidence?: number // Type classification confidence (from entity.confidence)
|
||||
|
|
@ -158,6 +166,14 @@ export interface AddParams<T = any> {
|
|||
data: any | Vector
|
||||
/** Entity type classification (required) */
|
||||
type: NounType
|
||||
/**
|
||||
* Per-product sub-classification within the NounType (e.g. a `Person` entity might have
|
||||
* `subtype: 'employee'` or `'customer'`; a `Document` might have `subtype: 'invoice'`).
|
||||
* Flat string, no hierarchy — consumers choose the vocabulary. Indexed and rolled up into
|
||||
* per-NounType statistics for fast `find({ type, subtype })` filtering and `groupBy:['subtype']`
|
||||
* aggregation on the standard-field fast path.
|
||||
*/
|
||||
subtype?: string
|
||||
/** Structured queryable fields — indexed by MetadataIndex, used in `where` filters */
|
||||
metadata?: T
|
||||
/** Custom entity ID (auto-generated UUID v4 if not provided) */
|
||||
|
|
@ -181,6 +197,7 @@ export interface UpdateParams<T = any> {
|
|||
id: string // Entity to update
|
||||
data?: any // New content to re-embed
|
||||
type?: NounType // Change type
|
||||
subtype?: string // Change subtype (set to '' or null-equivalent via dedicated unset is future work)
|
||||
metadata?: Partial<T> // Metadata to update
|
||||
merge?: boolean // Merge or replace metadata (default: true)
|
||||
vector?: Vector // New pre-computed vector
|
||||
|
|
@ -252,6 +269,13 @@ export interface FindParams<T = any> {
|
|||
// Metadata Intelligence
|
||||
/** Filter by entity type(s). Alias for `where.noun`. */
|
||||
type?: NounType | NounType[]
|
||||
/**
|
||||
* Filter by per-product subtype (top-level standard field — uses the fast path, not the
|
||||
* metadata fallback). Pass a single string for equality, or an array for set membership
|
||||
* (e.g. `subtype: ['employee', 'contractor']`). For operator-form predicates (e.g.
|
||||
* `{ exists: true }`, `{ missing: true }`) use `where: { subtype: { …operators… } }`.
|
||||
*/
|
||||
subtype?: string | string[]
|
||||
/** Metadata filters using BFO operators (e.g., `{ year: { greaterThan: 2020 } }`) */
|
||||
where?: Partial<T>
|
||||
|
||||
|
|
|
|||
|
|
@ -373,7 +373,15 @@ export function validateUpdateParams(params: UpdateParams): void {
|
|||
}
|
||||
|
||||
// Universal truth: must update something
|
||||
if (!params.data && !params.metadata && !params.type && !params.vector) {
|
||||
if (
|
||||
!params.data &&
|
||||
!params.metadata &&
|
||||
!params.type &&
|
||||
!params.vector &&
|
||||
params.subtype === undefined &&
|
||||
params.confidence === undefined &&
|
||||
params.weight === undefined
|
||||
) {
|
||||
throw new Error('must specify at least one field to update')
|
||||
}
|
||||
|
||||
|
|
|
|||
333
tests/integration/subtype-and-facets.test.ts
Normal file
333
tests/integration/subtype-and-facets.test.ts
Normal file
|
|
@ -0,0 +1,333 @@
|
|||
/**
|
||||
* @module tests/integration/subtype-and-facets
|
||||
* @description Integration coverage for the 7.29.0 subtype-and-facets primitive
|
||||
* (Layer 1 top-level `subtype` field + rollup + counts API, Layer 2
|
||||
* `brain.trackField()`, Layer 3 `brain.migrateField()`).
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { Brainy } from '../../src/brainy'
|
||||
import { NounType } from '../../src/types/graphTypes'
|
||||
|
||||
describe('Subtype + Facets (7.29.0)', () => {
|
||||
let brain: Brainy<any>
|
||||
|
||||
beforeEach(async () => {
|
||||
brain = new Brainy({ storage: { type: 'memory' }, silent: true })
|
||||
await brain.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
describe('Layer 1: subtype as top-level standard field', () => {
|
||||
it('persists subtype on add() and round-trips through get()', async () => {
|
||||
const id = await brain.add({
|
||||
data: 'Avery operates the AI lab',
|
||||
type: NounType.Person,
|
||||
subtype: 'operator',
|
||||
metadata: { department: 'ai-lab' }
|
||||
})
|
||||
|
||||
const entity = await brain.get(id)
|
||||
expect(entity).not.toBeNull()
|
||||
expect(entity!.subtype).toBe('operator')
|
||||
expect(entity!.type).toBe(NounType.Person)
|
||||
// subtype lives at top-level, NOT in metadata
|
||||
expect((entity!.metadata as any).subtype).toBeUndefined()
|
||||
expect((entity!.metadata as any).department).toBe('ai-lab')
|
||||
})
|
||||
|
||||
it('allows omitting subtype (optional field)', async () => {
|
||||
const id = await brain.add({ data: 'plain entity', type: NounType.Thing })
|
||||
const entity = await brain.get(id)
|
||||
expect(entity!.subtype).toBeUndefined()
|
||||
})
|
||||
|
||||
it('updates subtype via update() without disturbing other fields', async () => {
|
||||
const id = await brain.add({
|
||||
data: 'switched roles',
|
||||
type: NounType.Person,
|
||||
subtype: 'contractor',
|
||||
metadata: { department: 'eng' }
|
||||
})
|
||||
|
||||
await brain.update({ id, subtype: 'employee' })
|
||||
const entity = await brain.get(id)
|
||||
expect(entity!.subtype).toBe('employee')
|
||||
expect((entity!.metadata as any).department).toBe('eng')
|
||||
})
|
||||
|
||||
it('preserves existing subtype when update() omits it', async () => {
|
||||
const id = await brain.add({
|
||||
data: 'unchanged subtype',
|
||||
type: NounType.Person,
|
||||
subtype: 'employee'
|
||||
})
|
||||
await brain.update({ id, metadata: { tag: 'updated' } })
|
||||
const entity = await brain.get(id)
|
||||
expect(entity!.subtype).toBe('employee')
|
||||
})
|
||||
|
||||
it('find({ type, subtype }) hits the fast path and matches', async () => {
|
||||
await brain.add({ data: 'a', type: NounType.Person, subtype: 'employee' })
|
||||
await brain.add({ data: 'b', type: NounType.Person, subtype: 'employee' })
|
||||
await brain.add({ data: 'c', type: NounType.Person, subtype: 'customer' })
|
||||
await brain.add({ data: 'd', type: NounType.Document, subtype: 'employee' })
|
||||
|
||||
const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })
|
||||
expect(employees).toHaveLength(2)
|
||||
for (const r of employees) {
|
||||
expect(r.subtype).toBe('employee')
|
||||
expect(r.type).toBe(NounType.Person)
|
||||
}
|
||||
})
|
||||
|
||||
it('find({ subtype: [...] }) matches set membership', async () => {
|
||||
await brain.add({ data: 'a', type: NounType.Person, subtype: 'employee' })
|
||||
await brain.add({ data: 'b', type: NounType.Person, subtype: 'contractor' })
|
||||
await brain.add({ data: 'c', type: NounType.Person, subtype: 'customer' })
|
||||
|
||||
const both = await brain.find({
|
||||
type: NounType.Person,
|
||||
subtype: ['employee', 'contractor']
|
||||
})
|
||||
expect(both).toHaveLength(2)
|
||||
const subtypes = both.map(r => r.subtype).sort()
|
||||
expect(subtypes).toEqual(['contractor', 'employee'])
|
||||
})
|
||||
|
||||
it('Result objects expose subtype at the top level', async () => {
|
||||
await brain.add({ data: 'a', type: NounType.Person, subtype: 'customer' })
|
||||
const results = await brain.find({ subtype: 'customer' })
|
||||
expect(results[0].subtype).toBe('customer')
|
||||
expect(results[0].entity?.subtype).toBe('customer')
|
||||
})
|
||||
})
|
||||
|
||||
describe('Layer 1: counts API', () => {
|
||||
beforeEach(async () => {
|
||||
await brain.add({ data: 'a', type: NounType.Person, subtype: 'employee' })
|
||||
await brain.add({ data: 'b', type: NounType.Person, subtype: 'employee' })
|
||||
await brain.add({ data: 'c', type: NounType.Person, subtype: 'customer' })
|
||||
await brain.add({ data: 'd', type: NounType.Person, subtype: 'customer' })
|
||||
await brain.add({ data: 'e', type: NounType.Person, subtype: 'customer' })
|
||||
await brain.add({ data: 'f', type: NounType.Document, subtype: 'invoice' })
|
||||
})
|
||||
|
||||
it('counts.bySubtype(type) returns the full breakdown', async () => {
|
||||
const counts = brain.counts.bySubtype(NounType.Person)
|
||||
expect(counts).toEqual({ employee: 2, customer: 3 })
|
||||
})
|
||||
|
||||
it('counts.bySubtype(type, subtype) returns the O(1) point count', async () => {
|
||||
expect(brain.counts.bySubtype(NounType.Person, 'employee')).toBe(2)
|
||||
expect(brain.counts.bySubtype(NounType.Person, 'customer')).toBe(3)
|
||||
expect(brain.counts.bySubtype(NounType.Person, 'never-existed')).toBe(0)
|
||||
})
|
||||
|
||||
it('counts.topSubtypes ranks by count, respects N', async () => {
|
||||
const top1 = brain.counts.topSubtypes(NounType.Person, 1)
|
||||
expect(top1).toEqual([['customer', 3]])
|
||||
|
||||
const all = brain.counts.topSubtypes(NounType.Person, 10)
|
||||
expect(all).toEqual([
|
||||
['customer', 3],
|
||||
['employee', 2]
|
||||
])
|
||||
})
|
||||
|
||||
it('brain.subtypesOf(type) returns sorted distinct subtypes', async () => {
|
||||
expect(brain.subtypesOf(NounType.Person)).toEqual(['customer', 'employee'])
|
||||
expect(brain.subtypesOf(NounType.Document)).toEqual(['invoice'])
|
||||
})
|
||||
|
||||
it('rollup decrements on delete', async () => {
|
||||
const id = await brain.add({ data: 'g', type: NounType.Person, subtype: 'employee' })
|
||||
expect(brain.counts.bySubtype(NounType.Person, 'employee')).toBe(3)
|
||||
|
||||
await brain.delete(id)
|
||||
expect(brain.counts.bySubtype(NounType.Person, 'employee')).toBe(2)
|
||||
})
|
||||
|
||||
it('rollup re-routes on subtype change via update', async () => {
|
||||
const id = await brain.add({ data: 'h', type: NounType.Person, subtype: 'employee' })
|
||||
expect(brain.counts.bySubtype(NounType.Person, 'employee')).toBe(3)
|
||||
expect(brain.counts.bySubtype(NounType.Person, 'customer')).toBe(3)
|
||||
|
||||
await brain.update({ id, subtype: 'customer' })
|
||||
expect(brain.counts.bySubtype(NounType.Person, 'employee')).toBe(2)
|
||||
expect(brain.counts.bySubtype(NounType.Person, 'customer')).toBe(4)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Layer 2: trackField() + counts.byField()', () => {
|
||||
it('tracks a metadata field and returns value frequencies', async () => {
|
||||
brain.trackField('status')
|
||||
await brain.add({ data: 'a', type: NounType.Task, metadata: { status: 'todo' } })
|
||||
await brain.add({ data: 'b', type: NounType.Task, metadata: { status: 'todo' } })
|
||||
await brain.add({ data: 'c', type: NounType.Task, metadata: { status: 'done' } })
|
||||
await brain.add({ data: 'd', type: NounType.Task, metadata: { status: 'done' } })
|
||||
await brain.add({ data: 'e', type: NounType.Task, metadata: { status: 'done' } })
|
||||
|
||||
const byStatus = await brain.counts.byField('status')
|
||||
expect(byStatus).toEqual({ todo: 2, done: 3 })
|
||||
})
|
||||
|
||||
it('perType breakdown returns per-NounType counts when filtered', async () => {
|
||||
brain.trackField('status', { perType: true })
|
||||
await brain.add({ data: 'a', type: NounType.Task, metadata: { status: 'todo' } })
|
||||
await brain.add({ data: 'b', type: NounType.Task, metadata: { status: 'done' } })
|
||||
await brain.add({ data: 'c', type: NounType.Event, metadata: { status: 'todo' } })
|
||||
|
||||
const taskStatuses = await brain.counts.byField('status', { type: NounType.Task })
|
||||
expect(taskStatuses).toEqual({ todo: 1, done: 1 })
|
||||
})
|
||||
|
||||
it('throws when per-type breakdown requested but field tracked without perType', async () => {
|
||||
brain.trackField('status')
|
||||
await expect(
|
||||
brain.counts.byField('status', { type: NounType.Task })
|
||||
).rejects.toThrow(/perType/)
|
||||
})
|
||||
|
||||
it('returns {} for fields never registered', async () => {
|
||||
const result = await brain.counts.byField('unregistered')
|
||||
expect(result).toEqual({})
|
||||
})
|
||||
|
||||
it('values whitelist rejects off-vocabulary writes', async () => {
|
||||
brain.trackField('priority', { values: ['low', 'medium', 'high'] })
|
||||
await expect(
|
||||
brain.add({ data: 'bad', type: NounType.Task, metadata: { priority: 'urgent' } })
|
||||
).rejects.toThrow(/priority.*urgent.*vocabulary/i)
|
||||
})
|
||||
|
||||
it('values whitelist accepts on-vocabulary writes', async () => {
|
||||
brain.trackField('priority', { values: ['low', 'medium', 'high'] })
|
||||
const id = await brain.add({
|
||||
data: 'ok',
|
||||
type: NounType.Task,
|
||||
metadata: { priority: 'high' }
|
||||
})
|
||||
const entity = await brain.get(id)
|
||||
expect((entity!.metadata as any).priority).toBe('high')
|
||||
})
|
||||
})
|
||||
|
||||
describe('Layer 3: migrateField()', () => {
|
||||
it('migrates metadata.X → top-level subtype and clears the source', async () => {
|
||||
// Set up the legacy shape: store kind inside metadata.
|
||||
const id1 = await brain.add({
|
||||
data: 'a',
|
||||
type: NounType.Person,
|
||||
metadata: { kind: 'employee', department: 'eng' }
|
||||
})
|
||||
const id2 = await brain.add({
|
||||
data: 'b',
|
||||
type: NounType.Person,
|
||||
metadata: { kind: 'customer' }
|
||||
})
|
||||
|
||||
const result = await brain.migrateField({
|
||||
from: 'metadata.kind',
|
||||
to: 'subtype'
|
||||
})
|
||||
expect(result.errors).toEqual([])
|
||||
expect(result.migrated).toBe(2)
|
||||
expect(result.scanned).toBeGreaterThanOrEqual(2)
|
||||
|
||||
const e1 = await brain.get(id1)
|
||||
expect(e1!.subtype).toBe('employee')
|
||||
expect((e1!.metadata as any).kind).toBeUndefined()
|
||||
expect((e1!.metadata as any).department).toBe('eng')
|
||||
|
||||
const e2 = await brain.get(id2)
|
||||
expect(e2!.subtype).toBe('customer')
|
||||
expect((e2!.metadata as any).kind).toBeUndefined()
|
||||
})
|
||||
|
||||
it('readBoth: true preserves the source field for the deprecation window', async () => {
|
||||
const id = await brain.add({
|
||||
data: 'a',
|
||||
type: NounType.Person,
|
||||
metadata: { kind: 'employee' }
|
||||
})
|
||||
|
||||
const result = await brain.migrateField({
|
||||
from: 'metadata.kind',
|
||||
to: 'subtype',
|
||||
readBoth: true
|
||||
})
|
||||
expect(result.migrated).toBe(1)
|
||||
|
||||
const entity = await brain.get(id)
|
||||
expect(entity!.subtype).toBe('employee')
|
||||
expect((entity!.metadata as any).kind).toBe('employee')
|
||||
})
|
||||
|
||||
it('is idempotent — re-running on already-migrated data is a no-op', async () => {
|
||||
await brain.add({
|
||||
data: 'a',
|
||||
type: NounType.Person,
|
||||
metadata: { kind: 'employee' }
|
||||
})
|
||||
|
||||
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
|
||||
const second = await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
|
||||
// All entities have already migrated → every entity is skipped on the second pass
|
||||
expect(second.migrated).toBe(0)
|
||||
expect(second.skipped).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
it('migrates data.X → subtype (when data is an object)', async () => {
|
||||
const id = await brain.add({
|
||||
data: { description: 'thing', kind: 'character' } as any,
|
||||
type: NounType.Person
|
||||
})
|
||||
|
||||
const result = await brain.migrateField({ from: 'data.kind', to: 'subtype' })
|
||||
expect(result.migrated).toBe(1)
|
||||
expect(result.errors).toEqual([])
|
||||
|
||||
const entity = await brain.get(id)
|
||||
expect(entity!.subtype).toBe('character')
|
||||
expect((entity!.data as any).description).toBe('thing')
|
||||
expect((entity!.data as any).kind).toBeUndefined()
|
||||
})
|
||||
|
||||
it('throws on identical from/to', async () => {
|
||||
await expect(
|
||||
brain.migrateField({ from: 'subtype', to: 'subtype' })
|
||||
).rejects.toThrow(/identical/)
|
||||
})
|
||||
|
||||
it('rejects unsupported path prefixes', async () => {
|
||||
await expect(
|
||||
brain.migrateField({ from: 'metadata.kind', to: 'connections.foo' })
|
||||
).rejects.toThrow(/unsupported path prefix/)
|
||||
})
|
||||
|
||||
it('migrates with the rollup tracking the new subtype counts', async () => {
|
||||
await brain.add({
|
||||
data: 'a',
|
||||
type: NounType.Person,
|
||||
metadata: { kind: 'employee' }
|
||||
})
|
||||
await brain.add({
|
||||
data: 'b',
|
||||
type: NounType.Person,
|
||||
metadata: { kind: 'customer' }
|
||||
})
|
||||
|
||||
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
|
||||
|
||||
// Rollup should now reflect the new subtype values
|
||||
expect(brain.counts.bySubtype(NounType.Person)).toEqual({
|
||||
employee: 1,
|
||||
customer: 1
|
||||
})
|
||||
})
|
||||
})
|
||||
})
|
||||
Loading…
Add table
Add a link
Reference in a new issue