feat: subtype top-level field + trackField + migrateField
Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).
Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
BaseStorage, mirrored after nounCountsByType with the same self-heal
rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path
Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
cardinality + per-NounType breakdown stats. Backed by the aggregation
engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update
Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
streams every entity, copies the value from one path to another, and
(unless readBoth) clears the source. Supports top-level standard fields,
metadata.X, and data.X paths. Idempotent — safe to re-run.
Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
quick-start all treat subtype as a core primitive with anonymous example
vocabularies (employee/customer/invoice/milestone).
Tests
- 26 new integration tests covering write/read/update/delete round-trips,
counts rollup decrement + re-route on mutation, trackField + byField
with and without perType, vocabulary whitelist enforcement, and
migrateField for metadata.X → subtype and data.X → subtype paths
including readBoth deprecation-window semantics.
Unit suite: 1468/1468 passing. Type-check + build clean.
This commit is contained in:
parent
e47fea0917
commit
2cdf70ee0f
14 changed files with 1698 additions and 21 deletions
|
|
@ -109,11 +109,12 @@ Add a single entity to the database.
|
|||
|
||||
```typescript
|
||||
const id = await brain.add({
|
||||
data: 'JavaScript is a programming language', // Text or pre-computed vector
|
||||
type: NounType.Concept, // Required: Entity type
|
||||
metadata: { // Optional metadata
|
||||
category: 'programming',
|
||||
year: 1995
|
||||
data: 'JavaScript is a programming language', // Text or pre-computed vector
|
||||
type: NounType.Concept, // Required: Entity type
|
||||
subtype: 'language', // Optional: sub-classification
|
||||
metadata: { // Optional: queryable fields
|
||||
category: 'programming',
|
||||
year: 1995
|
||||
}
|
||||
})
|
||||
```
|
||||
|
|
@ -121,6 +122,7 @@ const id = await brain.add({
|
|||
**Parameters:**
|
||||
- `data`: `string | number[]` - Content to embed (text auto-embeds) or pre-computed vector
|
||||
- `type`: `NounType` - Entity type (required)
|
||||
- `subtype?`: `string` - Per-product sub-classification within the NounType (top-level standard field, indexed on the fast path). See [Subtypes & Facets](../guides/subtypes-and-facets.md).
|
||||
- `metadata?`: `object` - Structured queryable fields (indexed by MetadataIndex, used in `where` filters)
|
||||
- `id?`: `string` - Custom ID (auto-generated UUID if not provided)
|
||||
- `vector?`: `number[]` - Pre-computed vector (skips auto-embedding)
|
||||
|
|
@ -158,15 +160,20 @@ Update an existing entity.
|
|||
```typescript
|
||||
await brain.update({
|
||||
id: entityId,
|
||||
data: 'Updated content', // Optional: new data
|
||||
metadata: { updated: true } // Optional: new metadata (merges)
|
||||
data: 'Updated content', // Optional: new data
|
||||
subtype: 'archived', // Optional: change sub-classification
|
||||
metadata: { updated: true } // Optional: new metadata (merges)
|
||||
})
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `id`: `string` - Entity ID
|
||||
- `data?`: `string | number[]` - New data/vector
|
||||
- `metadata?`: `object` - Metadata to merge
|
||||
- `type?`: `NounType` - Change entity type
|
||||
- `subtype?`: `string` - Change subtype (omit to preserve existing)
|
||||
- `metadata?`: `object` - Metadata to merge (or replace with `merge: false`)
|
||||
- `confidence?`: `number` - Update classification confidence
|
||||
- `weight?`: `number` - Update entity importance
|
||||
|
||||
**Returns:** `Promise<void>`
|
||||
|
||||
|
|
@ -221,6 +228,7 @@ const results = await brain.find({
|
|||
**FindParams:**
|
||||
- `query?`: `string` - Text for semantic + hybrid search (searches `data` via HNSW + text index)
|
||||
- `type?`: `NounType | NounType[]` - Filter by entity type(s). Alias for `where.noun`.
|
||||
- `subtype?`: `string | string[]` - Filter by sub-classification (top-level standard field, fast path). Single string for equality, array for set membership.
|
||||
- `where?`: `object` - Metadata filters. See **[Query Operators](../QUERY_OPERATORS.md)** for all operators.
|
||||
- `connected?`: `object` - Graph traversal options
|
||||
- `to?`: `string` - Target entity ID
|
||||
|
|
@ -1928,6 +1936,86 @@ const count = await brain.getVerbCount()
|
|||
|
||||
---
|
||||
|
||||
### Subtype & facet APIs
|
||||
|
||||
Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**.
|
||||
|
||||
#### `counts.bySubtype(type, subtype?)` → `Record<string, number> | number`
|
||||
|
||||
O(1) subtype counts for a NounType (backed by the persisted rollup).
|
||||
|
||||
```typescript
|
||||
brain.counts.bySubtype(NounType.Person)
|
||||
// → { employee: 12, customer: 847, vendor: 34 }
|
||||
|
||||
brain.counts.bySubtype(NounType.Person, 'employee')
|
||||
// → 12
|
||||
```
|
||||
|
||||
#### `counts.topSubtypes(type, n=10)` → `Array<[subtype, count]>`
|
||||
|
||||
Top N subtypes ranked by count.
|
||||
|
||||
```typescript
|
||||
brain.counts.topSubtypes(NounType.Person, 3)
|
||||
// → [['customer', 847], ['employee', 12], ['vendor', 34]]
|
||||
```
|
||||
|
||||
#### `subtypesOf(type)` → `string[]`
|
||||
|
||||
Sorted distinct subtypes seen for a NounType.
|
||||
|
||||
```typescript
|
||||
brain.subtypesOf(NounType.Person)
|
||||
// → ['customer', 'employee', 'vendor']
|
||||
```
|
||||
|
||||
#### `trackField(name, options?)` → `void`
|
||||
|
||||
Register a metadata field for cardinality + per-NounType breakdown stats. With `values: [...]`, validates against the whitelist on `add()`/`update()`.
|
||||
|
||||
```typescript
|
||||
brain.trackField('status') // basic
|
||||
brain.trackField('status', { perType: true }) // with per-NounType breakdown
|
||||
brain.trackField('priority', { values: ['low', 'med', 'high'] }) // strict vocabulary
|
||||
```
|
||||
|
||||
#### `counts.byField(name, options?)` → `Promise<Record<string, number>>`
|
||||
|
||||
Counts by value for a tracked field. Requires `perType: true` registration if filtering by NounType.
|
||||
|
||||
```typescript
|
||||
await brain.counts.byField('status')
|
||||
// → { todo: 12, doing: 3, done: 47 }
|
||||
|
||||
await brain.counts.byField('status', { type: NounType.Task })
|
||||
// → { todo: 8, doing: 2, done: 30 }
|
||||
```
|
||||
|
||||
#### `migrateField(options)` → `Promise<MigrationSummary>`
|
||||
|
||||
Stream-and-rewrite a field across the brain. Supports `metadata.X`, `data.X`, and top-level paths. Idempotent.
|
||||
|
||||
```typescript
|
||||
// One-shot rewrite
|
||||
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
|
||||
|
||||
// Deprecation window — keep source field readable
|
||||
await brain.migrateField({ from: 'data.kind', to: 'subtype', readBoth: true })
|
||||
|
||||
// With progress reporting
|
||||
await brain.migrateField({
|
||||
from: 'metadata.kind',
|
||||
to: 'subtype',
|
||||
batchSize: 500,
|
||||
onProgress: ({ scanned, migrated }) => console.log(`${scanned} / ${migrated}`)
|
||||
})
|
||||
```
|
||||
|
||||
Returns `{ scanned: number, migrated: number, skipped: number, errors: Array<{id, error}> }`.
|
||||
|
||||
---
|
||||
|
||||
### `embed(data)` → `Promise<number[]>` ✨
|
||||
|
||||
Generate embedding vector from text or data.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue