feat: subtype top-level field + trackField + migrateField

Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).

Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
  BaseStorage, mirrored after nounCountsByType with the same self-heal
  rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path

Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
  cardinality + per-NounType breakdown stats. Backed by the aggregation
  engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update

Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
  streams every entity, copies the value from one path to another, and
  (unless readBoth) clears the source. Supports top-level standard fields,
  metadata.X, and data.X paths. Idempotent — safe to re-run.

Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
  quick-start all treat subtype as a core primitive with anonymous example
  vocabularies (employee/customer/invoice/milestone).

Tests
- 26 new integration tests covering write/read/update/delete round-trips,
  counts rollup decrement + re-route on mutation, trackField + byField
  with and without perType, vocabulary whitelist enforcement, and
  migrateField for metadata.X → subtype and data.X → subtype paths
  including readBoth deprecation-window semantics.

Unit suite: 1468/1468 passing. Type-check + build clean.
This commit is contained in:
David Snelling 2026-06-04 17:24:36 -07:00
parent e47fea0917
commit 2cdf70ee0f
14 changed files with 1698 additions and 21 deletions

View file

@ -11,6 +11,139 @@ Collective. The SDK wraps it — most products never call Brainy directly. Read
---
## v7.29.0 — 2026-06-04
**Affected products:** anyone modeling entities with per-product sub-classification — every
consumer that's been reaching for `metadata.kind`, a custom `metadata.subtype` field, a
`data.kind` shape inside the payload, or similar ad-hoc conventions. Additive; drop-in from
7.28.x. No deprecations.
### New — `subtype` promoted to a top-level standard field
`subtype?: string` is now a first-class standard field on every entity, alongside `type` /
`confidence` / `weight`. Use it as the platform-standard primitive for sub-classifying entities
within a NounType (`Person``'employee'` / `'customer'`; `Document``'invoice'` / `'contract'`;
`Event``'milestone'` / `'meeting'`):
```typescript
await brain.add({
data: 'Avery Brooks — runs the AI lab',
type: NounType.Person,
subtype: 'employee', // top-level write param
metadata: { department: 'ai-lab' }
})
// Fast-path filter — column-store hit, not metadata fallback:
const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })
// Set membership:
const internal = await brain.find({
type: NounType.Person,
subtype: ['employee', 'contractor']
})
```
Flat string, no hierarchy — your vocabulary, your choice. Brainy stores and counts, never validates.
### New — O(1) subtype counts via the persisted rollup
A new `_system/subtype-statistics.json` rollup is maintained incrementally as entities are added,
updated, and deleted — mirroring the existing `nounCountsByType` machinery with the same self-heal
behavior on poison detection. Counts are O(1) at billion scale:
```typescript
brain.counts.bySubtype(NounType.Person)
// → { employee: 12, customer: 847, vendor: 34 }
brain.counts.bySubtype(NounType.Person, 'employee') // O(1) point count
// → 12
brain.counts.topSubtypes(NounType.Person, 3)
// → [['customer', 847], ['employee', 12], ['vendor', 34]]
brain.subtypesOf(NounType.Person)
// → ['customer', 'employee', 'vendor']
```
### New — `brain.trackField()` for other metadata facets
For facets that aren't the *primary* sub-classification (`status`, `source`, `role`, `paradigm`),
`trackField` registers a field for cardinality + per-NounType breakdown stats without promoting
each one to a top-level slot. Piggybacks on the existing aggregation engine, so backfill-on-define
applies — registering on a populated brain scans existing entities on the first query:
```typescript
brain.trackField('status', { perType: true })
await brain.counts.byField('status')
// → { todo: 12, doing: 3, done: 47 }
await brain.counts.byField('status', { type: NounType.Task })
// → { todo: 8, doing: 2, done: 30 }
// Opt-in vocabulary validation:
brain.trackField('priority', { values: ['low', 'medium', 'high'] })
// brain.add({ ..., metadata: { priority: 'urgent' } }) → throws
```
### New — generic `brain.migrateField()` for one-shot rewrites
Streams every entity and copies a value from one field path to another. Supports top-level
standard fields, `metadata.*`, and `data.*` paths; idempotent; with optional `readBoth: true`
deprecation window that preserves the source field alongside the new one:
```typescript
// Phase 1: dual-populate (legacy readers still work)
await brain.migrateField({
from: 'metadata.kind',
to: 'subtype',
readBoth: true
})
// ...readers migrate to subtype at their own pace...
// Phase 2: clear the source field
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
// → { scanned: 1500, migrated: 1500, skipped: 0, errors: [] }
```
Supports `batchSize` and `onProgress` for large brains.
### Aggregation composition
`subtype` is a standard-field group-by dimension out of the box — no `where:` wrapper, no
metadata-fallback overhead:
```typescript
await brain.find({
aggregate: {
groupBy: ['type', 'subtype'],
metrics: { count: { op: 'COUNT' } }
}
})
// [
// { groupKey: { type: 'person', subtype: 'customer' }, count: 847 },
// { groupKey: { type: 'person', subtype: 'employee' }, count: 12 },
// { groupKey: { type: 'document', subtype: 'invoice' }, count: 2103 },
// ...
// ]
```
### Cortex compatibility
Subtype works under Cortex out of the box via auto-field-indexing. Cortex will land its own
native-side query-planner recognition + `subtypeCountsByType` native rollup in the next Cortex
release for full parity with `type`'s fast path. Not a Brainy blocker — subtype reads/writes
function correctly with current Cortex.
### Docs
Full guide: `docs/guides/subtypes-and-facets.md`. Subtype is documented as a core primitive
throughout `README.md`, `docs/DATA_MODEL.md`, `docs/architecture/finite-type-system.md`,
`docs/api/README.md`, and `docs/QUERY_OPERATORS.md`.
---
## v7.24.0 — 2026-05-26
**Affected products:** aggregation/reporting users + anyone calling `extractEntities` on