David Snelling 1f7e365a4e chore(8.0): final pre-RC1 sweep — API consistency, named errors, orphans, zero-cast codebase

2026-06-11 14:51:00 -07:00

21 KiB

Raw Blame History

title

slug

public

Subtypes & Facets

Sub-classify entities within a NounType, track arbitrary metadata facets, and migrate field names without downtime.

Why this exists

Brainy's NounType (Person, Document, Event, Concept, Task, …) is a stable, flat 42-type taxonomy. It's deliberately coarse — every product needs a way to further classify entities within a type. Is this Person an employee or a customer? Is this Document an invoice or a contract? Is this Event a meeting or a milestone?

Three layers solve this:

Layer	Use it for	Shape
`subtype`	The primary sub-classification of an entity, one value per entity	Top-level standard field
`trackField()`	Other facets you want to count or filter on (`status`, `source`, `role`, `paradigm`)	Registered metadata field
`migrateField()`	Renaming or restructuring fields across an existing dataset	One-shot stream-and-rewrite

Layer 1 — `subtype`

subtype is a top-level standard field on every entity, alongside type / confidence / weight. Flat string, no hierarchy. The vocabulary is your choice — Brainy stores and counts, never validates.

Write

import { Brainy, NounType } from '@soulcraft/brainy'

const brain = new Brainy()
await brain.init()

await brain.add({
  data: 'Avery Brooks — runs the AI lab',
  type: NounType.Person,
  subtype: 'employee',                  // top-level write param
  metadata: { department: 'ai-lab' }
})

Read

// Top-level filter — standard-field fast path, no `where` wrapper:
const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })

// Set membership:
const internal = await brain.find({
  type: NounType.Person,
  subtype: ['employee', 'contractor']
})

// Operator-form predicates use `where`:
const typed = await brain.find({
  type: NounType.Person,
  where: { subtype: { exists: true } }
})

What it looks like

{
  "id": "01HZK3M7TW...",
  "type": "person",
  "subtype": "employee",
  "data": "Avery Brooks — runs the AI lab",
  "metadata": { "department": "ai-lab" }
}

subtype lives at the top level — NOT inside metadata, NOT inside data. That's what makes the fast path possible: queries on subtype hit the column-store index directly, never the metadata fallback.

Counts (O(1))

// All subtypes for a NounType
brain.counts.bySubtype(NounType.Person)
// → { employee: 12, customer: 847, vendor: 34 }

// Point count
brain.counts.bySubtype(NounType.Person, 'employee')
// → 12

// Top N
brain.counts.topSubtypes(NounType.Person, 3)
// → [['customer', 847], ['employee', 12], ['vendor', 34]]

// Distinct subtypes for a NounType
brain.subtypesOf(NounType.Person)
// → ['customer', 'employee', 'vendor']

These are O(1) lookups backed by _system/subtype-statistics.json — no scan, no storage round-trip. The rollup is incrementally maintained as entities are added, updated, and deleted.

Aggregation

subtype is a first-class group-by dimension:

const rows = await brain.find({
  aggregate: {
    groupBy: ['type', 'subtype'],
    metrics: { count: { op: 'COUNT' } }
  }
})
// [
//   { groupKey: { type: 'person',   subtype: 'customer' }, count: 847 },
//   { groupKey: { type: 'person',   subtype: 'employee' }, count: 12 },
//   { groupKey: { type: 'document', subtype: 'invoice'  }, count: 2103 },
//   ...
// ]

Layer 2 — `trackField()`

Some metadata fields aren't the sub-classification of an entity but are still worth counting: status, source, role, paradigm. Promoting each one to top-level would clutter the contract. trackField() registers a metadata field for cardinality + per-NounType breakdown stats without the contract growth.

Register and query

// Track a single facet
brain.trackField('status')

await brain.add({ data: 'Ship subtype', type: NounType.Task, metadata: { status: 'todo' } })
await brain.add({ data: 'Write docs',   type: NounType.Task, metadata: { status: 'done' } })

await brain.counts.byField('status')
// → { todo: 1, done: 1 }

Per-NounType breakdown

brain.trackField('status', { perType: true })

await brain.counts.byField('status', { type: NounType.Task })
// → { todo: 1, done: 1 }

Vocabulary whitelist (opt-in validation)

brain.trackField('priority', { values: ['low', 'medium', 'high'] })

// Throws — 'urgent' isn't in the vocabulary:
await brain.add({
  data: 'Fix bug',
  type: NounType.Task,
  metadata: { priority: 'urgent' }
})

trackField piggybacks on the existing aggregation engine (see the Aggregation guide for the underlying mechanism). Backfill-on-define means the first call to counts.byField() scans existing entities; subsequent calls are O(groups).

subtype vs trackField — when to use which

subtype when there's one primary sub-classification per entity. Limit yourself to one per NounType. Examples: Person→employee/customer/vendor; Document→invoice/contract/policy.
trackField for anything else you want to count or filter on. No limit on how many you register. Examples: status, source, role, paradigm.

Layer 3 — `migrateField()`

Use this when you need to rename or restructure a field across an entire dataset — for example, moving a metadata.kind convention up to the top-level subtype standard field.

One-shot rewrite

// Starting state: every entity has metadata.kind
const result = await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype'
})

console.log(result)
// {
//   scanned: 1500,
//   migrated: 1500,
//   skipped: 0,
//   errors: []
// }

After this returns, every entity has subtype populated from the old metadata.kind value, and metadata.kind is cleared.

Deprecation window — keep both fields readable

When you can't coordinate all readers and the migration in a single deploy, use readBoth: true to preserve the source field alongside the new one:

// Phase 1: dual-populate (existing readers still work against metadata.kind):
await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype',
  readBoth: true
})

// ... readers migrate to query subtype at their own pace ...

// Phase 2: clear the source field when ready:
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })

Supported paths

Path form	Refers to
`'subtype'`, `'type'`, `'confidence'`	Top-level standard fields
`'metadata.X'`	A key under `entity.metadata`
`'data.X'`	A key under `entity.data` (when `data` is an object)
`'X'` (bare, non-standard)	Shorthand for `metadata.X`

Idempotent

migrateField is safe to re-run. Entities where the source is absent, or where the destination already holds the same value, are skipped. This makes it safe to use in a deploy-once-then-cleanup workflow.

Progress reporting

await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype',
  batchSize: 500,
  onProgress: ({ scanned, migrated }) => {
    console.log(`${scanned} scanned, ${migrated} migrated`)
  }
})

Putting it together

A realistic adoption sequence for a brain that started without these primitives:

import { Brainy, NounType } from '@soulcraft/brainy'

const brain = new Brainy({ storage: { type: 'filesystem', options: { path: './brain-data' } } })
await brain.init()

// 1. Migrate any existing metadata.kind convention to the new top-level subtype
await brain.migrateField({ from: 'metadata.kind', to: 'subtype', readBoth: true })

// 2. Register the other facets you want counted
brain.trackField('status', { perType: true })
brain.trackField('source')

// 3. Use subtype on every new write
await brain.add({
  data: 'Quarterly review',
  type: NounType.Event,
  subtype: 'milestone',
  metadata: { status: 'todo', source: 'planning-session' }
})

// 4. Query the breakdowns
brain.counts.bySubtype(NounType.Event)
// → { milestone: 14, meeting: 203, deadline: 7 }

await brain.counts.byField('status', { type: NounType.Event })
// → { todo: 12, done: 212 }

// 5. Once all readers are on the new field, drop the source:
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })

Layer V — `subtype` on relationships (verbs)

Relationships are first-class citizens too. Every verb (VerbType) gets the same subtype primitive: a ReportsTo relationship might carry subtype: 'direct' vs 'dotted-line'; a RelatedTo edge might carry 'spouse' / 'sibling' / 'colleague'. Same shape as the noun side: flat string, no hierarchy, top-level standard field, indexed on the fast path.

Write

const ceoId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Avery' })
const vpId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Jordan' })
const matrixId = await brain.add({ type: NounType.Person, subtype: 'contractor', data: 'Sam' })

await brain.relate({
  from: ceoId,
  to: vpId,
  type: VerbType.ReportsTo,
  subtype: 'direct'
})

await brain.relate({
  from: ceoId,
  to: matrixId,
  type: VerbType.ReportsTo,
  subtype: 'dotted-line'
})

Read & filter

// Direct reports — fast path filter (column-store hit, not metadata fallback)
const direct = await brain.related({
  from: ceoId,
  type: VerbType.ReportsTo,
  subtype: 'direct'
})

// Set membership
const allReports = await brain.related({
  from: ceoId,
  type: VerbType.ReportsTo,
  subtype: ['direct', 'dotted-line']
})

Update (new — `updateRelation()`)

Verbs previously had no update method — the only way to change a relationship was delete-then-recreate. 7.30 closes that gap:

// Promote a dotted-line report to direct without losing the edge id
await brain.updateRelation({ id: relationId, subtype: 'direct' })

// Or change weight/confidence
await brain.updateRelation({ id: relationId, weight: 0.5, confidence: 0.9 })

Traversal filter

find({ connected, subtype }) filters traversal edges by their subtype. Composes with via (verb-type filter):

// All direct reports two hops deep (will be supported in Cortex; for now depth-1)
const directChain = await brain.find({
  connected: {
    from: ceoId,
    via: VerbType.ReportsTo,
    subtype: 'direct',
    depth: 1
  }
})

Multi-hop subtype filtering (depth > 1) lights up on the Cortex native path; the JS path throws today rather than return incorrect partial results.

Counts

Same shape as the noun-side counts API:

brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
// → { direct: 12, 'dotted-line': 3 }

brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct')   // O(1) point
// → 12

brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 3)
// → [['direct', 12], ['dotted-line', 3]]

brain.relationshipSubtypesOf(VerbType.ReportsTo)
// → ['direct', 'dotted-line']

These are O(1) lookups backed by the persisted _system/verb-subtype-statistics.json rollup — same self-heal machinery as the noun-side rollup.

Migrate verb fields

migrateField() now walks verbs too:

// Migrate verb-side metadata.kind → top-level subtype
await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype',
  entityKind: 'verb'
})

// Or migrate both nouns and verbs in one pass
await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype',
  entityKind: 'both'
})

Default is entityKind: 'noun' (backward-compatible with 7.29).

Enforcement — `requireSubtype()` + brain-wide strict mode

By default (7.30) subtype is optional. Two complementary opt-in mechanisms let you enforce the pairing of type + subtype on every write:

Per-type registration

Mark a specific NounType or VerbType as requiring a subtype, optionally with a fixed vocabulary:

brain.requireSubtype(NounType.Person, {
  values: ['employee', 'customer', 'vendor'],
  required: true
})

brain.requireSubtype(VerbType.ReportsTo, {
  values: ['direct', 'dotted-line'],
  required: true
})

// Now this throws — Person requires a subtype:
await brain.add({ type: NounType.Person, data: 'no subtype' })

// And this throws — 'matrix' isn't in the registered vocabulary:
await brain.relate({
  from: a,
  to: b,
  type: VerbType.ReportsTo,
  subtype: 'matrix'
})

Brain-wide strict mode

Enforce on every write across the whole brain:

const brain = new Brainy({ requireSubtype: true })

// Allow specific types to omit subtype (e.g. catch-all `Thing`)
const brain2 = new Brainy({
  requireSubtype: { except: [NounType.Thing, NounType.Custom] }
})

When strict mode is on:

Every add() / addMany() / update() / relate() / relateMany() / updateRelation() rejects writes missing a subtype on a non-exempt type.
addMany() and relateMany() validate every item BEFORE any storage write — atomic-fail semantics, no partial writes.
Per-type rules registered via requireSubtype() compose with the brain-wide flag; specific rules win when both apply.
Brainy's own internal writes (VFS root, VFS directories, VFS file entities) bypass enforcement via the metadata.isVFSEntity: true infrastructure marker.

Migrating to strict mode on an existing brain

brain.migrateField() is your friend — populate subtype from an existing convention before flipping strict mode on:

// Step 1: backfill subtype from your existing metadata.kind convention
await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype',
  entityKind: 'both'
})

// Step 2: register the vocabulary
brain.requireSubtype(NounType.Person, {
  values: ['employee', 'customer', 'vendor'],
  required: true
})

// Step 3: future writes must have a subtype matching the vocabulary
await brain.add({ type: NounType.Person, subtype: 'employee', data: '...' })

Strict mode in practice (for SDK-style vocabulary consumers)

When a platform layer like the Soulcraft SDK registers requireSubtype() rules on behalf of every consumer's brain, every downstream product that calls brain.add() / brain.relate() against those types must pass a matching subtype. Skipping the field — or passing one outside the registered vocabulary — throws at the boundary.

This pattern is powerful but surfaces a class of latent bug: any brain.add() call site that was written before strict-mode adoption starts rejecting writes. A representative production incident: a booking flow started returning 500 on every request because a service method called brain.add({ type: NounType.Event, ... }) without subtype, and an SDK layer had just registered requireSubtype() for NounType.Event on every brain instance.

The fix is a four-step migration recipe — and Brainy 7.30.1+ ships diagnostic tools to make it deterministic.

Migration recipe

Inventory the gap with brain.audit() — returns the deterministic list of which NounTypes and VerbTypes have entities/relationships missing subtype, grouped by type:
```
const report = await brain.audit()
//   {
//     entitiesWithoutSubtype: { event: 24, document: 3, ... },
//     relationshipsWithoutSubtype: { relatedTo: 1402 },
//     total: 1429,
//     scanned: 8400,
//     recommendation: 'Found 1429 entries without subtype. ...'
//   }
```
By default, VFS infrastructure entities are excluded (they bypass enforcement anyway via the metadata.isVFSEntity marker). Pass { includeVFS: true } to surface them too.

Bulk-migrate any existing convention with brain.migrateField() if a legacy field can be lifted:

// Common pattern: subtype mirrors a discriminator field already in metadata
await brain.migrateField({
  from: 'metadata.entityType',
  to: 'subtype',
  readBoth: true  // safety: keep the source field readable during cutover
})

Hand-fix the remaining call sites. The exact list is in report.entitiesWithoutSubtype. For each call site, add subtype: '<value>' to the brain.add() / brain.relate() params. Choose a stable convention (e.g. mirror metadata.entityType if you have one; any rule that's deterministic from the data works).
Verify with brain.audit() again. Re-run; total should be 0. If you turn on brain-wide strict mode at this point, all future writes are protected.

Brainy's own infrastructure subtype labels (reference)

Brainy's internal write paths set subtype on every entity and edge they create. Consumers don't need to do anything for these — they're documented here so you understand the data shape:

Code path	NounType / VerbType	Subtype label
VFS root directory `/`	`NounType.Collection`	`'vfs-root'`
VFS subdirectories	`NounType.Collection`	`'vfs-directory'`
VFS files	(mime-driven, e.g. `Document`/`Code`/`Image`)	`'vfs-file'`
VFS symlinks	`NounType.File`	`'vfs-symlink'`
VFS Contains edges	`VerbType.Contains`	`'vfs-contains'`
Aggregation materialized output	`NounType.Measurement`	`'materialized-aggregate'`
Import-document provenance entity	`NounType.Document`	`'import-source'`
Importer-extracted entities (no caller default)	extractor-driven	`'imported'`
Importer placeholder targets	`NounType.Thing`	`'import-placeholder'`
Neural extraction (no caller default)	extractor-driven	`'extracted'`
GoogleSheets API entity writes	request-driven	`'imported-from-sheets'`
OData API entity writes	request-driven	`'imported-from-odata'`
MCP client message storage	`NounType.Message`	`'mcp-message'`
`brainy add` CLI (no `--subtype` flag)	user-supplied type	`'cli-add'`
`brainy relate` CLI (no `--subtype` flag)	user-supplied verb	`'cli-relate'`

You can query these directly: await brain.find({ subtype: 'vfs-file' }) returns every VFS-managed file regardless of NounType. await brain.counts.bySubtype(NounType.Document) shows you the import-source / imported / extracted / vfs-file breakdown.

Importer and extraction paths accept a caller-supplied defaultSubtype option so you can tag a whole batch with your own provenance label (e.g. 'customer-upload-2026q2') instead of the Brainy default 'imported' / 'extracted'.

Looking ahead — Brainy 8.0

Brainy 8.0 ships:

brain.fillSubtypes(rules) — the bulk migration helper that pairs with audit(). Given caller-supplied rules per NounType / VerbType, it walks the brain and fills in missing subtypes via update(). Pre-8.0 brains run this once before upgrading to clear migration debt.
subtype: string (non-optional) on AddParams<T> and RelateParams<T>. TypeScript catches missing subtype at compile time, not just runtime.
new Brainy({ requireSubtype: true }) becomes the default. Consumers explicitly opt out with { requireSubtype: false } during migration.

7.30.1's audit() is the diagnostic; 8.0's fillSubtypes() is the bulk fixer. Together they close the migration gap deterministically.

Reference

Layer 1 — `subtype` (nouns)

brain.add({ ..., subtype: 'value' }) — write the field
brain.update({ id, subtype: 'value' }) — change the field
brain.find({ type, subtype }) — filter (fast path)
brain.find({ subtype: ['a', 'b'] }) — set membership
brain.counts.bySubtype(type, subtype?) — O(1) counts
brain.counts.topSubtypes(type, n?) — top N by count
brain.subtypesOf(type) — distinct subtype list

Layer V — `subtype` (verbs / relationships)

brain.relate({ ..., subtype: 'value' }) — write the field
brain.updateRelation({ id, subtype, type?, weight?, ... }) — change a relationship
brain.related({ subtype }) — filter (fast path)
brain.related({ subtype: ['a', 'b'] }) — set membership
brain.find({ connected: { via, subtype, depth } }) — traversal filter
brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) counts
brain.counts.topRelationshipSubtypes(verb, n?) — top N by count
brain.relationshipSubtypesOf(verb) — distinct subtype list

Layer 2 — generic facets

brain.trackField(name, { perType?, values? }) — register a facet
brain.counts.byField(name, { type? }) — facet counts

Layer 3 — migration

brain.migrateField({ from, to, readBoth?, batchSize?, onProgress?, entityKind? }) — rewrite a field (nouns, verbs, or both)

Enforcement

brain.requireSubtype(type, { values?, required }) — per-NounType / VerbType rule
new Brainy({ requireSubtype: true }) — brain-wide strict mode
new Brainy({ requireSubtype: { except: [type, ...] } }) — strict with exemptions

21 KiB Raw Blame History

Subtypes & Facets

Why this exists

Layer 1 — subtype

Write

Read

What it looks like

Counts (O(1))

Aggregation

Layer 2 — trackField()

Register and query

Per-NounType breakdown

Vocabulary whitelist (opt-in validation)

subtype vs trackField — when to use which

Layer 3 — migrateField()

One-shot rewrite

Deprecation window — keep both fields readable

Supported paths

Idempotent

Progress reporting

Putting it together

Layer V — subtype on relationships (verbs)

Write

Read & filter

Update (new — updateRelation())

Traversal filter

Counts

Migrate verb fields

Enforcement — requireSubtype() + brain-wide strict mode

Per-type registration

Brain-wide strict mode

Migrating to strict mode on an existing brain

Strict mode in practice (for SDK-style vocabulary consumers)

Migration recipe

Brainy's own infrastructure subtype labels (reference)

Looking ahead — Brainy 8.0

Reference

Layer 1 — subtype (nouns)

Layer V — subtype (verbs / relationships)

Layer 2 — generic facets

Layer 3 — migration

Enforcement

21 KiB

Raw Blame History

Layer 1 — `subtype`

Layer 2 — `trackField()`

Layer 3 — `migrateField()`

Layer V — `subtype` on relationships (verbs)

Update (new — `updateRelation()`)

Enforcement — `requireSubtype()` + brain-wide strict mode

Layer 1 — `subtype` (nouns)

Layer V — `subtype` (verbs / relationships)