brainy/docs/guides/subtypes-and-facets.md

title	slug	public	category	template	order	description	next
Subtypes & Facets	guides/subtypes-and-facets	true	guides	guide	7	Use the top-level `subtype` field to sub-classify entities within a NounType, track other metadata facets like status or role, and migrate field names without downtime.	guides/aggregation api/reference

Subtypes & Facets

Sub-classify entities within a NounType, track arbitrary metadata facets, and migrate field names without downtime.

Why this exists

Brainy's NounType (Person, Document, Event, Concept, Task, …) is a stable, flat 42-type taxonomy. It's deliberately coarse — every product needs a way to further classify entities within a type. Is this Person an employee or a customer? Is this Document an invoice or a contract? Is this Event a meeting or a milestone?

Three layers solve this:

Layer	Use it for	Shape
subtype	The primary sub-classification of an entity, one value per entity	Top-level standard field
trackField()	Other facets you want to count or filter on (status, source, role, paradigm)	Registered metadata field
migrateField()	Renaming or restructuring fields across an existing dataset	One-shot stream-and-rewrite

Layer 1 — subtype

subtype is a top-level standard field on every entity, alongside type / confidence / weight. Flat string, no hierarchy. The vocabulary is your choice — Brainy stores and counts, never validates.

Write

import { Brainy, NounType } from '@soulcraft/brainy'

const brain = new Brainy()
await brain.init()

await brain.add({
  data: 'Avery Brooks — runs the AI lab',
  type: NounType.Person,
  subtype: 'employee',                  // top-level write param
  metadata: { department: 'ai-lab' }
})

Read

// Top-level filter — standard-field fast path, no `where` wrapper:
const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })

// Set membership:
const internal = await brain.find({
  type: NounType.Person,
  subtype: ['employee', 'contractor']
})

// Operator-form predicates use `where`:
const typed = await brain.find({
  type: NounType.Person,
  where: { subtype: { exists: true } }
})

What it looks like

{
  "id": "01HZK3M7TW...",
  "type": "person",
  "subtype": "employee",
  "data": "Avery Brooks — runs the AI lab",
  "metadata": { "department": "ai-lab" }
}

subtype lives at the top level — NOT inside metadata, NOT inside data. That's what makes the fast path possible: queries on subtype hit the column-store index directly, never the metadata fallback.

Counts (O(1))

// All subtypes for a NounType
brain.counts.bySubtype(NounType.Person)
// → { employee: 12, customer: 847, vendor: 34 }

// Point count
brain.counts.bySubtype(NounType.Person, 'employee')
// → 12

// Top N
brain.counts.topSubtypes(NounType.Person, 3)
// → [['customer', 847], ['employee', 12], ['vendor', 34]]

// Distinct subtypes for a NounType
brain.subtypesOf(NounType.Person)
// → ['customer', 'employee', 'vendor']

These are O(1) lookups backed by _system/subtype-statistics.json — no scan, no storage round-trip. The rollup is incrementally maintained as entities are added, updated, and deleted.

Aggregation

subtype is a first-class group-by dimension:

const rows = await brain.find({
  aggregate: {
    groupBy: ['type', 'subtype'],
    metrics: { count: { op: 'COUNT' } }
  }
})
// [
//   { groupKey: { type: 'person',   subtype: 'customer' }, count: 847 },
//   { groupKey: { type: 'person',   subtype: 'employee' }, count: 12 },
//   { groupKey: { type: 'document', subtype: 'invoice'  }, count: 2103 },
//   ...
// ]

Layer 2 — trackField()

Some metadata fields aren't the sub-classification of an entity but are still worth counting: status, source, role, paradigm. Promoting each one to top-level would clutter the contract. trackField() registers a metadata field for cardinality + per-NounType breakdown stats without the contract growth.

Register and query

// Track a single facet
brain.trackField('status')

await brain.add({ data: 'Ship subtype', type: NounType.Task, metadata: { status: 'todo' } })
await brain.add({ data: 'Write docs',   type: NounType.Task, metadata: { status: 'done' } })

await brain.counts.byField('status')
// → { todo: 1, done: 1 }

Per-NounType breakdown

brain.trackField('status', { perType: true })

await brain.counts.byField('status', { type: NounType.Task })
// → { todo: 1, done: 1 }

Vocabulary whitelist (opt-in validation)

brain.trackField('priority', { values: ['low', 'medium', 'high'] })

// Throws — 'urgent' isn't in the vocabulary:
await brain.add({
  data: 'Fix bug',
  type: NounType.Task,
  metadata: { priority: 'urgent' }
})

trackField piggybacks on the existing aggregation engine (see the Aggregation guide for the underlying mechanism). Backfill-on-define means the first call to counts.byField() scans existing entities; subsequent calls are O(groups).

subtype vs trackField — when to use which

• subtype when there's one primary sub-classification per entity. Limit yourself to one per NounType. Examples: Person→employee/customer/vendor; Document→invoice/contract/policy.
• trackField for anything else you want to count or filter on. No limit on how many you register. Examples: status, source, role, paradigm.

Layer 3 — migrateField()

Use this when you need to rename or restructure a field across an entire dataset — for example, moving a metadata.kind convention up to the top-level subtype standard field.

One-shot rewrite

// Starting state: every entity has metadata.kind
const result = await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype'
})

console.log(result)
// {
//   scanned: 1500,
//   migrated: 1500,
//   skipped: 0,
//   errors: []
// }

After this returns, every entity has subtype populated from the old metadata.kind value, and metadata.kind is cleared.

Deprecation window — keep both fields readable

When you can't coordinate all readers and the migration in a single deploy, use readBoth: true to preserve the source field alongside the new one:

// Phase 1: dual-populate (existing readers still work against metadata.kind):
await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype',
  readBoth: true
})

// ... readers migrate to query subtype at their own pace ...

// Phase 2: clear the source field when ready:
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })

Supported paths

Path form	Refers to
'subtype', 'type', 'confidence'	Top-level standard fields
'metadata.X'	A key under entity.metadata
'data.X'	A key under entity.data (when data is an object)
'X' (bare, non-standard)	Shorthand for metadata.X

Idempotent

migrateField is safe to re-run. Entities where the source is absent, or where the destination already holds the same value, are skipped. This makes it safe to use in a deploy-once-then-cleanup workflow.

Progress reporting

await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype',
  batchSize: 500,
  onProgress: ({ scanned, migrated }) => {
    console.log(`${scanned} scanned, ${migrated} migrated`)
  }
})

Putting it together

A realistic adoption sequence for a brain that started without these primitives:

import { Brainy, NounType } from '@soulcraft/brainy'

const brain = new Brainy({ storage: { type: 'filesystem', options: { path: './brain-data' } } })
await brain.init()

// 1. Migrate any existing metadata.kind convention to the new top-level subtype
await brain.migrateField({ from: 'metadata.kind', to: 'subtype', readBoth: true })

// 2. Register the other facets you want counted
brain.trackField('status', { perType: true })
brain.trackField('source')

// 3. Use subtype on every new write
await brain.add({
  data: 'Quarterly review',
  type: NounType.Event,
  subtype: 'milestone',
  metadata: { status: 'todo', source: 'planning-session' }
})

// 4. Query the breakdowns
brain.counts.bySubtype(NounType.Event)
// → { milestone: 14, meeting: 203, deadline: 7 }

await brain.counts.byField('status', { type: NounType.Event })
// → { todo: 12, done: 212 }

// 5. Once all readers are on the new field, drop the source:
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })

Layer V — subtype on relationships (verbs)

Relationships are first-class citizens too. Every verb (VerbType) gets the same subtype primitive: a ReportsTo relationship might carry subtype: 'direct' vs 'dotted-line'; a RelatedTo edge might carry 'spouse' / 'sibling' / 'colleague'. Same shape as the noun side: flat string, no hierarchy, top-level standard field, indexed on the fast path.

Write

const ceoId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Avery' })
const vpId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Jordan' })
const matrixId = await brain.add({ type: NounType.Person, subtype: 'contractor', data: 'Sam' })

await brain.relate({
  from: ceoId,
  to: vpId,
  type: VerbType.ReportsTo,
  subtype: 'direct'
})

await brain.relate({
  from: ceoId,
  to: matrixId,
  type: VerbType.ReportsTo,
  subtype: 'dotted-line'
})

Read & filter

// Direct reports — fast path filter (column-store hit, not metadata fallback)
const direct = await brain.getRelations({
  from: ceoId,
  type: VerbType.ReportsTo,
  subtype: 'direct'
})

// Set membership
const allReports = await brain.getRelations({
  from: ceoId,
  type: VerbType.ReportsTo,
  subtype: ['direct', 'dotted-line']
})

Update (new — updateRelation())

Verbs previously had no update method — the only way to change a relationship was delete-then-recreate. 7.30 closes that gap:

// Promote a dotted-line report to direct without losing the edge id
await brain.updateRelation({ id: relationId, subtype: 'direct' })

// Or change weight/confidence
await brain.updateRelation({ id: relationId, weight: 0.5, confidence: 0.9 })

Traversal filter

find({ connected, subtype }) filters traversal edges by their subtype. Composes with via (verb-type filter):

// All direct reports two hops deep (will be supported in Cortex; for now depth-1)
const directChain = await brain.find({
  connected: {
    from: ceoId,
    via: VerbType.ReportsTo,
    subtype: 'direct',
    depth: 1
  }
})

Multi-hop subtype filtering (depth > 1) lights up on the Cortex native path; the JS path throws today rather than return incorrect partial results.

Counts

Same shape as the noun-side counts API:

brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
// → { direct: 12, 'dotted-line': 3 }

brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct')   // O(1) point
// → 12

brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 3)
// → [['direct', 12], ['dotted-line', 3]]

brain.relationshipSubtypesOf(VerbType.ReportsTo)
// → ['direct', 'dotted-line']

These are O(1) lookups backed by the persisted _system/verb-subtype-statistics.json rollup — same self-heal machinery as the noun-side rollup.

Migrate verb fields

migrateField() now walks verbs too:

// Migrate verb-side metadata.kind → top-level subtype
await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype',
  entityKind: 'verb'
})

// Or migrate both nouns and verbs in one pass
await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype',
  entityKind: 'both'
})

Default is entityKind: 'noun' (backward-compatible with 7.29).

Enforcement — requireSubtype() + brain-wide strict mode

By default (7.30) subtype is optional. Two complementary opt-in mechanisms let you enforce the pairing of type + subtype on every write:

Per-type registration

Mark a specific NounType or VerbType as requiring a subtype, optionally with a fixed vocabulary:

brain.requireSubtype(NounType.Person, {
  values: ['employee', 'customer', 'vendor'],
  required: true
})

brain.requireSubtype(VerbType.ReportsTo, {
  values: ['direct', 'dotted-line'],
  required: true
})

// Now this throws — Person requires a subtype:
await brain.add({ type: NounType.Person, data: 'no subtype' })

// And this throws — 'matrix' isn't in the registered vocabulary:
await brain.relate({
  from: a,
  to: b,
  type: VerbType.ReportsTo,
  subtype: 'matrix'
})

Brain-wide strict mode

Enforce on every write across the whole brain:

const brain = new Brainy({ requireSubtype: true })

// Allow specific types to omit subtype (e.g. catch-all `Thing`)
const brain2 = new Brainy({
  requireSubtype: { except: [NounType.Thing, NounType.Custom] }
})

When strict mode is on:

• Every add() / addMany() / update() / relate() / relateMany() / updateRelation() rejects writes missing a subtype on a non-exempt type.
• addMany() and relateMany() validate every item BEFORE any storage write — atomic-fail semantics, no partial writes.
• Per-type rules registered via requireSubtype() compose with the brain-wide flag; specific rules win when both apply.
• Brainy's own internal writes (VFS root, VFS directories, VFS file entities) bypass enforcement via the metadata.isVFSEntity: true infrastructure marker.

Migrating to strict mode on an existing brain

brain.migrateField() is your friend — populate subtype from an existing convention before flipping strict mode on:

// Step 1: backfill subtype from your existing metadata.kind convention
await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype',
  entityKind: 'both'
})

// Step 2: register the vocabulary
brain.requireSubtype(NounType.Person, {
  values: ['employee', 'customer', 'vendor'],
  required: true
})

// Step 3: future writes must have a subtype matching the vocabulary
await brain.add({ type: NounType.Person, subtype: 'employee', data: '...' })

Strict mode in practice (for SDK-style vocabulary consumers)

When a platform layer like the Soulcraft SDK registers requireSubtype() rules on behalf of every consumer's brain, every downstream product that calls brain.add() / brain.relate() against those types must pass a matching subtype. Skipping the field — or passing one outside the registered vocabulary — throws at the boundary.

This pattern is powerful but surfaces a class of latent bug: any brain.add() call site that was written before strict-mode adoption starts rejecting writes. A representative production incident: a booking flow started returning 500 on every request because a service method called brain.add({ type: NounType.Event, ... }) without subtype, and an SDK layer had just registered requireSubtype() for NounType.Event on every brain instance.

The fix is a four-step migration recipe — and Brainy 7.30.1+ ships diagnostic tools to make it deterministic.

Migration recipe

1. Inventory the gap with brain.audit() — returns the deterministic list of which NounTypes and VerbTypes have entities/relationships missing subtype, grouped by type:

   const report = await brain.audit()
   //   {
   //     entitiesWithoutSubtype: { event: 24, document: 3, ... },
   //     relationshipsWithoutSubtype: { relatedTo: 1402 },
   //     total: 1429,
   //     scanned: 8400,
   //     recommendation: 'Found 1429 entries without subtype. ...'
   //   }
   

   By default, VFS infrastructure entities are excluded (they bypass enforcement anyway via the metadata.isVFSEntity marker). Pass { includeVFS: true } to surface them too.

2. Bulk-migrate any existing convention with brain.migrateField() if a legacy field can be lifted:

   // Common pattern: subtype mirrors a discriminator field already in metadata
   await brain.migrateField({
     from: 'metadata.entityType',
     to: 'subtype',
     readBoth: true  // safety: keep the source field readable during cutover
   })
   

3. Hand-fix the remaining call sites. The exact list is in report.entitiesWithoutSubtype. For each call site, add subtype: '<value>' to the brain.add() / brain.relate() params. Choose a stable convention (e.g. mirror metadata.entityType if you have one; any rule that's deterministic from the data works).

4. Verify with brain.audit() again. Re-run; total should be 0. If you turn on brain-wide strict mode at this point, all future writes are protected.

Brainy's own infrastructure subtype labels (reference)

Brainy's internal write paths set subtype on every entity and edge they create. Consumers don't need to do anything for these — they're documented here so you understand the data shape:

Code path	NounType / VerbType	Subtype label
VFS root directory /	NounType.Collection	'vfs-root'
VFS subdirectories	NounType.Collection	'vfs-directory'
VFS files	(mime-driven, e.g. Document/Code/Image)	'vfs-file'
VFS symlinks	NounType.File	'vfs-symlink'
VFS Contains edges	VerbType.Contains	'vfs-contains'
Aggregation materialized output	NounType.Measurement	'materialized-aggregate'
Import-document provenance entity	NounType.Document	'import-source'
Importer-extracted entities (no caller default)	extractor-driven	'imported'
Importer placeholder targets	NounType.Thing	'import-placeholder'
Neural extraction (no caller default)	extractor-driven	'extracted'
GoogleSheets API entity writes	request-driven	'imported-from-sheets'
OData API entity writes	request-driven	'imported-from-odata'
MCP client message storage	NounType.Message	'mcp-message'
brainy add CLI (no --subtype flag)	user-supplied type	'cli-add'
brainy relate CLI (no --subtype flag)	user-supplied verb	'cli-relate'

You can query these directly: await brain.find({ subtype: 'vfs-file' }) returns every VFS-managed file regardless of NounType. await brain.counts.bySubtype(NounType.Document) shows you the import-source / imported / extracted / vfs-file breakdown.

Importer and extraction paths accept a caller-supplied defaultSubtype option so you can tag a whole batch with your own provenance label (e.g. 'customer-upload-2026q2') instead of the Brainy default 'imported' / 'extracted'.

Looking ahead — Brainy 8.0

Brainy 8.0 ships:

• brain.fillSubtypes(rules) — the bulk migration helper that pairs with audit(). Given caller-supplied rules per NounType / VerbType, it walks the brain and fills in missing subtypes via update(). Pre-8.0 brains run this once before upgrading to clear migration debt.
• subtype: string (non-optional) on AddParams<T> and RelateParams<T>. TypeScript catches missing subtype at compile time, not just runtime.
• new Brainy({ requireSubtype: true }) becomes the default. Consumers explicitly opt out with { requireSubtype: false } during migration.

7.30.1's audit() is the diagnostic; 8.0's fillSubtypes() is the bulk fixer. Together they close the migration gap deterministically.

Reference

Layer 1 — subtype (nouns)

• brain.add({ ..., subtype: 'value' }) — write the field
• brain.update({ id, subtype: 'value' }) — change the field
• brain.find({ type, subtype }) — filter (fast path)
• brain.find({ subtype: ['a', 'b'] }) — set membership
• brain.counts.bySubtype(type, subtype?) — O(1) counts
• brain.counts.topSubtypes(type, n?) — top N by count
• brain.subtypesOf(type) — distinct subtype list

Layer V — subtype (verbs / relationships)

• brain.relate({ ..., subtype: 'value' }) — write the field
• brain.updateRelation({ id, subtype, type?, weight?, ... }) — change a relationship
• brain.getRelations({ subtype }) — filter (fast path)
• brain.getRelations({ subtype: ['a', 'b'] }) — set membership
• brain.find({ connected: { via, subtype, depth } }) — traversal filter
• brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) counts
• brain.counts.topRelationshipSubtypes(verb, n?) — top N by count
• brain.relationshipSubtypesOf(verb) — distinct subtype list

Layer 2 — generic facets

• brain.trackField(name, { perType?, values? }) — register a facet
• brain.counts.byField(name, { type? }) — facet counts

Layer 3 — migration

• brain.migrateField({ from, to, readBoth?, batchSize?, onProgress?, entityKind? }) — rewrite a field (nouns, verbs, or both)

Enforcement

• brain.requireSubtype(type, { values?, required }) — per-NounType / VerbType rule
• new Brainy({ requireSubtype: true }) — brain-wide strict mode
• new Brainy({ requireSubtype: { except: [type, ...] } }) — strict with exemptions