David Snelling 2cdf70ee0f feat: subtype top-level field + trackField + migrateField

Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).

Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
  BaseStorage, mirrored after nounCountsByType with the same self-heal
  rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path

Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
  cardinality + per-NounType breakdown stats. Backed by the aggregation
  engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update

Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
  streams every entity, copies the value from one path to another, and
  (unless readBoth) clears the source. Supports top-level standard fields,
  metadata.X, and data.X paths. Idempotent — safe to re-run.

Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
  quick-start all treat subtype as a core primitive with anonymous example
  vocabularies (employee/customer/invoice/milestone).

Tests
- 26 new integration tests covering write/read/update/delete round-trips,
  counts rollup decrement + re-route on mutation, trackField + byField
  with and without perType, vocabulary whitelist enforcement, and
  migrateField for metadata.X → subtype and data.X → subtype paths
  including readBoth deprecation-window semantics.

Unit suite: 1468/1468 passing. Type-check + build clean.

2026-06-04 17:25:28 -07:00

3.3 KiB

Raw Blame History

title

slug

public

Quick Start

Get Brainy running in under a minute.

1. Install

npm install @soulcraft/brainy

2. Initialize

import { Brainy, NounType, VerbType } from '@soulcraft/brainy'

const brain = new Brainy()
await brain.init()

That's it. Brainy auto-configures storage, loads the embedding model, and builds the indexes.

3. Add Knowledge

// Text is automatically embedded into 384-dim vectors
const reactId: string = await brain.add({
  data: 'React is a JavaScript library for building user interfaces',
  type: NounType.Concept,
  subtype: 'library',                                  // Sub-classification within Concept
  metadata: { category: 'frontend', year: 2013 }
})

const nextId: string = await brain.add({
  data: 'Next.js framework for React with server-side rendering',
  type: NounType.Concept,
  subtype: 'framework',
  metadata: { category: 'framework', year: 2016 }
})

type is one of Brainy's 42 stable NounTypes. subtype is your free-form sub-classification within that type — flat string, no hierarchy, indexed on the fast path. See Subtypes & Facets for the full guide.

4. Create Relationships

// Typed graph relationships
await brain.relate({
  from: nextId,
  to: reactId,
  type: VerbType.BuiltOn
})

5. Query with Triple Intelligence

import type { FindResult } from '@soulcraft/brainy'

// All three search paradigms in one call
const results: FindResult[] = await brain.find({
  query: 'modern frontend frameworks',    // Vector similarity search
  where: { year: { greaterThan: 2015 } }, // Metadata filtering
  connected: { to: reactId, depth: 2 }   // Graph traversal
})

console.log(results[0].data)   // 'Next.js framework for React...'
console.log(results[0].score)  // 0.94

What Just Happened

Every entity you add() lives in three indexes simultaneously:

Index	What it stores	Query with
Vector	384-dim embedding of `data`	`find({ query: '...' })`
Metadata	All `metadata` fields	`find({ where: { ... } })`
Graph	Typed relationships from `relate()`	`find({ connected: { ... } })`

find() queries all three in parallel and fuses the results.

Natural Language Queries

Brainy understands 220+ natural language patterns:

// These all work without any configuration
await brain.find({ query: 'recent documents about machine learning' })
await brain.find({ query: 'articles created this week' })
await brain.find({ query: 'people who work at Anthropic' })

Next Steps

Triple Intelligence — understand how the query engine works
The Find System — advanced queries, operators, and graph traversal
API Reference — complete method documentation
Storage Adapters — S3, GCS, Azure, filesystem, OPFS

3.3 KiB Raw Blame History