2026-02-19 17:04:05 -08:00
---
title: API Reference
slug: api/reference
public: true
category: api
template: api
order: 1
2026-06-11 08:37:26 -07:00
description: Complete API reference for all Brainy methods — add, find, relate, update, delete, batch operations, the Db API (transactions, snapshots, time travel), VFS, neural API, and more.
2026-02-19 17:04:05 -08:00
next:
- getting-started/quick-start
- guides/find-system
---
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
# 🧠 Brainy API Reference
2025-08-26 12:32:21 -07:00
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
> **Complete API documentation for Brainy**
2026-06-11 08:37:26 -07:00
> Zero Configuration • Triple Intelligence • Database as a Value • Atomic Transactions • Time Travel
2025-11-02 10:58:52 -08:00
2026-06-11 08:37:26 -07:00
**Updated:** 2026-06-11
2025-11-02 10:58:52 -08:00
**All APIs verified against actual code**
---
2025-08-26 12:32:21 -07:00
## Quick Start
```typescript
2025-11-02 10:58:52 -08:00
import { Brainy, NounType, VerbType } from '@soulcraft/brainy '
2025-08-26 12:32:21 -07:00
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
const brain = new Brainy() // Zero config!
await brain.init() // VFS auto-initialized!
2025-08-26 12:32:21 -07:00
// Add data (text auto-embeds!)
2025-11-02 10:58:52 -08:00
const id = await brain.add({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
data: 'The future of AI is here',
type: NounType.Concept,
metadata: { category: 'technology' }
2025-11-02 10:58:52 -08:00
})
2025-08-26 12:32:21 -07:00
// Search with Triple Intelligence
const results = await brain.find({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
query: 'artificial intelligence',
where: { year: { greaterThan: 2020 } },
connected: { from: id, depth: 2 }
2025-08-26 12:32:21 -07:00
})
2025-11-02 10:58:52 -08:00
2026-06-11 08:37:26 -07:00
// Pin the current state as an immutable value
const db = brain.now()
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
// Commit an atomic multi-write batch (all-or-nothing)
await brain.transact([
{ op: 'update', id, metadata: { category: 'AI' } }
], { meta: { author: 'docs-example' } })
await db.get(id) // still sees the pre-transaction state — snapshot isolation
await db.release()
// Time travel: query any past state
const yesterday = await brain.asOf(new Date(Date.now() - 86_400_000))
2025-08-26 12:32:21 -07:00
```
2025-11-02 10:58:52 -08:00
---
2025-08-26 12:32:21 -07:00
## Core Concepts
2025-11-02 10:58:52 -08:00
### 🧬 Entities (Nouns)
Semantic vectors with metadata and relationships - the fundamental data unit in Brainy.
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
### 🔗 Relationships (Verbs)
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
Typed connections between entities with optional `data` and `metadata` - building knowledge graphs.
### 📊 Data vs Metadata
docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
2026-06-09 16:13:35 -07:00
- **`data` **: Content embedded into vectors. Searchable via **semantic similarity** (vector index) and **hybrid text+semantic** search. NOT queryable via `where` filters.
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
- **`metadata` **: Structured fields indexed by MetadataIndex. Queryable via `where` filters in `find()` .
See ** [Data Model ](../DATA_MODEL.md )** for the full explanation.
2025-08-26 12:32:21 -07:00
### 🧠 Triple Intelligence
Vector search + Graph traversal + Metadata filtering in one unified query.
2026-06-11 08:37:26 -07:00
### 🧊 Database Values (Db)
The whole store pinned as an immutable value — snapshot isolation, atomic `transact()` batches, time travel with `asOf()` , instant hard-link snapshots with `persist()` . See the [consistency model ](../concepts/consistency-model.md ).
2025-11-04 11:19:02 -08:00
2025-08-26 12:32:21 -07:00
---
2025-11-02 10:58:52 -08:00
## Table of Contents
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
- [Core CRUD Operations ](#core-crud-operations )
- [Search & Query ](#search--query )
feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
- [Aggregation Engine ](#aggregation-engine )
2025-11-02 10:58:52 -08:00
- [Relationships ](#relationships )
- [Batch Operations ](#batch-operations )
2026-06-11 08:37:26 -07:00
- [Database Values & Time Travel (Db API) ](#database-values--time-travel-db-api )
2025-11-02 10:58:52 -08:00
- [Virtual Filesystem (VFS) ](#virtual-filesystem-vfs )
- [Neural API ](#neural-api )
- [Import & Export ](#import--export )
- [Configuration ](#configuration )
- [Storage Adapters ](#storage-adapters )
- [Utility Methods ](#utility-methods )
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
- [Embedding & Analysis APIs ](#embedding--analysis-apis )
docs: complete Stage 3 CANONICAL taxonomy documentation (v5.5.0)
Added comprehensive documentation for Stage 3 taxonomy migration:
API Documentation Updates:
- Updated version header from v5.4.0 to v5.5.0
- Fixed all deprecated NounType.Content examples → Document, Concept
- Added versions.asOf() method documentation with examples
- Added comprehensive Type System Reference section
Type System Reference:
- Documented all 42 noun types organized by category
- Documented 127 verb types organized by functional groups
- Added migration guide from v5.4.0 (deprecated types)
- Breaking changes: User→Person, Topic→Concept, Content→Document
- Stage 3 adds +11 nouns, +87 verbs (169 total types)
Build Status:
- ✅ TypeScript build passes with zero errors (stale cache issue resolved)
- ✅ All Stage 2 references updated to Stage 3 CANONICAL
- ✅ Type embeddings current (42 nouns + 127 verbs = 338KB)
Tested with confidential Tales from Talifar Glossary (NOT in repo):
- Successfully imported 2,284 entities across 7 noun types
- Created 2,280 relationships (contains verb)
- Noun type distribution: document (50%), thing (16%), person (16%),
location (9%), concept (9%), collection (1%), file (<1%)
- Demonstrates Stage 3 taxonomy handles rich fantasy world data
Type System Coverage:
- 7/42 noun types used (16.7%) - appropriate for glossary domain
- 1/127 verb types used (0.8%) - opportunity for richer relationships
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-06 10:28:47 -08:00
- [Type System Reference ](#type-system-reference )
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
---
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
## Core CRUD Operations
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
### `add(params)` → `Promise<string>`
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
Add a single entity to the database.
2025-08-26 12:32:21 -07:00
```typescript
2025-11-02 10:58:52 -08:00
const id = await brain.add({
feat: subtype top-level field + trackField + migrateField
Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).
Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
BaseStorage, mirrored after nounCountsByType with the same self-heal
rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path
Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
cardinality + per-NounType breakdown stats. Backed by the aggregation
engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update
Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
streams every entity, copies the value from one path to another, and
(unless readBoth) clears the source. Supports top-level standard fields,
metadata.X, and data.X paths. Idempotent — safe to re-run.
Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
quick-start all treat subtype as a core primitive with anonymous example
vocabularies (employee/customer/invoice/milestone).
Tests
- 26 new integration tests covering write/read/update/delete round-trips,
counts rollup decrement + re-route on mutation, trackField + byField
with and without perType, vocabulary whitelist enforcement, and
migrateField for metadata.X → subtype and data.X → subtype paths
including readBoth deprecation-window semantics.
Unit suite: 1468/1468 passing. Type-check + build clean.
2026-06-04 17:24:36 -07:00
data: 'JavaScript is a programming language', // Text or pre-computed vector
type: NounType.Concept, // Required: Entity type
subtype: 'language', // Optional: sub-classification
metadata: { // Optional: queryable fields
category: 'programming',
year: 1995
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
}
2025-08-26 12:32:21 -07:00
})
```
2025-11-02 10:58:52 -08:00
**Parameters:**
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
- `data` : `string | number[]` - Content to embed (text auto-embeds) or pre-computed vector
2025-11-02 10:58:52 -08:00
- `type` : `NounType` - Entity type (required)
feat: subtype top-level field + trackField + migrateField
Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).
Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
BaseStorage, mirrored after nounCountsByType with the same self-heal
rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path
Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
cardinality + per-NounType breakdown stats. Backed by the aggregation
engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update
Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
streams every entity, copies the value from one path to another, and
(unless readBoth) clears the source. Supports top-level standard fields,
metadata.X, and data.X paths. Idempotent — safe to re-run.
Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
quick-start all treat subtype as a core primitive with anonymous example
vocabularies (employee/customer/invoice/milestone).
Tests
- 26 new integration tests covering write/read/update/delete round-trips,
counts rollup decrement + re-route on mutation, trackField + byField
with and without perType, vocabulary whitelist enforcement, and
migrateField for metadata.X → subtype and data.X → subtype paths
including readBoth deprecation-window semantics.
Unit suite: 1468/1468 passing. Type-check + build clean.
2026-06-04 17:24:36 -07:00
- `subtype?` : `string` - Per-product sub-classification within the NounType (top-level standard field, indexed on the fast path). See [Subtypes & Facets ](../guides/subtypes-and-facets.md ).
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
- `metadata?` : `object` - Structured queryable fields (indexed by MetadataIndex, used in `where` filters)
- `id?` : `string` - Custom ID (auto-generated UUID if not provided)
- `vector?` : `number[]` - Pre-computed vector (skips auto-embedding)
- `confidence?` : `number` - Type classification confidence (0-1)
- `weight?` : `number` - Entity importance/salience (0-1)
feat: per-entity _rev + update({ ifRev }) CAS + add({ ifAbsent })
Optimistic concurrency for multi-writer coordination — the read-then-CAS
lock pattern + idempotent-bootstrap singleton inserts. Lands the surface the
SDK scheduler asked for in BRAINY-EXPOSE-TRANSACTIONS, scoped to what ships
cleanly without painting the 8.0 transact() API into a corner.
A. PER-ENTITY _rev FIELD
- src/coreTypes.ts — HNSWNounWithMetadata gains optional `_rev?: number`;
STANDARD_ENTITY_FIELDS includes it so resolveEntityField routes correctly.
- src/types/brainy.types.ts — Entity<T> gains optional `_rev?: number`;
Result<T> mirrors it on the convenience flatten layer.
- src/brainy.ts add() — initializes `_rev: 1` in storageMetadata. New entities
always start at 1.
- src/brainy.ts update() — reads currentRev off the persisted metadata
(falls back to 1 for pre-7.31.0 entities with no _rev), writes
`_rev: currentRev + 1` into the updated metadata. Every successful update()
bumps by exactly 1.
- src/brainy.ts convertNounToEntity + convertMetadataToEntity — both pull
_rev out of the storage metadata and surface it at the top level of Entity.
Pre-7.31.0 entities (no _rev in storage) read as `_rev: 1` so consumers
see a consistent value.
- src/storage/baseStorage.ts — six destructure sites updated to pull _rev
out of the metadata bag so it doesn't leak into customMetadata. Noun-side
returns surface _rev; verb-side destructures correctly but doesn't expose
it on HNSWVerbWithMetadata (verb CAS is future work).
- src/brainy.ts createResult() — flattens entity._rev onto the Result for
backward compat with the existing convenience-field layer.
B. update({ ifRev }) OPTIMISTIC CONCURRENCY
- src/types/brainy.types.ts — UpdateParams<T> gains optional `ifRev?: number`.
- src/transaction/RevisionConflictError.ts (NEW) — carries `{ id, expected,
actual }`. Message names the recipe: refetch with brain.get() and retry
with the latest _rev. Subclass of Error.
- src/brainy.ts update() — when params.ifRev is provided, compares against
currentRev (the rev we just read) and throws RevisionConflictError on
mismatch before any storage write. Omitting ifRev keeps the prior
unconditional-update behavior; existing consumers see no change.
- src/transaction/index.ts — exports RevisionConflictError.
- src/index.ts — public export of RevisionConflictError.
C. add({ ifAbsent }) BY-ID IDEMPOTENT INSERT
- src/types/brainy.types.ts — AddParams<T> gains optional `ifAbsent?: boolean`;
AddManyParams<T> mirrors it as a batch-level flag.
- src/brainy.ts add() — when params.id AND params.ifAbsent, pre-reads
storage.getNounMetadata(id); if present, returns the existing id without
writing. No throw, no overwrite. Ignored when id is omitted (a fresh UUID
can never collide).
- src/brainy.ts addMany() — propagates the batch-level ifAbsent to each
item's add() call. Per-item ifAbsent takes precedence so callers can
override individual rows.
WHAT'S NOT SHIPPED (and why)
A public brain.transaction(fn) wrapper was on the table but was cut. The
internal TransactionManager exposes raw Operation classes (SaveNounMetadata,
etc.) that take StorageAdapter as a constructor argument. A clean high-level
facade in 7.31.0 would have meant either:
(a) Delegate to brain.add() / update() / relate(). Each of those opens its
own internal transaction and commits before the closure returns. Looks
atomic, isn't — a footgun.
(b) Thread an optional `tx?` through every internal write site in
brainy.ts (8 transactionManager.executeTransaction sites). ~2 days of
real refactor with regression surface, and the API shape changes again
in 8.0 anyway.
The SDK scheduler's actual ask (BRAINY-EXPOSE-TRANSACTIONS) is the
read-then-CAS lock pattern. _rev + ifRev solves it completely. Multi-write
atomicity is the 8.0 brain.transact() use case where the Datomic-style
immutable Db makes it atomic by construction — that's the right home.
TESTS
- New tests/integration/rev-and-ifabsent.test.ts (18 tests):
- _rev initialization (1 on add) and surface on get fast/full paths + find
- _rev auto-bump on update across multiple writes
- update({ ifRev }) pass / fail / message format / omitted / legacy-no-rev
- add({ ifAbsent }) writes when absent / no-op when present / id-required
- addMany({ ifAbsent }) propagation + per-item override
- SDK-scheduler scenario: two concurrent CAS updates, one wins one throws
- Unit suite unchanged: 1468/1468.
- All integration subtype + verb + strict + find-limits + new rev suites
pass: 96/96.
DOCS
- New docs/guides/optimistic-concurrency.md (public: true) — full reference:
the lock pattern, read-modify-write retry, idempotent bootstrap, how _rev
interacts with brain.versions, branches/fork, and VFS, what's coming in 8.0.
- docs/api/README.md — add() and update() entries get the new params + tips
pointing at the new guide.
- RELEASES.md v7.31.0 entry.
CORTEX COMPATIBILITY
Zero changes required. _rev is a metadata column already supported by
NativeColumnStore. The auto-bump runs in Brainy JS before any storage call;
Cortex never sees the per-entity counter.
8.0 FORWARD-COMPAT
_rev, ifRev, RevisionConflictError, and ifAbsent survive the 8.0 Db redesign
unchanged. 8.0 layers brain.transact(tx, { ifAtGeneration }) for whole-tx CAS
on top of the same per-entity mechanism — per-entity for single-record
patterns (job locks, idempotent state machines), generation-based for
"did the world move under me." Locked in .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md § C-6.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All integration suites including new rev-and-ifabsent: 96/96
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-09 10:04:24 -07:00
- `ifAbsent?` : `boolean` - By-ID idempotent insert. When `true` AND a custom `id` is supplied AND an entity with that `id` already exists, returns the existing `id` without writing (no throw, no overwrite). Ignored without `id` . See [guides/optimistic-concurrency ](../guides/optimistic-concurrency.md ).
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
> **`data`** is embedded into vectors for semantic search. **`metadata`** is indexed for `where` filters. See [Data Model](../DATA_MODEL.md).
2025-11-02 10:58:52 -08:00
fix: internal subtype consistency + brain.audit() diagnostic + improved enforcement errors
Brainy 7.30 shipped opt-in subtype enforcement; SDK 3.20.0 then registered
SDK_CORE_VOCABULARY on every consumer's brain (Event, Collection, Message,
Contract, Media, Document NounTypes). On 2026-06-08 Venue's /book flow went 500
because their brain.add({ type: NounType.Event, ... }) call sites lacked
subtype. An audit of Brainy's OWN source revealed 14 HIGH-risk internal write
paths that also omit subtype — any consumer running the same vocabulary would
have hit Brainy's infrastructure paths next. 7.30.1 closes both gaps before
8.0 makes strict mode the default.
Additive across the board. Zero behavior change for consumers not using strict
mode. Every change is JS-side — Cortex needs no work for 7.30.1.
NEW — brain.audit() diagnostic
- Read-only method walking storage.getNouns() / getVerbs() pagination
- Returns { entitiesWithoutSubtype: { type: count }, relationshipsWithoutSubtype,
total, scanned, recommendation }
- VFS infrastructure entities excluded by default (they bypass enforcement via
isVFSEntity marker); pass { includeVFS: true } to surface them
- The companion to migrateField (7.x) and fillSubtypes (8.0): tells consumers
exactly what would break under strict enforcement, deterministically
NEW — Improved enforcement error messages
- Caller's source location extracted from Error().stack so users see their own
call site, not a Brainy internal frame
- Specific guidance branches: registered vocabulary → "Pass one of: a, b, c";
brain-wide strict mode → mentions the except clause; otherwise → registration
recipe via brain.requireSubtype()
- Documentation link to the canonical migration recipe
- Same shape for noun and verb enforcement
NEW — CLI --subtype flag
- brainy add and brainy relate gain -s/--subtype <value>
- Defaults to 'cli-add' / 'cli-relate' so the CLI works against strict-mode
brains without the user needing to know the vocabulary in advance
INTERNAL — every Brainy write path now sets subtype
- VFS Contains edges (5 sites at lines 503/905/1694/1772/1886) → 'vfs-contains'
- VFS symlink entity → 'vfs-symlink' (NEW — distinct from 'vfs-file')
- VFS copy-file → preserves source subtype, falls back to 'vfs-file'
- VFS symlink also adopts the isVFSEntity infrastructure marker so it bypasses
enforcement in strict mode
- Aggregation materializer (Measurement entities) → 'materialized-aggregate'
- ImportCoordinator (3 sites): document → 'import-source'; entities →
options.defaultSubtype ?? 'imported'; placeholder → 'import-placeholder'
- SmartImportOrchestrator (4 entity sites + 2 batch relate sites): same
precedence (extractor → options.defaultSubtype → 'imported')
- EntityDeduplicator → candidate.subtype ?? 'imported'
- UniversalImportAPI → extractor → 'extracted' for both entities and relations
- NeuralImport → adds defaultSubtype to NeuralImportOptions; precedence same
- GoogleSheetsIntegration → request body 'subtype' ?? 'imported-from-sheets'
- ODataIntegration → request body 'Subtype' ?? 'imported-from-odata'
- MCP client message storage → 'mcp-message' (also fixes pre-existing missing
data field and missing type by aliasing from the prior text field)
Side-effect fix: storage.getNouns() paginated now surfaces subtype to top-level
- Single-noun getNoun() already did this in 7.30; the paginated path was missed
- Without this fix brain.audit() saw missing subtype on entities that actually
had one (caught by the strict-mode self-test before release)
NEW — tests/integration/strict-mode-self-test.test.ts (13 tests)
- Creates a brain under the exact SDK_CORE_VOCABULARY shape Venue hit + brain-
wide strict mode
- Exercises every internal Brainy path: VFS root + mkdir + writeFile + cp + mv
+ ln + symlink; aggregation engine; audit diagnostic with includeVFS toggle
- Validates error message UX: caller location, vocabulary guidance, brain-wide
strict mode guidance, off-vocabulary value reporting
Docs
- New "Strict mode in practice" section in docs/guides/subtypes-and-facets.md
covering the SDK_CORE_VOCABULARY pattern, 4-step migration recipe
(audit → migrateField → hand-fix → re-audit), the Brainy-internal label
reference table, and an 8.0 forward-look on fillSubtypes()
- docs/api/README.md: new audit() entry, strict-mode tips on add() and relate()
- RELEASES.md: full 7.30.1 entry
Cortex parity (forward-looking, not blocking 7.30.1)
- 6th open question added to .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md: native
fast path for audit() and fillSubtypes() via column-store null-subtype
bitmap for billion-scale brains
- Cortex should add a parity test mirroring strict-mode-self-test.test.ts
against their native paths to catch any latent bug where native writes
bypass JS validation
- Brainy-internal subtype labels become a documented part of the 8.0 contract
(useful for Cortex telemetry surfacing Brainy-managed infrastructure %)
Verification
- npx tsc --noEmit: clean
- npm test: 1468/1468 unit
- 7.29 noun integration suite: 26/26 (no regression)
- 7.30 verb subtype + enforcement integration suite: 30/30 (no regression)
- New strict-mode-self-test integration suite: 13/13
- npm run build: clean
- Closed-source product reference audit: clean
Addresses VE-SUBTYPE-MIGRATION (Venue's reported request) and ships internal
labels Venue did NOT ask for but that would have broken them next under their
own vocabulary registration.
2026-06-08 11:31:47 -07:00
> **Strict-mode tip:** if a vocabulary is registered for your `type` (via `brain.requireSubtype()` or by an SDK that wraps Brainy), you must pass a matching `subtype`. Run `await brain.audit()` to inventory pre-existing gaps before enabling strict mode; see the [migration recipe](../guides/subtypes-and-facets.md#strict-mode-in-practice-for-sdk-style-vocabulary-consumers).
2025-11-02 10:58:52 -08:00
**Returns:** `Promise<string>` - Entity ID
2025-08-26 12:32:21 -07:00
---
2025-11-02 10:58:52 -08:00
### `get(id)` → `Promise<Entity | null>`
Retrieve a single entity by ID.
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
```typescript
const entity = await brain.get(id)
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
console.log(entity?.data) // Original data
console.log(entity?.metadata) // Metadata
console.log(entity?.vector) // Embedding vector
2025-11-02 10:58:52 -08:00
```
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
**Parameters:**
- `id` : `string` - Entity ID
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
**Returns:** `Promise<Entity | null>` - Entity or null if not found
2025-08-26 12:32:21 -07:00
---
2025-11-02 10:58:52 -08:00
### `update(params)` → `Promise<void>`
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
Update an existing entity.
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
```typescript
await brain.update({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
id: entityId,
feat: subtype top-level field + trackField + migrateField
Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).
Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
BaseStorage, mirrored after nounCountsByType with the same self-heal
rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path
Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
cardinality + per-NounType breakdown stats. Backed by the aggregation
engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update
Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
streams every entity, copies the value from one path to another, and
(unless readBoth) clears the source. Supports top-level standard fields,
metadata.X, and data.X paths. Idempotent — safe to re-run.
Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
quick-start all treat subtype as a core primitive with anonymous example
vocabularies (employee/customer/invoice/milestone).
Tests
- 26 new integration tests covering write/read/update/delete round-trips,
counts rollup decrement + re-route on mutation, trackField + byField
with and without perType, vocabulary whitelist enforcement, and
migrateField for metadata.X → subtype and data.X → subtype paths
including readBoth deprecation-window semantics.
Unit suite: 1468/1468 passing. Type-check + build clean.
2026-06-04 17:24:36 -07:00
data: 'Updated content', // Optional: new data
subtype: 'archived', // Optional: change sub-classification
metadata: { updated: true } // Optional: new metadata (merges)
2025-11-02 10:58:52 -08:00
})
```
**Parameters:**
- `id` : `string` - Entity ID
- `data?` : `string | number[]` - New data/vector
feat: subtype top-level field + trackField + migrateField
Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).
Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
BaseStorage, mirrored after nounCountsByType with the same self-heal
rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path
Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
cardinality + per-NounType breakdown stats. Backed by the aggregation
engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update
Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
streams every entity, copies the value from one path to another, and
(unless readBoth) clears the source. Supports top-level standard fields,
metadata.X, and data.X paths. Idempotent — safe to re-run.
Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
quick-start all treat subtype as a core primitive with anonymous example
vocabularies (employee/customer/invoice/milestone).
Tests
- 26 new integration tests covering write/read/update/delete round-trips,
counts rollup decrement + re-route on mutation, trackField + byField
with and without perType, vocabulary whitelist enforcement, and
migrateField for metadata.X → subtype and data.X → subtype paths
including readBoth deprecation-window semantics.
Unit suite: 1468/1468 passing. Type-check + build clean.
2026-06-04 17:24:36 -07:00
- `type?` : `NounType` - Change entity type
- `subtype?` : `string` - Change subtype (omit to preserve existing)
- `metadata?` : `object` - Metadata to merge (or replace with `merge: false` )
- `confidence?` : `number` - Update classification confidence
- `weight?` : `number` - Update entity importance
feat: per-entity _rev + update({ ifRev }) CAS + add({ ifAbsent })
Optimistic concurrency for multi-writer coordination — the read-then-CAS
lock pattern + idempotent-bootstrap singleton inserts. Lands the surface the
SDK scheduler asked for in BRAINY-EXPOSE-TRANSACTIONS, scoped to what ships
cleanly without painting the 8.0 transact() API into a corner.
A. PER-ENTITY _rev FIELD
- src/coreTypes.ts — HNSWNounWithMetadata gains optional `_rev?: number`;
STANDARD_ENTITY_FIELDS includes it so resolveEntityField routes correctly.
- src/types/brainy.types.ts — Entity<T> gains optional `_rev?: number`;
Result<T> mirrors it on the convenience flatten layer.
- src/brainy.ts add() — initializes `_rev: 1` in storageMetadata. New entities
always start at 1.
- src/brainy.ts update() — reads currentRev off the persisted metadata
(falls back to 1 for pre-7.31.0 entities with no _rev), writes
`_rev: currentRev + 1` into the updated metadata. Every successful update()
bumps by exactly 1.
- src/brainy.ts convertNounToEntity + convertMetadataToEntity — both pull
_rev out of the storage metadata and surface it at the top level of Entity.
Pre-7.31.0 entities (no _rev in storage) read as `_rev: 1` so consumers
see a consistent value.
- src/storage/baseStorage.ts — six destructure sites updated to pull _rev
out of the metadata bag so it doesn't leak into customMetadata. Noun-side
returns surface _rev; verb-side destructures correctly but doesn't expose
it on HNSWVerbWithMetadata (verb CAS is future work).
- src/brainy.ts createResult() — flattens entity._rev onto the Result for
backward compat with the existing convenience-field layer.
B. update({ ifRev }) OPTIMISTIC CONCURRENCY
- src/types/brainy.types.ts — UpdateParams<T> gains optional `ifRev?: number`.
- src/transaction/RevisionConflictError.ts (NEW) — carries `{ id, expected,
actual }`. Message names the recipe: refetch with brain.get() and retry
with the latest _rev. Subclass of Error.
- src/brainy.ts update() — when params.ifRev is provided, compares against
currentRev (the rev we just read) and throws RevisionConflictError on
mismatch before any storage write. Omitting ifRev keeps the prior
unconditional-update behavior; existing consumers see no change.
- src/transaction/index.ts — exports RevisionConflictError.
- src/index.ts — public export of RevisionConflictError.
C. add({ ifAbsent }) BY-ID IDEMPOTENT INSERT
- src/types/brainy.types.ts — AddParams<T> gains optional `ifAbsent?: boolean`;
AddManyParams<T> mirrors it as a batch-level flag.
- src/brainy.ts add() — when params.id AND params.ifAbsent, pre-reads
storage.getNounMetadata(id); if present, returns the existing id without
writing. No throw, no overwrite. Ignored when id is omitted (a fresh UUID
can never collide).
- src/brainy.ts addMany() — propagates the batch-level ifAbsent to each
item's add() call. Per-item ifAbsent takes precedence so callers can
override individual rows.
WHAT'S NOT SHIPPED (and why)
A public brain.transaction(fn) wrapper was on the table but was cut. The
internal TransactionManager exposes raw Operation classes (SaveNounMetadata,
etc.) that take StorageAdapter as a constructor argument. A clean high-level
facade in 7.31.0 would have meant either:
(a) Delegate to brain.add() / update() / relate(). Each of those opens its
own internal transaction and commits before the closure returns. Looks
atomic, isn't — a footgun.
(b) Thread an optional `tx?` through every internal write site in
brainy.ts (8 transactionManager.executeTransaction sites). ~2 days of
real refactor with regression surface, and the API shape changes again
in 8.0 anyway.
The SDK scheduler's actual ask (BRAINY-EXPOSE-TRANSACTIONS) is the
read-then-CAS lock pattern. _rev + ifRev solves it completely. Multi-write
atomicity is the 8.0 brain.transact() use case where the Datomic-style
immutable Db makes it atomic by construction — that's the right home.
TESTS
- New tests/integration/rev-and-ifabsent.test.ts (18 tests):
- _rev initialization (1 on add) and surface on get fast/full paths + find
- _rev auto-bump on update across multiple writes
- update({ ifRev }) pass / fail / message format / omitted / legacy-no-rev
- add({ ifAbsent }) writes when absent / no-op when present / id-required
- addMany({ ifAbsent }) propagation + per-item override
- SDK-scheduler scenario: two concurrent CAS updates, one wins one throws
- Unit suite unchanged: 1468/1468.
- All integration subtype + verb + strict + find-limits + new rev suites
pass: 96/96.
DOCS
- New docs/guides/optimistic-concurrency.md (public: true) — full reference:
the lock pattern, read-modify-write retry, idempotent bootstrap, how _rev
interacts with brain.versions, branches/fork, and VFS, what's coming in 8.0.
- docs/api/README.md — add() and update() entries get the new params + tips
pointing at the new guide.
- RELEASES.md v7.31.0 entry.
CORTEX COMPATIBILITY
Zero changes required. _rev is a metadata column already supported by
NativeColumnStore. The auto-bump runs in Brainy JS before any storage call;
Cortex never sees the per-entity counter.
8.0 FORWARD-COMPAT
_rev, ifRev, RevisionConflictError, and ifAbsent survive the 8.0 Db redesign
unchanged. 8.0 layers brain.transact(tx, { ifAtGeneration }) for whole-tx CAS
on top of the same per-entity mechanism — per-entity for single-record
patterns (job locks, idempotent state machines), generation-based for
"did the world move under me." Locked in .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md § C-6.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All integration suites including new rev-and-ifabsent: 96/96
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-09 10:04:24 -07:00
- `ifRev?` : `number` - Optimistic-concurrency check. When provided, the update throws `RevisionConflictError` if the persisted entity's `_rev` no longer equals `ifRev` . See [guides/optimistic-concurrency ](../guides/optimistic-concurrency.md ).
2025-11-02 10:58:52 -08:00
**Returns:** `Promise<void>`
2025-08-26 12:32:21 -07:00
feat: per-entity _rev + update({ ifRev }) CAS + add({ ifAbsent })
Optimistic concurrency for multi-writer coordination — the read-then-CAS
lock pattern + idempotent-bootstrap singleton inserts. Lands the surface the
SDK scheduler asked for in BRAINY-EXPOSE-TRANSACTIONS, scoped to what ships
cleanly without painting the 8.0 transact() API into a corner.
A. PER-ENTITY _rev FIELD
- src/coreTypes.ts — HNSWNounWithMetadata gains optional `_rev?: number`;
STANDARD_ENTITY_FIELDS includes it so resolveEntityField routes correctly.
- src/types/brainy.types.ts — Entity<T> gains optional `_rev?: number`;
Result<T> mirrors it on the convenience flatten layer.
- src/brainy.ts add() — initializes `_rev: 1` in storageMetadata. New entities
always start at 1.
- src/brainy.ts update() — reads currentRev off the persisted metadata
(falls back to 1 for pre-7.31.0 entities with no _rev), writes
`_rev: currentRev + 1` into the updated metadata. Every successful update()
bumps by exactly 1.
- src/brainy.ts convertNounToEntity + convertMetadataToEntity — both pull
_rev out of the storage metadata and surface it at the top level of Entity.
Pre-7.31.0 entities (no _rev in storage) read as `_rev: 1` so consumers
see a consistent value.
- src/storage/baseStorage.ts — six destructure sites updated to pull _rev
out of the metadata bag so it doesn't leak into customMetadata. Noun-side
returns surface _rev; verb-side destructures correctly but doesn't expose
it on HNSWVerbWithMetadata (verb CAS is future work).
- src/brainy.ts createResult() — flattens entity._rev onto the Result for
backward compat with the existing convenience-field layer.
B. update({ ifRev }) OPTIMISTIC CONCURRENCY
- src/types/brainy.types.ts — UpdateParams<T> gains optional `ifRev?: number`.
- src/transaction/RevisionConflictError.ts (NEW) — carries `{ id, expected,
actual }`. Message names the recipe: refetch with brain.get() and retry
with the latest _rev. Subclass of Error.
- src/brainy.ts update() — when params.ifRev is provided, compares against
currentRev (the rev we just read) and throws RevisionConflictError on
mismatch before any storage write. Omitting ifRev keeps the prior
unconditional-update behavior; existing consumers see no change.
- src/transaction/index.ts — exports RevisionConflictError.
- src/index.ts — public export of RevisionConflictError.
C. add({ ifAbsent }) BY-ID IDEMPOTENT INSERT
- src/types/brainy.types.ts — AddParams<T> gains optional `ifAbsent?: boolean`;
AddManyParams<T> mirrors it as a batch-level flag.
- src/brainy.ts add() — when params.id AND params.ifAbsent, pre-reads
storage.getNounMetadata(id); if present, returns the existing id without
writing. No throw, no overwrite. Ignored when id is omitted (a fresh UUID
can never collide).
- src/brainy.ts addMany() — propagates the batch-level ifAbsent to each
item's add() call. Per-item ifAbsent takes precedence so callers can
override individual rows.
WHAT'S NOT SHIPPED (and why)
A public brain.transaction(fn) wrapper was on the table but was cut. The
internal TransactionManager exposes raw Operation classes (SaveNounMetadata,
etc.) that take StorageAdapter as a constructor argument. A clean high-level
facade in 7.31.0 would have meant either:
(a) Delegate to brain.add() / update() / relate(). Each of those opens its
own internal transaction and commits before the closure returns. Looks
atomic, isn't — a footgun.
(b) Thread an optional `tx?` through every internal write site in
brainy.ts (8 transactionManager.executeTransaction sites). ~2 days of
real refactor with regression surface, and the API shape changes again
in 8.0 anyway.
The SDK scheduler's actual ask (BRAINY-EXPOSE-TRANSACTIONS) is the
read-then-CAS lock pattern. _rev + ifRev solves it completely. Multi-write
atomicity is the 8.0 brain.transact() use case where the Datomic-style
immutable Db makes it atomic by construction — that's the right home.
TESTS
- New tests/integration/rev-and-ifabsent.test.ts (18 tests):
- _rev initialization (1 on add) and surface on get fast/full paths + find
- _rev auto-bump on update across multiple writes
- update({ ifRev }) pass / fail / message format / omitted / legacy-no-rev
- add({ ifAbsent }) writes when absent / no-op when present / id-required
- addMany({ ifAbsent }) propagation + per-item override
- SDK-scheduler scenario: two concurrent CAS updates, one wins one throws
- Unit suite unchanged: 1468/1468.
- All integration subtype + verb + strict + find-limits + new rev suites
pass: 96/96.
DOCS
- New docs/guides/optimistic-concurrency.md (public: true) — full reference:
the lock pattern, read-modify-write retry, idempotent bootstrap, how _rev
interacts with brain.versions, branches/fork, and VFS, what's coming in 8.0.
- docs/api/README.md — add() and update() entries get the new params + tips
pointing at the new guide.
- RELEASES.md v7.31.0 entry.
CORTEX COMPATIBILITY
Zero changes required. _rev is a metadata column already supported by
NativeColumnStore. The auto-bump runs in Brainy JS before any storage call;
Cortex never sees the per-entity counter.
8.0 FORWARD-COMPAT
_rev, ifRev, RevisionConflictError, and ifAbsent survive the 8.0 Db redesign
unchanged. 8.0 layers brain.transact(tx, { ifAtGeneration }) for whole-tx CAS
on top of the same per-entity mechanism — per-entity for single-record
patterns (job locks, idempotent state machines), generation-based for
"did the world move under me." Locked in .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md § C-6.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All integration suites including new rev-and-ifabsent: 96/96
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-09 10:04:24 -07:00
> **Tip — read-then-CAS.** Every entity returned by `get()` / `find()` / `search()` carries `entity._rev` (a monotonic counter Brainy auto-bumps on every successful `update()`). Pass it back as `ifRev` to make multi-writer coordination safe without an external lock service. Full guide: [guides/optimistic-concurrency](../guides/optimistic-concurrency.md).
2025-08-26 12:32:21 -07:00
---
2026-06-11 14:51:00 -07:00
### `remove(id)` → `Promise<void>`
2025-11-02 10:58:52 -08:00
2026-06-11 14:51:00 -07:00
Remove a single entity (and every relationship where it is source or target).
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
```typescript
2026-06-11 14:51:00 -07:00
await brain.remove(id)
2025-11-02 10:58:52 -08:00
```
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
**Parameters:**
- `id` : `string` - Entity ID
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
**Returns:** `Promise<void>`
2025-08-26 12:32:21 -07:00
---
2025-11-02 10:58:52 -08:00
## Search & Query
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
### `find(query)` → `Promise<Result[]>`
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
**Triple Intelligence** - Vector + Graph + Metadata in ONE query.
2025-08-26 12:32:21 -07:00
```typescript
2025-11-02 10:58:52 -08:00
// Simple text search
const results = await brain.find('machine learning')
// Advanced Triple Intelligence query
const results = await brain.find({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
query: 'artificial intelligence', // Vector similarity
where: { // Metadata filtering
year: { greaterThan: 2020 },
category: { oneOf: ['AI', 'ML'] }
},
connected: { // Graph traversal
to: conceptId,
depth: 2,
type: VerbType.RelatedTo
},
limit: 10
2025-11-02 10:58:52 -08:00
})
2025-08-26 12:32:21 -07:00
```
2025-11-02 10:58:52 -08:00
**Parameters:**
- `query` : `string | FindParams`
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
- **Simple:** Just text for vector search
- **Advanced:** Object with vector + graph + metadata filters
2025-11-02 10:58:52 -08:00
**FindParams:**
docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
2026-06-09 16:13:35 -07:00
- `query?` : `string` - Text for semantic + hybrid search (searches `data` via the vector index + text index)
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
- `type?` : `NounType | NounType[]` - Filter by entity type(s). Alias for `where.noun` .
feat: subtype top-level field + trackField + migrateField
Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).
Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
BaseStorage, mirrored after nounCountsByType with the same self-heal
rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path
Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
cardinality + per-NounType breakdown stats. Backed by the aggregation
engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update
Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
streams every entity, copies the value from one path to another, and
(unless readBoth) clears the source. Supports top-level standard fields,
metadata.X, and data.X paths. Idempotent — safe to re-run.
Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
quick-start all treat subtype as a core primitive with anonymous example
vocabularies (employee/customer/invoice/milestone).
Tests
- 26 new integration tests covering write/read/update/delete round-trips,
counts rollup decrement + re-route on mutation, trackField + byField
with and without perType, vocabulary whitelist enforcement, and
migrateField for metadata.X → subtype and data.X → subtype paths
including readBoth deprecation-window semantics.
Unit suite: 1468/1468 passing. Type-check + build clean.
2026-06-04 17:24:36 -07:00
- `subtype?` : `string | string[]` - Filter by sub-classification (top-level standard field, fast path). Single string for equality, array for set membership.
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
- `where?` : `object` - Metadata filters. See ** [Query Operators ](../QUERY_OPERATORS.md )** for all operators.
2025-11-02 10:58:52 -08:00
- `connected?` : `object` - Graph traversal options
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
- `to?` : `string` - Target entity ID
- `from?` : `string` - Source entity ID
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
- `via?` : `VerbType | VerbType[]` - Relationship type(s) to traverse
- `type?` : `VerbType | VerbType[]` - Alias for `via`
- `depth?` : `number` - Traversal depth (default: 1)
- `direction?` : `'in' | 'out' | 'both'` - Traversal direction (default: 'both')
2025-11-02 10:58:52 -08:00
- `limit?` : `number` - Max results (default: 10)
- `offset?` : `number` - Skip results
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
- `orderBy?` : `string` - Field to sort by (e.g., 'createdAt', 'metadata.priority')
- `order?` : `'asc' | 'desc'` - Sort direction (default: 'asc')
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
- `searchMode?` : `'auto' | 'text' | 'semantic' | 'hybrid'` - Search strategy:
- `'auto'` (default): Zero-config hybrid combining text + semantic search
- `'text'` : Pure keyword/text matching
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
- `'semantic'` /`'vector'` : Pure vector similarity
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
- `'hybrid'` : Explicit hybrid mode
2026-01-26 17:16:18 -08:00
- `hybridAlpha?` : `number` - Balance between text (0.0) and semantic (1.0) search. Auto-detected by query length if not specified.
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
- `excludeVFS?` : `boolean` - Exclude VFS entities from results (default: false)
2025-11-02 10:58:52 -08:00
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
> **`limit` tip:** Brainy caps `limit` against an auto-configured maximum (based on container/free memory, ~25 KB per result). Above the cap you get a one-time warning per call site; above 2× the cap it throws. To raise the cap, pass `new Brainy({ maxQueryLimit: N })` or `{ reservedQueryMemory: bytes }`. For queries that need ALL matches, paginate with `{ limit, offset }` — that's the only pattern guaranteed to keep working across Brainy versions. See [Query Limits & Pagination](../guides/find-limits.md).
2025-11-02 10:58:52 -08:00
**Returns:** `Promise<Result[]>` - Matching entities with scores
2025-08-26 12:32:21 -07:00
---
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
### Hybrid Search
2026-01-26 17:16:18 -08:00
Brainy automatically combines text (keyword) and semantic (vector) search for optimal results. No configuration needed.
```typescript
// Zero-config hybrid search (just works)
const results = await brain.find({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
query: 'David Smith' // Finds both exact text matches AND semantically similar
2026-01-26 17:16:18 -08:00
})
// Force text-only search (exact keyword matching)
const textResults = await brain.find({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
query: 'exact keyword',
searchMode: 'text'
2026-01-26 17:16:18 -08:00
})
// Force semantic-only search (vector similarity)
const semanticResults = await brain.find({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
query: 'artificial intelligence concepts',
searchMode: 'semantic'
2026-01-26 17:16:18 -08:00
})
// Custom hybrid weighting (0 = text only, 1 = semantic only)
const customResults = await brain.find({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
query: 'David Smith',
hybridAlpha: 0.3 // Favor text matching
2026-01-26 17:16:18 -08:00
})
```
**How it works:**
- Short queries (1-2 words) automatically favor text matching
- Long queries (5+ words) automatically favor semantic search
- Results are combined using Reciprocal Rank Fusion (RRF)
---
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
### Match Visibility
2026-01-26 17:16:18 -08:00
Search results include detailed match information:
```typescript
const results = await brain.find({ query: 'david the warrior' })
// Each result now includes:
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
results[0].textMatches // ["david", "warrior"] - exact query words found
results[0].textScore // 0.25 - text match quality (0-1)
results[0].semanticScore // 0.87 - semantic similarity (0-1)
results[0].matchSource // 'both' | 'text' | 'semantic'
2026-01-26 17:16:18 -08:00
```
**Use cases:**
- Highlight exact matches in UI (textMatches)
- Explain why a result ranked high (matchSource)
- Debug search behavior (separate scores)
---
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
### `highlight(params)` → `Promise<Highlight[]>` ✨
2026-01-26 17:16:18 -08:00
Zero-config highlighting for both exact matches AND semantic concepts.
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
Handles plain text, rich-text JSON (TipTap, Slate, Lexical, Draft.js, Quill), HTML, and Markdown automatically.
2026-01-26 17:16:18 -08:00
```typescript
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
// Plain text (works as before)
2026-01-26 17:16:18 -08:00
const highlights = await brain.highlight({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
query: "david the warrior",
text: "David Smith is a brave fighter who battles dragons"
2026-01-26 17:16:18 -08:00
})
// [
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
// { text: "David", score: 1.0, position: [0, 5], matchType: 'text' },
// { text: "fighter", score: 0.78, position: [25, 32], matchType: 'semantic' },
// { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' }
2026-01-26 17:16:18 -08:00
// ]
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
// Rich-text JSON (auto-detected)
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
const highlights = await brain.highlight({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
query: "david the warrior",
text: JSON.stringify(tiptapDocument) // TipTap, Slate, Lexical, Draft.js, Quill
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
})
// Extracts text from nodes, annotates with contentCategory:
// [
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
// { text: "David", score: 1.0, matchType: 'text', contentCategory: 'title' },
// { text: "fighter", score: 0.78, matchType: 'semantic', contentCategory: 'content' }
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
// ]
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
// HTML input (auto-detected)
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
const highlights = await brain.highlight({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
query: "warrior",
text: "< h1 > David the Warrior< / h1 > < p > A brave fighter.< / p > "
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
})
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
// Custom extractor for proprietary formats
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
const highlights = await brain.highlight({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
query: "function",
text: sourceCode,
contentExtractor: (text) => treeSitterParse(text) // Your custom parser
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
})
2026-01-26 17:16:18 -08:00
```
**Parameters:**
- `query` : `string` - The search query
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
- `text` : `string` - Text to highlight (plain text, JSON, HTML, or Markdown)
2026-01-26 17:16:18 -08:00
- `granularity?` : `'word' | 'phrase' | 'sentence'` - Highlight unit (default: 'word')
- `threshold?` : `number` - Min similarity for semantic matches (default: 0.5)
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
- `contentType?` : `ContentType` - Optional hint: `'plaintext' | 'richtext-json' | 'html' | 'markdown'` . Skips auto-detection when provided.
- `contentExtractor?` : `(text: string) => ExtractedSegment[]` - Custom parser. Bypasses built-in detection entirely.
2026-01-26 17:16:18 -08:00
**Returns:** `Promise<Highlight[]>`
- `text` - The matched text
- `score` - Match score (1.0 for text matches, varies for semantic)
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
- `position` - [start, end] indices in extracted text
2026-01-26 17:16:18 -08:00
- `matchType` - `'text'` (exact) or `'semantic'` (concept)
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
- `contentCategory?` - `'title' | 'annotation' | 'content' | 'value' | 'code' | 'structural'` — Role of the source text. Built-in extractors produce `'title'` , `'content'` , `'code'` . All 6 categories are available for custom parsers.
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
**Supported Rich-Text Formats:**
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
| Format | Detection | Text nodes |
|--------|-----------|------------|
| TipTap / ProseMirror | `{ type: 'doc', content: [...] }` | `{ type: 'text', text }` |
| Slate.js | `[{ type, children }]` | `{ text }` |
| Lexical | `{ root: { children } }` | `{ type: 'text', text }` |
| Draft.js | `{ blocks: [{ text }] }` | `{ text }` in block |
| Quill Delta | `{ ops: [{ insert }] }` | `{ insert }` |
| HTML | Tags like `<h1>` , `<p>` , `<code>` | Visible text content |
| Markdown | `#` headings, ` ` `` ` code blocks | Stripped markup |
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
**Timeout Protection:**
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
Semantic matching has a 10-second timeout. If embedding takes too long (e.g., WASM stall), `highlight()` returns text-only matches instead of hanging.
2026-01-26 17:16:18 -08:00
**UI Pattern:**
```typescript
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
// Style differently based on match type and content category
2026-01-26 17:16:18 -08:00
highlights.forEach(h => {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
const style = h.matchType === 'text' ? 'font-weight: bold' : 'background: yellow'
if (h.contentCategory === 'title') { /* render as heading highlight */ }
if (h.contentCategory === 'code') { /* render with code styling */ }
if (h.contentCategory === 'annotation') { /* render as comment/caption */ }
// Apply style from h.position[0] to h.position[1]
2026-01-26 17:16:18 -08:00
})
```
---
2025-11-02 10:58:52 -08:00
### Query Operators
2025-08-26 12:32:21 -07:00
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
Brainy uses clean, readable operators (BFO — Brainy Field Operators):
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
| Operator | Description | Example |
|----------|-------------|---------|
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
| `equals` / `eq` | Exact match | `{age: {equals: 25}}` |
| `notEquals` / `ne` | Not equal | `{status: {notEquals: 'deleted'}}` |
| `greaterThan` / `gt` | Greater than | `{age: {greaterThan: 18}}` |
refactor(8.0): remove the 4 deprecated query-operator aliases (clean break)
8.0 keeps the canonical operators (eq/ne/gt/gte/lt/lte) and their clean
long-form aliases (equals/notEquals/greaterThan/greaterThanOrEqual/lessThan/
lessThanOrEqual), and drops the four redundant deprecated spellings:
is → eq, isNot → ne, greaterEqual → gte, lessEqual → lte
Removed from every evaluator (metadataIndex criteria + range switches,
metadataFilter, the db whereMatcher egress path) and from the
BrainyFieldOperators type, the unsupported-operator error message, the docs
(QUERY_OPERATORS / api README / VFS projection + semantic guides), and the
whereMatcher alias tests. Also migrated Brainy's own internal use — the VFS
TemporalProjection queried with greaterEqual/lessEqual, which would have
silently broken — to gte/lte.
Full gate green: build, unit 1512, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 10:29:20 -07:00
| `gte` / `greaterThanOrEqual` | Greater or equal | `{score: {gte: 90}}` |
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
| `lessThan` / `lt` | Less than | `{price: {lessThan: 100}}` |
refactor(8.0): remove the 4 deprecated query-operator aliases (clean break)
8.0 keeps the canonical operators (eq/ne/gt/gte/lt/lte) and their clean
long-form aliases (equals/notEquals/greaterThan/greaterThanOrEqual/lessThan/
lessThanOrEqual), and drops the four redundant deprecated spellings:
is → eq, isNot → ne, greaterEqual → gte, lessEqual → lte
Removed from every evaluator (metadataIndex criteria + range switches,
metadataFilter, the db whereMatcher egress path) and from the
BrainyFieldOperators type, the unsupported-operator error message, the docs
(QUERY_OPERATORS / api README / VFS projection + semantic guides), and the
whereMatcher alias tests. Also migrated Brainy's own internal use — the VFS
TemporalProjection queried with greaterEqual/lessEqual, which would have
silently broken — to gte/lte.
Full gate green: build, unit 1512, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 10:29:20 -07:00
| `lte` / `lessThanOrEqual` | Less or equal | `{rating: {lte: 3}}` |
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
| `between` | Inclusive range | `{year: {between: [2020, 2025]}}` |
| `oneOf` / `in` | In array | `{color: {oneOf: ['red', 'blue']}}` |
| `noneOf` | Not in array | `{status: {noneOf: ['deleted']}}` |
| `contains` | Array contains value | `{tags: {contains: 'ai'}}` |
| `exists` / `missing` | Field existence | `{email: {exists: true}}` |
2025-08-26 12:32:21 -07:00
| `startsWith` | String prefix | `{name: {startsWith: 'John'}}` |
| `endsWith` | String suffix | `{email: {endsWith: '@gmail.com'}}` |
| `matches` | Pattern match | `{text: {matches: /^[A-Z]/}}` |
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
| `allOf` | AND combinator | `{allOf: [{active: true}, {role: 'admin'}]}` |
| `anyOf` | OR combinator | `{anyOf: [{role: 'admin'}, {role: 'owner'}]}` |
**[Complete Operator Reference → ](../QUERY_OPERATORS.md )** — all operators, aliases, indexed vs in-memory support matrix, and practical examples.
2025-08-26 12:32:21 -07:00
---
feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
## Aggregation Engine
Brainy's aggregation engine maintains **incremental running totals** at write time, delivering O(1) aggregate reads regardless of dataset size. Define aggregates once, and every `add()` , `update()` , and `delete()` automatically updates the running metrics.
### `defineAggregate(definition)` → `void`
Register a named aggregate for incremental computation.
```typescript
brain.defineAggregate({
name: 'monthly_spending',
source: {
type: NounType.Event,
2026-06-11 14:51:00 -07:00
where: { domain: 'financial' } // matches custom metadata fields
feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
},
groupBy: [
'category',
{ field: 'date', window: 'month' } // Time-windowed dimension
],
metrics: {
total: { op: 'sum', field: 'amount' },
count: { op: 'count' },
average: { op: 'avg', field: 'amount' },
highest: { op: 'max', field: 'amount' },
2026-02-17 17:04:11 -08:00
lowest: { op: 'min', field: 'amount' },
spread: { op: 'stddev', field: 'amount' } // Welford's online algorithm
feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
},
materialize: true // Optional: write results as NounType.Measurement entities
})
```
**Parameters:**
| Field | Type | Description |
|-------|------|-------------|
| `name` | `string` | Unique identifier for this aggregate |
| `source.type` | `NounType \| NounType[]` | Entity types that feed into this aggregate |
2026-06-11 14:51:00 -07:00
| `source.where` | `Record<string, unknown>` | Filter on custom **metadata** fields (matched against the entity's `metadata` bag) |
feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
| `source.service` | `string` | Multi-tenancy filter |
2026-06-11 14:51:00 -07:00
| `groupBy` | `GroupByDimension[]` | Dimensions to group by — plain field names, `{ field, window }` for time bucketing, or `{ field, unnest: true }` for array fields (one contribution per element) |
| `metrics` | `Record<string, AggregateMetricDef>` | Named metrics with `op` (`sum` , `count` , `avg` , `min` , `max` , `stddev` , `variance` , `percentile` , `distinctCount` ) and optional `field` . `percentile` additionally requires `p` in `[0, 1]` . |
feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
| `materialize` | `boolean \| object` | Write results as `NounType.Measurement` entities (auto-visible in OData/Sheets/SSE) |
**Time window granularities:** `'hour'` , `'day'` , `'week'` , `'month'` , `'quarter'` , `'year'` , or `{ seconds: number }` for custom intervals.
### `removeAggregate(name)` → `void`
Remove a named aggregate and clean up its state.
```typescript
brain.removeAggregate('monthly_spending')
```
### Querying Aggregates via `find()`
Aggregate results are queried through the standard `find()` method using the `aggregate` parameter:
```typescript
// Simple: query by name
const results = await brain.find({ aggregate: 'monthly_spending' })
// With filtering on group keys
const foodOnly = await brain.find({
aggregate: 'monthly_spending',
where: { category: 'food' }
})
// With sorting and pagination
const topCategories = await brain.find({
aggregate: {
name: 'monthly_spending',
orderBy: 'total',
order: 'desc',
limit: 10
}
})
// Combine find-level params (where, orderBy, limit, offset merge automatically)
const recentFood = await brain.find({
aggregate: 'monthly_spending',
where: { category: 'food' },
orderBy: 'total',
order: 'desc',
limit: 12
})
```
**Result format:** Returns `Result<T>[]` with `type: NounType.Measurement` . Each result contains:
```typescript
{
id: string, // Aggregate group ID (or materialized entity ID)
score: 1.0, // Always 1.0 for aggregates
type: NounType.Measurement,
metadata: {
__aggregate: 'monthly_spending', // Source aggregate name
category: 'food', // Group key values
date: '2024-01', // Time window bucket
total: 342.50, // Computed metrics
count: 28,
average: 12.23,
highest: 45.00,
lowest: 2.50
},
entity: Entity // Full entity structure
}
```
2026-06-11 14:51:00 -07:00
### `queryAggregate(name, params?)` → `Promise<AggregateResult[]>`
The first-class analytics path — returns plain group rows (`{ groupKey, metrics, count }` ) instead of `find()` -style `Result` wrappers. Supports `where` (group-key filter), `having` (SQL-HAVING metric filter), `orderBy` , `order` , `limit` , `offset` :
```typescript
const rows = await brain.queryAggregate('monthly_spending', {
having: { total: { greaterThan: 100 } }, // filter by computed metrics
orderBy: 'total',
order: 'desc',
limit: 10
})
// [{ groupKey: { category: 'food', date: '2024-01' }, metrics: { total: 342.5, count: 28 }, count: 28 }, ...]
```
feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
### How It Works
Aggregation hooks run **outside transactions** on every write operation:
- **`add()` **: If the new entity matches any aggregate's `source` filter, its values are added to the matching group's running totals.
- **`update()` **: The old entity's contribution is reversed and the new entity's contribution is applied (handles group key changes, source filter changes).
- **`delete()` **: The deleted entity's contribution is reversed from its group.
**Performance:** O(A × G × M) per write where A = matching aggregates, G = groupBy dimensions, M = metrics. For typical configurations (2-5 aggregates, 1-3 dimensions, 3-5 metrics), this is effectively O(1) — measured at **10,000 entities in 13ms** in unit tests.
**Infinite loop prevention:** Materialized `NounType.Measurement` entities (with `service: 'brainy:aggregation'` or `metadata.__aggregate` ) are automatically excluded from all aggregate source matching.
**Persistence:** Definitions and running state are persisted to storage on `flush()` /`close()` and reloaded on `init()` . Definition changes are detected via FNV-1a hashing — only changed aggregates reset their state.
**Native acceleration:** Register an `'aggregation'` provider via the plugin system to replace the TypeScript engine with a custom native implementation for higher throughput at scale.
### Financial Data Modeling
2026-06-11 14:51:00 -07:00
Brainy supports financial analytics through **subtypes and metadata conventions** on existing NounTypes — no custom types needed:
feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
```typescript
2026-06-11 14:51:00 -07:00
// Transaction = NounType.Event + 'transaction' subtype + financial metadata
feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
await brain.add({
data: 'Coffee at Blue Bottle',
type: NounType.Event,
2026-06-11 14:51:00 -07:00
subtype: 'transaction', // top-level standard field (reserved — never in metadata)
feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
metadata: {
domain: 'financial',
amount: 5.50,
currency: 'USD',
category: 'food',
date: Date.now(),
merchant: 'Blue Bottle Coffee'
}
})
2026-06-11 14:51:00 -07:00
// Account = NounType.Collection + 'account' subtype + financial metadata
feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
await brain.add({
data: 'Checking Account',
type: NounType.Collection,
2026-06-11 14:51:00 -07:00
subtype: 'account',
feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
metadata: {
domain: 'financial',
accountType: 'checking',
currency: 'USD',
institution: 'Chase'
}
})
2026-06-11 14:51:00 -07:00
// Invoice = NounType.Document + 'invoice' subtype + financial metadata
feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
await brain.add({
data: 'Invoice #1234 from Acme Corp',
type: NounType.Document,
2026-06-11 14:51:00 -07:00
subtype: 'invoice',
feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
metadata: {
domain: 'financial',
amount: 15000,
currency: 'USD',
status: 'pending',
dueDate: Date.UTC(2024, 2, 15),
vendor: 'Acme Corp'
}
})
```
---
2025-11-02 10:58:52 -08:00
## Relationships
### `relate(params)` → `Promise<string>`
Create a typed relationship between entities.
```typescript
const relId = await brain.relate({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
from: sourceId,
to: targetId,
feat: verb subtype + updateRelation + requireSubtype enforcement
Brings verbs to first-class parity with nouns. The 7.29.0 subtype primitive
shipped for entities only; this release ships the symmetric verb mirror plus
the enforcement layer for ensuring every entity AND every relationship has
both type AND subtype.
Layer V1 — verb subtype mirror
- HNSWVerbWithMetadata.subtype + STANDARD_VERB_FIELDS set + resolveVerbField()
- Relation<T>.subtype, RelateParams<T>.subtype, UpdateRelationParams<T> extended,
GetRelationsParams.subtype, GraphConstraints.subtype (for find connected)
- relate() persists subtype on verbMetadata + GraphVerb + transaction ops
- getRelations({ type, subtype }) fast-path filter with set membership
- find({ connected: { via, subtype, depth } }) traversal filter (depth-1 on
the JS path; explicit error on depth > 1 pointing at Cortex native)
- verbsToRelations + storage destructure sites surface subtype to top-level
- All three graph-index fast-path queries (getVerbsBySource/ByTarget) enrich
with subtype from metadata
Layer V2 — updateRelation() closes a pre-7.30 gap
- New first-class verb update method (parallel to update() for nouns)
- Changes subtype/type/weight/confidence/data/metadata in place
- Re-indexes in graph adjacency when verb type changes; id preserved
- validateUpdateRelationParams enforces id + at-least-one-field-to-update
Layer V3 — verb subtype storage rollup
- verbSubtypeCountsByType: Map<number, Map<string, number>> on BaseStorage
- verbSubtypeByIdCache for self-heal during update/delete
- incrementVerbSubtypeCount + decrementVerbSubtypeCount maintain state
- loadVerbSubtypeStatistics + saveVerbSubtypeStatistics persist to
_system/verb-subtype-statistics.json (mirrors noun-side shape)
- rebuildVerbSubtypeCounts for poison recovery / explicit repair
- getVerbSubtypeCountsByType accessor for the public counts API
- Wired into init() / flushCounts() / saveVerbMetadata / deleteVerbMetadata
Layer V4 — verb counts API + relationshipSubtypesOf
- brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) breakdown or point
- brain.counts.topRelationshipSubtypes(verb, n) — top N by count
- brain.relationshipSubtypesOf(verb) — sorted distinct subtypes
Layer V5 — migrateField extended to verbs
- New entityKind?: 'noun' | 'verb' | 'both' option (default 'noun')
- Mirror verb iteration via storage.getVerbs() with same path semantics
- verbToRelationLike + buildRelationMigrationUpdate helpers project the
storage verb shape onto the Entity<T>-shaped surface readPath understands
- Routes through new updateRelation() for the verb-side rewrite
Enforcement (opt-in in 7.30, default in 8.0)
- brain.requireSubtype(type, options) — unified API for NounType OR VerbType.
Registers per-type rules with optional values whitelist; composes with the
brain-wide flag.
- new Brainy({ requireSubtype: true }) — brain-wide strict mode. Every public
write path validates the pairing guarantee.
- { except: [NounType.Thing, ...] } form for catch-all type exemptions
- Atomic-fail semantics on addMany / relateMany — pre-validate every item
before any storage write, throw on first failure with item index
- Per-type rules + brain-wide flag both throw with descriptive messages
- VFS infrastructure bypass via metadata.isVFSEntity / isVFS markers so
brain's own VFS writes don't get rejected when strict mode is on
VFS labeling — concrete subtypes for infrastructure entities
- VFS root: NounType.Collection + subtype: 'vfs-root' (was bare Collection)
- VFS directories: subtype: 'vfs-directory'
- VFS files: subtype: 'vfs-file' (NounType still mime-based)
- VFS containment edges: VerbType.Contains + subtype: 'vfs-contains'
- Lets consumers cleanly enumerate VFS state via find({ subtype: 'vfs-file' })
and distinguish Brainy's VFS Collections from user-created Collections
Docs
- docs/guides/subtypes-and-facets.md extended with Layer V (Verbs) section +
Enforcement section. New full reference at the bottom split into Layer 1
(nouns), Layer V (verbs), Layer 2 (facets), Layer 3 (migration), Enforcement.
- docs/api/README.md adds updateRelation(), getRelations({ subtype }), the
three verb-side counts methods, requireSubtype(), and the brain-wide
constructor option. relate() params include subtype.
- docs/DATA_MODEL.md adds a Subtype-for-VerbType section + STANDARD_VERB_FIELDS
- docs/architecture/finite-type-system.md extends Principle 1a to verbs
- docs/QUERY_OPERATORS.md adds a verb-subtype filter section covering
getRelations and find({connected, subtype}) traversal
- README.md "Subtypes" section now shows both noun + verb in one example +
the enforcement APIs
- RELEASES.md v7.30.0 entry with the full noun/verb capability parity matrix
Tests
- tests/integration/verb-subtype-and-enforcement.test.ts — 30 new tests
covering V1 round-trips, V1 set membership, updateRelation in place,
updateRelation preservation, V2 counts breakdown + point + topN + distinct,
V2 decrements on unrelate, V2 re-routes on updateRelation, V3 depth-1
traversal filter, V3 depth>1 explicit error, V4 verb migration, V4 both
entity kinds, V4 readBoth preservation, V5 per-type required rejection,
V5 vocabulary rejection, V5 on-vocab acceptance, V5 verb-side enforcement,
V5 addMany atomic-fail, V5 relateMany atomic-fail, V5 update enforcement,
V5 updateRelation enforcement, V5 brain-wide strict mode, V5 except clause.
Verification
- Unit suite: 1468/1468 passing
- Noun subtype integration (7.29 carryover): 26/26 passing
- Verb subtype + enforcement integration: 30/30 passing
- Type-check: clean
- Build: clean
- Public closed-source reference audit: clean
Internal 8.0 spec
- .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md (gitignored, not in npm artifact)
documents the contract upgrade Cortex 3.0 implements against: required-by-
default subtype, SubtypeRegistry typing hook, native simplification,
multi-hop traversal native fast path, brain.fillSubtypes() migration helper.
Coordinated via PLATFORM-HANDOFF rows CTX-SUBTYPE-PARITY-V2 (7.30 parallel
work) and CTX-SUBTYPE-8.0-CONTRACT (8.0 spec).
2026-06-05 11:15:52 -07:00
type: VerbType.ReportsTo,
subtype: 'direct', // Optional: sub-classification
data: 'Collaborated on the research paper', // Optional: content for this edge
metadata: { // Optional: structured edge fields
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
strength: 0.9,
role: 'primary author'
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
}
2025-11-02 10:58:52 -08:00
})
```
**Parameters:**
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
- `from` : `string` - Source entity ID (must exist)
- `to` : `string` - Target entity ID (must exist)
2025-11-02 10:58:52 -08:00
- `type` : `VerbType` - Relationship type
feat: verb subtype + updateRelation + requireSubtype enforcement
Brings verbs to first-class parity with nouns. The 7.29.0 subtype primitive
shipped for entities only; this release ships the symmetric verb mirror plus
the enforcement layer for ensuring every entity AND every relationship has
both type AND subtype.
Layer V1 — verb subtype mirror
- HNSWVerbWithMetadata.subtype + STANDARD_VERB_FIELDS set + resolveVerbField()
- Relation<T>.subtype, RelateParams<T>.subtype, UpdateRelationParams<T> extended,
GetRelationsParams.subtype, GraphConstraints.subtype (for find connected)
- relate() persists subtype on verbMetadata + GraphVerb + transaction ops
- getRelations({ type, subtype }) fast-path filter with set membership
- find({ connected: { via, subtype, depth } }) traversal filter (depth-1 on
the JS path; explicit error on depth > 1 pointing at Cortex native)
- verbsToRelations + storage destructure sites surface subtype to top-level
- All three graph-index fast-path queries (getVerbsBySource/ByTarget) enrich
with subtype from metadata
Layer V2 — updateRelation() closes a pre-7.30 gap
- New first-class verb update method (parallel to update() for nouns)
- Changes subtype/type/weight/confidence/data/metadata in place
- Re-indexes in graph adjacency when verb type changes; id preserved
- validateUpdateRelationParams enforces id + at-least-one-field-to-update
Layer V3 — verb subtype storage rollup
- verbSubtypeCountsByType: Map<number, Map<string, number>> on BaseStorage
- verbSubtypeByIdCache for self-heal during update/delete
- incrementVerbSubtypeCount + decrementVerbSubtypeCount maintain state
- loadVerbSubtypeStatistics + saveVerbSubtypeStatistics persist to
_system/verb-subtype-statistics.json (mirrors noun-side shape)
- rebuildVerbSubtypeCounts for poison recovery / explicit repair
- getVerbSubtypeCountsByType accessor for the public counts API
- Wired into init() / flushCounts() / saveVerbMetadata / deleteVerbMetadata
Layer V4 — verb counts API + relationshipSubtypesOf
- brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) breakdown or point
- brain.counts.topRelationshipSubtypes(verb, n) — top N by count
- brain.relationshipSubtypesOf(verb) — sorted distinct subtypes
Layer V5 — migrateField extended to verbs
- New entityKind?: 'noun' | 'verb' | 'both' option (default 'noun')
- Mirror verb iteration via storage.getVerbs() with same path semantics
- verbToRelationLike + buildRelationMigrationUpdate helpers project the
storage verb shape onto the Entity<T>-shaped surface readPath understands
- Routes through new updateRelation() for the verb-side rewrite
Enforcement (opt-in in 7.30, default in 8.0)
- brain.requireSubtype(type, options) — unified API for NounType OR VerbType.
Registers per-type rules with optional values whitelist; composes with the
brain-wide flag.
- new Brainy({ requireSubtype: true }) — brain-wide strict mode. Every public
write path validates the pairing guarantee.
- { except: [NounType.Thing, ...] } form for catch-all type exemptions
- Atomic-fail semantics on addMany / relateMany — pre-validate every item
before any storage write, throw on first failure with item index
- Per-type rules + brain-wide flag both throw with descriptive messages
- VFS infrastructure bypass via metadata.isVFSEntity / isVFS markers so
brain's own VFS writes don't get rejected when strict mode is on
VFS labeling — concrete subtypes for infrastructure entities
- VFS root: NounType.Collection + subtype: 'vfs-root' (was bare Collection)
- VFS directories: subtype: 'vfs-directory'
- VFS files: subtype: 'vfs-file' (NounType still mime-based)
- VFS containment edges: VerbType.Contains + subtype: 'vfs-contains'
- Lets consumers cleanly enumerate VFS state via find({ subtype: 'vfs-file' })
and distinguish Brainy's VFS Collections from user-created Collections
Docs
- docs/guides/subtypes-and-facets.md extended with Layer V (Verbs) section +
Enforcement section. New full reference at the bottom split into Layer 1
(nouns), Layer V (verbs), Layer 2 (facets), Layer 3 (migration), Enforcement.
- docs/api/README.md adds updateRelation(), getRelations({ subtype }), the
three verb-side counts methods, requireSubtype(), and the brain-wide
constructor option. relate() params include subtype.
- docs/DATA_MODEL.md adds a Subtype-for-VerbType section + STANDARD_VERB_FIELDS
- docs/architecture/finite-type-system.md extends Principle 1a to verbs
- docs/QUERY_OPERATORS.md adds a verb-subtype filter section covering
getRelations and find({connected, subtype}) traversal
- README.md "Subtypes" section now shows both noun + verb in one example +
the enforcement APIs
- RELEASES.md v7.30.0 entry with the full noun/verb capability parity matrix
Tests
- tests/integration/verb-subtype-and-enforcement.test.ts — 30 new tests
covering V1 round-trips, V1 set membership, updateRelation in place,
updateRelation preservation, V2 counts breakdown + point + topN + distinct,
V2 decrements on unrelate, V2 re-routes on updateRelation, V3 depth-1
traversal filter, V3 depth>1 explicit error, V4 verb migration, V4 both
entity kinds, V4 readBoth preservation, V5 per-type required rejection,
V5 vocabulary rejection, V5 on-vocab acceptance, V5 verb-side enforcement,
V5 addMany atomic-fail, V5 relateMany atomic-fail, V5 update enforcement,
V5 updateRelation enforcement, V5 brain-wide strict mode, V5 except clause.
Verification
- Unit suite: 1468/1468 passing
- Noun subtype integration (7.29 carryover): 26/26 passing
- Verb subtype + enforcement integration: 30/30 passing
- Type-check: clean
- Build: clean
- Public closed-source reference audit: clean
Internal 8.0 spec
- .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md (gitignored, not in npm artifact)
documents the contract upgrade Cortex 3.0 implements against: required-by-
default subtype, SubtypeRegistry typing hook, native simplification,
multi-hop traversal native fast path, brain.fillSubtypes() migration helper.
Coordinated via PLATFORM-HANDOFF rows CTX-SUBTYPE-PARITY-V2 (7.30 parallel
work) and CTX-SUBTYPE-8.0-CONTRACT (8.0 spec).
2026-06-05 11:15:52 -07:00
- `subtype?` : `string` - Per-product sub-classification within the VerbType (top-level standard field, fast-path indexed). See [Subtypes & Facets ](../guides/subtypes-and-facets.md ).
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
- `data?` : `any` - Content for the relationship (overrides auto-computed vector)
- `metadata?` : `object` - Structured edge fields
- `weight?` : `number` - Connection strength (0-1, default: 1.0)
- `bidirectional?` : `boolean` - Create reverse edge too (default: false)
- `confidence?` : `number` - Relationship certainty (0-1)
2025-11-02 10:58:52 -08:00
fix: internal subtype consistency + brain.audit() diagnostic + improved enforcement errors
Brainy 7.30 shipped opt-in subtype enforcement; SDK 3.20.0 then registered
SDK_CORE_VOCABULARY on every consumer's brain (Event, Collection, Message,
Contract, Media, Document NounTypes). On 2026-06-08 Venue's /book flow went 500
because their brain.add({ type: NounType.Event, ... }) call sites lacked
subtype. An audit of Brainy's OWN source revealed 14 HIGH-risk internal write
paths that also omit subtype — any consumer running the same vocabulary would
have hit Brainy's infrastructure paths next. 7.30.1 closes both gaps before
8.0 makes strict mode the default.
Additive across the board. Zero behavior change for consumers not using strict
mode. Every change is JS-side — Cortex needs no work for 7.30.1.
NEW — brain.audit() diagnostic
- Read-only method walking storage.getNouns() / getVerbs() pagination
- Returns { entitiesWithoutSubtype: { type: count }, relationshipsWithoutSubtype,
total, scanned, recommendation }
- VFS infrastructure entities excluded by default (they bypass enforcement via
isVFSEntity marker); pass { includeVFS: true } to surface them
- The companion to migrateField (7.x) and fillSubtypes (8.0): tells consumers
exactly what would break under strict enforcement, deterministically
NEW — Improved enforcement error messages
- Caller's source location extracted from Error().stack so users see their own
call site, not a Brainy internal frame
- Specific guidance branches: registered vocabulary → "Pass one of: a, b, c";
brain-wide strict mode → mentions the except clause; otherwise → registration
recipe via brain.requireSubtype()
- Documentation link to the canonical migration recipe
- Same shape for noun and verb enforcement
NEW — CLI --subtype flag
- brainy add and brainy relate gain -s/--subtype <value>
- Defaults to 'cli-add' / 'cli-relate' so the CLI works against strict-mode
brains without the user needing to know the vocabulary in advance
INTERNAL — every Brainy write path now sets subtype
- VFS Contains edges (5 sites at lines 503/905/1694/1772/1886) → 'vfs-contains'
- VFS symlink entity → 'vfs-symlink' (NEW — distinct from 'vfs-file')
- VFS copy-file → preserves source subtype, falls back to 'vfs-file'
- VFS symlink also adopts the isVFSEntity infrastructure marker so it bypasses
enforcement in strict mode
- Aggregation materializer (Measurement entities) → 'materialized-aggregate'
- ImportCoordinator (3 sites): document → 'import-source'; entities →
options.defaultSubtype ?? 'imported'; placeholder → 'import-placeholder'
- SmartImportOrchestrator (4 entity sites + 2 batch relate sites): same
precedence (extractor → options.defaultSubtype → 'imported')
- EntityDeduplicator → candidate.subtype ?? 'imported'
- UniversalImportAPI → extractor → 'extracted' for both entities and relations
- NeuralImport → adds defaultSubtype to NeuralImportOptions; precedence same
- GoogleSheetsIntegration → request body 'subtype' ?? 'imported-from-sheets'
- ODataIntegration → request body 'Subtype' ?? 'imported-from-odata'
- MCP client message storage → 'mcp-message' (also fixes pre-existing missing
data field and missing type by aliasing from the prior text field)
Side-effect fix: storage.getNouns() paginated now surfaces subtype to top-level
- Single-noun getNoun() already did this in 7.30; the paginated path was missed
- Without this fix brain.audit() saw missing subtype on entities that actually
had one (caught by the strict-mode self-test before release)
NEW — tests/integration/strict-mode-self-test.test.ts (13 tests)
- Creates a brain under the exact SDK_CORE_VOCABULARY shape Venue hit + brain-
wide strict mode
- Exercises every internal Brainy path: VFS root + mkdir + writeFile + cp + mv
+ ln + symlink; aggregation engine; audit diagnostic with includeVFS toggle
- Validates error message UX: caller location, vocabulary guidance, brain-wide
strict mode guidance, off-vocabulary value reporting
Docs
- New "Strict mode in practice" section in docs/guides/subtypes-and-facets.md
covering the SDK_CORE_VOCABULARY pattern, 4-step migration recipe
(audit → migrateField → hand-fix → re-audit), the Brainy-internal label
reference table, and an 8.0 forward-look on fillSubtypes()
- docs/api/README.md: new audit() entry, strict-mode tips on add() and relate()
- RELEASES.md: full 7.30.1 entry
Cortex parity (forward-looking, not blocking 7.30.1)
- 6th open question added to .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md: native
fast path for audit() and fillSubtypes() via column-store null-subtype
bitmap for billion-scale brains
- Cortex should add a parity test mirroring strict-mode-self-test.test.ts
against their native paths to catch any latent bug where native writes
bypass JS validation
- Brainy-internal subtype labels become a documented part of the 8.0 contract
(useful for Cortex telemetry surfacing Brainy-managed infrastructure %)
Verification
- npx tsc --noEmit: clean
- npm test: 1468/1468 unit
- 7.29 noun integration suite: 26/26 (no regression)
- 7.30 verb subtype + enforcement integration suite: 30/30 (no regression)
- New strict-mode-self-test integration suite: 13/13
- npm run build: clean
- Closed-source product reference audit: clean
Addresses VE-SUBTYPE-MIGRATION (Venue's reported request) and ships internal
labels Venue did NOT ask for but that would have broken them next under their
own vocabulary registration.
2026-06-08 11:31:47 -07:00
> **Strict-mode tip:** same as `add()` — if a vocabulary is registered for your `type`, pass a matching `subtype`. Run `await brain.audit()` first to surface pre-existing gaps.
2025-11-02 10:58:52 -08:00
**Returns:** `Promise<string>` - Relationship ID
---
feat: verb subtype + updateRelation + requireSubtype enforcement
Brings verbs to first-class parity with nouns. The 7.29.0 subtype primitive
shipped for entities only; this release ships the symmetric verb mirror plus
the enforcement layer for ensuring every entity AND every relationship has
both type AND subtype.
Layer V1 — verb subtype mirror
- HNSWVerbWithMetadata.subtype + STANDARD_VERB_FIELDS set + resolveVerbField()
- Relation<T>.subtype, RelateParams<T>.subtype, UpdateRelationParams<T> extended,
GetRelationsParams.subtype, GraphConstraints.subtype (for find connected)
- relate() persists subtype on verbMetadata + GraphVerb + transaction ops
- getRelations({ type, subtype }) fast-path filter with set membership
- find({ connected: { via, subtype, depth } }) traversal filter (depth-1 on
the JS path; explicit error on depth > 1 pointing at Cortex native)
- verbsToRelations + storage destructure sites surface subtype to top-level
- All three graph-index fast-path queries (getVerbsBySource/ByTarget) enrich
with subtype from metadata
Layer V2 — updateRelation() closes a pre-7.30 gap
- New first-class verb update method (parallel to update() for nouns)
- Changes subtype/type/weight/confidence/data/metadata in place
- Re-indexes in graph adjacency when verb type changes; id preserved
- validateUpdateRelationParams enforces id + at-least-one-field-to-update
Layer V3 — verb subtype storage rollup
- verbSubtypeCountsByType: Map<number, Map<string, number>> on BaseStorage
- verbSubtypeByIdCache for self-heal during update/delete
- incrementVerbSubtypeCount + decrementVerbSubtypeCount maintain state
- loadVerbSubtypeStatistics + saveVerbSubtypeStatistics persist to
_system/verb-subtype-statistics.json (mirrors noun-side shape)
- rebuildVerbSubtypeCounts for poison recovery / explicit repair
- getVerbSubtypeCountsByType accessor for the public counts API
- Wired into init() / flushCounts() / saveVerbMetadata / deleteVerbMetadata
Layer V4 — verb counts API + relationshipSubtypesOf
- brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) breakdown or point
- brain.counts.topRelationshipSubtypes(verb, n) — top N by count
- brain.relationshipSubtypesOf(verb) — sorted distinct subtypes
Layer V5 — migrateField extended to verbs
- New entityKind?: 'noun' | 'verb' | 'both' option (default 'noun')
- Mirror verb iteration via storage.getVerbs() with same path semantics
- verbToRelationLike + buildRelationMigrationUpdate helpers project the
storage verb shape onto the Entity<T>-shaped surface readPath understands
- Routes through new updateRelation() for the verb-side rewrite
Enforcement (opt-in in 7.30, default in 8.0)
- brain.requireSubtype(type, options) — unified API for NounType OR VerbType.
Registers per-type rules with optional values whitelist; composes with the
brain-wide flag.
- new Brainy({ requireSubtype: true }) — brain-wide strict mode. Every public
write path validates the pairing guarantee.
- { except: [NounType.Thing, ...] } form for catch-all type exemptions
- Atomic-fail semantics on addMany / relateMany — pre-validate every item
before any storage write, throw on first failure with item index
- Per-type rules + brain-wide flag both throw with descriptive messages
- VFS infrastructure bypass via metadata.isVFSEntity / isVFS markers so
brain's own VFS writes don't get rejected when strict mode is on
VFS labeling — concrete subtypes for infrastructure entities
- VFS root: NounType.Collection + subtype: 'vfs-root' (was bare Collection)
- VFS directories: subtype: 'vfs-directory'
- VFS files: subtype: 'vfs-file' (NounType still mime-based)
- VFS containment edges: VerbType.Contains + subtype: 'vfs-contains'
- Lets consumers cleanly enumerate VFS state via find({ subtype: 'vfs-file' })
and distinguish Brainy's VFS Collections from user-created Collections
Docs
- docs/guides/subtypes-and-facets.md extended with Layer V (Verbs) section +
Enforcement section. New full reference at the bottom split into Layer 1
(nouns), Layer V (verbs), Layer 2 (facets), Layer 3 (migration), Enforcement.
- docs/api/README.md adds updateRelation(), getRelations({ subtype }), the
three verb-side counts methods, requireSubtype(), and the brain-wide
constructor option. relate() params include subtype.
- docs/DATA_MODEL.md adds a Subtype-for-VerbType section + STANDARD_VERB_FIELDS
- docs/architecture/finite-type-system.md extends Principle 1a to verbs
- docs/QUERY_OPERATORS.md adds a verb-subtype filter section covering
getRelations and find({connected, subtype}) traversal
- README.md "Subtypes" section now shows both noun + verb in one example +
the enforcement APIs
- RELEASES.md v7.30.0 entry with the full noun/verb capability parity matrix
Tests
- tests/integration/verb-subtype-and-enforcement.test.ts — 30 new tests
covering V1 round-trips, V1 set membership, updateRelation in place,
updateRelation preservation, V2 counts breakdown + point + topN + distinct,
V2 decrements on unrelate, V2 re-routes on updateRelation, V3 depth-1
traversal filter, V3 depth>1 explicit error, V4 verb migration, V4 both
entity kinds, V4 readBoth preservation, V5 per-type required rejection,
V5 vocabulary rejection, V5 on-vocab acceptance, V5 verb-side enforcement,
V5 addMany atomic-fail, V5 relateMany atomic-fail, V5 update enforcement,
V5 updateRelation enforcement, V5 brain-wide strict mode, V5 except clause.
Verification
- Unit suite: 1468/1468 passing
- Noun subtype integration (7.29 carryover): 26/26 passing
- Verb subtype + enforcement integration: 30/30 passing
- Type-check: clean
- Build: clean
- Public closed-source reference audit: clean
Internal 8.0 spec
- .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md (gitignored, not in npm artifact)
documents the contract upgrade Cortex 3.0 implements against: required-by-
default subtype, SubtypeRegistry typing hook, native simplification,
multi-hop traversal native fast path, brain.fillSubtypes() migration helper.
Coordinated via PLATFORM-HANDOFF rows CTX-SUBTYPE-PARITY-V2 (7.30 parallel
work) and CTX-SUBTYPE-8.0-CONTRACT (8.0 spec).
2026-06-05 11:15:52 -07:00
### `updateRelation(params)` → `Promise<void>`
Update an existing relationship. Mirror of `update()` for verbs — closed a long-standing gap (verbs had no update path before 7.30).
```typescript
// Change the subtype on an existing relationship
await brain.updateRelation({ id: relId, subtype: 'dotted-line' })
// Update weight + confidence
await brain.updateRelation({ id: relId, weight: 0.7, confidence: 0.9 })
// Change verb type (re-indexes in graph adjacency, id preserved)
await brain.updateRelation({ id: relId, type: VerbType.WorksWith })
```
**Parameters:**
- `id` : `string` - Relationship ID (required)
- `type?` : `VerbType` - Change verb type (re-indexes in graph adjacency)
- `subtype?` : `string` - Change sub-classification (omit to preserve existing)
- `weight?` : `number` - New weight (0-1)
- `confidence?` : `number` - New confidence (0-1)
- `data?` : `any` - New content
- `metadata?` : `object` - Metadata to merge (or replace with `merge: false` )
- `merge?` : `boolean` - Merge or replace metadata (default: true)
**Returns:** `Promise<void>`
---
2026-06-11 14:51:00 -07:00
### `related(params)` → `Promise<Relation[]>`
2025-11-02 10:58:52 -08:00
2026-06-11 14:51:00 -07:00
Get relationships for an entity. Same name and surface as `db.related()` on a
pinned `Db` view.
2025-08-26 12:32:21 -07:00
```typescript
2025-11-02 10:58:52 -08:00
// Get all relationships FROM an entity
2026-06-11 14:51:00 -07:00
const outgoing = await brain.related({ from: entityId })
2025-11-02 10:58:52 -08:00
// Get all relationships TO an entity
2026-06-11 14:51:00 -07:00
const incoming = await brain.related({ to: entityId })
2025-11-02 10:58:52 -08:00
// Filter by type
2026-06-11 14:51:00 -07:00
const contains = await brain.related({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
from: entityId,
type: VerbType.Contains
2025-08-26 12:32:21 -07:00
})
feat: verb subtype + updateRelation + requireSubtype enforcement
Brings verbs to first-class parity with nouns. The 7.29.0 subtype primitive
shipped for entities only; this release ships the symmetric verb mirror plus
the enforcement layer for ensuring every entity AND every relationship has
both type AND subtype.
Layer V1 — verb subtype mirror
- HNSWVerbWithMetadata.subtype + STANDARD_VERB_FIELDS set + resolveVerbField()
- Relation<T>.subtype, RelateParams<T>.subtype, UpdateRelationParams<T> extended,
GetRelationsParams.subtype, GraphConstraints.subtype (for find connected)
- relate() persists subtype on verbMetadata + GraphVerb + transaction ops
- getRelations({ type, subtype }) fast-path filter with set membership
- find({ connected: { via, subtype, depth } }) traversal filter (depth-1 on
the JS path; explicit error on depth > 1 pointing at Cortex native)
- verbsToRelations + storage destructure sites surface subtype to top-level
- All three graph-index fast-path queries (getVerbsBySource/ByTarget) enrich
with subtype from metadata
Layer V2 — updateRelation() closes a pre-7.30 gap
- New first-class verb update method (parallel to update() for nouns)
- Changes subtype/type/weight/confidence/data/metadata in place
- Re-indexes in graph adjacency when verb type changes; id preserved
- validateUpdateRelationParams enforces id + at-least-one-field-to-update
Layer V3 — verb subtype storage rollup
- verbSubtypeCountsByType: Map<number, Map<string, number>> on BaseStorage
- verbSubtypeByIdCache for self-heal during update/delete
- incrementVerbSubtypeCount + decrementVerbSubtypeCount maintain state
- loadVerbSubtypeStatistics + saveVerbSubtypeStatistics persist to
_system/verb-subtype-statistics.json (mirrors noun-side shape)
- rebuildVerbSubtypeCounts for poison recovery / explicit repair
- getVerbSubtypeCountsByType accessor for the public counts API
- Wired into init() / flushCounts() / saveVerbMetadata / deleteVerbMetadata
Layer V4 — verb counts API + relationshipSubtypesOf
- brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) breakdown or point
- brain.counts.topRelationshipSubtypes(verb, n) — top N by count
- brain.relationshipSubtypesOf(verb) — sorted distinct subtypes
Layer V5 — migrateField extended to verbs
- New entityKind?: 'noun' | 'verb' | 'both' option (default 'noun')
- Mirror verb iteration via storage.getVerbs() with same path semantics
- verbToRelationLike + buildRelationMigrationUpdate helpers project the
storage verb shape onto the Entity<T>-shaped surface readPath understands
- Routes through new updateRelation() for the verb-side rewrite
Enforcement (opt-in in 7.30, default in 8.0)
- brain.requireSubtype(type, options) — unified API for NounType OR VerbType.
Registers per-type rules with optional values whitelist; composes with the
brain-wide flag.
- new Brainy({ requireSubtype: true }) — brain-wide strict mode. Every public
write path validates the pairing guarantee.
- { except: [NounType.Thing, ...] } form for catch-all type exemptions
- Atomic-fail semantics on addMany / relateMany — pre-validate every item
before any storage write, throw on first failure with item index
- Per-type rules + brain-wide flag both throw with descriptive messages
- VFS infrastructure bypass via metadata.isVFSEntity / isVFS markers so
brain's own VFS writes don't get rejected when strict mode is on
VFS labeling — concrete subtypes for infrastructure entities
- VFS root: NounType.Collection + subtype: 'vfs-root' (was bare Collection)
- VFS directories: subtype: 'vfs-directory'
- VFS files: subtype: 'vfs-file' (NounType still mime-based)
- VFS containment edges: VerbType.Contains + subtype: 'vfs-contains'
- Lets consumers cleanly enumerate VFS state via find({ subtype: 'vfs-file' })
and distinguish Brainy's VFS Collections from user-created Collections
Docs
- docs/guides/subtypes-and-facets.md extended with Layer V (Verbs) section +
Enforcement section. New full reference at the bottom split into Layer 1
(nouns), Layer V (verbs), Layer 2 (facets), Layer 3 (migration), Enforcement.
- docs/api/README.md adds updateRelation(), getRelations({ subtype }), the
three verb-side counts methods, requireSubtype(), and the brain-wide
constructor option. relate() params include subtype.
- docs/DATA_MODEL.md adds a Subtype-for-VerbType section + STANDARD_VERB_FIELDS
- docs/architecture/finite-type-system.md extends Principle 1a to verbs
- docs/QUERY_OPERATORS.md adds a verb-subtype filter section covering
getRelations and find({connected, subtype}) traversal
- README.md "Subtypes" section now shows both noun + verb in one example +
the enforcement APIs
- RELEASES.md v7.30.0 entry with the full noun/verb capability parity matrix
Tests
- tests/integration/verb-subtype-and-enforcement.test.ts — 30 new tests
covering V1 round-trips, V1 set membership, updateRelation in place,
updateRelation preservation, V2 counts breakdown + point + topN + distinct,
V2 decrements on unrelate, V2 re-routes on updateRelation, V3 depth-1
traversal filter, V3 depth>1 explicit error, V4 verb migration, V4 both
entity kinds, V4 readBoth preservation, V5 per-type required rejection,
V5 vocabulary rejection, V5 on-vocab acceptance, V5 verb-side enforcement,
V5 addMany atomic-fail, V5 relateMany atomic-fail, V5 update enforcement,
V5 updateRelation enforcement, V5 brain-wide strict mode, V5 except clause.
Verification
- Unit suite: 1468/1468 passing
- Noun subtype integration (7.29 carryover): 26/26 passing
- Verb subtype + enforcement integration: 30/30 passing
- Type-check: clean
- Build: clean
- Public closed-source reference audit: clean
Internal 8.0 spec
- .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md (gitignored, not in npm artifact)
documents the contract upgrade Cortex 3.0 implements against: required-by-
default subtype, SubtypeRegistry typing hook, native simplification,
multi-hop traversal native fast path, brain.fillSubtypes() migration helper.
Coordinated via PLATFORM-HANDOFF rows CTX-SUBTYPE-PARITY-V2 (7.30 parallel
work) and CTX-SUBTYPE-8.0-CONTRACT (8.0 spec).
2026-06-05 11:15:52 -07:00
// Filter by subtype (fast path, column-store hit)
2026-06-11 14:51:00 -07:00
const direct = await brain.related({
feat: verb subtype + updateRelation + requireSubtype enforcement
Brings verbs to first-class parity with nouns. The 7.29.0 subtype primitive
shipped for entities only; this release ships the symmetric verb mirror plus
the enforcement layer for ensuring every entity AND every relationship has
both type AND subtype.
Layer V1 — verb subtype mirror
- HNSWVerbWithMetadata.subtype + STANDARD_VERB_FIELDS set + resolveVerbField()
- Relation<T>.subtype, RelateParams<T>.subtype, UpdateRelationParams<T> extended,
GetRelationsParams.subtype, GraphConstraints.subtype (for find connected)
- relate() persists subtype on verbMetadata + GraphVerb + transaction ops
- getRelations({ type, subtype }) fast-path filter with set membership
- find({ connected: { via, subtype, depth } }) traversal filter (depth-1 on
the JS path; explicit error on depth > 1 pointing at Cortex native)
- verbsToRelations + storage destructure sites surface subtype to top-level
- All three graph-index fast-path queries (getVerbsBySource/ByTarget) enrich
with subtype from metadata
Layer V2 — updateRelation() closes a pre-7.30 gap
- New first-class verb update method (parallel to update() for nouns)
- Changes subtype/type/weight/confidence/data/metadata in place
- Re-indexes in graph adjacency when verb type changes; id preserved
- validateUpdateRelationParams enforces id + at-least-one-field-to-update
Layer V3 — verb subtype storage rollup
- verbSubtypeCountsByType: Map<number, Map<string, number>> on BaseStorage
- verbSubtypeByIdCache for self-heal during update/delete
- incrementVerbSubtypeCount + decrementVerbSubtypeCount maintain state
- loadVerbSubtypeStatistics + saveVerbSubtypeStatistics persist to
_system/verb-subtype-statistics.json (mirrors noun-side shape)
- rebuildVerbSubtypeCounts for poison recovery / explicit repair
- getVerbSubtypeCountsByType accessor for the public counts API
- Wired into init() / flushCounts() / saveVerbMetadata / deleteVerbMetadata
Layer V4 — verb counts API + relationshipSubtypesOf
- brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) breakdown or point
- brain.counts.topRelationshipSubtypes(verb, n) — top N by count
- brain.relationshipSubtypesOf(verb) — sorted distinct subtypes
Layer V5 — migrateField extended to verbs
- New entityKind?: 'noun' | 'verb' | 'both' option (default 'noun')
- Mirror verb iteration via storage.getVerbs() with same path semantics
- verbToRelationLike + buildRelationMigrationUpdate helpers project the
storage verb shape onto the Entity<T>-shaped surface readPath understands
- Routes through new updateRelation() for the verb-side rewrite
Enforcement (opt-in in 7.30, default in 8.0)
- brain.requireSubtype(type, options) — unified API for NounType OR VerbType.
Registers per-type rules with optional values whitelist; composes with the
brain-wide flag.
- new Brainy({ requireSubtype: true }) — brain-wide strict mode. Every public
write path validates the pairing guarantee.
- { except: [NounType.Thing, ...] } form for catch-all type exemptions
- Atomic-fail semantics on addMany / relateMany — pre-validate every item
before any storage write, throw on first failure with item index
- Per-type rules + brain-wide flag both throw with descriptive messages
- VFS infrastructure bypass via metadata.isVFSEntity / isVFS markers so
brain's own VFS writes don't get rejected when strict mode is on
VFS labeling — concrete subtypes for infrastructure entities
- VFS root: NounType.Collection + subtype: 'vfs-root' (was bare Collection)
- VFS directories: subtype: 'vfs-directory'
- VFS files: subtype: 'vfs-file' (NounType still mime-based)
- VFS containment edges: VerbType.Contains + subtype: 'vfs-contains'
- Lets consumers cleanly enumerate VFS state via find({ subtype: 'vfs-file' })
and distinguish Brainy's VFS Collections from user-created Collections
Docs
- docs/guides/subtypes-and-facets.md extended with Layer V (Verbs) section +
Enforcement section. New full reference at the bottom split into Layer 1
(nouns), Layer V (verbs), Layer 2 (facets), Layer 3 (migration), Enforcement.
- docs/api/README.md adds updateRelation(), getRelations({ subtype }), the
three verb-side counts methods, requireSubtype(), and the brain-wide
constructor option. relate() params include subtype.
- docs/DATA_MODEL.md adds a Subtype-for-VerbType section + STANDARD_VERB_FIELDS
- docs/architecture/finite-type-system.md extends Principle 1a to verbs
- docs/QUERY_OPERATORS.md adds a verb-subtype filter section covering
getRelations and find({connected, subtype}) traversal
- README.md "Subtypes" section now shows both noun + verb in one example +
the enforcement APIs
- RELEASES.md v7.30.0 entry with the full noun/verb capability parity matrix
Tests
- tests/integration/verb-subtype-and-enforcement.test.ts — 30 new tests
covering V1 round-trips, V1 set membership, updateRelation in place,
updateRelation preservation, V2 counts breakdown + point + topN + distinct,
V2 decrements on unrelate, V2 re-routes on updateRelation, V3 depth-1
traversal filter, V3 depth>1 explicit error, V4 verb migration, V4 both
entity kinds, V4 readBoth preservation, V5 per-type required rejection,
V5 vocabulary rejection, V5 on-vocab acceptance, V5 verb-side enforcement,
V5 addMany atomic-fail, V5 relateMany atomic-fail, V5 update enforcement,
V5 updateRelation enforcement, V5 brain-wide strict mode, V5 except clause.
Verification
- Unit suite: 1468/1468 passing
- Noun subtype integration (7.29 carryover): 26/26 passing
- Verb subtype + enforcement integration: 30/30 passing
- Type-check: clean
- Build: clean
- Public closed-source reference audit: clean
Internal 8.0 spec
- .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md (gitignored, not in npm artifact)
documents the contract upgrade Cortex 3.0 implements against: required-by-
default subtype, SubtypeRegistry typing hook, native simplification,
multi-hop traversal native fast path, brain.fillSubtypes() migration helper.
Coordinated via PLATFORM-HANDOFF rows CTX-SUBTYPE-PARITY-V2 (7.30 parallel
work) and CTX-SUBTYPE-8.0-CONTRACT (8.0 spec).
2026-06-05 11:15:52 -07:00
from: entityId,
type: VerbType.ReportsTo,
subtype: 'direct'
})
// Set membership on subtype
2026-06-11 14:51:00 -07:00
const all = await brain.related({
feat: verb subtype + updateRelation + requireSubtype enforcement
Brings verbs to first-class parity with nouns. The 7.29.0 subtype primitive
shipped for entities only; this release ships the symmetric verb mirror plus
the enforcement layer for ensuring every entity AND every relationship has
both type AND subtype.
Layer V1 — verb subtype mirror
- HNSWVerbWithMetadata.subtype + STANDARD_VERB_FIELDS set + resolveVerbField()
- Relation<T>.subtype, RelateParams<T>.subtype, UpdateRelationParams<T> extended,
GetRelationsParams.subtype, GraphConstraints.subtype (for find connected)
- relate() persists subtype on verbMetadata + GraphVerb + transaction ops
- getRelations({ type, subtype }) fast-path filter with set membership
- find({ connected: { via, subtype, depth } }) traversal filter (depth-1 on
the JS path; explicit error on depth > 1 pointing at Cortex native)
- verbsToRelations + storage destructure sites surface subtype to top-level
- All three graph-index fast-path queries (getVerbsBySource/ByTarget) enrich
with subtype from metadata
Layer V2 — updateRelation() closes a pre-7.30 gap
- New first-class verb update method (parallel to update() for nouns)
- Changes subtype/type/weight/confidence/data/metadata in place
- Re-indexes in graph adjacency when verb type changes; id preserved
- validateUpdateRelationParams enforces id + at-least-one-field-to-update
Layer V3 — verb subtype storage rollup
- verbSubtypeCountsByType: Map<number, Map<string, number>> on BaseStorage
- verbSubtypeByIdCache for self-heal during update/delete
- incrementVerbSubtypeCount + decrementVerbSubtypeCount maintain state
- loadVerbSubtypeStatistics + saveVerbSubtypeStatistics persist to
_system/verb-subtype-statistics.json (mirrors noun-side shape)
- rebuildVerbSubtypeCounts for poison recovery / explicit repair
- getVerbSubtypeCountsByType accessor for the public counts API
- Wired into init() / flushCounts() / saveVerbMetadata / deleteVerbMetadata
Layer V4 — verb counts API + relationshipSubtypesOf
- brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) breakdown or point
- brain.counts.topRelationshipSubtypes(verb, n) — top N by count
- brain.relationshipSubtypesOf(verb) — sorted distinct subtypes
Layer V5 — migrateField extended to verbs
- New entityKind?: 'noun' | 'verb' | 'both' option (default 'noun')
- Mirror verb iteration via storage.getVerbs() with same path semantics
- verbToRelationLike + buildRelationMigrationUpdate helpers project the
storage verb shape onto the Entity<T>-shaped surface readPath understands
- Routes through new updateRelation() for the verb-side rewrite
Enforcement (opt-in in 7.30, default in 8.0)
- brain.requireSubtype(type, options) — unified API for NounType OR VerbType.
Registers per-type rules with optional values whitelist; composes with the
brain-wide flag.
- new Brainy({ requireSubtype: true }) — brain-wide strict mode. Every public
write path validates the pairing guarantee.
- { except: [NounType.Thing, ...] } form for catch-all type exemptions
- Atomic-fail semantics on addMany / relateMany — pre-validate every item
before any storage write, throw on first failure with item index
- Per-type rules + brain-wide flag both throw with descriptive messages
- VFS infrastructure bypass via metadata.isVFSEntity / isVFS markers so
brain's own VFS writes don't get rejected when strict mode is on
VFS labeling — concrete subtypes for infrastructure entities
- VFS root: NounType.Collection + subtype: 'vfs-root' (was bare Collection)
- VFS directories: subtype: 'vfs-directory'
- VFS files: subtype: 'vfs-file' (NounType still mime-based)
- VFS containment edges: VerbType.Contains + subtype: 'vfs-contains'
- Lets consumers cleanly enumerate VFS state via find({ subtype: 'vfs-file' })
and distinguish Brainy's VFS Collections from user-created Collections
Docs
- docs/guides/subtypes-and-facets.md extended with Layer V (Verbs) section +
Enforcement section. New full reference at the bottom split into Layer 1
(nouns), Layer V (verbs), Layer 2 (facets), Layer 3 (migration), Enforcement.
- docs/api/README.md adds updateRelation(), getRelations({ subtype }), the
three verb-side counts methods, requireSubtype(), and the brain-wide
constructor option. relate() params include subtype.
- docs/DATA_MODEL.md adds a Subtype-for-VerbType section + STANDARD_VERB_FIELDS
- docs/architecture/finite-type-system.md extends Principle 1a to verbs
- docs/QUERY_OPERATORS.md adds a verb-subtype filter section covering
getRelations and find({connected, subtype}) traversal
- README.md "Subtypes" section now shows both noun + verb in one example +
the enforcement APIs
- RELEASES.md v7.30.0 entry with the full noun/verb capability parity matrix
Tests
- tests/integration/verb-subtype-and-enforcement.test.ts — 30 new tests
covering V1 round-trips, V1 set membership, updateRelation in place,
updateRelation preservation, V2 counts breakdown + point + topN + distinct,
V2 decrements on unrelate, V2 re-routes on updateRelation, V3 depth-1
traversal filter, V3 depth>1 explicit error, V4 verb migration, V4 both
entity kinds, V4 readBoth preservation, V5 per-type required rejection,
V5 vocabulary rejection, V5 on-vocab acceptance, V5 verb-side enforcement,
V5 addMany atomic-fail, V5 relateMany atomic-fail, V5 update enforcement,
V5 updateRelation enforcement, V5 brain-wide strict mode, V5 except clause.
Verification
- Unit suite: 1468/1468 passing
- Noun subtype integration (7.29 carryover): 26/26 passing
- Verb subtype + enforcement integration: 30/30 passing
- Type-check: clean
- Build: clean
- Public closed-source reference audit: clean
Internal 8.0 spec
- .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md (gitignored, not in npm artifact)
documents the contract upgrade Cortex 3.0 implements against: required-by-
default subtype, SubtypeRegistry typing hook, native simplification,
multi-hop traversal native fast path, brain.fillSubtypes() migration helper.
Coordinated via PLATFORM-HANDOFF rows CTX-SUBTYPE-PARITY-V2 (7.30 parallel
work) and CTX-SUBTYPE-8.0-CONTRACT (8.0 spec).
2026-06-05 11:15:52 -07:00
from: entityId,
type: VerbType.ReportsTo,
subtype: ['direct', 'dotted-line']
})
2025-11-02 10:58:52 -08:00
```
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
**Parameters:**
- `from?` : `string` - Source entity ID
- `to?` : `string` - Target entity ID
feat: verb subtype + updateRelation + requireSubtype enforcement
Brings verbs to first-class parity with nouns. The 7.29.0 subtype primitive
shipped for entities only; this release ships the symmetric verb mirror plus
the enforcement layer for ensuring every entity AND every relationship has
both type AND subtype.
Layer V1 — verb subtype mirror
- HNSWVerbWithMetadata.subtype + STANDARD_VERB_FIELDS set + resolveVerbField()
- Relation<T>.subtype, RelateParams<T>.subtype, UpdateRelationParams<T> extended,
GetRelationsParams.subtype, GraphConstraints.subtype (for find connected)
- relate() persists subtype on verbMetadata + GraphVerb + transaction ops
- getRelations({ type, subtype }) fast-path filter with set membership
- find({ connected: { via, subtype, depth } }) traversal filter (depth-1 on
the JS path; explicit error on depth > 1 pointing at Cortex native)
- verbsToRelations + storage destructure sites surface subtype to top-level
- All three graph-index fast-path queries (getVerbsBySource/ByTarget) enrich
with subtype from metadata
Layer V2 — updateRelation() closes a pre-7.30 gap
- New first-class verb update method (parallel to update() for nouns)
- Changes subtype/type/weight/confidence/data/metadata in place
- Re-indexes in graph adjacency when verb type changes; id preserved
- validateUpdateRelationParams enforces id + at-least-one-field-to-update
Layer V3 — verb subtype storage rollup
- verbSubtypeCountsByType: Map<number, Map<string, number>> on BaseStorage
- verbSubtypeByIdCache for self-heal during update/delete
- incrementVerbSubtypeCount + decrementVerbSubtypeCount maintain state
- loadVerbSubtypeStatistics + saveVerbSubtypeStatistics persist to
_system/verb-subtype-statistics.json (mirrors noun-side shape)
- rebuildVerbSubtypeCounts for poison recovery / explicit repair
- getVerbSubtypeCountsByType accessor for the public counts API
- Wired into init() / flushCounts() / saveVerbMetadata / deleteVerbMetadata
Layer V4 — verb counts API + relationshipSubtypesOf
- brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) breakdown or point
- brain.counts.topRelationshipSubtypes(verb, n) — top N by count
- brain.relationshipSubtypesOf(verb) — sorted distinct subtypes
Layer V5 — migrateField extended to verbs
- New entityKind?: 'noun' | 'verb' | 'both' option (default 'noun')
- Mirror verb iteration via storage.getVerbs() with same path semantics
- verbToRelationLike + buildRelationMigrationUpdate helpers project the
storage verb shape onto the Entity<T>-shaped surface readPath understands
- Routes through new updateRelation() for the verb-side rewrite
Enforcement (opt-in in 7.30, default in 8.0)
- brain.requireSubtype(type, options) — unified API for NounType OR VerbType.
Registers per-type rules with optional values whitelist; composes with the
brain-wide flag.
- new Brainy({ requireSubtype: true }) — brain-wide strict mode. Every public
write path validates the pairing guarantee.
- { except: [NounType.Thing, ...] } form for catch-all type exemptions
- Atomic-fail semantics on addMany / relateMany — pre-validate every item
before any storage write, throw on first failure with item index
- Per-type rules + brain-wide flag both throw with descriptive messages
- VFS infrastructure bypass via metadata.isVFSEntity / isVFS markers so
brain's own VFS writes don't get rejected when strict mode is on
VFS labeling — concrete subtypes for infrastructure entities
- VFS root: NounType.Collection + subtype: 'vfs-root' (was bare Collection)
- VFS directories: subtype: 'vfs-directory'
- VFS files: subtype: 'vfs-file' (NounType still mime-based)
- VFS containment edges: VerbType.Contains + subtype: 'vfs-contains'
- Lets consumers cleanly enumerate VFS state via find({ subtype: 'vfs-file' })
and distinguish Brainy's VFS Collections from user-created Collections
Docs
- docs/guides/subtypes-and-facets.md extended with Layer V (Verbs) section +
Enforcement section. New full reference at the bottom split into Layer 1
(nouns), Layer V (verbs), Layer 2 (facets), Layer 3 (migration), Enforcement.
- docs/api/README.md adds updateRelation(), getRelations({ subtype }), the
three verb-side counts methods, requireSubtype(), and the brain-wide
constructor option. relate() params include subtype.
- docs/DATA_MODEL.md adds a Subtype-for-VerbType section + STANDARD_VERB_FIELDS
- docs/architecture/finite-type-system.md extends Principle 1a to verbs
- docs/QUERY_OPERATORS.md adds a verb-subtype filter section covering
getRelations and find({connected, subtype}) traversal
- README.md "Subtypes" section now shows both noun + verb in one example +
the enforcement APIs
- RELEASES.md v7.30.0 entry with the full noun/verb capability parity matrix
Tests
- tests/integration/verb-subtype-and-enforcement.test.ts — 30 new tests
covering V1 round-trips, V1 set membership, updateRelation in place,
updateRelation preservation, V2 counts breakdown + point + topN + distinct,
V2 decrements on unrelate, V2 re-routes on updateRelation, V3 depth-1
traversal filter, V3 depth>1 explicit error, V4 verb migration, V4 both
entity kinds, V4 readBoth preservation, V5 per-type required rejection,
V5 vocabulary rejection, V5 on-vocab acceptance, V5 verb-side enforcement,
V5 addMany atomic-fail, V5 relateMany atomic-fail, V5 update enforcement,
V5 updateRelation enforcement, V5 brain-wide strict mode, V5 except clause.
Verification
- Unit suite: 1468/1468 passing
- Noun subtype integration (7.29 carryover): 26/26 passing
- Verb subtype + enforcement integration: 30/30 passing
- Type-check: clean
- Build: clean
- Public closed-source reference audit: clean
Internal 8.0 spec
- .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md (gitignored, not in npm artifact)
documents the contract upgrade Cortex 3.0 implements against: required-by-
default subtype, SubtypeRegistry typing hook, native simplification,
multi-hop traversal native fast path, brain.fillSubtypes() migration helper.
Coordinated via PLATFORM-HANDOFF rows CTX-SUBTYPE-PARITY-V2 (7.30 parallel
work) and CTX-SUBTYPE-8.0-CONTRACT (8.0 spec).
2026-06-05 11:15:52 -07:00
- `type?` : `VerbType | VerbType[]` - Filter by relationship type
- `subtype?` : `string | string[]` - Filter by VerbType subtype (top-level standard field, fast path)
- `service?` : `string` - Multi-tenancy filter
- `limit?` : `number` - Pagination limit (default: 100)
- `offset?` : `number` - Pagination offset
2025-08-26 12:32:21 -07:00
feat: verb subtype + updateRelation + requireSubtype enforcement
Brings verbs to first-class parity with nouns. The 7.29.0 subtype primitive
shipped for entities only; this release ships the symmetric verb mirror plus
the enforcement layer for ensuring every entity AND every relationship has
both type AND subtype.
Layer V1 — verb subtype mirror
- HNSWVerbWithMetadata.subtype + STANDARD_VERB_FIELDS set + resolveVerbField()
- Relation<T>.subtype, RelateParams<T>.subtype, UpdateRelationParams<T> extended,
GetRelationsParams.subtype, GraphConstraints.subtype (for find connected)
- relate() persists subtype on verbMetadata + GraphVerb + transaction ops
- getRelations({ type, subtype }) fast-path filter with set membership
- find({ connected: { via, subtype, depth } }) traversal filter (depth-1 on
the JS path; explicit error on depth > 1 pointing at Cortex native)
- verbsToRelations + storage destructure sites surface subtype to top-level
- All three graph-index fast-path queries (getVerbsBySource/ByTarget) enrich
with subtype from metadata
Layer V2 — updateRelation() closes a pre-7.30 gap
- New first-class verb update method (parallel to update() for nouns)
- Changes subtype/type/weight/confidence/data/metadata in place
- Re-indexes in graph adjacency when verb type changes; id preserved
- validateUpdateRelationParams enforces id + at-least-one-field-to-update
Layer V3 — verb subtype storage rollup
- verbSubtypeCountsByType: Map<number, Map<string, number>> on BaseStorage
- verbSubtypeByIdCache for self-heal during update/delete
- incrementVerbSubtypeCount + decrementVerbSubtypeCount maintain state
- loadVerbSubtypeStatistics + saveVerbSubtypeStatistics persist to
_system/verb-subtype-statistics.json (mirrors noun-side shape)
- rebuildVerbSubtypeCounts for poison recovery / explicit repair
- getVerbSubtypeCountsByType accessor for the public counts API
- Wired into init() / flushCounts() / saveVerbMetadata / deleteVerbMetadata
Layer V4 — verb counts API + relationshipSubtypesOf
- brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) breakdown or point
- brain.counts.topRelationshipSubtypes(verb, n) — top N by count
- brain.relationshipSubtypesOf(verb) — sorted distinct subtypes
Layer V5 — migrateField extended to verbs
- New entityKind?: 'noun' | 'verb' | 'both' option (default 'noun')
- Mirror verb iteration via storage.getVerbs() with same path semantics
- verbToRelationLike + buildRelationMigrationUpdate helpers project the
storage verb shape onto the Entity<T>-shaped surface readPath understands
- Routes through new updateRelation() for the verb-side rewrite
Enforcement (opt-in in 7.30, default in 8.0)
- brain.requireSubtype(type, options) — unified API for NounType OR VerbType.
Registers per-type rules with optional values whitelist; composes with the
brain-wide flag.
- new Brainy({ requireSubtype: true }) — brain-wide strict mode. Every public
write path validates the pairing guarantee.
- { except: [NounType.Thing, ...] } form for catch-all type exemptions
- Atomic-fail semantics on addMany / relateMany — pre-validate every item
before any storage write, throw on first failure with item index
- Per-type rules + brain-wide flag both throw with descriptive messages
- VFS infrastructure bypass via metadata.isVFSEntity / isVFS markers so
brain's own VFS writes don't get rejected when strict mode is on
VFS labeling — concrete subtypes for infrastructure entities
- VFS root: NounType.Collection + subtype: 'vfs-root' (was bare Collection)
- VFS directories: subtype: 'vfs-directory'
- VFS files: subtype: 'vfs-file' (NounType still mime-based)
- VFS containment edges: VerbType.Contains + subtype: 'vfs-contains'
- Lets consumers cleanly enumerate VFS state via find({ subtype: 'vfs-file' })
and distinguish Brainy's VFS Collections from user-created Collections
Docs
- docs/guides/subtypes-and-facets.md extended with Layer V (Verbs) section +
Enforcement section. New full reference at the bottom split into Layer 1
(nouns), Layer V (verbs), Layer 2 (facets), Layer 3 (migration), Enforcement.
- docs/api/README.md adds updateRelation(), getRelations({ subtype }), the
three verb-side counts methods, requireSubtype(), and the brain-wide
constructor option. relate() params include subtype.
- docs/DATA_MODEL.md adds a Subtype-for-VerbType section + STANDARD_VERB_FIELDS
- docs/architecture/finite-type-system.md extends Principle 1a to verbs
- docs/QUERY_OPERATORS.md adds a verb-subtype filter section covering
getRelations and find({connected, subtype}) traversal
- README.md "Subtypes" section now shows both noun + verb in one example +
the enforcement APIs
- RELEASES.md v7.30.0 entry with the full noun/verb capability parity matrix
Tests
- tests/integration/verb-subtype-and-enforcement.test.ts — 30 new tests
covering V1 round-trips, V1 set membership, updateRelation in place,
updateRelation preservation, V2 counts breakdown + point + topN + distinct,
V2 decrements on unrelate, V2 re-routes on updateRelation, V3 depth-1
traversal filter, V3 depth>1 explicit error, V4 verb migration, V4 both
entity kinds, V4 readBoth preservation, V5 per-type required rejection,
V5 vocabulary rejection, V5 on-vocab acceptance, V5 verb-side enforcement,
V5 addMany atomic-fail, V5 relateMany atomic-fail, V5 update enforcement,
V5 updateRelation enforcement, V5 brain-wide strict mode, V5 except clause.
Verification
- Unit suite: 1468/1468 passing
- Noun subtype integration (7.29 carryover): 26/26 passing
- Verb subtype + enforcement integration: 30/30 passing
- Type-check: clean
- Build: clean
- Public closed-source reference audit: clean
Internal 8.0 spec
- .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md (gitignored, not in npm artifact)
documents the contract upgrade Cortex 3.0 implements against: required-by-
default subtype, SubtypeRegistry typing hook, native simplification,
multi-hop traversal native fast path, brain.fillSubtypes() migration helper.
Coordinated via PLATFORM-HANDOFF rows CTX-SUBTYPE-PARITY-V2 (7.30 parallel
work) and CTX-SUBTYPE-8.0-CONTRACT (8.0 spec).
2026-06-05 11:15:52 -07:00
**Returns:** `Promise<Relation[]>` - Matching relationships (each with `subtype` at top level when set)
2025-11-02 10:58:52 -08:00
---
## Batch Operations
### `addMany(params)` → `Promise<BatchResult<string>>`
Add multiple entities in one operation.
```typescript
const result = await brain.addMany({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
items: [
{ data: 'Entity 1', type: NounType.Document },
{ data: 'Entity 2', type: NounType.Concept }
]
2025-11-02 10:58:52 -08:00
})
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
console.log(result.successful) // Array of IDs
console.log(result.failed) // Array of errors
2025-11-02 10:58:52 -08:00
```
**Returns:** `Promise<BatchResult<string>>` - Success/failure results
---
2026-06-11 14:51:00 -07:00
### `removeMany(params)` → `Promise<BatchResult<string>>`
2025-11-02 10:58:52 -08:00
2026-06-11 14:51:00 -07:00
Remove multiple entities.
2025-11-02 10:58:52 -08:00
```typescript
2026-06-11 14:51:00 -07:00
const result = await brain.removeMany({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
ids: [id1, id2, id3]
2025-11-02 10:58:52 -08:00
})
```
---
### `updateMany(params)` → `Promise<BatchResult<string>>`
Update multiple entities.
```typescript
const result = await brain.updateMany({
2026-06-11 14:51:00 -07:00
items: [
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
{ id: id1, metadata: { updated: true } },
{ id: id2, data: 'New content' }
]
2025-11-02 10:58:52 -08:00
})
```
---
### `relateMany(params)` → `Promise<string[]>`
Create multiple relationships.
```typescript
const ids = await brain.relateMany({
2026-06-11 14:51:00 -07:00
items: [
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
{ from: id1, to: id2, type: VerbType.RelatedTo },
{ from: id1, to: id3, type: VerbType.Contains }
]
2025-11-02 10:58:52 -08:00
})
```
---
2026-06-11 08:37:26 -07:00
## Database Values & Time Travel (Db API)
2025-11-02 10:58:52 -08:00
2026-06-11 08:37:26 -07:00
Brainy 8.0's generational MVCC exposes the whole store as an immutable
value: the ** `Db` **. Pin the current state in O(1), commit atomic
multi-write batches, query any past generation with the full query surface,
cut instant snapshots, and ask what-if questions in memory. The exact
guarantees live in the ** [consistency model ](../concepts/consistency-model.md )**;
recipes live in ** [Snapshots & Time Travel ](../guides/snapshots-and-time-travel.md )**.
2025-11-02 10:58:52 -08:00
2026-06-11 08:37:26 -07:00
### `generation()` → `number`
2025-11-02 10:58:52 -08:00
2026-06-11 08:37:26 -07:00
The store's current generation — a monotonic watermark advanced once per
committed `transact()` batch and once per single-operation write. Never
reissued, including across restarts and `restore()` .
docs: complete Stage 3 CANONICAL taxonomy documentation (v5.5.0)
Added comprehensive documentation for Stage 3 taxonomy migration:
API Documentation Updates:
- Updated version header from v5.4.0 to v5.5.0
- Fixed all deprecated NounType.Content examples → Document, Concept
- Added versions.asOf() method documentation with examples
- Added comprehensive Type System Reference section
Type System Reference:
- Documented all 42 noun types organized by category
- Documented 127 verb types organized by functional groups
- Added migration guide from v5.4.0 (deprecated types)
- Breaking changes: User→Person, Topic→Concept, Content→Document
- Stage 3 adds +11 nouns, +87 verbs (169 total types)
Build Status:
- ✅ TypeScript build passes with zero errors (stale cache issue resolved)
- ✅ All Stage 2 references updated to Stage 3 CANONICAL
- ✅ Type embeddings current (42 nouns + 127 verbs = 338KB)
Tested with confidential Tales from Talifar Glossary (NOT in repo):
- Successfully imported 2,284 entities across 7 noun types
- Created 2,280 relationships (contains verb)
- Noun type distribution: document (50%), thing (16%), person (16%),
location (9%), concept (9%), collection (1%), file (<1%)
- Demonstrates Stage 3 taxonomy handles rich fantasy world data
Type System Coverage:
- 7/42 noun types used (16.7%) - appropriate for glossary domain
- 1/127 verb types used (0.8%) - opportunity for richer relationships
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-06 10:28:47 -08:00
```typescript
2026-06-11 08:37:26 -07:00
const g = brain.generation()
docs: complete Stage 3 CANONICAL taxonomy documentation (v5.5.0)
Added comprehensive documentation for Stage 3 taxonomy migration:
API Documentation Updates:
- Updated version header from v5.4.0 to v5.5.0
- Fixed all deprecated NounType.Content examples → Document, Concept
- Added versions.asOf() method documentation with examples
- Added comprehensive Type System Reference section
Type System Reference:
- Documented all 42 noun types organized by category
- Documented 127 verb types organized by functional groups
- Added migration guide from v5.4.0 (deprecated types)
- Breaking changes: User→Person, Topic→Concept, Content→Document
- Stage 3 adds +11 nouns, +87 verbs (169 total types)
Build Status:
- ✅ TypeScript build passes with zero errors (stale cache issue resolved)
- ✅ All Stage 2 references updated to Stage 3 CANONICAL
- ✅ Type embeddings current (42 nouns + 127 verbs = 338KB)
Tested with confidential Tales from Talifar Glossary (NOT in repo):
- Successfully imported 2,284 entities across 7 noun types
- Created 2,280 relationships (contains verb)
- Noun type distribution: document (50%), thing (16%), person (16%),
location (9%), concept (9%), collection (1%), file (<1%)
- Demonstrates Stage 3 taxonomy handles rich fantasy world data
Type System Coverage:
- 7/42 noun types used (16.7%) - appropriate for glossary domain
- 1/127 verb types used (0.8%) - opportunity for richer relationships
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-06 10:28:47 -08:00
```
---
2026-06-11 08:37:26 -07:00
### `now()` → `Db`
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
Pin the current generation and return an immutable view — O(1), no I/O.
The view keeps reading exactly this state no matter what commits afterwards.
2025-11-04 11:19:02 -08:00
```typescript
2026-06-11 08:37:26 -07:00
const db = brain.now()
await brain.update({ id, metadata: { v: 2 } })
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
await db.get(id) // still sees v: 1 — pinned
await brain.get(id) // sees v: 2 — live
await db.release() // unpin (enables history compaction)
2025-11-04 11:19:02 -08:00
```
2026-06-11 08:37:26 -07:00
**Returns:** `Db` — release it when done; pins gate `compactHistory()` .
2025-11-04 11:19:02 -08:00
---
2026-06-11 08:37:26 -07:00
### `transact(ops, options?)` → `Promise<Db>`
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
Execute a declarative operation batch **atomically** : either every
operation applies and the store advances exactly one generation, or none
apply and the store is byte-identical to its pre-transaction state. The
commit point is an atomic manifest rename; a crash anywhere before it rolls
back to the exact pre-transaction bytes on the next open.
2025-11-04 11:19:02 -08:00
```typescript
2026-06-11 08:37:26 -07:00
const db = await brain.transact([
{ op: 'add', id: orderId, type: NounType.Document, subtype: 'order', data: 'Order #1042 ' },
{ op: 'update', id: customerId, metadata: { lastOrderAt: Date.now() }, ifRev: customer._rev },
{ op: 'relate', from: customerId, to: orderId, type: VerbType.Creates, subtype: 'purchase' },
{ op: 'remove', id: staleDraftId },
{ op: 'unrelate', id: oldRelationId }
], {
meta: { author: 'order-service', requestId: 'req-9f2' }, // reified, durable
ifAtGeneration: expectedGeneration // whole-store CAS
2025-11-04 11:19:02 -08:00
})
2026-06-11 08:37:26 -07:00
db.receipt.ids // resolved id per operation, in input order
db.receipt.generation // the committed generation
2025-11-04 11:19:02 -08:00
```
2026-06-11 08:37:26 -07:00
**Operations** (`op` discriminates; parameters mirror the single-operation methods):
- `{ op: 'add', ... }` — same parameters as `add()` ; optional explicit `id`
- `{ op: 'update', ... }` — same parameters as `update()` , including per-entity `ifRev` CAS
- `{ op: 'remove', id }` — deletes the entity plus its relationships (same cascade as `delete()` )
- `{ op: 'relate', ... }` — same parameters as `relate()` , including `bidirectional` ; duplicates dedupe to the existing relationship id
- `{ op: 'unrelate', id }` — deletes a relationship by id
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
Operations may reference ids created earlier in the same batch.
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
**Options:**
- `meta?` : `Record<string, unknown>` — transaction metadata, recorded durably in the transaction log (audit fields: author, reason, request id)
- `ifAtGeneration?` : `number` — whole-store compare-and-swap; commits only if the store is still at this generation
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
**Returns:** `Promise<Db>` — pinned at the freshly committed generation, carrying a `receipt` .
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
**Throws:**
- `GenerationConflictError` — `ifAtGeneration` did not match (nothing staged, generation unchanged)
- `RevisionConflictError` — an `ifRev` did not match (whole batch rejected)
2025-11-04 11:19:02 -08:00
---
2026-06-11 08:37:26 -07:00
### `asOf(target)` → `Promise<Db>`
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
Open an immutable view of **past** state:
2025-11-04 11:19:02 -08:00
```typescript
2026-06-11 08:37:26 -07:00
const atGen = await brain.asOf(1041) // generation number
const lastWeek = await brain.asOf(new Date(Date.now() - 7 * 86_400_000)) // wall-clock
const fromSnapshot = await brain.asOf('/backups/2026-06-01') // snapshot directory
2025-11-04 11:19:02 -08:00
```
2026-06-11 08:37:26 -07:00
- **`number` ** — pins that generation; reads resolve through the immutable record layer.
- **`Date` ** — resolved via the transaction log to the newest generation committed at or before it.
- **`string` ** — a snapshot directory from `db.persist()` , opened as a self-contained read-only store (equivalent to `Brainy.load()` ).
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
Historical views serve the **full query surface** . Metadata-level reads are
free; the first index-accelerated query (semantic search, traversal,
cursors, aggregation) builds an in-memory index materialization — O(n at
that generation), once per `Db` , freed on `release()` .
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
**Throws:** `GenerationCompactedError` when the generation's records were reclaimed by `compactHistory()` .
2025-11-04 11:19:02 -08:00
feat(8.0): Model-B per-write generation-stamping + adaptive retention knob
Every write — transact() AND single-op add/update/remove/relate — is now its
own immutable generation (Model-B), so a now() pin always freezes and
asOf/since/diff/history/transactionLog reflect single-ops exactly like
transacts. Closes the Model-A hole where pins did not freeze against single-op
writes.
Generation-stamping:
- GenerationStore.commitSingleOp: a one-operation commitTransaction with
deferred durability. Wired into add/update/remove/relate/updateRelation/
unrelate + removeMany (the *Many and VFS paths delegate to these).
- Async group-commit (flushPendingSingleOps): the live write is acknowledged
immediately; its before-image is buffered in an in-memory pending tier that
resolveAt/chains/changedBetween/tx-log read like on-disk generations, so the
synchronous now() freezes with no forced flush. One fsync per window
(triggers: size / 50ms timer / flush / close / transact / compactHistory).
- Crash recovery is drop-without-restore for group-commit generations (marked
groupCommit:true): a crash mid-flush discards the partial generation and
never restores its before-images, which would otherwise revert the
already-acknowledged live write.
- Init-time infrastructure (the VFS root) is the un-versioned generation-0
baseline: a fresh brain reports generation()===0 and an empty
transactionLog(); the first user write is generation 1.
- Historical find()/related() overlay bound is the full reserved watermark
(generation()), so un-flushed single-op writes are overlaid too.
Retention knob:
- config `history` -> `retention`: 'all' | 'adaptive' |
{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }; unset ->
adaptive (disk/RAM pressure, zero-config). CompactHistoryOptions floors ->
caps (retainGenerations->maxGenerations, retainMs->maxAge, +maxBytes):
reclaim oldest-unpinned while ANY cap is exceeded; pins always exempt.
- brain.setRetentionBudget(bytes) drives the adaptive byte budget at runtime
(a coordinator's fair-share input). Per-generation bytes recorded in each
delta enable historyBytes() introspection without a storage size API.
Tests: per-write generation resolution, pin freeze vs add/update/remove,
drop-without-restore corruption-trap (fault injector), clean-reopen replay,
maxBytes/maxAge/no-cap reclamation, retention-then-reopen. 107 db/generation/
temporal tests green, tsc clean. Docs (ADR-001, consistency-model, snapshots
guide, api reference, RELEASES) updated to per-write granularity + retention.
2026-06-22 15:19:58 -07:00
**History granularity:** every write is its own immutable generation —
`transact()` batches AND single-operation writes — so a pin always freezes and
every write is addressable via `asOf()` . See the
[consistency model ](../concepts/consistency-model.md ).
2025-11-04 11:19:02 -08:00
---
2026-06-11 08:37:26 -07:00
### `transactionLog(options?)` → `Promise<TxLogEntry[]>`
2025-11-04 11:19:02 -08:00
feat(8.0): Model-B per-write generation-stamping + adaptive retention knob
Every write — transact() AND single-op add/update/remove/relate — is now its
own immutable generation (Model-B), so a now() pin always freezes and
asOf/since/diff/history/transactionLog reflect single-ops exactly like
transacts. Closes the Model-A hole where pins did not freeze against single-op
writes.
Generation-stamping:
- GenerationStore.commitSingleOp: a one-operation commitTransaction with
deferred durability. Wired into add/update/remove/relate/updateRelation/
unrelate + removeMany (the *Many and VFS paths delegate to these).
- Async group-commit (flushPendingSingleOps): the live write is acknowledged
immediately; its before-image is buffered in an in-memory pending tier that
resolveAt/chains/changedBetween/tx-log read like on-disk generations, so the
synchronous now() freezes with no forced flush. One fsync per window
(triggers: size / 50ms timer / flush / close / transact / compactHistory).
- Crash recovery is drop-without-restore for group-commit generations (marked
groupCommit:true): a crash mid-flush discards the partial generation and
never restores its before-images, which would otherwise revert the
already-acknowledged live write.
- Init-time infrastructure (the VFS root) is the un-versioned generation-0
baseline: a fresh brain reports generation()===0 and an empty
transactionLog(); the first user write is generation 1.
- Historical find()/related() overlay bound is the full reserved watermark
(generation()), so un-flushed single-op writes are overlaid too.
Retention knob:
- config `history` -> `retention`: 'all' | 'adaptive' |
{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }; unset ->
adaptive (disk/RAM pressure, zero-config). CompactHistoryOptions floors ->
caps (retainGenerations->maxGenerations, retainMs->maxAge, +maxBytes):
reclaim oldest-unpinned while ANY cap is exceeded; pins always exempt.
- brain.setRetentionBudget(bytes) drives the adaptive byte budget at runtime
(a coordinator's fair-share input). Per-generation bytes recorded in each
delta enable historyBytes() introspection without a storage size API.
Tests: per-write generation resolution, pin freeze vs add/update/remove,
drop-without-restore corruption-trap (fault injector), clean-reopen replay,
maxBytes/maxAge/no-cap reclamation, retention-then-reopen. 107 db/generation/
temporal tests green, tsc clean. Docs (ADR-001, consistency-model, snapshots
guide, api reference, RELEASES) updated to per-write granularity + retention.
2026-06-22 15:19:58 -07:00
Read the reified transaction log — one entry per committed generation (every
`transact()` AND single-op write), newest first: `{ generation, timestamp,
meta? }`. Single-op generations carry no ` meta` (it is a ` transact()`-only field).
2025-11-04 11:19:02 -08:00
```typescript
2026-06-11 08:37:26 -07:00
const [latest] = await brain.transactionLog({ limit: 1 })
latest.meta // { author: 'order-service', requestId: 'req-9f2' }
2025-11-04 11:19:02 -08:00
```
---
2026-06-11 08:37:26 -07:00
### `compactHistory(options?)` → `Promise<CompactHistoryResult>`
2025-11-04 11:19:02 -08:00
feat(8.0): Model-B per-write generation-stamping + adaptive retention knob
Every write — transact() AND single-op add/update/remove/relate — is now its
own immutable generation (Model-B), so a now() pin always freezes and
asOf/since/diff/history/transactionLog reflect single-ops exactly like
transacts. Closes the Model-A hole where pins did not freeze against single-op
writes.
Generation-stamping:
- GenerationStore.commitSingleOp: a one-operation commitTransaction with
deferred durability. Wired into add/update/remove/relate/updateRelation/
unrelate + removeMany (the *Many and VFS paths delegate to these).
- Async group-commit (flushPendingSingleOps): the live write is acknowledged
immediately; its before-image is buffered in an in-memory pending tier that
resolveAt/chains/changedBetween/tx-log read like on-disk generations, so the
synchronous now() freezes with no forced flush. One fsync per window
(triggers: size / 50ms timer / flush / close / transact / compactHistory).
- Crash recovery is drop-without-restore for group-commit generations (marked
groupCommit:true): a crash mid-flush discards the partial generation and
never restores its before-images, which would otherwise revert the
already-acknowledged live write.
- Init-time infrastructure (the VFS root) is the un-versioned generation-0
baseline: a fresh brain reports generation()===0 and an empty
transactionLog(); the first user write is generation 1.
- Historical find()/related() overlay bound is the full reserved watermark
(generation()), so un-flushed single-op writes are overlaid too.
Retention knob:
- config `history` -> `retention`: 'all' | 'adaptive' |
{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }; unset ->
adaptive (disk/RAM pressure, zero-config). CompactHistoryOptions floors ->
caps (retainGenerations->maxGenerations, retainMs->maxAge, +maxBytes):
reclaim oldest-unpinned while ANY cap is exceeded; pins always exempt.
- brain.setRetentionBudget(bytes) drives the adaptive byte budget at runtime
(a coordinator's fair-share input). Per-generation bytes recorded in each
delta enable historyBytes() introspection without a storage size API.
Tests: per-write generation resolution, pin freeze vs add/update/remove,
drop-without-restore corruption-trap (fault injector), clean-reopen replay,
maxBytes/maxAge/no-cap reclamation, retention-then-reopen. 107 db/generation/
temporal tests green, tsc clean. Docs (ADR-001, consistency-model, snapshots
guide, api reference, RELEASES) updated to per-write granularity + retention.
2026-06-22 15:19:58 -07:00
Reclaim historical record-sets that no retention cap and no live `Db` pin
protects. Pinned reads stay correct across compaction, always. (Auto-compaction
on `flush()` /`close()` is governed by the constructor `retention` knob — unset →
adaptive, `'all'` → unbounded, `{ … }` → explicit caps.)
2025-11-04 11:19:02 -08:00
```typescript
2026-06-11 08:37:26 -07:00
await brain.compactHistory({
feat(8.0): Model-B per-write generation-stamping + adaptive retention knob
Every write — transact() AND single-op add/update/remove/relate — is now its
own immutable generation (Model-B), so a now() pin always freezes and
asOf/since/diff/history/transactionLog reflect single-ops exactly like
transacts. Closes the Model-A hole where pins did not freeze against single-op
writes.
Generation-stamping:
- GenerationStore.commitSingleOp: a one-operation commitTransaction with
deferred durability. Wired into add/update/remove/relate/updateRelation/
unrelate + removeMany (the *Many and VFS paths delegate to these).
- Async group-commit (flushPendingSingleOps): the live write is acknowledged
immediately; its before-image is buffered in an in-memory pending tier that
resolveAt/chains/changedBetween/tx-log read like on-disk generations, so the
synchronous now() freezes with no forced flush. One fsync per window
(triggers: size / 50ms timer / flush / close / transact / compactHistory).
- Crash recovery is drop-without-restore for group-commit generations (marked
groupCommit:true): a crash mid-flush discards the partial generation and
never restores its before-images, which would otherwise revert the
already-acknowledged live write.
- Init-time infrastructure (the VFS root) is the un-versioned generation-0
baseline: a fresh brain reports generation()===0 and an empty
transactionLog(); the first user write is generation 1.
- Historical find()/related() overlay bound is the full reserved watermark
(generation()), so un-flushed single-op writes are overlaid too.
Retention knob:
- config `history` -> `retention`: 'all' | 'adaptive' |
{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }; unset ->
adaptive (disk/RAM pressure, zero-config). CompactHistoryOptions floors ->
caps (retainGenerations->maxGenerations, retainMs->maxAge, +maxBytes):
reclaim oldest-unpinned while ANY cap is exceeded; pins always exempt.
- brain.setRetentionBudget(bytes) drives the adaptive byte budget at runtime
(a coordinator's fair-share input). Per-generation bytes recorded in each
delta enable historyBytes() introspection without a storage size API.
Tests: per-write generation resolution, pin freeze vs add/update/remove,
drop-without-restore corruption-trap (fault injector), clean-reopen replay,
maxBytes/maxAge/no-cap reclamation, retention-then-reopen. 107 db/generation/
temporal tests green, tsc clean. Docs (ADR-001, consistency-model, snapshots
guide, api reference, RELEASES) updated to per-write granularity + retention.
2026-06-22 15:19:58 -07:00
maxGenerations: 100, // keep at most the 100 most recent generations
maxAge: 7 * 24 * 60 * 60 * 1000 // and only those from the last 7 days
2025-11-04 11:19:02 -08:00
})
```
2026-06-11 08:37:26 -07:00
**Returns:** `{ removedGenerations, horizon }` — `asOf()` below the horizon throws `GenerationCompactedError` .
2025-11-04 11:19:02 -08:00
---
2026-06-11 08:37:26 -07:00
### `restore(path, { confirm: true })` → `Promise<void>`
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
Replace the store's **entire** state from a snapshot directory. Destructive
— requires `{ confirm: true }` . All indexes are rebuilt; the generation
counter is floored so observed generation numbers are never reissued; live
pins do not survive.
2025-11-04 11:19:02 -08:00
```typescript
2026-06-11 08:37:26 -07:00
await brain.restore('/backups/2026-06-01', { confirm: true })
2025-11-04 11:19:02 -08:00
```
---
2026-06-11 08:37:26 -07:00
### `Brainy.load(path)` → `Promise<Db>` (static)
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
Open a persisted snapshot as a self-contained **read-only** store with the
full query surface, including vector search. Releasing the returned `Db`
closes the underlying instance.
2025-11-04 11:19:02 -08:00
```typescript
2026-06-11 08:37:26 -07:00
const db = await Brainy.load('/backups/2026-06-01')
feat(8.0): API simplification — remove neural()/Db.search, one storage `path` key, integration→0
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
2026-06-20 13:31:11 -07:00
const hits = await db.find({ query: 'quarterly invoices' })
2026-06-11 08:37:26 -07:00
await db.release()
2025-11-04 11:19:02 -08:00
```
---
2026-06-11 08:37:26 -07:00
### The `Db` value
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
Every `Db` is pinned at one generation and serves the full query surface at
exactly that state.
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
**Properties:**
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
| Property | Type | Meaning |
|---|---|---|
| `generation` | `number` | The pinned generation |
| `timestamp` | `number` | Pin time (`now()` ), commit time (`transact()` ), or resolved commit time (`asOf()` ) |
| `receipt` | `TransactReceipt?` | Present only on `transact()` results |
| `speculative` | `boolean` | Whether this view carries a `with()` overlay |
| `released` | `boolean` | Whether `release()` has been called |
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
**Methods:**
2025-11-04 11:19:02 -08:00
```typescript
2026-06-11 08:37:26 -07:00
await db.get(id) // entity as of this generation
await db.find({ where: { status: 'open' } }) // full find() surface
feat(8.0): API simplification — remove neural()/Db.search, one storage `path` key, integration→0
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
2026-06-20 13:31:11 -07:00
await db.find({ query: 'unpaid invoices' }) // semantic search as of this generation
2026-06-11 08:37:26 -07:00
await db.related(entityId) // relationships as of this generation
await db.since(olderDb) // ids changed between two views
const whatIf = await db.with(ops) // speculative in-memory overlay
await db.persist('/backups/today') // self-contained hard-link snapshot
await db.release() // unpin + free cached materialization
2025-11-04 11:19:02 -08:00
```
2026-06-11 08:37:26 -07:00
- **`with(ops)` ** — applies `transact()` -style operations **in memory** on
top of the view; nothing touches disk, the generation counter, or index
providers. Overlay entities carry no embeddings, so index-accelerated
queries and `persist()` on overlays throw `SpeculativeOverlayError` ;
`get()` , metadata-filter `find()` , and filter-based `related()` work
fully. Commit the same ops with `transact()` for the full surface.
- **`persist(path)` ** — cuts an instant snapshot (hard links on filesystem
storage; byte copies across devices; in-memory stores serialize to the
same layout). Requires the view to still be the store's latest generation
— otherwise `GenerationConflictError` .
- **`release()` ** — idempotent; after release every read throws. A
`FinalizationRegistry` backstop releases leaked pins at GC, but explicit
release is what makes `compactHistory()` deterministic.
2025-12-09 16:13:45 -08:00
2026-06-11 08:37:26 -07:00
### Db API errors
2025-12-09 16:13:45 -08:00
2026-06-11 08:37:26 -07:00
All exported from `@soulcraft/brainy` :
2025-12-09 16:13:45 -08:00
2026-06-11 08:37:26 -07:00
| Error | Thrown by | Meaning |
|---|---|---|
| `GenerationConflictError` | `transact({ ifAtGeneration })` , `db.persist()` | The store moved past the expected generation — re-read and retry |
| `RevisionConflictError` | `update({ ifRev })` , `transact()` update ops | Per-entity revision moved — see [optimistic concurrency ](../guides/optimistic-concurrency.md ) |
| `GenerationCompactedError` | `asOf()` | The requested generation's records were reclaimed — persist what you must keep |
| `SpeculativeOverlayError` | index-accelerated reads / `persist()` on `with()` overlays | Honest boundary: overlay entities carry no embeddings |
2025-11-04 11:19:02 -08:00
---
2025-11-02 10:58:52 -08:00
## Virtual Filesystem (VFS)
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
Access via `brain.vfs` (property, not method). Auto-initialized during `brain.init()` .
2025-11-02 10:58:52 -08:00
2025-11-04 11:19:02 -08:00
### Filtering VFS Entities
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
All VFS entities (files/folders) have `metadata.isVFSEntity: true` set automatically.
2025-11-04 11:19:02 -08:00
Use this to filter VFS entities from semantic search results:
```typescript
// Exclude VFS entities from semantic search
const semanticOnly = await brain.find({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
query: 'artificial intelligence',
where: {
isVFSEntity: { notEquals: true } // Only semantic entities
}
2025-11-04 11:19:02 -08:00
})
// Or filter to ONLY VFS entities
const vfsOnly = await brain.find({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
where: {
isVFSEntity: { equals: true } // Only VFS files/folders
}
2025-11-04 11:19:02 -08:00
})
// Check if an entity is a VFS entity
if (entity.metadata.isVFSEntity === true) {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
console.log('This is a VFS file or folder')
2025-11-04 11:19:02 -08:00
}
```
**Why this matters:** Without filtering, VFS files/folders can appear in concept explorers and semantic search results where they don't belong.
---
2025-11-02 10:58:52 -08:00
### Basic File Operations
#### `vfs.readFile(path, options?)` → `Promise<Buffer>`
Read file content.
2025-08-26 12:32:21 -07:00
```typescript
2025-11-02 10:58:52 -08:00
const content = await brain.vfs.readFile('/docs/README.md')
console.log(content.toString())
```
---
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
#### `vfs.writeFile(path, data, options?)` → `Promise<void>`
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
Write file content.
```typescript
await brain.vfs.writeFile('/docs/README.md', 'New content', {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
encoding: 'utf-8'
2025-08-26 12:32:21 -07:00
})
```
2025-11-02 10:58:52 -08:00
---
#### `vfs.unlink(path)` → `Promise<void>`
Delete a file.
2025-08-26 12:32:21 -07:00
```typescript
2025-11-02 10:58:52 -08:00
await brain.vfs.unlink('/docs/old-file.md')
```
---
### Directory Operations
#### `vfs.mkdir(path, options?)` → `Promise<void>`
Create directory.
```typescript
await brain.vfs.mkdir('/projects/new-app', { recursive: true })
```
---
#### `vfs.readdir(path, options?)` → `Promise<string[] | Dirent[]>`
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
List directory contents.
```typescript
const files = await brain.vfs.readdir('/projects')
// With file types
const entries = await brain.vfs.readdir('/projects', { withFileTypes: true })
entries.forEach(entry => {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
console.log(entry.name, entry.isDirectory() ? 'DIR' : 'FILE')
2025-08-26 12:32:21 -07:00
})
2025-11-02 10:58:52 -08:00
```
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
---
#### `vfs.rmdir(path, options?)` → `Promise<void>`
Remove directory.
```typescript
await brain.vfs.rmdir('/old-project', { recursive: true })
```
---
#### `vfs.stat(path)` → `Promise<Stats>`
Get file/directory stats.
```typescript
const stats = await brain.vfs.stat('/docs/README.md')
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
console.log(stats.size) // File size
console.log(stats.mtime) // Modified time
console.log(stats.isDirectory()) // Is directory?
2025-11-02 10:58:52 -08:00
```
---
### Semantic Operations
#### `vfs.search(query, options?)` → `Promise<SearchResult[]>`
Semantic file search.
```typescript
const results = await brain.vfs.search('React components with hooks', {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
path: '/src',
limit: 10
2025-11-02 10:58:52 -08:00
})
```
---
#### `vfs.findSimilar(path, options?)` → `Promise<SearchResult[]>`
Find similar files.
```typescript
const similar = await brain.vfs.findSimilar('/src/App.tsx', {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
limit: 5,
threshold: 0.7
2025-11-02 10:58:52 -08:00
})
```
---
### Tree Operations
#### `vfs.getTreeStructure(path, options?)` → `Promise<TreeNode>`
Get directory tree (prevents infinite recursion).
```typescript
const tree = await brain.vfs.getTreeStructure('/projects', {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
maxDepth: 3
2025-11-02 10:58:52 -08:00
})
```
---
#### `vfs.getDescendants(path, options?)` → `Promise<VFSEntity[]>`
Get all descendants with optional filtering.
```typescript
const files = await brain.vfs.getDescendants('/src', {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
filter: (entity) => entity.name.endsWith('.tsx')
2025-08-26 12:32:21 -07:00
})
```
---
2025-11-02 10:58:52 -08:00
### Metadata & Relationships
#### `vfs.getMetadata(path)` → `Promise<Metadata>`
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
Get file metadata.
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
```typescript
const meta = await brain.vfs.getMetadata('/src/App.tsx')
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
console.log(meta.todos) // Extracted TODOs
console.log(meta.tags) // Tags
2025-11-02 10:58:52 -08:00
```
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
---
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
#### `vfs.getRelationships(path)` → `Promise<Relation[]>`
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
Get file relationships.
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
```typescript
const rels = await brain.vfs.getRelationships('/src/App.tsx')
// Returns: imports, references, dependencies
```
2025-08-26 12:32:21 -07:00
---
2025-11-02 10:58:52 -08:00
#### `vfs.getTodos(path)` → `Promise<Todo[]>`
Get TODOs from a file.
```typescript
const todos = await brain.vfs.getTodos('/src/App.tsx')
```
---
2026-06-11 14:51:00 -07:00
#### `vfs.searchEntities(query)` → `Promise<Array<{ id, path, type, metadata }>>`
2025-11-02 10:58:52 -08:00
2026-06-11 14:51:00 -07:00
Search for semantic entities tracked by the VFS, filtered by type, name, or metadata.
2025-11-02 10:58:52 -08:00
```typescript
2026-06-11 14:51:00 -07:00
const people = await brain.vfs.searchEntities({
type: 'person', // entity type filter
name: 'Ada', // semantic name search
where: { role: 'author' }, // metadata filters
limit: 50
2025-11-02 10:58:52 -08:00
})
```
---
**[📖 Complete VFS Documentation → ](../vfs/QUICK_START.md )**
---
## Import & Export
### `import(source, options?)` → `Promise<ImportResult>`
Smart import with auto-detection (CSV, Excel, PDF, JSON, URLs).
```typescript
// CSV import
await brain.import('data.csv', {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
format: 'csv',
createEntities: true
2025-11-02 10:58:52 -08:00
})
2026-06-11 14:51:00 -07:00
// Excel import (all sheets processed automatically)
2025-11-02 10:58:52 -08:00
await brain.import('sales.xlsx', {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
format: 'excel',
2026-06-11 14:51:00 -07:00
vfsPath: '/imports/sales', // optional: mirror into the VFS
groupBy: 'sheet'
2025-11-02 10:58:52 -08:00
})
2026-06-11 14:51:00 -07:00
// PDF import (tables extracted automatically)
await brain.import('research.pdf', { format: 'pdf' })
2025-11-02 10:58:52 -08:00
// URL import
await brain.import('https://api.example.com/data.json')
```
**Parameters:**
- `source` : `string | Buffer | object` - File path, URL, buffer, or object
- `options?` : Import configuration
2026-06-11 14:51:00 -07:00
- `format?` : `'excel' | 'pdf' | 'csv' | 'json' | 'markdown' | 'yaml' | 'docx' | 'image'` - Auto-detected if omitted
- `vfsPath?` : `string` - Mirror imported content into the VFS at this path
- `groupBy?` : `'type' | 'sheet' | 'flat' | 'custom'` - VFS grouping strategy
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
- `createEntities?` : `boolean` - Create entities from rows
2026-06-11 14:51:00 -07:00
- `createRelationships?` : `boolean` - Create relationships between extracted entities
- `preserveSource?` : `boolean` - Save the original file in the VFS
- `enableNeuralExtraction?` : `boolean` - Extract entity names via AI
- `enableRelationshipInference?` : `boolean` - Infer relationships via AI
- `enableConceptExtraction?` : `boolean` - Extract entity types via AI
- `confidenceThreshold?` : `number` - Minimum confidence for extracted entities
- `onProgress?` : `(progress) => void` - Progress callback (stage, counts, throughput, ETA)
2025-11-02 10:58:52 -08:00
**Returns:** `Promise<ImportResult>` - Import statistics
**[📖 Complete Import Guide → ](../guides/import-anything.md )**
---
2026-06-17 11:24:57 -07:00
### Export & Import (portable) + Snapshots (native)
refactor(8.0): rename BackupData → PortableGraph (the type is interchange, not a backup)
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
2026-06-19 12:09:19 -07:00
**Portable graph export/import** — `brain.export()` / `brain.import()` (`PortableGraph` v1, versioned
2026-06-17 11:24:57 -07:00
JSON, partial-or-whole, cross-version). `export()` lives on the immutable `Db` , so it composes
with `now()` /`asOf()` /`with()` :
```typescript
// Export part or all of the brain to a portable, versioned document
refactor(8.0): rename BackupData → PortableGraph (the type is interchange, not a backup)
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
2026-06-19 12:09:19 -07:00
const graph = await brain.export({ ids }, { includeVectors: true })
2026-06-17 11:24:57 -07:00
refactor(8.0): rename BackupData → PortableGraph (the type is interchange, not a backup)
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
2026-06-19 12:09:19 -07:00
// Restore it — import() routes a PortableGraph to the graph round-trip (merge by id)
await otherBrain.import(graph, { onConflict: 'merge' })
2026-06-17 11:24:57 -07:00
// Time-travel export (serialize a past generation) / what-if export (a speculative state)
const past = await brain.asOf(gen)
const asWas = await past.export({ collection: id })
await past.release()
await brain.now().with(ops).export({ ids })
```
Selectors: `{ ids }` , `{ collection }` (alias `memberOf` ), `{ connected: { from, depth } }` ,
`{ vfsPath }` , predicate (`{ type, subtype, where, service }` ), or whole brain (omit). See the
**[Export & Import guide ](../guides/export-and-import.md )**. Distinct from `brain.import(file)`
refactor(8.0): rename BackupData → PortableGraph (the type is interchange, not a backup)
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
2026-06-19 12:09:19 -07:00
(CSV/PDF/Excel/JSON ingestion — `import()` dispatches on whether you pass a `PortableGraph` or a file).
2026-06-17 11:24:57 -07:00
**Native whole-brain snapshot** (generation-preserving, not portable JSON):
2025-11-02 10:58:52 -08:00
```typescript
2026-06-11 08:37:26 -07:00
// Instant hard-link snapshot via the Db API
const pin = brain.now()
await pin.persist('/backups/2026-06-11')
await pin.release()
2025-11-02 10:58:52 -08:00
2026-06-11 08:37:26 -07:00
// Time-travel to a past generation or timestamp
const snapshot = await brain.asOf(new Date('2026-06-01'))
feat: ID-first storage architecture + remove memory-unsafe APIs (v6.0.0)
BREAKING CHANGES:
**ID-First Storage Paths**
- Direct O(1) entity access without type lookups
- Before: entities/nouns/{TYPE}/metadata/{SHARD}/{ID}.json
- After: entities/nouns/{SHARD}/{ID}/metadata.json
- Migration handled automatically on first init()
**Removed Memory-Unsafe APIs**
- Removed brain.merge() - loaded all entities into memory
- Removed brain.diff() - loaded all entities into memory
- Removed brain.data().backup() - loaded all entities into memory
- Removed brain.data().restore() - depended on backup()
- Removed CLI commands: backup, restore, cow merge
**Migration Paths**
- merge() → Use checkout() or manually copy entities with pagination
- diff() → Use asOf() with manual paginated comparison
- backup() → Use fork() for instant COW snapshots
- restore() → Use checkout() to switch to snapshot branch
Core Improvements:
- ✅ All 8 storage adapters properly call super.init()
- ✅ GraphAdjacencyIndex integration in BaseStorage.init()
- ✅ Fixed ID-first path bugs (vector.json → vectors.json)
- ✅ Fixed MemoryStorage.initializeCounts() for ID-first paths
- ✅ New VFS APIs: du(), access(), find()
- ✅ Comprehensive documentation with migration guides
Storage Adapters Fixed:
- MemoryStorage, FileSystemStorage, AzureBlobStorage
- GCSStorage, R2Storage, S3CompatibleStorage
- OPFSStorage, HistoricalStorageAdapter
Files Changed: 28 files, +1,075/-1,933 lines (net -858)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-19 16:46:11 -08:00
const entities = await snapshot.find({ limit: 100 })
2026-06-11 08:37:26 -07:00
await snapshot.release()
2025-11-02 10:58:52 -08:00
```
---
## Configuration
### Constructor Options
```typescript
const brain = new Brainy({
docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
2026-06-09 16:13:35 -07:00
// Storage configuration
storage: {
type: 'filesystem', // 'memory' | 'filesystem' | 'auto'
feat(8.0): API simplification — remove neural()/Db.search, one storage `path` key, integration→0
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
2026-06-20 13:31:11 -07:00
path: './brainy-data'
docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
2026-06-09 16:13:35 -07:00
},
2025-11-02 10:58:52 -08:00
refactor(8.0)!: remove orphaned zero-config subsystem + dead cloud/progressive-init storage vestige
The old config-generation subsystem (src/config/ + autoConfiguration.ts) was
superseded during the 8.0 rework and never wired into init(): it emitted settings
for a partitioning subsystem that no longer exists and probed deleted cloud env
vars. The live zero-config path is inline — recall preset → HNSW knobs, storage
auto-detect, auto persistMode, container-memory-aware cache sizing.
The storage progressive-init / cloud-detection cluster was equally dead after the
cloud adapters were dropped: isCloudStorage() is permanently false (no overriders),
scheduleBackgroundInit/runBackgroundInit were never called (the latter an empty
body), initMode was never assigned, and Brainy.isFullyInitialized()/
awaitBackgroundInit() were always-trivial with zero callers. scheduleCountPersist()
collapses to its only-ever-taken immediate write-through path.
Removed:
- src/config/{index,zeroConfig,storageAutoConfig,modelAutoConfig,sharedConfigManager}.ts
- src/utils/autoConfiguration.ts + the inert BrainyZeroConfig export
- Brainy.isFullyInitialized()/awaitBackgroundInit() (+ BrainyInterface decls)
- InitMode type, isCloudStorage/detectCloudEnvironment/resolveInitMode,
scheduleBackgroundInit/runBackgroundInit/ensureValidatedForWrite and their state
- Dead cloud env-var probes (K_SERVICE/K_REVISION/AWS_LAMBDA_FUNCTION_NAME/
FUNCTIONS_TARGET/AZURE_FUNCTIONS_ENVIRONMENT)
Kept (verified live): production-detection logging (environment.ts), container-
memory cache sizing (memoryDetection/paramValidation), on-disk hash bucketing
(sharding.ts).
Docs: scrubbed deleted-subsystem references (JS quantization knobs, cloud/OPFS
adapters, partitioning, old zero-config API) across 14 files; deleted two wholly-
obsolete feature docs (complete-feature-list, v3-features); rewrote
architecture/zero-config for 8.0.
~3,700 LOC removed. Build clean; 1392 unit + 24 db-mvcc green.
2026-06-15 11:11:21 -07:00
// Vector index configuration (2 knobs)
docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
2026-06-09 16:13:35 -07:00
vector: {
recall: 'balanced', // 'fast' | 'balanced' | 'accurate'
persistMode: 'immediate' // 'immediate' | 'deferred'
},
2025-11-02 10:58:52 -08:00
docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
2026-06-09 16:13:35 -07:00
// Model configuration (embedded in WASM - zero config needed)
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
// Model: all-MiniLM-L6-v2 (384 dimensions)
// Device: CPU via WASM (works everywhere)
2025-11-02 10:58:52 -08:00
2026-06-11 14:51:00 -07:00
// Cache configuration — `true` /`false` , or an options object
cache: {
maxSize: 10000,
ttl: 3600000 // 1 hour in ms
}
2025-11-02 10:58:52 -08:00
})
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
await brain.init() // Required! VFS auto-initialized
2025-11-02 10:58:52 -08:00
```
---
## Storage Adapters
2026-06-11 08:37:26 -07:00
Brainy 8.0 ships two adapters — both support the full Db API (generational history, snapshots, restore).
2025-11-02 10:58:52 -08:00
docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
2026-06-09 16:13:35 -07:00
### Memory (Default for Tests)
2025-11-02 10:58:52 -08:00
```typescript
const brain = new Brainy({
docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
2026-06-09 16:13:35 -07:00
storage: { type: 'memory' }
2025-11-02 10:58:52 -08:00
})
```
**Use case:** Development, testing, prototyping
---
docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
2026-06-09 16:13:35 -07:00
### Filesystem (Default for Node)
2025-11-02 10:58:52 -08:00
```typescript
const brain = new Brainy({
docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
2026-06-09 16:13:35 -07:00
storage: {
type: 'filesystem',
feat(8.0): API simplification — remove neural()/Db.search, one storage `path` key, integration→0
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
2026-06-20 13:31:11 -07:00
path: './brainy-data'
docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
2026-06-09 16:13:35 -07:00
}
2025-11-02 10:58:52 -08:00
})
```
docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
2026-06-09 16:13:35 -07:00
**Use case:** Node.js applications, single-node production deployments
2025-11-02 10:58:52 -08:00
feat(8.0): API simplification — remove neural()/Db.search, one storage `path` key, integration→0
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
2026-06-20 13:31:11 -07:00
For off-site backup, snapshot `path` from your scheduler (`gsutil rsync` , `aws s3 sync` , `rclone` , or `tar` ) — Brainy itself doesn't reach out to cloud object stores.
2025-11-02 10:58:52 -08:00
---
docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
2026-06-09 16:13:35 -07:00
### Auto
2025-11-02 10:58:52 -08:00
```typescript
const brain = new Brainy({
feat(8.0): API simplification — remove neural()/Db.search, one storage `path` key, integration→0
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
2026-06-20 13:31:11 -07:00
storage: { type: 'auto', path: './brainy-data' }
2025-11-02 10:58:52 -08:00
})
```
feat(8.0): API simplification — remove neural()/Db.search, one storage `path` key, integration→0
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
2026-06-20 13:31:11 -07:00
Picks `'filesystem'` on Node with a writable `path` , falls back to `'memory'` otherwise.
2025-11-02 10:58:52 -08:00
---
## Utility Methods
### `clear()` → `Promise<void>`
Clear all data (entities and relationships).
```typescript
await brain.clear()
```
---
### `getNounCount()` → `Promise<number>`
Get total entity count.
```typescript
const count = await brain.getNounCount()
```
---
### `getVerbCount()` → `Promise<number>`
Get total relationship count.
```typescript
const count = await brain.getVerbCount()
```
---
feat: subtype top-level field + trackField + migrateField
Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).
Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
BaseStorage, mirrored after nounCountsByType with the same self-heal
rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path
Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
cardinality + per-NounType breakdown stats. Backed by the aggregation
engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update
Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
streams every entity, copies the value from one path to another, and
(unless readBoth) clears the source. Supports top-level standard fields,
metadata.X, and data.X paths. Idempotent — safe to re-run.
Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
quick-start all treat subtype as a core primitive with anonymous example
vocabularies (employee/customer/invoice/milestone).
Tests
- 26 new integration tests covering write/read/update/delete round-trips,
counts rollup decrement + re-route on mutation, trackField + byField
with and without perType, vocabulary whitelist enforcement, and
migrateField for metadata.X → subtype and data.X → subtype paths
including readBoth deprecation-window semantics.
Unit suite: 1468/1468 passing. Type-check + build clean.
2026-06-04 17:24:36 -07:00
### Subtype & facet APIs
Full guide: ** [Subtypes & Facets ](../guides/subtypes-and-facets.md )**.
#### `counts.bySubtype(type, subtype?)` → `Record<string, number> | number`
O(1) subtype counts for a NounType (backed by the persisted rollup).
```typescript
brain.counts.bySubtype(NounType.Person)
// → { employee: 12, customer: 847, vendor: 34 }
brain.counts.bySubtype(NounType.Person, 'employee')
// → 12
```
#### `counts.topSubtypes(type, n=10)` → `Array<[subtype, count]>`
Top N subtypes ranked by count.
```typescript
brain.counts.topSubtypes(NounType.Person, 3)
// → [['customer', 847], ['employee', 12], ['vendor', 34]]
```
#### `subtypesOf(type)` → `string[]`
Sorted distinct subtypes seen for a NounType.
```typescript
brain.subtypesOf(NounType.Person)
// → ['customer', 'employee', 'vendor']
```
feat: verb subtype + updateRelation + requireSubtype enforcement
Brings verbs to first-class parity with nouns. The 7.29.0 subtype primitive
shipped for entities only; this release ships the symmetric verb mirror plus
the enforcement layer for ensuring every entity AND every relationship has
both type AND subtype.
Layer V1 — verb subtype mirror
- HNSWVerbWithMetadata.subtype + STANDARD_VERB_FIELDS set + resolveVerbField()
- Relation<T>.subtype, RelateParams<T>.subtype, UpdateRelationParams<T> extended,
GetRelationsParams.subtype, GraphConstraints.subtype (for find connected)
- relate() persists subtype on verbMetadata + GraphVerb + transaction ops
- getRelations({ type, subtype }) fast-path filter with set membership
- find({ connected: { via, subtype, depth } }) traversal filter (depth-1 on
the JS path; explicit error on depth > 1 pointing at Cortex native)
- verbsToRelations + storage destructure sites surface subtype to top-level
- All three graph-index fast-path queries (getVerbsBySource/ByTarget) enrich
with subtype from metadata
Layer V2 — updateRelation() closes a pre-7.30 gap
- New first-class verb update method (parallel to update() for nouns)
- Changes subtype/type/weight/confidence/data/metadata in place
- Re-indexes in graph adjacency when verb type changes; id preserved
- validateUpdateRelationParams enforces id + at-least-one-field-to-update
Layer V3 — verb subtype storage rollup
- verbSubtypeCountsByType: Map<number, Map<string, number>> on BaseStorage
- verbSubtypeByIdCache for self-heal during update/delete
- incrementVerbSubtypeCount + decrementVerbSubtypeCount maintain state
- loadVerbSubtypeStatistics + saveVerbSubtypeStatistics persist to
_system/verb-subtype-statistics.json (mirrors noun-side shape)
- rebuildVerbSubtypeCounts for poison recovery / explicit repair
- getVerbSubtypeCountsByType accessor for the public counts API
- Wired into init() / flushCounts() / saveVerbMetadata / deleteVerbMetadata
Layer V4 — verb counts API + relationshipSubtypesOf
- brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) breakdown or point
- brain.counts.topRelationshipSubtypes(verb, n) — top N by count
- brain.relationshipSubtypesOf(verb) — sorted distinct subtypes
Layer V5 — migrateField extended to verbs
- New entityKind?: 'noun' | 'verb' | 'both' option (default 'noun')
- Mirror verb iteration via storage.getVerbs() with same path semantics
- verbToRelationLike + buildRelationMigrationUpdate helpers project the
storage verb shape onto the Entity<T>-shaped surface readPath understands
- Routes through new updateRelation() for the verb-side rewrite
Enforcement (opt-in in 7.30, default in 8.0)
- brain.requireSubtype(type, options) — unified API for NounType OR VerbType.
Registers per-type rules with optional values whitelist; composes with the
brain-wide flag.
- new Brainy({ requireSubtype: true }) — brain-wide strict mode. Every public
write path validates the pairing guarantee.
- { except: [NounType.Thing, ...] } form for catch-all type exemptions
- Atomic-fail semantics on addMany / relateMany — pre-validate every item
before any storage write, throw on first failure with item index
- Per-type rules + brain-wide flag both throw with descriptive messages
- VFS infrastructure bypass via metadata.isVFSEntity / isVFS markers so
brain's own VFS writes don't get rejected when strict mode is on
VFS labeling — concrete subtypes for infrastructure entities
- VFS root: NounType.Collection + subtype: 'vfs-root' (was bare Collection)
- VFS directories: subtype: 'vfs-directory'
- VFS files: subtype: 'vfs-file' (NounType still mime-based)
- VFS containment edges: VerbType.Contains + subtype: 'vfs-contains'
- Lets consumers cleanly enumerate VFS state via find({ subtype: 'vfs-file' })
and distinguish Brainy's VFS Collections from user-created Collections
Docs
- docs/guides/subtypes-and-facets.md extended with Layer V (Verbs) section +
Enforcement section. New full reference at the bottom split into Layer 1
(nouns), Layer V (verbs), Layer 2 (facets), Layer 3 (migration), Enforcement.
- docs/api/README.md adds updateRelation(), getRelations({ subtype }), the
three verb-side counts methods, requireSubtype(), and the brain-wide
constructor option. relate() params include subtype.
- docs/DATA_MODEL.md adds a Subtype-for-VerbType section + STANDARD_VERB_FIELDS
- docs/architecture/finite-type-system.md extends Principle 1a to verbs
- docs/QUERY_OPERATORS.md adds a verb-subtype filter section covering
getRelations and find({connected, subtype}) traversal
- README.md "Subtypes" section now shows both noun + verb in one example +
the enforcement APIs
- RELEASES.md v7.30.0 entry with the full noun/verb capability parity matrix
Tests
- tests/integration/verb-subtype-and-enforcement.test.ts — 30 new tests
covering V1 round-trips, V1 set membership, updateRelation in place,
updateRelation preservation, V2 counts breakdown + point + topN + distinct,
V2 decrements on unrelate, V2 re-routes on updateRelation, V3 depth-1
traversal filter, V3 depth>1 explicit error, V4 verb migration, V4 both
entity kinds, V4 readBoth preservation, V5 per-type required rejection,
V5 vocabulary rejection, V5 on-vocab acceptance, V5 verb-side enforcement,
V5 addMany atomic-fail, V5 relateMany atomic-fail, V5 update enforcement,
V5 updateRelation enforcement, V5 brain-wide strict mode, V5 except clause.
Verification
- Unit suite: 1468/1468 passing
- Noun subtype integration (7.29 carryover): 26/26 passing
- Verb subtype + enforcement integration: 30/30 passing
- Type-check: clean
- Build: clean
- Public closed-source reference audit: clean
Internal 8.0 spec
- .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md (gitignored, not in npm artifact)
documents the contract upgrade Cortex 3.0 implements against: required-by-
default subtype, SubtypeRegistry typing hook, native simplification,
multi-hop traversal native fast path, brain.fillSubtypes() migration helper.
Coordinated via PLATFORM-HANDOFF rows CTX-SUBTYPE-PARITY-V2 (7.30 parallel
work) and CTX-SUBTYPE-8.0-CONTRACT (8.0 spec).
2026-06-05 11:15:52 -07:00
#### `counts.byRelationshipSubtype(verb, subtype?)` → `Record<string, number> | number`
Verb-side mirror of `counts.bySubtype` . O(1) per-VerbType-per-subtype counts.
```typescript
brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
// → { direct: 12, 'dotted-line': 3 }
brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct')
// → 12
```
#### `counts.topRelationshipSubtypes(verb, n=10)` → `Array<[subtype, count]>`
Top N subtypes for a `VerbType` ranked by count.
```typescript
brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 3)
// → [['direct', 12], ['dotted-line', 3]]
```
#### `relationshipSubtypesOf(verb)` → `string[]`
Sorted distinct subtypes seen for a `VerbType` .
```typescript
brain.relationshipSubtypesOf(VerbType.ReportsTo)
// → ['direct', 'dotted-line']
```
fix: internal subtype consistency + brain.audit() diagnostic + improved enforcement errors
Brainy 7.30 shipped opt-in subtype enforcement; SDK 3.20.0 then registered
SDK_CORE_VOCABULARY on every consumer's brain (Event, Collection, Message,
Contract, Media, Document NounTypes). On 2026-06-08 Venue's /book flow went 500
because their brain.add({ type: NounType.Event, ... }) call sites lacked
subtype. An audit of Brainy's OWN source revealed 14 HIGH-risk internal write
paths that also omit subtype — any consumer running the same vocabulary would
have hit Brainy's infrastructure paths next. 7.30.1 closes both gaps before
8.0 makes strict mode the default.
Additive across the board. Zero behavior change for consumers not using strict
mode. Every change is JS-side — Cortex needs no work for 7.30.1.
NEW — brain.audit() diagnostic
- Read-only method walking storage.getNouns() / getVerbs() pagination
- Returns { entitiesWithoutSubtype: { type: count }, relationshipsWithoutSubtype,
total, scanned, recommendation }
- VFS infrastructure entities excluded by default (they bypass enforcement via
isVFSEntity marker); pass { includeVFS: true } to surface them
- The companion to migrateField (7.x) and fillSubtypes (8.0): tells consumers
exactly what would break under strict enforcement, deterministically
NEW — Improved enforcement error messages
- Caller's source location extracted from Error().stack so users see their own
call site, not a Brainy internal frame
- Specific guidance branches: registered vocabulary → "Pass one of: a, b, c";
brain-wide strict mode → mentions the except clause; otherwise → registration
recipe via brain.requireSubtype()
- Documentation link to the canonical migration recipe
- Same shape for noun and verb enforcement
NEW — CLI --subtype flag
- brainy add and brainy relate gain -s/--subtype <value>
- Defaults to 'cli-add' / 'cli-relate' so the CLI works against strict-mode
brains without the user needing to know the vocabulary in advance
INTERNAL — every Brainy write path now sets subtype
- VFS Contains edges (5 sites at lines 503/905/1694/1772/1886) → 'vfs-contains'
- VFS symlink entity → 'vfs-symlink' (NEW — distinct from 'vfs-file')
- VFS copy-file → preserves source subtype, falls back to 'vfs-file'
- VFS symlink also adopts the isVFSEntity infrastructure marker so it bypasses
enforcement in strict mode
- Aggregation materializer (Measurement entities) → 'materialized-aggregate'
- ImportCoordinator (3 sites): document → 'import-source'; entities →
options.defaultSubtype ?? 'imported'; placeholder → 'import-placeholder'
- SmartImportOrchestrator (4 entity sites + 2 batch relate sites): same
precedence (extractor → options.defaultSubtype → 'imported')
- EntityDeduplicator → candidate.subtype ?? 'imported'
- UniversalImportAPI → extractor → 'extracted' for both entities and relations
- NeuralImport → adds defaultSubtype to NeuralImportOptions; precedence same
- GoogleSheetsIntegration → request body 'subtype' ?? 'imported-from-sheets'
- ODataIntegration → request body 'Subtype' ?? 'imported-from-odata'
- MCP client message storage → 'mcp-message' (also fixes pre-existing missing
data field and missing type by aliasing from the prior text field)
Side-effect fix: storage.getNouns() paginated now surfaces subtype to top-level
- Single-noun getNoun() already did this in 7.30; the paginated path was missed
- Without this fix brain.audit() saw missing subtype on entities that actually
had one (caught by the strict-mode self-test before release)
NEW — tests/integration/strict-mode-self-test.test.ts (13 tests)
- Creates a brain under the exact SDK_CORE_VOCABULARY shape Venue hit + brain-
wide strict mode
- Exercises every internal Brainy path: VFS root + mkdir + writeFile + cp + mv
+ ln + symlink; aggregation engine; audit diagnostic with includeVFS toggle
- Validates error message UX: caller location, vocabulary guidance, brain-wide
strict mode guidance, off-vocabulary value reporting
Docs
- New "Strict mode in practice" section in docs/guides/subtypes-and-facets.md
covering the SDK_CORE_VOCABULARY pattern, 4-step migration recipe
(audit → migrateField → hand-fix → re-audit), the Brainy-internal label
reference table, and an 8.0 forward-look on fillSubtypes()
- docs/api/README.md: new audit() entry, strict-mode tips on add() and relate()
- RELEASES.md: full 7.30.1 entry
Cortex parity (forward-looking, not blocking 7.30.1)
- 6th open question added to .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md: native
fast path for audit() and fillSubtypes() via column-store null-subtype
bitmap for billion-scale brains
- Cortex should add a parity test mirroring strict-mode-self-test.test.ts
against their native paths to catch any latent bug where native writes
bypass JS validation
- Brainy-internal subtype labels become a documented part of the 8.0 contract
(useful for Cortex telemetry surfacing Brainy-managed infrastructure %)
Verification
- npx tsc --noEmit: clean
- npm test: 1468/1468 unit
- 7.29 noun integration suite: 26/26 (no regression)
- 7.30 verb subtype + enforcement integration suite: 30/30 (no regression)
- New strict-mode-self-test integration suite: 13/13
- npm run build: clean
- Closed-source product reference audit: clean
Addresses VE-SUBTYPE-MIGRATION (Venue's reported request) and ships internal
labels Venue did NOT ask for but that would have broken them next under their
own vocabulary registration.
2026-06-08 11:31:47 -07:00
#### `audit(options?)` → `Promise<AuditReport>` (7.30.1+)
Diagnostic — find entities and relationships missing a `subtype` value, grouped by type. The companion to `migrateField()` / `fillSubtypes()` — answers "what would break if I enabled strict subtype enforcement?".
```typescript
const report = await brain.audit()
// {
// entitiesWithoutSubtype: { event: 24, document: 3 },
// relationshipsWithoutSubtype: { relatedTo: 1402 },
// total: 1429,
// scanned: 8400,
// recommendation: 'Found 1429 entries without subtype. ...'
// }
```
**Parameters:**
- `options.includeVFS?` : `boolean` — When `false` (default), VFS infrastructure entities (`metadata.isVFSEntity` / `metadata.isVFS` ) are excluded. They bypass enforcement anyway, so counting them is noise.
- `options.batchSize?` : `number` — Pagination batch size (default 200).
- `options.onProgress?` : `(progress: { scanned, missingSubtype }) => void` — Progress callback per batch.
Run before adopting an SDK that registers `requireSubtype()` rules, or before upgrading to Brainy 8.0 (which makes strict mode the default). See the [Strict mode in practice ](../guides/subtypes-and-facets.md#strict-mode-in-practice-for-sdk-style-vocabulary-consumers ) guide for the full migration recipe.
feat: verb subtype + updateRelation + requireSubtype enforcement
Brings verbs to first-class parity with nouns. The 7.29.0 subtype primitive
shipped for entities only; this release ships the symmetric verb mirror plus
the enforcement layer for ensuring every entity AND every relationship has
both type AND subtype.
Layer V1 — verb subtype mirror
- HNSWVerbWithMetadata.subtype + STANDARD_VERB_FIELDS set + resolveVerbField()
- Relation<T>.subtype, RelateParams<T>.subtype, UpdateRelationParams<T> extended,
GetRelationsParams.subtype, GraphConstraints.subtype (for find connected)
- relate() persists subtype on verbMetadata + GraphVerb + transaction ops
- getRelations({ type, subtype }) fast-path filter with set membership
- find({ connected: { via, subtype, depth } }) traversal filter (depth-1 on
the JS path; explicit error on depth > 1 pointing at Cortex native)
- verbsToRelations + storage destructure sites surface subtype to top-level
- All three graph-index fast-path queries (getVerbsBySource/ByTarget) enrich
with subtype from metadata
Layer V2 — updateRelation() closes a pre-7.30 gap
- New first-class verb update method (parallel to update() for nouns)
- Changes subtype/type/weight/confidence/data/metadata in place
- Re-indexes in graph adjacency when verb type changes; id preserved
- validateUpdateRelationParams enforces id + at-least-one-field-to-update
Layer V3 — verb subtype storage rollup
- verbSubtypeCountsByType: Map<number, Map<string, number>> on BaseStorage
- verbSubtypeByIdCache for self-heal during update/delete
- incrementVerbSubtypeCount + decrementVerbSubtypeCount maintain state
- loadVerbSubtypeStatistics + saveVerbSubtypeStatistics persist to
_system/verb-subtype-statistics.json (mirrors noun-side shape)
- rebuildVerbSubtypeCounts for poison recovery / explicit repair
- getVerbSubtypeCountsByType accessor for the public counts API
- Wired into init() / flushCounts() / saveVerbMetadata / deleteVerbMetadata
Layer V4 — verb counts API + relationshipSubtypesOf
- brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) breakdown or point
- brain.counts.topRelationshipSubtypes(verb, n) — top N by count
- brain.relationshipSubtypesOf(verb) — sorted distinct subtypes
Layer V5 — migrateField extended to verbs
- New entityKind?: 'noun' | 'verb' | 'both' option (default 'noun')
- Mirror verb iteration via storage.getVerbs() with same path semantics
- verbToRelationLike + buildRelationMigrationUpdate helpers project the
storage verb shape onto the Entity<T>-shaped surface readPath understands
- Routes through new updateRelation() for the verb-side rewrite
Enforcement (opt-in in 7.30, default in 8.0)
- brain.requireSubtype(type, options) — unified API for NounType OR VerbType.
Registers per-type rules with optional values whitelist; composes with the
brain-wide flag.
- new Brainy({ requireSubtype: true }) — brain-wide strict mode. Every public
write path validates the pairing guarantee.
- { except: [NounType.Thing, ...] } form for catch-all type exemptions
- Atomic-fail semantics on addMany / relateMany — pre-validate every item
before any storage write, throw on first failure with item index
- Per-type rules + brain-wide flag both throw with descriptive messages
- VFS infrastructure bypass via metadata.isVFSEntity / isVFS markers so
brain's own VFS writes don't get rejected when strict mode is on
VFS labeling — concrete subtypes for infrastructure entities
- VFS root: NounType.Collection + subtype: 'vfs-root' (was bare Collection)
- VFS directories: subtype: 'vfs-directory'
- VFS files: subtype: 'vfs-file' (NounType still mime-based)
- VFS containment edges: VerbType.Contains + subtype: 'vfs-contains'
- Lets consumers cleanly enumerate VFS state via find({ subtype: 'vfs-file' })
and distinguish Brainy's VFS Collections from user-created Collections
Docs
- docs/guides/subtypes-and-facets.md extended with Layer V (Verbs) section +
Enforcement section. New full reference at the bottom split into Layer 1
(nouns), Layer V (verbs), Layer 2 (facets), Layer 3 (migration), Enforcement.
- docs/api/README.md adds updateRelation(), getRelations({ subtype }), the
three verb-side counts methods, requireSubtype(), and the brain-wide
constructor option. relate() params include subtype.
- docs/DATA_MODEL.md adds a Subtype-for-VerbType section + STANDARD_VERB_FIELDS
- docs/architecture/finite-type-system.md extends Principle 1a to verbs
- docs/QUERY_OPERATORS.md adds a verb-subtype filter section covering
getRelations and find({connected, subtype}) traversal
- README.md "Subtypes" section now shows both noun + verb in one example +
the enforcement APIs
- RELEASES.md v7.30.0 entry with the full noun/verb capability parity matrix
Tests
- tests/integration/verb-subtype-and-enforcement.test.ts — 30 new tests
covering V1 round-trips, V1 set membership, updateRelation in place,
updateRelation preservation, V2 counts breakdown + point + topN + distinct,
V2 decrements on unrelate, V2 re-routes on updateRelation, V3 depth-1
traversal filter, V3 depth>1 explicit error, V4 verb migration, V4 both
entity kinds, V4 readBoth preservation, V5 per-type required rejection,
V5 vocabulary rejection, V5 on-vocab acceptance, V5 verb-side enforcement,
V5 addMany atomic-fail, V5 relateMany atomic-fail, V5 update enforcement,
V5 updateRelation enforcement, V5 brain-wide strict mode, V5 except clause.
Verification
- Unit suite: 1468/1468 passing
- Noun subtype integration (7.29 carryover): 26/26 passing
- Verb subtype + enforcement integration: 30/30 passing
- Type-check: clean
- Build: clean
- Public closed-source reference audit: clean
Internal 8.0 spec
- .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md (gitignored, not in npm artifact)
documents the contract upgrade Cortex 3.0 implements against: required-by-
default subtype, SubtypeRegistry typing hook, native simplification,
multi-hop traversal native fast path, brain.fillSubtypes() migration helper.
Coordinated via PLATFORM-HANDOFF rows CTX-SUBTYPE-PARITY-V2 (7.30 parallel
work) and CTX-SUBTYPE-8.0-CONTRACT (8.0 spec).
2026-06-05 11:15:52 -07:00
#### `requireSubtype(type, options?)` → `void`
Register subtype enforcement for a specific `NounType` or `VerbType` . Unified API for nouns and verbs. Composes with the brain-wide `requireSubtype` constructor flag.
```typescript
// Lock down Person sub-classification
brain.requireSubtype(NounType.Person, {
values: ['employee', 'customer', 'vendor'],
required: true
})
// Lock down management edges
brain.requireSubtype(VerbType.ReportsTo, {
values: ['direct', 'dotted-line'],
required: true
})
```
**Parameters:**
- `type` : `NounType | VerbType` - The type to register
- `options.values?` : `string[]` - Vocabulary whitelist (rejects off-vocab values)
- `options.required?` : `boolean` - Whether subtype is required (default: `true` )
#### Brain-wide strict mode — `new Brainy({ requireSubtype })`
Constructor option that enforces subtype on every `add()` / `addMany()` / `update()` / `relate()` / `relateMany()` / `updateRelation()` for every type:
```typescript
// Every write must include subtype
const brain = new Brainy({ requireSubtype: true })
// Exempt specific types (e.g. catch-all Thing)
const brain2 = new Brainy({
requireSubtype: { except: [NounType.Thing, NounType.Custom] }
})
```
When strict mode is on:
- Every public write path checks the pairing guarantee.
- `addMany()` / `relateMany()` validate all items BEFORE any storage write — atomic-fail, no partial writes.
- Brainy's own VFS infrastructure writes bypass via the `metadata.isVFSEntity: true` marker.
- Per-type registrations always apply regardless of the brain-wide flag.
2026-06-11 14:51:00 -07:00
The default since 8.0.0 — pass `requireSubtype: false` to opt out while migrating pre-8.0 data.
feat: verb subtype + updateRelation + requireSubtype enforcement
Brings verbs to first-class parity with nouns. The 7.29.0 subtype primitive
shipped for entities only; this release ships the symmetric verb mirror plus
the enforcement layer for ensuring every entity AND every relationship has
both type AND subtype.
Layer V1 — verb subtype mirror
- HNSWVerbWithMetadata.subtype + STANDARD_VERB_FIELDS set + resolveVerbField()
- Relation<T>.subtype, RelateParams<T>.subtype, UpdateRelationParams<T> extended,
GetRelationsParams.subtype, GraphConstraints.subtype (for find connected)
- relate() persists subtype on verbMetadata + GraphVerb + transaction ops
- getRelations({ type, subtype }) fast-path filter with set membership
- find({ connected: { via, subtype, depth } }) traversal filter (depth-1 on
the JS path; explicit error on depth > 1 pointing at Cortex native)
- verbsToRelations + storage destructure sites surface subtype to top-level
- All three graph-index fast-path queries (getVerbsBySource/ByTarget) enrich
with subtype from metadata
Layer V2 — updateRelation() closes a pre-7.30 gap
- New first-class verb update method (parallel to update() for nouns)
- Changes subtype/type/weight/confidence/data/metadata in place
- Re-indexes in graph adjacency when verb type changes; id preserved
- validateUpdateRelationParams enforces id + at-least-one-field-to-update
Layer V3 — verb subtype storage rollup
- verbSubtypeCountsByType: Map<number, Map<string, number>> on BaseStorage
- verbSubtypeByIdCache for self-heal during update/delete
- incrementVerbSubtypeCount + decrementVerbSubtypeCount maintain state
- loadVerbSubtypeStatistics + saveVerbSubtypeStatistics persist to
_system/verb-subtype-statistics.json (mirrors noun-side shape)
- rebuildVerbSubtypeCounts for poison recovery / explicit repair
- getVerbSubtypeCountsByType accessor for the public counts API
- Wired into init() / flushCounts() / saveVerbMetadata / deleteVerbMetadata
Layer V4 — verb counts API + relationshipSubtypesOf
- brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) breakdown or point
- brain.counts.topRelationshipSubtypes(verb, n) — top N by count
- brain.relationshipSubtypesOf(verb) — sorted distinct subtypes
Layer V5 — migrateField extended to verbs
- New entityKind?: 'noun' | 'verb' | 'both' option (default 'noun')
- Mirror verb iteration via storage.getVerbs() with same path semantics
- verbToRelationLike + buildRelationMigrationUpdate helpers project the
storage verb shape onto the Entity<T>-shaped surface readPath understands
- Routes through new updateRelation() for the verb-side rewrite
Enforcement (opt-in in 7.30, default in 8.0)
- brain.requireSubtype(type, options) — unified API for NounType OR VerbType.
Registers per-type rules with optional values whitelist; composes with the
brain-wide flag.
- new Brainy({ requireSubtype: true }) — brain-wide strict mode. Every public
write path validates the pairing guarantee.
- { except: [NounType.Thing, ...] } form for catch-all type exemptions
- Atomic-fail semantics on addMany / relateMany — pre-validate every item
before any storage write, throw on first failure with item index
- Per-type rules + brain-wide flag both throw with descriptive messages
- VFS infrastructure bypass via metadata.isVFSEntity / isVFS markers so
brain's own VFS writes don't get rejected when strict mode is on
VFS labeling — concrete subtypes for infrastructure entities
- VFS root: NounType.Collection + subtype: 'vfs-root' (was bare Collection)
- VFS directories: subtype: 'vfs-directory'
- VFS files: subtype: 'vfs-file' (NounType still mime-based)
- VFS containment edges: VerbType.Contains + subtype: 'vfs-contains'
- Lets consumers cleanly enumerate VFS state via find({ subtype: 'vfs-file' })
and distinguish Brainy's VFS Collections from user-created Collections
Docs
- docs/guides/subtypes-and-facets.md extended with Layer V (Verbs) section +
Enforcement section. New full reference at the bottom split into Layer 1
(nouns), Layer V (verbs), Layer 2 (facets), Layer 3 (migration), Enforcement.
- docs/api/README.md adds updateRelation(), getRelations({ subtype }), the
three verb-side counts methods, requireSubtype(), and the brain-wide
constructor option. relate() params include subtype.
- docs/DATA_MODEL.md adds a Subtype-for-VerbType section + STANDARD_VERB_FIELDS
- docs/architecture/finite-type-system.md extends Principle 1a to verbs
- docs/QUERY_OPERATORS.md adds a verb-subtype filter section covering
getRelations and find({connected, subtype}) traversal
- README.md "Subtypes" section now shows both noun + verb in one example +
the enforcement APIs
- RELEASES.md v7.30.0 entry with the full noun/verb capability parity matrix
Tests
- tests/integration/verb-subtype-and-enforcement.test.ts — 30 new tests
covering V1 round-trips, V1 set membership, updateRelation in place,
updateRelation preservation, V2 counts breakdown + point + topN + distinct,
V2 decrements on unrelate, V2 re-routes on updateRelation, V3 depth-1
traversal filter, V3 depth>1 explicit error, V4 verb migration, V4 both
entity kinds, V4 readBoth preservation, V5 per-type required rejection,
V5 vocabulary rejection, V5 on-vocab acceptance, V5 verb-side enforcement,
V5 addMany atomic-fail, V5 relateMany atomic-fail, V5 update enforcement,
V5 updateRelation enforcement, V5 brain-wide strict mode, V5 except clause.
Verification
- Unit suite: 1468/1468 passing
- Noun subtype integration (7.29 carryover): 26/26 passing
- Verb subtype + enforcement integration: 30/30 passing
- Type-check: clean
- Build: clean
- Public closed-source reference audit: clean
Internal 8.0 spec
- .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md (gitignored, not in npm artifact)
documents the contract upgrade Cortex 3.0 implements against: required-by-
default subtype, SubtypeRegistry typing hook, native simplification,
multi-hop traversal native fast path, brain.fillSubtypes() migration helper.
Coordinated via PLATFORM-HANDOFF rows CTX-SUBTYPE-PARITY-V2 (7.30 parallel
work) and CTX-SUBTYPE-8.0-CONTRACT (8.0 spec).
2026-06-05 11:15:52 -07:00
feat: subtype top-level field + trackField + migrateField
Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).
Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
BaseStorage, mirrored after nounCountsByType with the same self-heal
rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path
Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
cardinality + per-NounType breakdown stats. Backed by the aggregation
engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update
Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
streams every entity, copies the value from one path to another, and
(unless readBoth) clears the source. Supports top-level standard fields,
metadata.X, and data.X paths. Idempotent — safe to re-run.
Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
quick-start all treat subtype as a core primitive with anonymous example
vocabularies (employee/customer/invoice/milestone).
Tests
- 26 new integration tests covering write/read/update/delete round-trips,
counts rollup decrement + re-route on mutation, trackField + byField
with and without perType, vocabulary whitelist enforcement, and
migrateField for metadata.X → subtype and data.X → subtype paths
including readBoth deprecation-window semantics.
Unit suite: 1468/1468 passing. Type-check + build clean.
2026-06-04 17:24:36 -07:00
#### `trackField(name, options?)` → `void`
Register a metadata field for cardinality + per-NounType breakdown stats. With `values: [...]` , validates against the whitelist on `add()` /`update()` .
```typescript
brain.trackField('status') // basic
brain.trackField('status', { perType: true }) // with per-NounType breakdown
brain.trackField('priority', { values: ['low', 'med', 'high'] }) // strict vocabulary
```
#### `counts.byField(name, options?)` → `Promise<Record<string, number>>`
Counts by value for a tracked field. Requires `perType: true` registration if filtering by NounType.
```typescript
await brain.counts.byField('status')
// → { todo: 12, doing: 3, done: 47 }
await brain.counts.byField('status', { type: NounType.Task })
// → { todo: 8, doing: 2, done: 30 }
```
#### `migrateField(options)` → `Promise<MigrationSummary>`
Stream-and-rewrite a field across the brain. Supports `metadata.X` , `data.X` , and top-level paths. Idempotent.
```typescript
// One-shot rewrite
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
// Deprecation window — keep source field readable
await brain.migrateField({ from: 'data.kind', to: 'subtype', readBoth: true })
// With progress reporting
await brain.migrateField({
from: 'metadata.kind',
to: 'subtype',
batchSize: 500,
onProgress: ({ scanned, migrated }) => console.log(`${scanned} / ${migrated}` )
})
```
Returns `{ scanned: number, migrated: number, skipped: number, errors: Array<{id, error}> }` .
2026-06-11 14:51:00 -07:00
#### `fillSubtypes(rules, options?)` → `Promise<FillSubtypesResult>` (8.0+)
Back-fill missing `subtype` values across entities AND relationships in one streaming pass — the migration companion to `audit()` . Keys are `NounType` /`VerbType` values; each rule is a literal subtype string or a function deriving one from the entry (return `undefined` to decline). Idempotent: entries that already carry a subtype are never touched, so a crashed run is resumed safely by re-running.
```typescript
const report = await brain.fillSubtypes({
[NounType.Person]: (e) => e.metadata?.kind ?? 'unspecified', // derived
[NounType.Document]: 'general', // literal default
[VerbType.RelatedTo]: 'unspecified' // relationship rule
})
// → { scanned, filled, skipped, errors, byType }
```
**Parameters:**
- `rules` : `FillSubtypeRules` - Map of NounType/VerbType → literal subtype or `(entry) => string | undefined`
- `options.includeVFS?` : `boolean` - Also fill VFS infrastructure entries (default `false` )
- `options.batchSize?` : `number` - Pagination batch size (default `200` )
- `options.onProgress?` : `(progress: { scanned, filled, skipped }) => void` - Per-batch callback
Returns `{ scanned, filled, skipped, errors, byType }` . After a clean run, `skipped` equals the remaining `audit().total` . See the [migration recipe ](../guides/subtypes-and-facets.md ).
feat: subtype top-level field + trackField + migrateField
Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).
Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
BaseStorage, mirrored after nounCountsByType with the same self-heal
rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path
Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
cardinality + per-NounType breakdown stats. Backed by the aggregation
engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update
Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
streams every entity, copies the value from one path to another, and
(unless readBoth) clears the source. Supports top-level standard fields,
metadata.X, and data.X paths. Idempotent — safe to re-run.
Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
quick-start all treat subtype as a core primitive with anonymous example
vocabularies (employee/customer/invoice/milestone).
Tests
- 26 new integration tests covering write/read/update/delete round-trips,
counts rollup decrement + re-route on mutation, trackField + byField
with and without perType, vocabulary whitelist enforcement, and
migrateField for metadata.X → subtype and data.X → subtype paths
including readBoth deprecation-window semantics.
Unit suite: 1468/1468 passing. Type-check + build clean.
2026-06-04 17:24:36 -07:00
---
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
### `embed(data)` → `Promise<number[]>` ✨
2025-11-02 10:58:52 -08:00
2026-01-06 14:52:12 -08:00
Generate embedding vector from text or data.
2025-11-02 10:58:52 -08:00
```typescript
const vector = await brain.embed('Hello world')
2026-01-06 14:52:12 -08:00
// 384-dimensional vector
console.log(vector.length) // 384
2025-11-02 10:58:52 -08:00
```
---
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
### `embedBatch(texts)` → `Promise<number[][]>` ✨
2026-01-06 14:52:12 -08:00
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
Batch embed multiple texts using native WASM batch API (single forward pass).
2026-01-06 14:52:12 -08:00
```typescript
const embeddings = await brain.embedBatch([
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
'Machine learning is fascinating',
'Deep neural networks',
'Natural language processing'
2026-01-06 14:52:12 -08:00
])
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
console.log(embeddings.length) // 3
2026-01-06 14:52:12 -08:00
console.log(embeddings[0].length) // 384
```
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
> Uses the WASM engine's native `embed_batch()` for a single model forward pass instead of N individual calls. This is the same batch API used internally by `highlight()`.
feat: add structured content extraction and batch embedding optimization to highlight()
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
2026-01-27 10:27:22 -08:00
2026-01-06 14:52:12 -08:00
---
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
### `similarity(textA, textB)` → `Promise<number>` ✨
2026-01-06 14:52:12 -08:00
Calculate semantic similarity between two texts.
```typescript
const score = await brain.similarity(
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
'The cat sat on the mat',
'A feline was resting on the rug'
2026-01-06 14:52:12 -08:00
)
console.log(score) // ~0.85 (high semantic similarity)
```
**Returns:** Score from 0 (different) to 1 (identical meaning)
---
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
### `neighbors(entityId, options?)` → `Promise<string[]>` ✨
2026-01-06 14:52:12 -08:00
Get graph neighbors of an entity.
```typescript
// Get all connected entities
const neighbors = await brain.neighbors(entityId)
// Get outgoing connections only
const outgoing = await brain.neighbors(entityId, {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
direction: 'outgoing',
limit: 10
2026-01-06 14:52:12 -08:00
})
// Multi-hop traversal
const extended = await brain.neighbors(entityId, {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
depth: 2,
direction: 'both'
2026-01-06 14:52:12 -08:00
})
```
**Options:**
- `direction` : `'outgoing' | 'incoming' | 'both'` (default: 'both')
- `depth` : `number` - Traversal depth (default: 1)
- `verbType` : `VerbType` - Filter by relationship type
- `limit` : `number` - Maximum neighbors to return
---
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
### `findDuplicates(options?)` → `Promise<DuplicateResult[]>` ✨
2026-01-06 14:52:12 -08:00
Find semantic duplicates in the database.
```typescript
// Find all duplicates
const duplicates = await brain.findDuplicates()
for (const group of duplicates) {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
console.log('Original:', group.entity.id)
for (const dup of group.duplicates) {
console.log(` Duplicate: ${dup.entity.id} (${dup.similarity.toFixed(2)})` )
}
2026-01-06 14:52:12 -08:00
}
// Find person duplicates with higher threshold
const personDupes = await brain.findDuplicates({
2026-06-11 14:51:00 -07:00
type: NounType.Person,
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
threshold: 0.9,
limit: 50
2026-01-06 14:52:12 -08:00
})
```
**Options:**
- `threshold` : `number` - Minimum similarity (default: 0.85)
- `type` : `NounType` - Filter by entity type
- `limit` : `number` - Maximum duplicate groups (default: 100)
---
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
### `indexStats()` → `Promise<IndexStats>` ✨
2026-01-06 14:52:12 -08:00
Get comprehensive index statistics.
```typescript
const stats = await brain.indexStats()
console.log(`Entities: ${stats.entities}` )
console.log(`Vectors: ${stats.vectors}` )
console.log(`Relationships: ${stats.relationships}` )
console.log(`Memory: ${(stats.memoryUsage.total / 1024 / 1024).toFixed(1)}MB` )
console.log(`Fields: ${stats.metadataFields.join(', ')}` )
```
**Returns:**
- `entities` - Total entity count
docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
2026-06-09 16:13:35 -07:00
- `vectors` - Total vectors in the vector index
2026-01-06 14:52:12 -08:00
- `relationships` - Total relationships in graph
- `metadataFields` - Indexed metadata fields
- `memoryUsage.vectors` - Vector memory (bytes)
- `memoryUsage.graph` - Graph memory (bytes)
- `memoryUsage.metadata` - Metadata index memory (bytes)
- `memoryUsage.total` - Total memory usage
---
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
### `cluster(options?)` → `Promise<ClusterResult[]>` ✨
2026-01-06 14:52:12 -08:00
Cluster entities by semantic similarity.
```typescript
// Find all clusters
const clusters = await brain.cluster()
for (const cluster of clusters) {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
console.log(`${cluster.clusterId}: ${cluster.entities.length} entities` )
2026-01-06 14:52:12 -08:00
}
// Find document clusters with centroids
const docClusters = await brain.cluster({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
type: NounType.Document,
threshold: 0.85,
minClusterSize: 3,
includeCentroid: true
2026-01-06 14:52:12 -08:00
})
```
**Options:**
- `threshold` : `number` - Similarity threshold (default: 0.8)
- `type` : `NounType` - Filter by entity type
- `minClusterSize` : `number` - Minimum cluster size (default: 2)
- `limit` : `number` - Maximum clusters to return (default: 100)
- `includeCentroid` : `boolean` - Calculate cluster centroids (default: false)
**Returns:**
- `clusterId` - Unique cluster identifier
- `entities` - Array of entities in the cluster
- `centroid` - Average embedding vector (if includeCentroid is true)
---
2026-06-11 14:51:00 -07:00
### `getStats(options?)` → `Promise<Statistics>`
2025-11-02 10:58:52 -08:00
2026-06-11 14:51:00 -07:00
Get complete entity/relationship statistics (convenience wrapper over `brain.counts` ).
2025-11-02 10:58:52 -08:00
```typescript
2026-06-11 14:51:00 -07:00
const stats = await brain.getStats()
console.log(stats.entities.total) // total entity count
console.log(stats.entities.byType) // counts per NounType
console.log(stats.relationships) // relationship stats
console.log(stats.density) // relationships per entity
// Exclude VFS infrastructure entities from the counts
const semanticOnly = await brain.getStats({ excludeVFS: true })
2025-11-02 10:58:52 -08:00
```
---
## Lifecycle
### Initialization
```typescript
const brain = new Brainy(config)
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
await brain.init() // Required! VFS auto-initialized here
2025-11-02 10:58:52 -08:00
```
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
VFS is auto-initialized during `brain.init()` - no separate `vfs.init()` needed!
2025-11-02 10:58:52 -08:00
---
### Shutdown
```typescript
2026-06-11 14:51:00 -07:00
await brain.close() // Graceful shutdown — flushes pending writes and releases the writer lock
2025-11-02 10:58:52 -08:00
```
---
## Examples
### Basic CRUD
```typescript
// Create
const id = await brain.add({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
data: 'Quantum computing breakthrough',
type: NounType.Concept,
metadata: { category: 'tech', year: 2024 }
2025-11-02 10:58:52 -08:00
})
// Read
const entity = await brain.get(id)
// Update
await brain.update({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
id,
metadata: { updated: true }
2025-11-02 10:58:52 -08:00
})
2026-06-11 14:51:00 -07:00
// Remove
await brain.remove(id)
2025-11-02 10:58:52 -08:00
```
---
### Knowledge Graphs
```typescript
// Create entities
const ai = await brain.add({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
data: 'Artificial Intelligence',
type: NounType.Concept
2025-11-02 10:58:52 -08:00
})
const ml = await brain.add({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
data: 'Machine Learning',
type: NounType.Concept
2025-11-02 10:58:52 -08:00
})
// Create relationship
await brain.relate({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
from: ml,
to: ai,
type: VerbType.IsA
2025-11-02 10:58:52 -08:00
})
// Traverse graph
const results = await brain.find({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
connected: { from: ai, depth: 2 }
2025-11-02 10:58:52 -08:00
})
```
---
### Triple Intelligence Query
```typescript
const results = await brain.find({
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
query: 'modern frontend frameworks', // 🔍 Vector
where: { // 📊 Document
year: { greaterThan: 2020 },
category: { oneOf: ['framework', 'library'] }
},
connected: { // 🕸️ Graph
to: reactId,
depth: 2,
type: VerbType.BuiltOn
},
limit: 10
2025-11-02 10:58:52 -08:00
})
```
---
2026-06-11 08:37:26 -07:00
### Database-as-a-Value Workflow
2025-11-02 10:58:52 -08:00
```typescript
2026-06-11 08:37:26 -07:00
// Speculate: what would this change look like? (nothing touches disk)
const base = brain.now()
const whatIf = await base.with([
2026-06-11 14:51:00 -07:00
{ op: 'add', type: NounType.Document, subtype: 'note', data: 'New feature', metadata: { draft: true } }
2026-06-11 08:37:26 -07:00
])
await whatIf.find({ where: { draft: true } })
await whatIf.release()
await base.release()
// Commit it for real — one atomic generation, with audit metadata
await brain.transact(
2026-06-11 14:51:00 -07:00
[{ op: 'add', type: NounType.Document, subtype: 'note', data: 'New feature', metadata: { draft: true } }],
2026-06-11 08:37:26 -07:00
{ meta: { author: 'dev@example .com', message: 'Add new feature' } }
)
2025-11-02 10:58:52 -08:00
```
---
### VFS File Management
```typescript
// Write files
await brain.vfs.writeFile('/docs/README.md', 'Project documentation')
await brain.vfs.mkdir('/src/components', { recursive: true })
// Read files
const content = await brain.vfs.readFile('/docs/README.md')
// Semantic search
const reactFiles = await brain.vfs.search('React components with hooks', {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
path: '/src'
2025-11-02 10:58:52 -08:00
})
// Get tree structure (safe, prevents infinite recursion)
const tree = await brain.vfs.getTreeStructure('/projects', {
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
maxDepth: 3
2025-11-02 10:58:52 -08:00
})
```
---
docs: complete Stage 3 CANONICAL taxonomy documentation (v5.5.0)
Added comprehensive documentation for Stage 3 taxonomy migration:
API Documentation Updates:
- Updated version header from v5.4.0 to v5.5.0
- Fixed all deprecated NounType.Content examples → Document, Concept
- Added versions.asOf() method documentation with examples
- Added comprehensive Type System Reference section
Type System Reference:
- Documented all 42 noun types organized by category
- Documented 127 verb types organized by functional groups
- Added migration guide from v5.4.0 (deprecated types)
- Breaking changes: User→Person, Topic→Concept, Content→Document
- Stage 3 adds +11 nouns, +87 verbs (169 total types)
Build Status:
- ✅ TypeScript build passes with zero errors (stale cache issue resolved)
- ✅ All Stage 2 references updated to Stage 3 CANONICAL
- ✅ Type embeddings current (42 nouns + 127 verbs = 338KB)
Tested with confidential Tales from Talifar Glossary (NOT in repo):
- Successfully imported 2,284 entities across 7 noun types
- Created 2,280 relationships (contains verb)
- Noun type distribution: document (50%), thing (16%), person (16%),
location (9%), concept (9%), collection (1%), file (<1%)
- Demonstrates Stage 3 taxonomy handles rich fantasy world data
Type System Coverage:
- 7/42 noun types used (16.7%) - appropriate for glossary domain
- 1/127 verb types used (0.8%) - opportunity for richer relationships
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-06 10:28:47 -08:00
## Type System Reference
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
Stage 3 CANONICAL taxonomy with 169 types (42 nouns + 127 verbs)
docs: complete Stage 3 CANONICAL taxonomy documentation (v5.5.0)
Added comprehensive documentation for Stage 3 taxonomy migration:
API Documentation Updates:
- Updated version header from v5.4.0 to v5.5.0
- Fixed all deprecated NounType.Content examples → Document, Concept
- Added versions.asOf() method documentation with examples
- Added comprehensive Type System Reference section
Type System Reference:
- Documented all 42 noun types organized by category
- Documented 127 verb types organized by functional groups
- Added migration guide from v5.4.0 (deprecated types)
- Breaking changes: User→Person, Topic→Concept, Content→Document
- Stage 3 adds +11 nouns, +87 verbs (169 total types)
Build Status:
- ✅ TypeScript build passes with zero errors (stale cache issue resolved)
- ✅ All Stage 2 references updated to Stage 3 CANONICAL
- ✅ Type embeddings current (42 nouns + 127 verbs = 338KB)
Tested with confidential Tales from Talifar Glossary (NOT in repo):
- Successfully imported 2,284 entities across 7 noun types
- Created 2,280 relationships (contains verb)
- Noun type distribution: document (50%), thing (16%), person (16%),
location (9%), concept (9%), collection (1%), file (<1%)
- Demonstrates Stage 3 taxonomy handles rich fantasy world data
Type System Coverage:
- 7/42 noun types used (16.7%) - appropriate for glossary domain
- 1/127 verb types used (0.8%) - opportunity for richer relationships
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-06 10:28:47 -08:00
### Noun Types (42)
Brainy uses a comprehensive noun type system covering 96-97% of human knowledge:
**Core Entity Types (7)**
- `NounType.Person` - Individual human entities
- `NounType.Organization` - Companies, institutions, collectives
- `NounType.Location` - Geographic and spatial entities
- `NounType.Thing` - Physical objects and artifacts
- `NounType.Concept` - Abstract ideas and principles
- `NounType.Event` - Temporal occurrences
- `NounType.Agent` - AI agents, bots, automated systems
**Digital/Content Types (4)**
- `NounType.Document` - Text-based files and written content
- `NounType.Media` - Audio, video, images
- `NounType.File` - Generic digital files
- `NounType.Message` - Communication content
**Business Types (4)**
- `NounType.Product` - Commercial products
- `NounType.Service` - Service offerings
- `NounType.Task` - Actions, todos, work items
- `NounType.Project` - Organized initiatives
**Scientific Types (2)**
- `NounType.Hypothesis` - Theories and propositions
- `NounType.Experiment` - Studies and investigations
**And 25 more types** including: `Organism` , `Substance` , `Quality` , `TimeInterval` , `Function` , `Proposition` , `Collection` , `Dataset` , `Process` , `State` , `Role` , `Language` , `Currency` , `Measurement` , `Contract` , `Regulation` , `Interface` , `Resource` , `Custom` , `SocialGroup` , `Institution` , `Norm` , `InformationContent` , `InformationBearer` , `Relationship`
### Verb Types (127)
Brainy supports 127 relationship types organized into categories:
**Foundational (7)**
- `VerbType.InstanceOf` , `VerbType.SubclassOf` , `VerbType.ParticipatesIn`
- `VerbType.RelatedTo` , `VerbType.Contains` , `VerbType.PartOf` , `VerbType.References`
**Spatial & Temporal (14)**
- Location: `LocatedAt` , `AdjacentTo` , `ContainsSpatially` , `OverlapsSpatially` , `Above` , `Below` , `Inside` , `Outside` , `Facing`
- Time: `Precedes` , `During` , `OccursAt` , `Overlaps` , `ImmediatelyAfter` , `SimultaneousWith`
**Causal & Dependency (11)**
- Direct: `Causes` , `Enables` , `Prevents` , `DependsOn` , `Requires`
- Modal: `CanCause` , `MustCause` , `WouldCauseIf` , `ProbablyCauses`
- Variations: `RigidlyDependsOn` , `FunctionallyDependsOn` , `HistoricallyDependsOn`
**Creation & Change (10)**
- Lifecycle: `Creates` , `Transforms` , `Becomes` , `Modifies` , `Consumes` , `Destroys`
- Properties: `GainsProperty` , `LosesProperty` , `RemainsSame` , `PersistsThrough`
**Social & Communication (8)**
- `MemberOf` , `WorksWith` , `FriendOf` , `Follows` , `Likes` , `ReportsTo` , `Mentors` , `Communicates`
**Epistemic & Modal (14)**
- Knowledge: `Knows` , `Doubts` , `Believes` , `Learns`
- Mental states: `Desires` , `Intends` , `Fears` , `Loves` , `Hates` , `Hopes` , `Perceives`
- Modality: `CouldBe` , `MustBe` , `Counterfactual`
**Measurement & Comparison (9)**
- `Measures` , `MeasuredIn` , `ConvertsTo` , `HasMagnitude` , `GreaterThan`
- `SimilarityDegree` , `ApproximatelyEquals` , `MoreXThan` , `HasDegree`
**And 54 more specialized verbs** including ownership, composition, uncertainty, deontic relationships (obligations/permissions), context-dependent truth, spatial/temporal variations, information theory, and meta-level relationships.
### Complete Reference
For the full taxonomy with all 169 types and their descriptions, see:
- **[Stage 3 CANONICAL Taxonomy ](../STAGE3-CANONICAL-TAXONOMY.md )** - Complete list with categories
- **[Noun-Verb Taxonomy Architecture ](../architecture/noun-verb-taxonomy.md )** - Design rationale
2026-06-11 14:51:00 -07:00
### Migration from pre-Stage-3 taxonomies
docs: complete Stage 3 CANONICAL taxonomy documentation (v5.5.0)
Added comprehensive documentation for Stage 3 taxonomy migration:
API Documentation Updates:
- Updated version header from v5.4.0 to v5.5.0
- Fixed all deprecated NounType.Content examples → Document, Concept
- Added versions.asOf() method documentation with examples
- Added comprehensive Type System Reference section
Type System Reference:
- Documented all 42 noun types organized by category
- Documented 127 verb types organized by functional groups
- Added migration guide from v5.4.0 (deprecated types)
- Breaking changes: User→Person, Topic→Concept, Content→Document
- Stage 3 adds +11 nouns, +87 verbs (169 total types)
Build Status:
- ✅ TypeScript build passes with zero errors (stale cache issue resolved)
- ✅ All Stage 2 references updated to Stage 3 CANONICAL
- ✅ Type embeddings current (42 nouns + 127 verbs = 338KB)
Tested with confidential Tales from Talifar Glossary (NOT in repo):
- Successfully imported 2,284 entities across 7 noun types
- Created 2,280 relationships (contains verb)
- Noun type distribution: document (50%), thing (16%), person (16%),
location (9%), concept (9%), collection (1%), file (<1%)
- Demonstrates Stage 3 taxonomy handles rich fantasy world data
Type System Coverage:
- 7/42 noun types used (16.7%) - appropriate for glossary domain
- 1/127 verb types used (0.8%) - opportunity for richer relationships
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-06 10:28:47 -08:00
**Breaking Changes:**
- `NounType.Content` removed → Use `Document` , `Message` , or `InformationContent`
- `NounType.User` removed → Use `Person` or `Agent`
- `NounType.Topic` removed → Use `Concept` or `Category`
**New Types Added:**
- **+11 noun types**: Agent, Organism, Substance, Quality, TimeInterval, Function, Proposition, Custom, SocialGroup, Institution, Norm, InformationContent, InformationBearer, Relationship
- **+87 verb types**: Extensive additions across all categories
---
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
## Key Features
2025-11-04 11:19:02 -08:00
2026-06-11 08:37:26 -07:00
- ✅ **Database as a Value** - `brain.now()` pins the whole store as an immutable `Db` in O(1)
- ✅ **Atomic Transactions** - `brain.transact()` commits multi-write batches all-or-nothing
- ✅ **Two-Level CAS** - per-entity `ifRev` and whole-store `ifAtGeneration`
- ✅ **Time Travel** - `brain.asOf()` serves the full query surface at any reachable past generation
- ✅ **Instant Snapshots** - `db.persist()` cuts hard-link snapshots; `Brainy.load()` opens them read-only
- ✅ **Speculative Writes** - `db.with()` answers what-if questions purely in memory
- ✅ **Reified Transaction Metadata** - audit fields recorded durably, readable via `transactionLog()`
- ✅ **VFS Entity Filtering** - All VFS entities have the `isVFSEntity: true` flag
- ✅ **VFS Auto-Initialization** - No separate `vfs.init()` calls
2025-11-02 10:58:52 -08:00
- ✅ **VFS Property Access** - Use `brain.vfs.method()` instead of `brain.vfs().method()`
2026-06-11 08:37:26 -07:00
- ✅ **Universal Storage Support** - Filesystem and memory adapters share one on-disk contract
2025-11-02 10:58:52 -08:00
---
## Support & Resources
- **📖 Documentation:** [Full Documentation ](../ )
- **🐛 Issues:** [GitHub Issues ](https://github.com/soulcraftlabs/brainy/issues )
- **💬 Discussions:** [GitHub Discussions ](https://github.com/soulcraftlabs/brainy/discussions )
- **📦 NPM:** [@soulcraft/brainy ](https://www.npmjs.com/package/@soulcraft/brainy )
- **⭐ GitHub:** [Star us ](https://github.com/soulcraftlabs/brainy )
---
## See Also
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
- **[Data Model ](../DATA_MODEL.md )** - Entity structure, data vs metadata, storage fields
- **[Query Operators ](../QUERY_OPERATORS.md )** - All BFO operators with examples and indexed vs in-memory matrix
2025-11-02 10:58:52 -08:00
- **[Triple Intelligence Architecture ](../architecture/triple-intelligence.md )** - How vector + graph + document work together
feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
2026-02-09 12:06:59 -08:00
- **[Find System ](../FIND_SYSTEM.md )** - Natural language find() details
2025-11-02 10:58:52 -08:00
- **[VFS Quick Start ](../vfs/QUICK_START.md )** - Complete VFS documentation
- **[Import Anything Guide ](../guides/import-anything.md )** - CSV, Excel, PDF, URL imports
2026-06-11 08:37:26 -07:00
- **[Consistency Model ](../concepts/consistency-model.md )** - The guarantees behind the Db API
- **[Snapshots & Time Travel ](../guides/snapshots-and-time-travel.md )** - Backup, restore, what-if, audit recipes
2025-11-02 10:58:52 -08:00
---
2025-08-26 12:32:21 -07:00
2025-11-02 10:58:52 -08:00
**License:** MIT © Brainy Contributors
2025-08-26 12:32:21 -07:00
---
feat: expand ContentCategory to universal 6-category set for highlight()
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
2026-01-27 13:44:58 -08:00
*Brainy - The Knowledge Operating System*
2026-06-11 08:37:26 -07:00
*From prototype to planet-scale • Zero configuration • Triple Intelligence™ • Database as a Value*