brainy/docs/DATA_MODEL.md

# Data Model

> How Brainy stores entities and relationships, and the critical distinction between `data` and `metadata`.

---

## Entity (Noun)

An entity is the fundamental data unit in Brainy. Every entity has:

| Field | Type | Indexed | Description |
|-------|------|---------|-------------|
| `id` | `string` | Primary key | UUID v4 (auto-generated or custom) |
| `data` | `any` | **HNSW vector index** | Content used for semantic/hybrid search. Strings auto-embed. |
| `metadata` | `object` | **MetadataIndex** | Structured queryable fields (tags, dates, flags, etc.) |
| `type` | `NounType` | MetadataIndex (as `noun`) | Entity type classification |
| `vector` | `number[]` | HNSW | 384-dim embedding (auto-computed from `data` or user-provided) |
| `confidence` | `number` | MetadataIndex | Type classification confidence (0-1) |
| `weight` | `number` | MetadataIndex | Entity importance/salience (0-1) |
| `service` | `string` | MetadataIndex | Multi-tenancy identifier |
| `createdAt` | `number` | MetadataIndex | Creation timestamp (ms since epoch) |
| `updatedAt` | `number` | MetadataIndex | Last update timestamp (ms since epoch) |
| `createdBy` | `object` | MetadataIndex | Source augmentation info |

### Example

```typescript
const id = await brain.add({
  data: 'John Smith is a software engineer at Acme Corp',  // → embedded into vector
  type: NounType.Person,
  metadata: {                   // → indexed, queryable via where filters
    role: 'engineer',
    department: 'backend',
    yearsExperience: 8
  },
  confidence: 0.95,
  weight: 0.7
})
```

---

## Relationship (Verb)

A relationship is a typed, directed edge connecting two entities.

| Field | Type | Indexed | Description |
|-------|------|---------|-------------|
| `id` | `string` | Primary key | UUID v4 (auto-generated) |
| `from` | `string` | **GraphAdjacencyIndex** | Source entity ID |
| `to` | `string` | **GraphAdjacencyIndex** | Target entity ID |
| `type` | `VerbType` | GraphAdjacencyIndex (as `verb`) | Relationship type classification |
| `data` | `any` | — | Opaque content (overrides auto-computed vector if provided) |
| `metadata` | `object` | — | Structured fields on the edge |
| `weight` | `number` | — | Connection strength (0-1, default: 1.0) |
| `confidence` | `number` | — | Relationship certainty (0-1) |
| `evidence` | `RelationEvidence` | — | Why this relationship was detected |
| `createdAt` | `number` | — | Creation timestamp (ms since epoch) |
| `updatedAt` | `number` | — | Last update timestamp (ms since epoch) |
| `service` | `string` | — | Multi-tenancy identifier |

### Example

```typescript
const relId = await brain.relate({
  from: personId,
  to: projectId,
  type: VerbType.WorksOn,
  data: 'Lead engineer on the AI module',   // Optional: content for this edge
  metadata: {                                // Optional: queryable edge fields
    role: 'lead',
    startDate: '2024-01-15'
  },
  weight: 0.9
})
```

---

## Data vs Metadata

This is the most important concept in Brainy's storage model:

### `data` — Content for Semantic Search

- Embedded into a 384-dimensional vector via the WASM embedding engine
- Searchable via **semantic similarity** (HNSW vector index) and **hybrid text+semantic** search
- Queried by passing `query` to `find()`:
  ```typescript
  brain.find({ query: 'machine learning algorithms' })
  ```
- **NOT** indexed by MetadataIndex — you cannot use `where` filters on `data`
- Stored opaquely: strings, objects, numbers — anything goes

### `metadata` — Structured Queryable Fields

- Indexed by MetadataIndex with O(1) lookups per field
- Queryable via `where` filters using [BFO operators](./QUERY_OPERATORS.md):
  ```typescript
  brain.find({
    where: {
      department: 'engineering',
      yearsExperience: { greaterThan: 5 },
      tags: { contains: 'senior' }
    }
  })
  ```
- **NOT** used for vector/semantic search
- Must be a flat or lightly nested object

### Quick Reference

| | `data` | `metadata` |
|---|---|---|
| **Purpose** | Content for embedding / semantic search | Structured fields for filtering |
| **Searched by** | `find({ query })` — vector similarity, hybrid text+semantic | `find({ where })` — exact, range, set operators |
| **Indexed by** | HNSW vector index | MetadataIndex |
| **Queryable with operators?** | No | Yes (`equals`, `greaterThan`, `oneOf`, etc.) |
| **Auto-embedded?** | Yes (strings → 384-dim vectors) | No |
| **Typical content** | Text descriptions, document content | Tags, dates, status flags, categories, numeric fields |

### Common Pattern

```typescript
// Add an article
await brain.add({
  data: 'A deep dive into transformer architectures and attention mechanisms',
  type: NounType.Document,
  metadata: {
    title: 'Transformer Deep Dive',
    author: 'Dr. Chen',
    publishedYear: 2024,
    tags: ['AI', 'transformers', 'NLP'],
    status: 'published'
  }
})

// Search by content (semantic — searches data)
const results = await brain.find({ query: 'neural network attention' })

// Filter by fields (exact — queries metadata)
const recent = await brain.find({
  where: {
    publishedYear: { greaterThan: 2023 },
    status: 'published'
  }
})

// Combine both (Triple Intelligence)
const precise = await brain.find({
  query: 'attention mechanisms',             // Semantic search on data
  where: { author: 'Dr. Chen' },            // Metadata filter
  connected: { from: authorId, depth: 1 }   // Graph traversal
})
```

---

## Storage Field Naming

Internally, Brainy uses different field names in storage vs the public API:

| Public API (Entity/Relation) | Storage (metadata object) | Notes |
|------------------------------|--------------------------|-------|
| `type` | `noun` | Entity type stored as `noun` |
| `from` | `sourceId` | Relationship source |
| `to` | `targetId` | Relationship target |
| `type` (on Relation) | `verb` | Relationship type stored as `verb` |

When querying with `find()`, you can use:
- `type` parameter (convenience alias, equivalent to `where.noun`)
- `where.noun` directly

```typescript
// These are equivalent:
brain.find({ type: NounType.Person })
brain.find({ where: { noun: NounType.Person } })
```

---

## Standard Metadata Fields

When you add an entity, Brainy stores these standard fields in the metadata object alongside your custom fields:

| Field | Set By | Description |
|-------|--------|-------------|
| `noun` | System | Entity type (NounType enum value) |
| `subtype` | User | Per-NounType sub-classification (e.g. `'employee'`, `'invoice'`, `'milestone'`). Flat string, no hierarchy. Indexed on the fast path and rolled into per-NounType statistics. |
| `data` | System | The raw `data` value (stored opaquely) |
| `createdAt` | System | Creation timestamp |
| `updatedAt` | System | Last update timestamp |
| `confidence` | User | Type classification confidence |
| `weight` | User | Entity importance |
| `service` | User | Multi-tenancy identifier |
| `createdBy` | User/System | Source augmentation |

On read, these standard fields are extracted to top-level Entity properties. The `metadata` field on the returned Entity contains **only your custom fields**.

### Subtype — sub-classification within a NounType

`type` (NounType) is a stable 42-value enum. `subtype` is the consumer-chosen string vocabulary *within* a type:

```typescript
// A Person who is an employee:
await brain.add({
  data: 'Avery Brooks — runs the AI lab',
  type: NounType.Person,
  subtype: 'employee',
  metadata: { department: 'ai-lab' }
})

// A Document that is an invoice:
await brain.add({
  data: 'INV-2026-001',
  type: NounType.Document,
  subtype: 'invoice',
  metadata: { amount: 1500 }
})
```

`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's how `find({ type, subtype })` routes through the standard-field fast path (column-store hit) instead of the metadata fallback. See **[Subtypes & Facets](./guides/subtypes-and-facets.md)** for the full guide including `trackField()` and `migrateField()`.

### Subtype — sub-classification within a VerbType (7.30+)

Relationships are first-class citizens too. Every verb (`VerbType`) gets the same `subtype` primitive — a `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'sibling'` / `'colleague'`. Same shape as the noun side: flat string, no hierarchy, top-level standard field on `HNSWVerbWithMetadata` and on the public `Relation<T>`:

```typescript
await brain.relate({
  from: ceoId,
  to: vpId,
  type: VerbType.ReportsTo,
  subtype: 'direct',                  // top-level standard field
  metadata: { since: '2025-Q1' }      // user-custom fields stay in metadata
})
```

Fast-path filter on the verb side:

```typescript
const direct = await brain.getRelations({
  from: ceoId,
  type: VerbType.ReportsTo,
  subtype: 'direct'
})
```

The verb-side rollup at `_system/verb-subtype-statistics.json` mirrors the noun-side `_system/subtype-statistics.json` — same shape, same self-heal machinery. Per-VerbType-per-subtype counts are O(1) via `brain.counts.byRelationshipSubtype()`.

Verbs and nouns now have full capability parity — every API on the noun side has a verb-side mirror, including the new `brain.updateRelation()` (which closed a pre-7.30 gap where relationships had no update path).

### Standard verb fields

The verb-side equivalent of `STANDARD_ENTITY_FIELDS` is `STANDARD_VERB_FIELDS`, exported from `src/coreTypes.ts`. Verb-specific standard fields:

| Field | Description |
|---|---|
| `verb` | The VerbType enum value |
| `sourceId` / `targetId` | The two endpoints of the relationship |
| `subtype` | Sub-classification within the VerbType (7.30+) |
| `confidence`, `weight`, `createdAt`, `updatedAt`, `service`, `createdBy`, `data` | Same semantics as the noun-side standard fields |

The companion `resolveVerbField(verb, field)` helper resolves field paths the same way `resolveEntityField` does for nouns: standard fields first, metadata fallback for everything else.

---

## See Also

- [API Reference](./api/README.md) — Complete API documentation
- [Query Operators](./QUERY_OPERATORS.md) — All BFO operators with examples
- [Find System](./FIND_SYSTEM.md) — Natural language find() details