brainy/docs/guides/schema-migrations.md

# Schema Migrations

Brainy includes a built-in migration system for transforming entity and verb metadata across storage versions. Migrations are pure functions that run once per storage instance, with optional snapshot backup (`backupTo`), resume support, and error tracking.

---

## Quick Start

### 1. Define a migration

Add your migration to the `MIGRATIONS` array in `src/migration/migrations.ts`:

```typescript
import type { Migration } from './types.js'

export const MIGRATIONS: Migration[] = [
  {
    id: '7.17.0-rename-status',
    version: '7.17.0',
    description: 'Rename "state" field to "status"',
    applies: 'nouns',
    transform: (m) => {
      if ('state' in m) {
        const { state, ...rest } = m
        return { ...rest, status: state }
      }
      return null // already migrated or not applicable
    }
  }
]
```

### 2. Ship the new version

That's it. Brainy detects pending migrations on `init()` and either runs them automatically or warns the user to call `brain.migrate()`.

---

## How It Works

When `brain.init()` runs:

1. **Detection** — reads migration state from storage (one key lookup). Compares completed migration IDs against the `MIGRATIONS` array. If nothing pending, cost is ~0ms.

2. **Small datasets** (`autoMigrate: true`, <10K entities) — migrates inline during `init()`.

3. **Large datasets or manual mode** — logs a warning. User calls `brain.migrate()` when ready.

When `brain.migrate()` runs:

1. **Backup (optional)** — with `backupTo`, a hard-link snapshot of the current generation is persisted before any transform runs. Rollback is `brain.restore(backupPath, { confirm: true })`.

2. **Transform** — iterates all nouns/verbs in paginated batches. For each entity, calls the `transform` function. If it returns a new object, saves it. If it returns `null`, skips. Vectors are never touched.

3. **Save state** — records each completed migration ID so it never re-runs.

4. **Rebuild indexes** — if any entities were modified, rebuilds the MetadataIndex.

---

## Writing Migrations

### The Migration interface

```typescript
interface Migration {
  id: string          // Unique ID, e.g. "7.17.0-rename-field"
  version: string     // Version that introduced this migration
  description: string // Human-readable description
  applies: 'nouns' | 'verbs' | 'both'
  transform: (metadata: Record<string, unknown>) => Record<string, unknown> | null
}
```

### Transform rules

- **Return a new object** to modify the entity's metadata.
- **Return `null`** to skip (no change needed).
- **Must be idempotent** — running the same transform twice on the same data should produce the same result (or return `null` the second time). This is required because interrupted runs resume and re-encounter already-migrated entities.
- **Must be pure** — no side effects, no async, no external state.
- Transforms only see metadata. Vectors, embeddings, and the `data` field stored inside metadata are available as properties on the metadata object.

### Ordering

Migrations run in array order. Add new migrations at the end of the `MIGRATIONS` array. Each migration runs independently per entity — migration 2 sees the output of migration 1.

### Validation

`MigrationRunner.validateMigrations()` checks migration definitions and will throw on:

- Duplicate IDs
- Invalid `applies` values (must be `'nouns'`, `'verbs'`, or `'both'`)
- Non-function `transform`
- Missing or empty `id`, `version`, or `description`

---

## API Reference

### `brain.migrate(options?)`

```typescript
// Dry-run: preview what would change without writing
const preview = await brain.migrate({ dryRun: true })
// preview.pendingMigrations — array of { id, description }
// preview.affectedEntities  — count of entities that would change
// preview.totalEntities     — count of entities scanned
// preview.sampleChanges     — up to 5 before/after samples
// preview.estimatedTime     — rough time estimate string

// Apply migrations (optionally with a pre-migration snapshot)
const result = await brain.migrate({ backupTo: '/backups/pre-migration' })
// result.backupPath        — snapshot path, or null when no backupTo was supplied
// result.migrationsApplied — array of migration IDs that ran
// result.entitiesProcessed — total entities scanned
// result.entitiesModified  — entities actually changed
// result.errors            — array of entity-level errors (non-fatal)
```

### Options

```typescript
interface MigrateOptions {
  dryRun?: boolean       // Preview without writing (default: false)
  maxErrors?: number     // Bail out after N entity errors (default: 100)
  onProgress?: (progress: {
    migrationId: string
    processed: number
    modified: number
    hasMore: boolean
  }) => void
}
```

---

## Error Handling

If a transform function throws on a specific entity, the error is recorded and migration continues to the next entity. The failed entity's metadata is left unchanged.

```typescript
const result = await brain.migrate()

if (result.errors.length > 0) {
  for (const err of result.errors) {
    console.warn(`Entity ${err.entityId} failed in ${err.migrationId}: ${err.error}`)
  }
}
```

If errors exceed `maxErrors` (default: 100), the migration stops early and returns partial results. Successfully migrated entities keep their changes; failed entities are unchanged.

```typescript
// Strict mode: fail fast on any error
const result = await brain.migrate({ maxErrors: 1 })

// Lenient mode: tolerate many errors
const result = await brain.migrate({ maxErrors: 10000 })
```

---

## Backup and Rollback

Pass `backupTo` and `brain.migrate()` persists a snapshot of the current generation **before any transform runs**. On filesystem storage the snapshot is a hard-link farm — created without copying entity data, and immune to later writes (see [Snapshots & Time Travel](./snapshots-and-time-travel.md)):

```typescript
const result = await brain.migrate({ backupTo: '/backups/pre-migration-8.0' })
console.log(result.backupPath) // '/backups/pre-migration-8.0' (null when no backupTo)
```

To roll back, restore the snapshot wholesale:

```typescript
await brain.restore('/backups/pre-migration-8.0', { confirm: true })
```

Without `backupTo`, no backup is taken — transforms are idempotent (they return `null` when already applied), but a pre-migration snapshot is the cheap insurance for anything destructive.

---

## Progress Tracking

For large datasets, use the `onProgress` callback:

```typescript
await brain.migrate({
  onProgress: ({ migrationId, processed, modified, hasMore }) => {
    console.log(`[${migrationId}] ${processed} scanned, ${modified} modified${hasMore ? '...' : ' (done)'}`)
  }
})
```

Progress is reported after each batch (batch size is determined by the storage adapter).

---

## Examples

### Rename a field

```typescript
{
  id: '7.17.0-rename-state-to-status',
  version: '7.17.0',
  description: 'Rename metadata.state to metadata.status',
  applies: 'nouns',
  transform: (m) => {
    if ('state' in m) {
      const { state, ...rest } = m
      return { ...rest, status: state }
    }
    return null
  }
}
```

### Add a default value

```typescript
{
  id: '7.18.0-add-priority-default',
  version: '7.18.0',
  description: 'Add priority field with default "normal"',
  applies: 'both',
  transform: (m) => {
    if (!('priority' in m)) {
      return { ...m, priority: 'normal' }
    }
    return null
  }
}
```

### Remove a deprecated field

```typescript
{
  id: '7.19.0-remove-legacy-flag',
  version: '7.19.0',
  description: 'Remove deprecated "legacy" field',
  applies: 'nouns',
  transform: (m) => {
    if ('legacy' in m) {
      const { legacy, ...rest } = m
      return rest
    }
    return null
  }
}
```

### Transform verb metadata

```typescript
{
  id: '7.20.0-normalize-verb-weights',
  version: '7.20.0',
  description: 'Normalize verb weights from 0-100 to 0-1 scale',
  applies: 'verbs',
  transform: (m) => {
    if (typeof m.weight === 'number' && m.weight > 1) {
      return { ...m, weight: m.weight / 100 }
    }
    return null
  }
}
```

---

## Storage Backend Compatibility

Migrations work identically across all storage backends (Memory, FileSystem). The system uses `BaseStorage` methods (`getNouns`, `saveNounMetadata`, `getVerbs`, `saveVerbMetadata`) which are implemented by every adapter.

Batch size and rate limiting are automatically configured per adapter — no tuning required.

---

## What Migrations Don't Do

- **Re-embedding** — migrations transform metadata only. If you change your embedding model or dimensions, that requires re-vectorizing data, which is a separate concern (not part of this system).
- **Vector modification** — the `vectors.json` files are never touched by migrations.
- **Schema enforcement** — migrations are opt-in transforms, not schema validators. Brainy's metadata is schemaless by design.