docs(8.0): consistency-model concept + snapshots guide — Db API replaces branching docs
This commit is contained in:
parent
e5feae4104
commit
cc8037db10
23 changed files with 1053 additions and 1871 deletions
|
|
@ -11,15 +11,12 @@
|
||||||
- **BaseStorage** (`src/storage/baseStorage.ts`): Base implementation with built-in type-aware partitioning (TypeAwareStorageAdapter was removed -- functionality merged into BaseStorage).
|
- **BaseStorage** (`src/storage/baseStorage.ts`): Base implementation with built-in type-aware partitioning (TypeAwareStorageAdapter was removed -- functionality merged into BaseStorage).
|
||||||
- **Adapters** (`src/storage/adapters/`):
|
- **Adapters** (`src/storage/adapters/`):
|
||||||
- `fileSystemStorage.ts` -- local filesystem
|
- `fileSystemStorage.ts` -- local filesystem
|
||||||
- `opfsStorage.ts` -- browser Origin Private File System (formerly browserStorage.ts)
|
|
||||||
- `memoryStorage.ts` -- in-memory
|
- `memoryStorage.ts` -- in-memory
|
||||||
- `s3CompatibleStorage.ts` -- generic S3-compatible (AWS, MinIO, etc.)
|
- `baseStorageAdapter.ts` -- shared adapter base (counts, batch ops)
|
||||||
- `r2Storage.ts` -- Cloudflare R2
|
- Cloud + OPFS adapters were removed in 8.0 (cloud backup is operator tooling)
|
||||||
- `gcsStorage.ts` -- Google Cloud Storage
|
- **Generational MVCC / Db API** (`src/db/`): immutable `Db` values over generation-stamped records
|
||||||
- `azureBlobStorage.ts` -- Azure Blob Storage
|
- `db.ts` (the `Db` value), `generationStore.ts` (record layer + commit protocol), `types.ts`, `errors.ts`, `whereMatcher.ts`
|
||||||
- Supporting: `batchS3Operations.ts`, `optimizedS3Search.ts`
|
- Design record: `docs/ADR-001-generational-mvcc.md`; replaced the pre-8.0 COW branching + versioning subsystems
|
||||||
- **COW** (`src/storage/cow/`): Copy-on-Write infrastructure for versioning and branching
|
|
||||||
- CommitLog, CommitObject, CommitBuilder, BlobStorage, RefManager, TreeObject
|
|
||||||
|
|
||||||
### Vector Search (`src/hnsw/`)
|
### Vector Search (`src/hnsw/`)
|
||||||
- `hnswIndex.ts` -- HNSW-based approximate nearest neighbor search
|
- `hnswIndex.ts` -- HNSW-based approximate nearest neighbor search
|
||||||
|
|
|
||||||
48
README.md
48
README.md
|
|
@ -144,38 +144,34 @@ const results = await brain.find({
|
||||||
})
|
})
|
||||||
```
|
```
|
||||||
|
|
||||||
### Git-Style Branching
|
### Database as a Value
|
||||||
|
|
||||||
Fork your entire database in <100ms. Snowflake-style copy-on-write.
|
The whole database, pinned as an immutable value. Snapshot isolation, time travel, atomic transactions, instant hard-link snapshots.
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
const experiment = await brain.fork('test-migration')
|
const db = brain.now() // Pin current state — O(1)
|
||||||
await experiment.add({ data: 'test data', type: NounType.Concept })
|
|
||||||
await experiment.commit({ message: 'Add test data', author: 'dev@co.com' })
|
|
||||||
await brain.checkout('test-migration')
|
|
||||||
|
|
||||||
// Time-travel: query at any past commit
|
// Atomic multi-write transaction (all-or-nothing, with CAS)
|
||||||
const snapshot = await brain.asOf(commitId)
|
await brain.transact([
|
||||||
const pastResults = await snapshot.find({ query: 'historical data' })
|
{ op: 'update', id: orderId, metadata: { status: 'paid' } },
|
||||||
await snapshot.close()
|
{ op: 'relate', from: invoiceId, to: orderId, type: VerbType.References, subtype: 'billing' }
|
||||||
|
], { meta: { author: 'billing-service' }, ifAtGeneration: db.generation })
|
||||||
|
|
||||||
|
await db.get(orderId) // Still 'pending' — pinned, forever
|
||||||
|
await brain.get(orderId) // 'paid' — live
|
||||||
|
|
||||||
|
// Time travel: full query surface at any past state
|
||||||
|
const yesterday = await brain.asOf(new Date(Date.now() - 86_400_000))
|
||||||
|
const past = await yesterday.find({ query: 'unpaid orders' })
|
||||||
|
|
||||||
|
// What-if: speculative writes, nothing touches disk
|
||||||
|
const whatIf = await db.with([{ op: 'remove', id: orderId }])
|
||||||
|
|
||||||
|
// Instant backup: hard-link snapshot, opens read-only with Brainy.load()
|
||||||
|
await brain.now().persist('/backups/today')
|
||||||
```
|
```
|
||||||
|
|
||||||
**[Branching Documentation](docs/features/instant-fork.md)**
|
**[Consistency Model](docs/concepts/consistency-model.md)** | **[Snapshots & Time Travel](docs/guides/snapshots-and-time-travel.md)**
|
||||||
|
|
||||||
### Entity Versioning
|
|
||||||
|
|
||||||
Save, restore, and compare entity snapshots.
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const userId = await brain.add({ data: 'Alice', type: NounType.Person })
|
|
||||||
await brain.versions.save(userId, { tag: 'v1.0' })
|
|
||||||
|
|
||||||
await brain.update(userId, { data: 'Alice Smith' })
|
|
||||||
await brain.versions.save(userId, { tag: 'v2.0' })
|
|
||||||
|
|
||||||
const diff = await brain.versions.compare(userId, 1, 2)
|
|
||||||
await brain.versions.restore(userId, 1)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Virtual Filesystem
|
### Virtual Filesystem
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -77,7 +77,7 @@ for (const [id, metadata] of metadataMap) {
|
||||||
**Features:**
|
**Features:**
|
||||||
- ✅ Direct O(1) path construction from ID (no type lookup needed!)
|
- ✅ Direct O(1) path construction from ID (no type lookup needed!)
|
||||||
- ✅ Sharding preservation (all paths include `{shard}/{id}`)
|
- ✅ Sharding preservation (all paths include `{shard}/{id}`)
|
||||||
- ✅ COW-aware (respects branch paths)
|
- ✅ Write-cache coherent (read-after-write consistency)
|
||||||
- ✅ 40x faster than v5.x type-first architecture
|
- ✅ 40x faster than v5.x type-first architecture
|
||||||
|
|
||||||
**Performance:**
|
**Performance:**
|
||||||
|
|
@ -126,32 +126,6 @@ for (const [sourceId, verbs] of results) {
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### 4. `storage.readBatchWithInheritance(paths, targetBranch?)`
|
|
||||||
|
|
||||||
COW-aware batch path resolution with branch inheritance.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const storage = brain.storage as BaseStorage
|
|
||||||
|
|
||||||
const paths = [
|
|
||||||
'entities/nouns/{shard}/id1/metadata.json',
|
|
||||||
'entities/nouns/{shard}/id2/metadata.json'
|
|
||||||
]
|
|
||||||
|
|
||||||
// Resolves to: branches/{branch}/entities/nouns/{shard}/{id}/metadata.json
|
|
||||||
const results: Map<string, any> = await storage.readBatchWithInheritance(paths, 'my-branch')
|
|
||||||
|
|
||||||
// Automatically inherits from parent branches for missing entities
|
|
||||||
```
|
|
||||||
|
|
||||||
**Features:**
|
|
||||||
- ✅ Branch path resolution (`branches/{branch}/...`)
|
|
||||||
- ✅ Write cache integration (read-after-write consistency)
|
|
||||||
- ✅ COW inheritance (fallback to parent commits for missing entities)
|
|
||||||
- ✅ Adapter-agnostic (works with all storage adapters)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## VFS Integration
|
## VFS Integration
|
||||||
|
|
||||||
VFS operations automatically use batch APIs for maximum performance.
|
VFS operations automatically use batch APIs for maximum performance.
|
||||||
|
|
@ -229,62 +203,21 @@ const path = `entities/nouns/${shard}/${id}/metadata.json`
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### ✅ COW (Copy-on-Write)
|
### ✅ Generational MVCC (8.0)
|
||||||
|
|
||||||
Batch operations respect branch isolation and time-travel:
|
Batch reads always serve the **live** generation through the fast paths
|
||||||
|
shown above. Point-in-time reads go through the Db API instead: a pinned
|
||||||
|
`Db` (`brain.now()`, `brain.asOf()`) resolves changed ids from immutable
|
||||||
|
generation records and unchanged ids from the same live paths batch reads
|
||||||
|
use — see the [consistency model](concepts/consistency-model.md).
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// Main branch
|
const db = brain.now() // pinned view
|
||||||
const brain = await Brainy.create({ enableCOW: true })
|
const entity = await db.get(id) // correct at the pinned generation
|
||||||
await brain.add({ type: 'document', data: 'Main' })
|
const results = await brain.batchGet(ids) // live state, batched
|
||||||
|
await db.release()
|
||||||
// Create fork
|
|
||||||
const fork = await brain.fork('experiment')
|
|
||||||
|
|
||||||
// Batch operations are isolated
|
|
||||||
await brain.batchGet([id1, id2]) // → Reads from: branches/main/...
|
|
||||||
await fork.batchGet([id1, id2]) // → Reads from: branches/experiment/...
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Inheritance:**
|
|
||||||
- Entities missing from child branch automatically inherit from parent commits
|
|
||||||
- `readBatchWithInheritance()` walks commit history for missing items
|
|
||||||
- Preserves fork semantics while maintaining performance
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### ✅ fork() and checkout()
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const fork = await brain.fork('my-branch')
|
|
||||||
await fork.add({ type: 'document', data: 'Fork entity' })
|
|
||||||
|
|
||||||
// Batch operations use correct branch
|
|
||||||
const results = await fork.batchGet([id1, id2])
|
|
||||||
// → Reads from: branches/my-branch/...
|
|
||||||
|
|
||||||
// Checkout changes active branch
|
|
||||||
await fork.checkout('main')
|
|
||||||
const mainResults = await fork.batchGet([id1, id2])
|
|
||||||
// → Reads from: branches/main/...
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### ✅ asOf() Time-Travel
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Create historical snapshot
|
|
||||||
await brain.commit('v1.0')
|
|
||||||
const snapshot = await brain.asOf('v1.0')
|
|
||||||
|
|
||||||
// Batch operations on historical data
|
|
||||||
const results = await snapshot.batchGet([id1, id2])
|
|
||||||
// → Reads from historical tree state
|
|
||||||
```
|
|
||||||
|
|
||||||
Historical queries use `HistoricalStorageAdapter` which wraps batch operations to point at specific commits.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Performance Benchmarks
|
## Performance Benchmarks
|
||||||
|
|
@ -490,8 +423,6 @@ High-Level API (src/brainy.ts)
|
||||||
↓
|
↓
|
||||||
Storage Layer (src/storage/baseStorage.ts)
|
Storage Layer (src/storage/baseStorage.ts)
|
||||||
↓
|
↓
|
||||||
COW Layer (readBatchWithInheritance)
|
|
||||||
↓
|
|
||||||
Adapter Layer (readBatchFromAdapter)
|
Adapter Layer (readBatchFromAdapter)
|
||||||
↓
|
↓
|
||||||
Storage Adapter (FileSystemStorage / MemoryStorage)
|
Storage Adapter (FileSystemStorage / MemoryStorage)
|
||||||
|
|
@ -509,7 +440,6 @@ return await Promise.all(resolvedPaths.map(path => this.read(path)))
|
||||||
**Shipped Adapters:**
|
**Shipped Adapters:**
|
||||||
- MemoryStorage
|
- MemoryStorage
|
||||||
- FileSystemStorage
|
- FileSystemStorage
|
||||||
- HistoricalStorageAdapter (delegates to underlying)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -518,7 +448,6 @@ return await Promise.all(resolvedPaths.map(path => this.read(path)))
|
||||||
- `brain.batchGet(ids, options?)` - High-level batch entity retrieval
|
- `brain.batchGet(ids, options?)` - High-level batch entity retrieval
|
||||||
- `storage.getNounMetadataBatch(ids)` - Storage-level metadata batch
|
- `storage.getNounMetadataBatch(ids)` - Storage-level metadata batch
|
||||||
- `storage.getVerbsBySourceBatch(sourceIds, verbType?)` - Batch relationship queries
|
- `storage.getVerbsBySourceBatch(sourceIds, verbType?)` - Batch relationship queries
|
||||||
- `storage.readBatchWithInheritance(paths, targetBranch?)` - COW-aware batch reads
|
|
||||||
|
|
||||||
**Performance Improvements:**
|
**Performance Improvements:**
|
||||||
- VFS operations: 90%+ faster than the naive per-entity loop
|
- VFS operations: 90%+ faster than the naive per-entity loop
|
||||||
|
|
@ -528,10 +457,8 @@ return await Promise.all(resolvedPaths.map(path => this.read(path)))
|
||||||
**Compatibility:**
|
**Compatibility:**
|
||||||
- ✅ ID-first storage
|
- ✅ ID-first storage
|
||||||
- ✅ Sharding (256 shards)
|
- ✅ Sharding (256 shards)
|
||||||
- ✅ COW (branch isolation, inheritance)
|
- ✅ Generational MVCC — batch reads serve the live generation; pinned `Db` views serve the past
|
||||||
- ✅ fork() and checkout()
|
- ✅ All indexes respected (vector, metadata, graph adjacency)
|
||||||
- ✅ asOf() time-travel
|
|
||||||
- ✅ All indexes respected (vector, type-aware vector, metadata, graph adjacency, version, deleted items)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -35,6 +35,7 @@ const results = await brain.find({
|
||||||
| **[Data Model](./DATA_MODEL.md)** | Entity structure, data vs metadata, storage fields |
|
| **[Data Model](./DATA_MODEL.md)** | Entity structure, data vs metadata, storage fields |
|
||||||
| **[Query Operators](./QUERY_OPERATORS.md)** | All BFO operators with examples and indexed/in-memory matrix |
|
| **[Query Operators](./QUERY_OPERATORS.md)** | All BFO operators with examples and indexed/in-memory matrix |
|
||||||
| [Find System](./FIND_SYSTEM.md) | Natural language `find()` and hybrid search details |
|
| [Find System](./FIND_SYSTEM.md) | Natural language `find()` and hybrid search details |
|
||||||
|
| [Consistency Model](./concepts/consistency-model.md) | The Db API guarantees — snapshot isolation, atomic transactions, time travel |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -70,6 +71,7 @@ See [vfs/](./vfs/) for the complete VFS documentation set.
|
||||||
| Document | Description |
|
| Document | Description |
|
||||||
|----------|-------------|
|
|----------|-------------|
|
||||||
| [Import Anything](./guides/import-anything.md) | CSV, Excel, PDF, URL imports |
|
| [Import Anything](./guides/import-anything.md) | CSV, Excel, PDF, URL imports |
|
||||||
|
| [Snapshots & Time Travel](./guides/snapshots-and-time-travel.md) | Backups, restore, what-if analysis, audit trails |
|
||||||
| [Natural Language](./guides/natural-language.md) | Query in plain English |
|
| [Natural Language](./guides/natural-language.md) | Query in plain English |
|
||||||
| [Neural API](./guides/neural-api.md) | AI-powered features |
|
| [Neural API](./guides/neural-api.md) | AI-powered features |
|
||||||
| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No limits, no tiers |
|
| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No limits, no tiers |
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,7 @@ public: true
|
||||||
category: api
|
category: api
|
||||||
template: api
|
template: api
|
||||||
order: 1
|
order: 1
|
||||||
description: Complete API reference for all Brainy methods — add, find, relate, update, delete, batch operations, branching, entity versioning, VFS, neural API, and more.
|
description: Complete API reference for all Brainy methods — add, find, relate, update, delete, batch operations, the Db API (transactions, snapshots, time travel), VFS, neural API, and more.
|
||||||
next:
|
next:
|
||||||
- getting-started/quick-start
|
- getting-started/quick-start
|
||||||
- guides/find-system
|
- guides/find-system
|
||||||
|
|
@ -14,9 +14,9 @@ next:
|
||||||
# 🧠 Brainy API Reference
|
# 🧠 Brainy API Reference
|
||||||
|
|
||||||
> **Complete API documentation for Brainy**
|
> **Complete API documentation for Brainy**
|
||||||
> Zero Configuration • Triple Intelligence • Git-Style Branching • Entity Versioning • Candle WASM Embeddings
|
> Zero Configuration • Triple Intelligence • Database as a Value • Atomic Transactions • Time Travel
|
||||||
|
|
||||||
**Updated:** 2026-01-06
|
**Updated:** 2026-06-11
|
||||||
**All APIs verified against actual code**
|
**All APIs verified against actual code**
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -43,15 +43,19 @@ const results = await brain.find({
|
||||||
connected: { from: id, depth: 2 }
|
connected: { from: id, depth: 2 }
|
||||||
})
|
})
|
||||||
|
|
||||||
// Fork for safe experimentation
|
// Pin the current state as an immutable value
|
||||||
const experiment = await brain.fork('test-feature')
|
const db = brain.now()
|
||||||
await experiment.add({ data: 'test', type: NounType.Document })
|
|
||||||
await experiment.commit({ message: 'Add test data' })
|
|
||||||
|
|
||||||
// Entity versioning
|
// Commit an atomic multi-write batch (all-or-nothing)
|
||||||
await brain.versions.save(id, { tag: 'v1.0', description: 'Initial version' })
|
await brain.transact([
|
||||||
await brain.update(id, { category: 'AI' })
|
{ op: 'update', id, metadata: { category: 'AI' } }
|
||||||
await brain.versions.save(id, { tag: 'v2.0' })
|
], { meta: { author: 'docs-example' } })
|
||||||
|
|
||||||
|
await db.get(id) // still sees the pre-transaction state — snapshot isolation
|
||||||
|
await db.release()
|
||||||
|
|
||||||
|
// Time travel: query any past state
|
||||||
|
const yesterday = await brain.asOf(new Date(Date.now() - 86_400_000))
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -73,11 +77,8 @@ See **[Data Model](../DATA_MODEL.md)** for the full explanation.
|
||||||
### 🧠 Triple Intelligence
|
### 🧠 Triple Intelligence
|
||||||
Vector search + Graph traversal + Metadata filtering in one unified query.
|
Vector search + Graph traversal + Metadata filtering in one unified query.
|
||||||
|
|
||||||
### 🌳 Git-Style Branching
|
### 🧊 Database Values (Db)
|
||||||
Fork, experiment, and commit - Snowflake-style copy-on-write isolation.
|
The whole store pinned as an immutable value — snapshot isolation, atomic `transact()` batches, time travel with `asOf()`, instant hard-link snapshots with `persist()`. See the [consistency model](../concepts/consistency-model.md).
|
||||||
|
|
||||||
### 📜 Entity Versioning
|
|
||||||
Time-travel and history tracking for individual entities - Git-like version control with content-addressable storage.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -88,8 +89,7 @@ Time-travel and history tracking for individual entities - Git-like version cont
|
||||||
- [Aggregation Engine](#aggregation-engine)
|
- [Aggregation Engine](#aggregation-engine)
|
||||||
- [Relationships](#relationships)
|
- [Relationships](#relationships)
|
||||||
- [Batch Operations](#batch-operations)
|
- [Batch Operations](#batch-operations)
|
||||||
- [Branch Management](#branch-management)
|
- [Database Values & Time Travel (Db API)](#database-values--time-travel-db-api)
|
||||||
- [Entity Versioning](#entity-versioning)
|
|
||||||
- [Virtual Filesystem (VFS)](#virtual-filesystem-vfs)
|
- [Virtual Filesystem (VFS)](#virtual-filesystem-vfs)
|
||||||
- [Neural API](#neural-api)
|
- [Neural API](#neural-api)
|
||||||
- [Import & Export](#import--export)
|
- [Import & Export](#import--export)
|
||||||
|
|
@ -781,576 +781,224 @@ const ids = await brain.relateMany({
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Branch Management
|
## Database Values & Time Travel (Db API)
|
||||||
|
|
||||||
Git-style branching with Snowflake-style copy-on-write.
|
Brainy 8.0's generational MVCC exposes the whole store as an immutable
|
||||||
|
value: the **`Db`**. Pin the current state in O(1), commit atomic
|
||||||
|
multi-write batches, query any past generation with the full query surface,
|
||||||
|
cut instant snapshots, and ask what-if questions in memory. The exact
|
||||||
|
guarantees live in the **[consistency model](../concepts/consistency-model.md)**;
|
||||||
|
recipes live in **[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md)**.
|
||||||
|
|
||||||
### `fork(branch?, options?)` → `Promise<Brainy>`
|
### `generation()` → `number`
|
||||||
|
|
||||||
Create an instant fork (<100ms) with full isolation.
|
The store's current generation — a monotonic watermark advanced once per
|
||||||
|
committed `transact()` batch and once per single-operation write. Never
|
||||||
|
reissued, including across restarts and `restore()`.
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// Create a fork
|
const g = brain.generation()
|
||||||
const experiment = await brain.fork('test-feature')
|
|
||||||
|
|
||||||
// Make changes safely in isolation
|
|
||||||
await experiment.add({ data: 'Test entity', type: NounType.Document })
|
|
||||||
await experiment.update({ id: someId, metadata: { modified: true } })
|
|
||||||
|
|
||||||
// Parent is unaffected!
|
|
||||||
const parentData = await brain.find({}) // Original data unchanged
|
|
||||||
```
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `branch?`: `string` - Branch name (auto-generated if omitted)
|
|
||||||
- `options?`: `object`
|
|
||||||
- `description?`: `string` - Branch description
|
|
||||||
|
|
||||||
**Returns:** `Promise<Brainy>` - New Brainy instance on forked branch
|
|
||||||
|
|
||||||
**How it works:** Snowflake-style COW shares the vector index, copies only modified nodes (10-20% memory overhead).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### `checkout(branch)` → `Promise<void>`
|
|
||||||
|
|
||||||
Switch to a different branch.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
await brain.checkout('main')
|
|
||||||
await brain.checkout('test-feature')
|
|
||||||
```
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `branch`: `string` - Branch name
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### `listBranches()` → `Promise<string[]>`
|
|
||||||
|
|
||||||
List all branches.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const branches = await brain.listBranches()
|
|
||||||
// ['main', 'test-feature', 'experiment-2']
|
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### `getCurrentBranch()` → `Promise<string>`
|
### `now()` → `Db`
|
||||||
|
|
||||||
Get current branch name.
|
Pin the current generation and return an immutable view — O(1), no I/O.
|
||||||
|
The view keeps reading exactly this state no matter what commits afterwards.
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
const current = await brain.getCurrentBranch()
|
const db = brain.now()
|
||||||
// 'main'
|
await brain.update({ id, metadata: { v: 2 } })
|
||||||
|
|
||||||
|
await db.get(id) // still sees v: 1 — pinned
|
||||||
|
await brain.get(id) // sees v: 2 — live
|
||||||
|
await db.release() // unpin (enables history compaction)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Returns:** `Db` — release it when done; pins gate `compactHistory()`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `transact(ops, options?)` → `Promise<Db>`
|
||||||
|
|
||||||
|
Execute a declarative operation batch **atomically**: either every
|
||||||
|
operation applies and the store advances exactly one generation, or none
|
||||||
|
apply and the store is byte-identical to its pre-transaction state. The
|
||||||
|
commit point is an atomic manifest rename; a crash anywhere before it rolls
|
||||||
|
back to the exact pre-transaction bytes on the next open.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const db = await brain.transact([
|
||||||
|
{ op: 'add', id: orderId, type: NounType.Document, subtype: 'order', data: 'Order #1042' },
|
||||||
|
{ op: 'update', id: customerId, metadata: { lastOrderAt: Date.now() }, ifRev: customer._rev },
|
||||||
|
{ op: 'relate', from: customerId, to: orderId, type: VerbType.Creates, subtype: 'purchase' },
|
||||||
|
{ op: 'remove', id: staleDraftId },
|
||||||
|
{ op: 'unrelate', id: oldRelationId }
|
||||||
|
], {
|
||||||
|
meta: { author: 'order-service', requestId: 'req-9f2' }, // reified, durable
|
||||||
|
ifAtGeneration: expectedGeneration // whole-store CAS
|
||||||
|
})
|
||||||
|
|
||||||
|
db.receipt.ids // resolved id per operation, in input order
|
||||||
|
db.receipt.generation // the committed generation
|
||||||
|
```
|
||||||
|
|
||||||
|
**Operations** (`op` discriminates; parameters mirror the single-operation methods):
|
||||||
|
- `{ op: 'add', ... }` — same parameters as `add()`; optional explicit `id`
|
||||||
|
- `{ op: 'update', ... }` — same parameters as `update()`, including per-entity `ifRev` CAS
|
||||||
|
- `{ op: 'remove', id }` — deletes the entity plus its relationships (same cascade as `delete()`)
|
||||||
|
- `{ op: 'relate', ... }` — same parameters as `relate()`, including `bidirectional`; duplicates dedupe to the existing relationship id
|
||||||
|
- `{ op: 'unrelate', id }` — deletes a relationship by id
|
||||||
|
|
||||||
|
Operations may reference ids created earlier in the same batch.
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `meta?`: `Record<string, unknown>` — transaction metadata, recorded durably in the transaction log (audit fields: author, reason, request id)
|
||||||
|
- `ifAtGeneration?`: `number` — whole-store compare-and-swap; commits only if the store is still at this generation
|
||||||
|
|
||||||
|
**Returns:** `Promise<Db>` — pinned at the freshly committed generation, carrying a `receipt`.
|
||||||
|
|
||||||
|
**Throws:**
|
||||||
|
- `GenerationConflictError` — `ifAtGeneration` did not match (nothing staged, generation unchanged)
|
||||||
|
- `RevisionConflictError` — an `ifRev` did not match (whole batch rejected)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `asOf(target)` → `Promise<Db>`
|
||||||
|
|
||||||
|
Open an immutable view of **past** state:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const atGen = await brain.asOf(1041) // generation number
|
||||||
|
const lastWeek = await brain.asOf(new Date(Date.now() - 7 * 86_400_000)) // wall-clock
|
||||||
|
const fromSnapshot = await brain.asOf('/backups/2026-06-01') // snapshot directory
|
||||||
|
```
|
||||||
|
|
||||||
|
- **`number`** — pins that generation; reads resolve through the immutable record layer.
|
||||||
|
- **`Date`** — resolved via the transaction log to the newest generation committed at or before it.
|
||||||
|
- **`string`** — a snapshot directory from `db.persist()`, opened as a self-contained read-only store (equivalent to `Brainy.load()`).
|
||||||
|
|
||||||
|
Historical views serve the **full query surface**. Metadata-level reads are
|
||||||
|
free; the first index-accelerated query (semantic search, traversal,
|
||||||
|
cursors, aggregation) builds an in-memory index materialization — O(n at
|
||||||
|
that generation), once per `Db`, freed on `release()`.
|
||||||
|
|
||||||
|
**Throws:** `GenerationCompactedError` when the generation's records were reclaimed by `compactHistory()`.
|
||||||
|
|
||||||
|
**History granularity:** only `transact()` batches produce historical
|
||||||
|
records; single-operation writes advance the clock but stay visible through
|
||||||
|
earlier pins. See the [consistency model](../concepts/consistency-model.md).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `transactionLog(options?)` → `Promise<TxLogEntry[]>`
|
||||||
|
|
||||||
|
Read the reified transaction log — one entry per committed `transact()`
|
||||||
|
batch, newest first: `{ generation, timestamp, meta? }`.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const [latest] = await brain.transactionLog({ limit: 1 })
|
||||||
|
latest.meta // { author: 'order-service', requestId: 'req-9f2' }
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### `commit(options?)` → `Promise<string>`
|
### `compactHistory(options?)` → `Promise<CompactHistoryResult>`
|
||||||
|
|
||||||
Create a commit snapshot.
|
Reclaim historical record-sets that no retention rule and no live `Db` pin
|
||||||
|
protects. Pinned reads stay correct across compaction, always.
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
const commitId = await brain.commit({
|
await brain.compactHistory({
|
||||||
message: 'Add new features',
|
retainGenerations: 100, // keep the 100 most recent commits
|
||||||
author: 'dev@example.com',
|
retainMs: 7 * 24 * 60 * 60 * 1000 // and everything from the last 7 days
|
||||||
metadata: { ticket: 'PROJ-123' }
|
|
||||||
})
|
})
|
||||||
```
|
```
|
||||||
|
|
||||||
**Parameters:**
|
**Returns:** `{ removedGenerations, horizon }` — `asOf()` below the horizon throws `GenerationCompactedError`.
|
||||||
- `message?`: `string` - Commit message
|
|
||||||
- `author?`: `string` - Author email
|
|
||||||
- `metadata?`: `object` - Additional commit metadata
|
|
||||||
|
|
||||||
**Returns:** `Promise<string>` - Commit ID
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
### `restore(path, { confirm: true })` → `Promise<void>`
|
||||||
|
|
||||||
### `deleteBranch(branch)` → `Promise<void>`
|
Replace the store's **entire** state from a snapshot directory. Destructive
|
||||||
|
— requires `{ confirm: true }`. All indexes are rebuilt; the generation
|
||||||
Delete a branch (cannot delete 'main').
|
counter is floored so observed generation numbers are never reissued; live
|
||||||
|
pins do not survive.
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
await brain.deleteBranch('old-experiment')
|
await brain.restore('/backups/2026-06-01', { confirm: true })
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### `getHistory(options?)` → `Promise<Commit[]>`
|
### `Brainy.load(path)` → `Promise<Db>` (static)
|
||||||
|
|
||||||
Get commit history.
|
Open a persisted snapshot as a self-contained **read-only** store with the
|
||||||
|
full query surface, including vector search. Releasing the returned `Db`
|
||||||
|
closes the underlying instance.
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
const history = await brain.getHistory({
|
const db = await Brainy.load('/backups/2026-06-01')
|
||||||
branch: 'main',
|
const hits = await db.search('quarterly invoices')
|
||||||
limit: 10
|
await db.release()
|
||||||
})
|
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### `asOf(commitId, options?)` → `Promise<Brainy>`
|
### The `Db` value
|
||||||
|
|
||||||
Create a read-only snapshot at a specific commit for time-travel queries.
|
Every `Db` is pinned at one generation and serves the full query surface at
|
||||||
|
exactly that state.
|
||||||
|
|
||||||
|
**Properties:**
|
||||||
|
|
||||||
|
| Property | Type | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `generation` | `number` | The pinned generation |
|
||||||
|
| `timestamp` | `number` | Pin time (`now()`), commit time (`transact()`), or resolved commit time (`asOf()`) |
|
||||||
|
| `receipt` | `TransactReceipt?` | Present only on `transact()` results |
|
||||||
|
| `speculative` | `boolean` | Whether this view carries a `with()` overlay |
|
||||||
|
| `released` | `boolean` | Whether `release()` has been called |
|
||||||
|
|
||||||
|
**Methods:**
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// Get commit ID from history
|
await db.get(id) // entity as of this generation
|
||||||
const commits = await brain.getHistory({ limit: 1 })
|
await db.find({ where: { status: 'open' } }) // full find() surface
|
||||||
const commitId = commits[0].id
|
await db.search('unpaid invoices') // semantic search as of this generation
|
||||||
|
await db.related(entityId) // relationships as of this generation
|
||||||
// Create snapshot (lazy-loading, no eager data loading)
|
await db.since(olderDb) // ids changed between two views
|
||||||
const snapshot = await brain.asOf(commitId, {
|
const whatIf = await db.with(ops) // speculative in-memory overlay
|
||||||
cacheSize: 10000 // LRU cache size (default: 10000)
|
await db.persist('/backups/today') // self-contained hard-link snapshot
|
||||||
})
|
await db.release() // unpin + free cached materialization
|
||||||
|
|
||||||
// Query historical state - full Triple Intelligence works!
|
|
||||||
const results = await snapshot.find({
|
|
||||||
query: 'AI research',
|
|
||||||
where: { category: 'technology' }
|
|
||||||
})
|
|
||||||
|
|
||||||
// Get historical relationships
|
|
||||||
const related = await snapshot.getRelated(entityId, { depth: 2 })
|
|
||||||
|
|
||||||
// MUST close when done to free memory
|
|
||||||
await snapshot.close()
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Parameters:**
|
- **`with(ops)`** — applies `transact()`-style operations **in memory** on
|
||||||
- `commitId`: `string` - Commit hash to snapshot from
|
top of the view; nothing touches disk, the generation counter, or index
|
||||||
- `options?`: `object`
|
providers. Overlay entities carry no embeddings, so index-accelerated
|
||||||
- `cacheSize?`: `number` - LRU cache size for lazy-loading (default: 10000)
|
queries and `persist()` on overlays throw `SpeculativeOverlayError`;
|
||||||
|
`get()`, metadata-filter `find()`, and filter-based `related()` work
|
||||||
**Returns:** `Promise<Brainy>` - Read-only Brainy instance with historical state
|
fully. Commit the same ops with `transact()` for the full surface.
|
||||||
|
- **`persist(path)`** — cuts an instant snapshot (hard links on filesystem
|
||||||
**Features:**
|
storage; byte copies across devices; in-memory stores serialize to the
|
||||||
- **Lazy-Loading** - Loads entities on-demand, not eagerly
|
same layout). Requires the view to still be the store's latest generation
|
||||||
- **Bounded Memory** - LRU cache prevents memory bloat
|
— otherwise `GenerationConflictError`.
|
||||||
- **Full Query Support** - All find(), getRelated(), etc. work on historical data
|
- **`release()`** — idempotent; after release every read throws. A
|
||||||
- **Read-Only** - Prevents accidental modifications to history
|
`FinalizationRegistry` backstop releases leaked pins at GC, but explicit
|
||||||
|
release is what makes `compactHistory()` deterministic.
|
||||||
**Important:** Always call `snapshot.close()` when done to release resources.
|
|
||||||
|
### Db API errors
|
||||||
---
|
|
||||||
|
All exported from `@soulcraft/brainy`:
|
||||||
## Entity Versioning
|
|
||||||
|
| Error | Thrown by | Meaning |
|
||||||
Git-style versioning for individual entities with content-addressable storage.
|
|---|---|---|
|
||||||
|
| `GenerationConflictError` | `transact({ ifAtGeneration })`, `db.persist()` | The store moved past the expected generation — re-read and retry |
|
||||||
### Overview
|
| `RevisionConflictError` | `update({ ifRev })`, `transact()` update ops | Per-entity revision moved — see [optimistic concurrency](../guides/optimistic-concurrency.md) |
|
||||||
|
| `GenerationCompactedError` | `asOf()` | The requested generation's records were reclaimed — persist what you must keep |
|
||||||
Entity Versioning provides time-travel and history tracking for individual entities:
|
| `SpeculativeOverlayError` | index-accelerated reads / `persist()` on `with()` overlays | Honest boundary: overlay entities carry no embeddings |
|
||||||
|
|
||||||
- **Content-Addressable Storage** - Deduplication via SHA-256 hashing
|
|
||||||
- **Zero-Config** - Lazy initialization, uses existing indexes
|
|
||||||
- **Branch-Isolated** - Versions isolated per branch
|
|
||||||
- **Selective Auto-Versioning** - Optional augmentation for automatic version creation
|
|
||||||
- **Production-Scale** - Designed for billions of entities
|
|
||||||
- **VFS File Support** - Full versioning for VFS files with actual blob content
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### `versions.save(entityId, options?)` → `Promise<EntityVersion>`
|
|
||||||
|
|
||||||
Save a new version of an entity.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Save version with tag
|
|
||||||
const version = await brain.versions.save('user-123', {
|
|
||||||
tag: 'v1.0',
|
|
||||||
description: 'Initial user profile',
|
|
||||||
metadata: { author: 'dev@example.com' }
|
|
||||||
})
|
|
||||||
|
|
||||||
console.log(version.version) // 1
|
|
||||||
console.log(version.contentHash) // SHA-256 hash
|
|
||||||
console.log(version.createdAt) // Timestamp
|
|
||||||
```
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `entityId`: `string` - Entity ID to version
|
|
||||||
- `options?`: `object`
|
|
||||||
- `tag?`: `string` - Version tag (e.g., 'v1.0', 'beta')
|
|
||||||
- `description?`: `string` - Version description
|
|
||||||
- `metadata?`: `object` - Additional version metadata
|
|
||||||
|
|
||||||
**Returns:** `Promise<EntityVersion>` - Created version
|
|
||||||
|
|
||||||
**Features:**
|
|
||||||
- Automatic deduplication (identical content = same version)
|
|
||||||
- Sequential version numbering (1, 2, 3, ...)
|
|
||||||
- Content-addressable storage (SHA-256)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### `versions.list(entityId, options?)` → `Promise<EntityVersion[]>`
|
|
||||||
|
|
||||||
List all versions of an entity.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const versions = await brain.versions.list('user-123', {
|
|
||||||
limit: 10,
|
|
||||||
offset: 0
|
|
||||||
})
|
|
||||||
|
|
||||||
versions.forEach(v => {
|
|
||||||
console.log(`Version ${v.version}: ${v.tag} - ${v.description}`)
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `entityId`: `string` - Entity ID
|
|
||||||
- `options?`: `object`
|
|
||||||
- `limit?`: `number` - Max versions to return
|
|
||||||
- `offset?`: `number` - Skip versions
|
|
||||||
|
|
||||||
**Returns:** `Promise<EntityVersion[]>` - Versions (newest first)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### `versions.restore(entityId, versionOrTag)` → `Promise<void>`
|
|
||||||
|
|
||||||
Restore entity to a previous version.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Restore by version number
|
|
||||||
await brain.versions.restore('user-123', 1)
|
|
||||||
|
|
||||||
// Restore by tag
|
|
||||||
await brain.versions.restore('user-123', 'beta')
|
|
||||||
```
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `entityId`: `string` - Entity ID
|
|
||||||
- `versionOrTag`: `number | string` - Version number or tag
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### `versions.compare(entityId, version1, version2)` → `Promise<VersionDiff>`
|
|
||||||
|
|
||||||
Compare two versions.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const diff = await brain.versions.compare('user-123', 1, 2)
|
|
||||||
|
|
||||||
console.log(diff.totalChanges) // Total changes
|
|
||||||
console.log(diff.modified) // Modified fields
|
|
||||||
console.log(diff.added) // Added fields
|
|
||||||
console.log(diff.removed) // Removed fields
|
|
||||||
|
|
||||||
// Check specific changes
|
|
||||||
const nameChange = diff.modified.find(c => c.path === 'metadata.name')
|
|
||||||
console.log(`${nameChange.oldValue} → ${nameChange.newValue}`)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Returns:** `Promise<VersionDiff>` - Detailed diff with field-level changes
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### `versions.getContent(entityId, versionOrTag)` → `Promise<EntitySnapshot>`
|
|
||||||
|
|
||||||
Get version content without restoring.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// View old version without changing current state
|
|
||||||
const v1Content = await brain.versions.getContent('user-123', 1)
|
|
||||||
console.log(v1Content.metadata.name) // Old name
|
|
||||||
|
|
||||||
// Current state unchanged
|
|
||||||
const current = await brain.get('user-123')
|
|
||||||
console.log(current.metadata.name) // Current name
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### `versions.undo(entityId)` → `Promise<void>`
|
|
||||||
|
|
||||||
Undo to previous version (shorthand for restore to latest-1).
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Make a bad change
|
|
||||||
await brain.update('user-123', { status: 'deleted' })
|
|
||||||
|
|
||||||
// Undo immediately
|
|
||||||
await brain.versions.undo('user-123')
|
|
||||||
```
|
|
||||||
|
|
||||||
**Alias:** `versions.revert(entityId)`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### `versions.prune(entityId, options)` → `Promise<PruneResult>`
|
|
||||||
|
|
||||||
Clean up old versions.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const result = await brain.versions.prune('user-123', {
|
|
||||||
keepRecent: 10, // Keep 10 most recent
|
|
||||||
keepTagged: true, // Always keep tagged versions
|
|
||||||
olderThan: Date.now() - 30 * 24 * 60 * 60 * 1000 // Older than 30 days
|
|
||||||
})
|
|
||||||
|
|
||||||
console.log(`Deleted ${result.deleted}, kept ${result.kept}`)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `keepRecent?`: `number` - Keep N most recent versions
|
|
||||||
- `keepTagged?`: `boolean` - Always keep tagged versions (default: true)
|
|
||||||
- `olderThan?`: `number` - Only prune versions older than timestamp
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### `versions.getLatest(entityId)` → `Promise<EntityVersion | null>`
|
|
||||||
|
|
||||||
Get latest version.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const latest = await brain.versions.getLatest('user-123')
|
|
||||||
if (latest) {
|
|
||||||
console.log(`Latest: v${latest.version} (${latest.tag})`)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### `versions.getVersionByTag(entityId, tag)` → `Promise<EntityVersion | null>`
|
|
||||||
|
|
||||||
Get version by tag.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const beta = await brain.versions.getVersionByTag('user-123', 'beta')
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### `versions.count(entityId)` → `Promise<number>`
|
|
||||||
|
|
||||||
Count versions for an entity.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const count = await brain.versions.count('user-123')
|
|
||||||
console.log(`${count} versions saved`)
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### `versions.hasVersions(entityId)` → `Promise<boolean>`
|
|
||||||
|
|
||||||
Check if entity has versions.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
if (await brain.versions.hasVersions('user-123')) {
|
|
||||||
console.log('Entity has version history')
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Auto-Versioning Augmentation
|
|
||||||
|
|
||||||
Automatically create versions on entity updates.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import { VersioningAugmentation } from '@soulcraft/brainy'
|
|
||||||
|
|
||||||
// Configure auto-versioning
|
|
||||||
const versioning = new VersioningAugmentation({
|
|
||||||
enabled: true,
|
|
||||||
onUpdate: true, // Version on update()
|
|
||||||
onDelete: false, // Don't version on delete
|
|
||||||
entities: ['user-*'], // Only version users
|
|
||||||
excludeEntities: ['temp-*'],
|
|
||||||
excludeTypes: ['temporary'],
|
|
||||||
keepRecent: 50, // Auto-prune old versions
|
|
||||||
keepTagged: true
|
|
||||||
})
|
|
||||||
|
|
||||||
// Apply augmentation
|
|
||||||
brain.augment(versioning)
|
|
||||||
|
|
||||||
// Now updates auto-create versions
|
|
||||||
await brain.update('user-123', { name: 'New Name' })
|
|
||||||
|
|
||||||
// Version automatically created!
|
|
||||||
const versions = await brain.versions.list('user-123')
|
|
||||||
console.log(`Auto-created version: ${versions[0].version}`)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Configuration:**
|
|
||||||
- `enabled`: `boolean` - Enable/disable augmentation
|
|
||||||
- `onUpdate`: `boolean` - Version on entity updates
|
|
||||||
- `onDelete`: `boolean` - Version before deletion
|
|
||||||
- `entities`: `string[]` - Entity ID patterns (glob-style)
|
|
||||||
- `excludeEntities`: `string[]` - Exclusion patterns
|
|
||||||
- `types`: `string[]` - Entity types to version
|
|
||||||
- `excludeTypes`: `string[]` - Types to exclude
|
|
||||||
- `keepRecent`: `number` - Auto-prune to keep N versions
|
|
||||||
- `keepTagged`: `boolean` - Always keep tagged versions
|
|
||||||
|
|
||||||
**Pattern Matching:**
|
|
||||||
- `['*']` - All entities
|
|
||||||
- `['user-*']` - All IDs starting with "user-"
|
|
||||||
- `['*-prod']` - All IDs ending with "-prod"
|
|
||||||
- `['user-*', 'account-*']` - Multiple patterns
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Branch Isolation
|
|
||||||
|
|
||||||
Versions are isolated per branch.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Save version on main
|
|
||||||
await brain.versions.save('doc-1', { tag: 'main-v1' })
|
|
||||||
|
|
||||||
// Fork and create version
|
|
||||||
const feature = await brain.fork('feature')
|
|
||||||
await feature.update('doc-1', { content: 'Feature update' })
|
|
||||||
await feature.versions.save('doc-1', { tag: 'feature-v1' })
|
|
||||||
|
|
||||||
// Versions are isolated
|
|
||||||
const mainVersions = await brain.versions.list('doc-1')
|
|
||||||
const featureVersions = await feature.versions.list('doc-1')
|
|
||||||
|
|
||||||
console.log(mainVersions.length !== featureVersions.length) // true
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Architecture
|
|
||||||
|
|
||||||
**Content-Addressable Storage:**
|
|
||||||
- SHA-256 hashing for deduplication
|
|
||||||
- Identical content = single storage blob
|
|
||||||
- Efficient for entities with few changes
|
|
||||||
|
|
||||||
**Metadata Indexing:**
|
|
||||||
- Leverages existing MetadataIndexManager
|
|
||||||
- Fast lookups by entity ID
|
|
||||||
- Version number indexing
|
|
||||||
|
|
||||||
**Storage Structure:**
|
|
||||||
```
|
|
||||||
_version:{entityId}:{versionNum}:{branch} // Version metadata
|
|
||||||
_version_blob:{contentHash} // Content blob (deduplicated)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Performance:**
|
|
||||||
- Version save: O(1) if duplicate, O(log N) for index update
|
|
||||||
- Version list: O(K) where K = version count
|
|
||||||
- Version restore: O(log N) lookup + O(1) restore
|
|
||||||
- Pruning: O(K) where K = versions pruned
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Examples
|
|
||||||
|
|
||||||
#### Basic Versioning Workflow
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Create entity
|
|
||||||
await brain.add({
|
|
||||||
data: 'User profile',
|
|
||||||
id: 'user-123',
|
|
||||||
type: 'user',
|
|
||||||
metadata: { name: 'Alice', email: 'alice@example.com' }
|
|
||||||
})
|
|
||||||
|
|
||||||
// Save v1
|
|
||||||
await brain.versions.save('user-123', { tag: 'v1.0' })
|
|
||||||
|
|
||||||
// Make changes
|
|
||||||
await brain.update('user-123', { name: 'Alice Smith' })
|
|
||||||
|
|
||||||
// Save v2
|
|
||||||
await brain.versions.save('user-123', { tag: 'v2.0' })
|
|
||||||
|
|
||||||
// Compare versions
|
|
||||||
const diff = await brain.versions.compare('user-123', 1, 2)
|
|
||||||
|
|
||||||
// Restore to v1 if needed
|
|
||||||
await brain.versions.restore('user-123', 'v1.0')
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Release Management
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Development workflow
|
|
||||||
await brain.update('app-config', { version: '1.0.0-alpha' })
|
|
||||||
await brain.versions.save('app-config', { tag: 'alpha' })
|
|
||||||
|
|
||||||
await brain.update('app-config', { version: '1.0.0-beta' })
|
|
||||||
await brain.versions.save('app-config', { tag: 'beta' })
|
|
||||||
|
|
||||||
await brain.update('app-config', { version: '1.0.0' })
|
|
||||||
await brain.versions.save('app-config', { tag: 'release' })
|
|
||||||
|
|
||||||
// Rollback to beta if issues found
|
|
||||||
await brain.versions.restore('app-config', 'beta')
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Audit Trail
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Track all changes
|
|
||||||
const versioning = new VersioningAugmentation({
|
|
||||||
enabled: true,
|
|
||||||
onUpdate: true,
|
|
||||||
entities: ['audit-*'],
|
|
||||||
keepRecent: 100 // Keep 100 versions for audit
|
|
||||||
})
|
|
||||||
|
|
||||||
brain.augment(versioning)
|
|
||||||
|
|
||||||
// All updates now tracked
|
|
||||||
await brain.update('audit-record-1', { status: 'modified' })
|
|
||||||
await brain.update('audit-record-1', { status: 'approved' })
|
|
||||||
|
|
||||||
// View complete history
|
|
||||||
const versions = await brain.versions.list('audit-record-1')
|
|
||||||
versions.forEach(v => {
|
|
||||||
console.log(`${v.createdAt}: ${v.description}`)
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
#### VFS File Versioning
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// VFS files can be versioned with actual blob content
|
|
||||||
await brain.vfs.writeFile('/docs/readme.md', 'Version 1 content')
|
|
||||||
|
|
||||||
// Get the file's entity ID
|
|
||||||
const stat = await brain.vfs.stat('/docs/readme.md')
|
|
||||||
|
|
||||||
// Save version 1
|
|
||||||
await brain.versions.save(stat.entityId, { tag: 'v1', description: 'Initial draft' })
|
|
||||||
|
|
||||||
// Modify the file
|
|
||||||
await brain.vfs.writeFile('/docs/readme.md', 'Version 2 - updated content')
|
|
||||||
|
|
||||||
// Save version 2
|
|
||||||
await brain.versions.save(stat.entityId, { tag: 'v2', description: 'Updated docs' })
|
|
||||||
|
|
||||||
// Compare versions - content is DIFFERENT
|
|
||||||
const v1 = await brain.versions.getContent(stat.entityId, 1)
|
|
||||||
const v2 = await brain.versions.getContent(stat.entityId, 2)
|
|
||||||
console.log(v1.data !== v2.data) // true
|
|
||||||
|
|
||||||
// Restore to v1 - writes content back to blob storage
|
|
||||||
await brain.versions.restore(stat.entityId, 'v1')
|
|
||||||
|
|
||||||
// File is now back to v1
|
|
||||||
const content = await brain.vfs.readFile('/docs/readme.md')
|
|
||||||
console.log(content.toString()) // 'Version 1 content'
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**[📖 Complete Versioning Guide →](../features/entity-versioning.md)**
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -1759,8 +1407,6 @@ await brain.import('https://api.example.com/data.json')
|
||||||
|
|
||||||
**Returns:** `Promise<ImportResult>` - Import statistics
|
**Returns:** `Promise<ImportResult>` - Import statistics
|
||||||
|
|
||||||
**Note:** Import always uses the current branch.
|
|
||||||
|
|
||||||
**[📖 Complete Import Guide →](../guides/import-anything.md)**
|
**[📖 Complete Import Guide →](../guides/import-anything.md)**
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -1771,12 +1417,15 @@ await brain.import('https://api.example.com/data.json')
|
||||||
// Export to file
|
// Export to file
|
||||||
await brain.export('/path/to/backup.brainy')
|
await brain.export('/path/to/backup.brainy')
|
||||||
|
|
||||||
// Create instant snapshot using COW fork
|
// Instant hard-link snapshot via the Db API
|
||||||
await brain.fork('backup-2025-01-19')
|
const pin = brain.now()
|
||||||
|
await pin.persist('/backups/2026-06-11')
|
||||||
|
await pin.release()
|
||||||
|
|
||||||
// Time-travel to specific commit
|
// Time-travel to a past generation or timestamp
|
||||||
const snapshot = await brain.asOf(commitId)
|
const snapshot = await brain.asOf(new Date('2026-06-01'))
|
||||||
const entities = await snapshot.find({ limit: 100 })
|
const entities = await snapshot.find({ limit: 100 })
|
||||||
|
await snapshot.release()
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -1819,7 +1468,7 @@ await brain.init() // Required! VFS auto-initialized
|
||||||
|
|
||||||
## Storage Adapters
|
## Storage Adapters
|
||||||
|
|
||||||
Brainy 8.0 ships two adapters — both support **copy-on-write branching**.
|
Brainy 8.0 ships two adapters — both support the full Db API (generational history, snapshots, restore).
|
||||||
|
|
||||||
### Memory (Default for Tests)
|
### Memory (Default for Tests)
|
||||||
|
|
||||||
|
|
@ -2346,26 +1995,23 @@ const results = await brain.find({
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### Git-Style Workflow
|
### Database-as-a-Value Workflow
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// Fork for experimentation
|
// Speculate: what would this change look like? (nothing touches disk)
|
||||||
const experiment = await brain.fork('test-migration')
|
const base = brain.now()
|
||||||
|
const whatIf = await base.with([
|
||||||
|
{ op: 'add', type: NounType.Document, subtype: 'note', data: 'New feature' }
|
||||||
|
])
|
||||||
|
await whatIf.find({ where: { draft: true } })
|
||||||
|
await whatIf.release()
|
||||||
|
await base.release()
|
||||||
|
|
||||||
// Make changes in isolation
|
// Commit it for real — one atomic generation, with audit metadata
|
||||||
await experiment.add({
|
await brain.transact(
|
||||||
data: 'New feature',
|
[{ op: 'add', type: NounType.Document, subtype: 'note', data: 'New feature' }],
|
||||||
type: NounType.Document
|
{ meta: { author: 'dev@example.com', message: 'Add new feature' } }
|
||||||
})
|
)
|
||||||
|
|
||||||
// Commit your work
|
|
||||||
await experiment.commit({
|
|
||||||
message: 'Add new feature',
|
|
||||||
author: 'dev@example.com'
|
|
||||||
})
|
|
||||||
|
|
||||||
// Switch to experimental branch to make it active
|
|
||||||
await brain.checkout('test-migration')
|
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -2483,22 +2129,17 @@ For the full taxonomy with all 169 types and their descriptions, see:
|
||||||
|
|
||||||
## Key Features
|
## Key Features
|
||||||
|
|
||||||
- ✅ **Entity Versioning** - Git-style versioning for individual entities
|
- ✅ **Database as a Value** - `brain.now()` pins the whole store as an immutable `Db` in O(1)
|
||||||
- ✅ **Content-Addressable Storage** - SHA-256 deduplication for versions
|
- ✅ **Atomic Transactions** - `brain.transact()` commits multi-write batches all-or-nothing
|
||||||
- ✅ **Auto-Versioning Augmentation** - Automatic version creation on updates
|
- ✅ **Two-Level CAS** - per-entity `ifRev` and whole-store `ifAtGeneration`
|
||||||
- ✅ **Branch-Isolated Versions** - Versions isolated per branch
|
- ✅ **Time Travel** - `brain.asOf()` serves the full query surface at any reachable past generation
|
||||||
- ✅ **VFS Entity Filtering** - All VFS entities now have `isVFSEntity: true` flag
|
- ✅ **Instant Snapshots** - `db.persist()` cuts hard-link snapshots; `Brainy.load()` opens them read-only
|
||||||
- ✅ **VFS Auto-Initialization** - No more separate `vfs.init()` calls
|
- ✅ **Speculative Writes** - `db.with()` answers what-if questions purely in memory
|
||||||
|
- ✅ **Reified Transaction Metadata** - audit fields recorded durably, readable via `transactionLog()`
|
||||||
|
- ✅ **VFS Entity Filtering** - All VFS entities have the `isVFSEntity: true` flag
|
||||||
|
- ✅ **VFS Auto-Initialization** - No separate `vfs.init()` calls
|
||||||
- ✅ **VFS Property Access** - Use `brain.vfs.method()` instead of `brain.vfs().method()`
|
- ✅ **VFS Property Access** - Use `brain.vfs.method()` instead of `brain.vfs().method()`
|
||||||
- ✅ **Complete COW Support** - All 20 TypeAware methods use COW helpers
|
- ✅ **Universal Storage Support** - Filesystem and memory adapters share one on-disk contract
|
||||||
- ✅ **Verified Import/Export** - Work correctly with current branch
|
|
||||||
- ✅ **Instant Fork** - Snowflake-style copy-on-write (<100ms fork time)
|
|
||||||
- ✅ **Git-Style Branching** - fork, commit, checkout, listBranches
|
|
||||||
- ✅ **Full Branch Isolation** - Parent and fork fully isolated
|
|
||||||
- ✅ **Read-Through Inheritance** - Forks see parent + own data
|
|
||||||
- ✅ **Universal Storage Support** - Filesystem and memory adapters both support branching
|
|
||||||
|
|
||||||
**[📖 Complete Changes →](../../.strategy/v5.1.0-CHANGES.md)**
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -2521,7 +2162,8 @@ For the full taxonomy with all 169 types and their descriptions, see:
|
||||||
- **[VFS Quick Start](../vfs/QUICK_START.md)** - Complete VFS documentation
|
- **[VFS Quick Start](../vfs/QUICK_START.md)** - Complete VFS documentation
|
||||||
- **[Import Anything Guide](../guides/import-anything.md)** - CSV, Excel, PDF, URL imports
|
- **[Import Anything Guide](../guides/import-anything.md)** - CSV, Excel, PDF, URL imports
|
||||||
- **[Cloud Deployment](../deployment/CLOUD_DEPLOYMENT_GUIDE.md)** - Production deployment
|
- **[Cloud Deployment](../deployment/CLOUD_DEPLOYMENT_GUIDE.md)** - Production deployment
|
||||||
- **[Instant Fork](../features/instant-fork.md)** - Git-style branching guide
|
- **[Consistency Model](../concepts/consistency-model.md)** - The guarantees behind the Db API
|
||||||
|
- **[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md)** - Backup, restore, what-if, audit recipes
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -2530,4 +2172,4 @@ For the full taxonomy with all 169 types and their descriptions, see:
|
||||||
---
|
---
|
||||||
|
|
||||||
*Brainy - The Knowledge Operating System*
|
*Brainy - The Knowledge Operating System*
|
||||||
*From prototype to planet-scale • Zero configuration • Triple Intelligence™ • Git-Style Branching*
|
*From prototype to planet-scale • Zero configuration • Triple Intelligence™ • Database as a Value*
|
||||||
|
|
|
||||||
|
|
@ -4,6 +4,17 @@
|
||||||
|
|
||||||
This document explains how Brainy stores, indexes, and scales data on disk and in memory.
|
This document explains how Brainy stores, indexes, and scales data on disk and in memory.
|
||||||
|
|
||||||
|
> **8.0 accuracy note (internal):** this reference describes the pre-8.0
|
||||||
|
> layout. In 8.0 the copy-on-write subsystem (`_cow/`, branch-scoped
|
||||||
|
> `branches/{branch}/...` paths) was replaced by generational MVCC:
|
||||||
|
> canonical entity files live at `entities/nouns/{shard}/{id}/...` with
|
||||||
|
> history under `_generations/` and the commit watermark in
|
||||||
|
> `_system/manifest.json`. The authoritative 8.0 records are
|
||||||
|
> [ADR-001 (generational MVCC)](../ADR-001-generational-mvcc.md) and
|
||||||
|
> [index-architecture.md](./index-architecture.md); the COW sections below
|
||||||
|
> are retained only as historical context until this document's full
|
||||||
|
> rewrite.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Table of Contents
|
## Table of Contents
|
||||||
|
|
|
||||||
|
|
@ -32,7 +32,7 @@ on `BaseStorage`.
|
||||||
|
|
||||||
`BaseStorage` already mixes several concerns:
|
`BaseStorage` already mixes several concerns:
|
||||||
- entity / verb CRUD primitives
|
- entity / verb CRUD primitives
|
||||||
- COW (copy-on-write) lifecycle
|
- generational record hooks (8.0 MVCC)
|
||||||
- type-statistics tracking
|
- type-statistics tracking
|
||||||
- count persistence
|
- count persistence
|
||||||
- multi-process safety (new)
|
- multi-process safety (new)
|
||||||
|
|
|
||||||
|
|
@ -275,10 +275,15 @@ class BackupAugmentation extends BaseAugmentation {
|
||||||
|
|
||||||
private async performBackup(brain?: any): Promise<void> {
|
private async performBackup(brain?: any): Promise<void> {
|
||||||
if (!brain) return
|
if (!brain) return
|
||||||
// Create instant COW snapshot
|
// Cut an instant hard-link snapshot via the Db API
|
||||||
const snapshotName = `backup-${Date.now()}`
|
const snapshotPath = `/backups/auto-${Date.now()}`
|
||||||
await brain.fork(snapshotName)
|
const pin = brain.now()
|
||||||
console.log(`Automatic snapshot created: ${snapshotName}`)
|
try {
|
||||||
|
await pin.persist(snapshotPath)
|
||||||
|
} finally {
|
||||||
|
await pin.release()
|
||||||
|
}
|
||||||
|
console.log(`Automatic snapshot created: ${snapshotPath}`)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
|
||||||
282
docs/concepts/consistency-model.md
Normal file
282
docs/concepts/consistency-model.md
Normal file
|
|
@ -0,0 +1,282 @@
|
||||||
|
---
|
||||||
|
title: Consistency Model
|
||||||
|
slug: concepts/consistency-model
|
||||||
|
public: true
|
||||||
|
category: concepts
|
||||||
|
template: concept
|
||||||
|
order: 4
|
||||||
|
description: The exact guarantees behind Brainy's Db API — snapshot isolation, atomic transactions, two levels of compare-and-swap, time travel, retention, snapshots, and crash recovery.
|
||||||
|
next:
|
||||||
|
- guides/snapshots-and-time-travel
|
||||||
|
- guides/optimistic-concurrency
|
||||||
|
---
|
||||||
|
|
||||||
|
# Consistency Model
|
||||||
|
|
||||||
|
Brainy 8.0's consistency story rests on one mechanism: **generational MVCC**
|
||||||
|
— multi-version concurrency control over immutable, generation-stamped
|
||||||
|
records. It is exposed through a single value type, the **`Db`**: an
|
||||||
|
immutable, point-in-time view of the whole store that you query like the
|
||||||
|
live brain.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const db = brain.now() // pin the current state — O(1), no I/O
|
||||||
|
|
||||||
|
await brain.transact([
|
||||||
|
{ op: 'update', id: invoiceId, metadata: { status: 'paid' } }
|
||||||
|
])
|
||||||
|
|
||||||
|
await db.get(invoiceId) // still 'pending' — pinned, forever
|
||||||
|
await brain.get(invoiceId) // 'paid' — live
|
||||||
|
await db.release() // unpin when done
|
||||||
|
```
|
||||||
|
|
||||||
|
This page states the guarantees precisely — what is promised, what it costs,
|
||||||
|
and where the honest limits are. The design record is
|
||||||
|
[ADR-001](../ADR-001-generational-mvcc.md); every guarantee below is proven
|
||||||
|
by a dedicated test in `tests/integration/db-mvcc.test.ts`.
|
||||||
|
|
||||||
|
## The generation clock
|
||||||
|
|
||||||
|
A **monotonic generation counter** is the store's logical clock:
|
||||||
|
|
||||||
|
- It advances **once per committed `transact()` batch** and once per
|
||||||
|
single-operation write (`add`/`update`/`delete`/`relate`/…).
|
||||||
|
- `brain.generation()` reads it; it is persisted in the data directory and
|
||||||
|
**never reissued** — not across restarts, and not across `restore()`
|
||||||
|
(the counter is floored at its pre-restore value).
|
||||||
|
|
||||||
|
Every `Db` is pinned at one generation. `db.generation` and `db.timestamp`
|
||||||
|
identify the view; `newerDb.since(olderDb)` returns exactly the entity and
|
||||||
|
relationship ids that committed transactions touched between two views.
|
||||||
|
|
||||||
|
## Snapshot isolation for reads
|
||||||
|
|
||||||
|
**Guarantee:** a `Db` reads exactly the state at its pinned generation, no
|
||||||
|
matter what commits afterwards — including deletes. There are no torn reads,
|
||||||
|
no partially applied batches, and no drift over time.
|
||||||
|
|
||||||
|
- `brain.now()` pins the current generation in O(1).
|
||||||
|
- `brain.transact()` returns a `Db` pinned at the freshly committed
|
||||||
|
generation.
|
||||||
|
- `brain.asOf(generation | Date | snapshotPath)` pins past state.
|
||||||
|
|
||||||
|
While nothing has committed past the pin, reads delegate to the live fast
|
||||||
|
paths — pinning is free until history actually moves. Once later
|
||||||
|
transactions commit, the view keeps serving the **full query surface** at
|
||||||
|
its generation (see "Reading the past" below).
|
||||||
|
|
||||||
|
Writers are never blocked by readers and readers never block writers: a
|
||||||
|
pinned view stays valid because nothing overwrites the immutable records it
|
||||||
|
resolves from (the LMDB reader-pin model).
|
||||||
|
|
||||||
|
## Transaction atomicity
|
||||||
|
|
||||||
|
`brain.transact(ops)` executes a declarative batch — `add`, `update`,
|
||||||
|
`remove`, `relate`, `unrelate` — **atomically as exactly one generation**:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const db = await brain.transact([
|
||||||
|
{ op: 'add', id: orderId, type: NounType.Document, subtype: 'order', data: 'Order #1042' },
|
||||||
|
{ op: 'add', id: itemId, type: NounType.Thing, subtype: 'line-item', data: 'Widget x3' },
|
||||||
|
{ op: 'relate', from: orderId, to: itemId, type: VerbType.Contains, subtype: 'order-line' }
|
||||||
|
], { meta: { author: 'order-service', requestId: 'req-9f2' } })
|
||||||
|
|
||||||
|
db.receipt.ids // resolved id per operation, in input order
|
||||||
|
```
|
||||||
|
|
||||||
|
Either every operation applies, or none do and the store is byte-identical
|
||||||
|
to its pre-transaction state. Operation semantics mirror the corresponding
|
||||||
|
single-operation methods — validation, subtype enforcement, relationship
|
||||||
|
deduplication, delete cascades — and later operations may reference ids
|
||||||
|
created earlier in the same batch.
|
||||||
|
|
||||||
|
**The commit point is one atomic rename.** The durability protocol:
|
||||||
|
|
||||||
|
1. Before-images of every touched id are staged into an immutable
|
||||||
|
generation directory and **fsynced**.
|
||||||
|
2. The batch executes through the transaction manager (which has its own
|
||||||
|
operation-level rollback for non-crash failures).
|
||||||
|
3. The store manifest is replaced via atomic temp-file rename and fsynced.
|
||||||
|
**The rename is the commit** — a generation is committed if and only if
|
||||||
|
the manifest says so.
|
||||||
|
|
||||||
|
**Crash recovery:** on the next open, any staged generation above the
|
||||||
|
manifest watermark is an uncommitted transaction; its before-images are
|
||||||
|
restored (idempotently — recovery can itself crash and rerun) and derived
|
||||||
|
indexes never observe the rolled-back state. A crash anywhere before the
|
||||||
|
rename rolls back to the exact pre-transaction bytes; a crash after it keeps
|
||||||
|
the transaction.
|
||||||
|
|
||||||
|
Transaction metadata (`meta`) is reified Datomic-style: recorded in an
|
||||||
|
append-only transaction log readable via `brain.transactionLog()` — audit
|
||||||
|
fields live in the database, not in commit messages.
|
||||||
|
|
||||||
|
## Two levels of compare-and-swap
|
||||||
|
|
||||||
|
Concurrent `transact()` calls commit serially (snapshot-isolated batches).
|
||||||
|
For lost-update protection across a read–modify–write cycle, Brainy offers
|
||||||
|
CAS at two granularities:
|
||||||
|
|
||||||
|
| Granularity | Mechanism | Conflict error | Use when |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **Per entity** | `_rev` + `{ op: 'update', ifRev }` (also on `brain.update()`) | `RevisionConflictError` | "This entity must not have changed since I read it." |
|
||||||
|
| **Whole store** | `transact(ops, { ifAtGeneration })` | `GenerationConflictError` | "*Nothing* may have committed since I read." |
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const view = brain.now()
|
||||||
|
const order = await view.get(orderId)
|
||||||
|
|
||||||
|
try {
|
||||||
|
await brain.transact(
|
||||||
|
[{ op: 'update', id: orderId, metadata: { total: recompute(order) }, ifRev: order._rev }],
|
||||||
|
{ ifAtGeneration: view.generation }
|
||||||
|
)
|
||||||
|
} catch (err) {
|
||||||
|
if (err instanceof GenerationConflictError) {
|
||||||
|
// Something committed since the pin — re-read and retry.
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
await view.release()
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
An `ifRev` conflict on any operation rejects the **whole batch**; an
|
||||||
|
`ifAtGeneration` conflict is detected before anything is staged. Both leave
|
||||||
|
the store untouched and the generation counter unchanged. See
|
||||||
|
[Optimistic concurrency with `_rev`](../guides/optimistic-concurrency.md)
|
||||||
|
for the per-entity pattern in depth.
|
||||||
|
|
||||||
|
## Reading the past
|
||||||
|
|
||||||
|
`brain.asOf()` accepts a generation number, a `Date` (resolved through the
|
||||||
|
transaction log to the newest generation committed at or before it), or a
|
||||||
|
snapshot directory path. Historical views serve the **full query surface**
|
||||||
|
— `get()`, `find()` in every mode, semantic search, graph traversal,
|
||||||
|
cursors, aggregation — through two complementary paths:
|
||||||
|
|
||||||
|
- **Record path** (free): `get()`, metadata-level `find()`, and
|
||||||
|
filter-based `related()` resolve directly through the immutable record
|
||||||
|
layer. Ids untouched since the pin still ride the live fast paths.
|
||||||
|
- **Index path** (paid once): index-accelerated queries — semantic/vector
|
||||||
|
search, graph traversal, cursors, aggregation — are served by an
|
||||||
|
**at-generation index materialization** built lazily on first use:
|
||||||
|
Brainy reconstructs in-memory indexes over the exact record set at that
|
||||||
|
generation. This costs O(n at the pinned generation) time and memory,
|
||||||
|
**once per `Db`**, cached until `release()`. That is the open-core price
|
||||||
|
of historical index queries, stated plainly.
|
||||||
|
|
||||||
|
A native index provider implementing the optional
|
||||||
|
`VersionedIndexProvider` plugin capability serves the same historical reads
|
||||||
|
from its retained index segments **without any rebuild** — the materializer
|
||||||
|
is the correctness baseline, the provider is the accelerator. Semantics are
|
||||||
|
identical on both paths.
|
||||||
|
|
||||||
|
### History granularity — the honest limit
|
||||||
|
|
||||||
|
Generation *records* are written per `transact()` batch only.
|
||||||
|
Single-operation writes (`add`/`update`/`delete`/`relate`/… outside
|
||||||
|
`transact()`) advance the generation counter — so watermarks and CAS stay
|
||||||
|
sound — but do **not** stage before-images: they remain visible through
|
||||||
|
earlier pins and are not reported by `db.since()`. Code that needs pinned
|
||||||
|
isolation across its own writes uses `transact()`. This is the documented
|
||||||
|
8.0 contract, not an accident.
|
||||||
|
|
||||||
|
## Speculative writes: `db.with()`
|
||||||
|
|
||||||
|
`db.with(ops)` returns a new `Db` whose reads see the operations applied
|
||||||
|
**in memory, on top of the view** — Datomic's `with`. Nothing touches disk,
|
||||||
|
the generation counter, or index providers:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const current = brain.now()
|
||||||
|
const whatIf = await current.with([
|
||||||
|
{ op: 'update', id: employeeId, metadata: { team: 'platform' } }
|
||||||
|
])
|
||||||
|
|
||||||
|
await whatIf.find({ where: { team: 'platform' } }) // sees the change
|
||||||
|
await brain.get(employeeId) // unchanged — nothing committed
|
||||||
|
```
|
||||||
|
|
||||||
|
**The one boundary:** overlay entities carry no embeddings (`with()` never
|
||||||
|
invokes the embedder), so index-accelerated queries and `persist()` on a
|
||||||
|
speculative view throw `SpeculativeOverlayError` rather than returning
|
||||||
|
silently incomplete results. `get()`, metadata-filter `find()`, and
|
||||||
|
filter-based `related()` work fully on overlays. To get the full surface,
|
||||||
|
commit the same operations with `brain.transact()`.
|
||||||
|
|
||||||
|
## Retention and compaction
|
||||||
|
|
||||||
|
Historical records cost disk space, so retention is explicit:
|
||||||
|
|
||||||
|
- Every live `Db` holds a refcounted **pin**; a record-set is never
|
||||||
|
reclaimed while any pin could need it — pinned reads stay correct across
|
||||||
|
compaction, always.
|
||||||
|
- `brain.compactHistory({ retainGenerations?, retainMs? })` reclaims
|
||||||
|
everything no retention rule and no pin protects, and records the
|
||||||
|
**horizon** — `asOf()` below it throws `GenerationCompactedError`,
|
||||||
|
explicitly, never partial data.
|
||||||
|
- To keep a state readable forever, `persist()` it first: snapshots are
|
||||||
|
self-contained and unaffected by compaction of the source store.
|
||||||
|
|
||||||
|
Release `Db` values you do not keep (including the ones `transact()`
|
||||||
|
returns). A `FinalizationRegistry` backstop releases leaked pins at garbage
|
||||||
|
collection, but explicit `release()` is what makes compaction
|
||||||
|
deterministic.
|
||||||
|
|
||||||
|
## Durability: snapshots and restore
|
||||||
|
|
||||||
|
`db.persist(path)` cuts a **self-contained snapshot** under the store's
|
||||||
|
commit mutex, so no commit or compaction can interleave. On filesystem
|
||||||
|
storage it is built from **hard links**: because every data file is
|
||||||
|
immutable-by-rename, linking is safe — the snapshot is created without
|
||||||
|
copying entity data, shares disk space with the source, and later writes to
|
||||||
|
the source can never alter it (rewrites swap inodes; the snapshot keeps the
|
||||||
|
old bytes). Cross-device targets fall back to byte copies; in-memory stores
|
||||||
|
serialize to the same directory layout, producing a real, durable store.
|
||||||
|
|
||||||
|
Two rules keep snapshots honest:
|
||||||
|
|
||||||
|
- `persist()` requires the view to still be the store's **latest**
|
||||||
|
generation (a snapshot captures current bytes); a view that history has
|
||||||
|
moved past throws `GenerationConflictError` instead of persisting the
|
||||||
|
wrong state.
|
||||||
|
- `brain.restore(path, { confirm: true })` replaces the store's entire
|
||||||
|
state from a snapshot via byte copy (never links — the snapshot stays
|
||||||
|
independent), rebuilds all indexes, and floors the generation counter so
|
||||||
|
observed generation numbers are never reissued. Live pins do not survive
|
||||||
|
a restore — release them first (a warning is logged when any exist).
|
||||||
|
|
||||||
|
`Brainy.load(path)` (or `brain.asOf(path)`) opens a snapshot as a
|
||||||
|
self-contained **read-only** store with the full query surface, including
|
||||||
|
vector search.
|
||||||
|
|
||||||
|
## What is not guaranteed
|
||||||
|
|
||||||
|
Stated plainly, so nothing surprises you in production:
|
||||||
|
|
||||||
|
- **Single-writer.** Brainy is a single-writer, many-reader database
|
||||||
|
([multi-process model](./multi-process.md)). Transactions are atomic
|
||||||
|
within one writer process — there is no distributed or cross-process
|
||||||
|
transaction coordination.
|
||||||
|
- **History granularity.** Only `transact()` batches produce historical
|
||||||
|
records; single-operation writes between commits stay visible through
|
||||||
|
earlier pins (see above).
|
||||||
|
- **Compacted history is gone.** `asOf()` below the compaction horizon
|
||||||
|
fails explicitly; persist what you must keep.
|
||||||
|
- **Counter persistence is coalesced for single-operation writes.** Durable
|
||||||
|
artifacts (records, manifests, snapshots) always persist the counter at
|
||||||
|
their own commit points, so a crash inside the coalescing window can lose
|
||||||
|
only counter values nothing durable ever referenced.
|
||||||
|
- **Speculative overlays are metadata-only readers.** Index-accelerated
|
||||||
|
queries on `with()` views throw rather than guess.
|
||||||
|
|
||||||
|
## Where to go next
|
||||||
|
|
||||||
|
- [Snapshots & Time Travel](../guides/snapshots-and-time-travel.md) — the
|
||||||
|
recipes: backup, restore, time-travel debugging, what-if analysis, audit
|
||||||
|
trails.
|
||||||
|
- [Optimistic concurrency with `_rev`](../guides/optimistic-concurrency.md)
|
||||||
|
— the per-entity CAS pattern.
|
||||||
|
- [ADR-001: Generational MVCC](../ADR-001-generational-mvcc.md) — the full
|
||||||
|
design record, including the persisted layout and the proof table.
|
||||||
|
|
@ -27,8 +27,8 @@ override, and the install-time failure modes that the defensive
|
||||||
```
|
```
|
||||||
BaseStorageAdapter (counts, batch ops, multi-tenancy hooks)
|
BaseStorageAdapter (counts, batch ops, multi-tenancy hooks)
|
||||||
↑
|
↑
|
||||||
BaseStorage (COW, type-statistics, lifecycle helpers,
|
BaseStorage (type-statistics, lifecycle helpers, generation
|
||||||
default no-op multi-process methods)
|
hooks, default no-op multi-process methods)
|
||||||
↑
|
↑
|
||||||
FileSystemStorage (real filesystem I/O, writer-lock implementation,
|
FileSystemStorage (real filesystem I/O, writer-lock implementation,
|
||||||
flush-request watcher, atomic writes)
|
flush-request watcher, atomic writes)
|
||||||
|
|
|
||||||
|
|
@ -59,7 +59,7 @@ Brainy can narrow any result set down by exact labels or ranges in the same brea
|
||||||
|
|
||||||
- **Live dashboards.** You can define running totals that Brainy keeps updated automatically — things like "total sales this month by region" or "average response time per service." Every time new data comes in, the numbers stay current with no manual recalculation.
|
- **Live dashboards.** You can define running totals that Brainy keeps updated automatically — things like "total sales this month by region" or "average response time per service." Every time new data comes in, the numbers stay current with no manual recalculation.
|
||||||
|
|
||||||
- **Safe experiments.** You can branch your entire knowledge base — like branching code in version control — make changes on the branch, and either keep them or throw them away without ever touching the original.
|
- **Time travel.** Every committed change becomes part of the database's history. You can pin the current state as a frozen view, see the whole knowledge base exactly as it was last week, try out changes in a scratch copy that never touches the real data, and take instant backups.
|
||||||
|
|
||||||
- **Universal vocabulary.** Brainy ships with a shared language of 42 kinds of things (Person, Document, Task, Concept, Event…) and 127 kinds of connections (Contains, DependsOn, Creates, RelatedTo…). This means data from different sources speaks the same language without you having to translate.
|
- **Universal vocabulary.** Brainy ships with a shared language of 42 kinds of things (Person, Document, Task, Concept, Event…) and 127 kinds of connections (Contains, DependsOn, Creates, RelatedTo…). This means data from different sources speaks the same language without you having to translate.
|
||||||
|
|
||||||
|
|
@ -101,11 +101,11 @@ Most applications that need to store and search knowledge end up stitching toget
|
||||||
- Plus glue code, sync jobs, ETL pipelines, and 3am incidents
|
- Plus glue code, sync jobs, ETL pipelines, and 3am incidents
|
||||||
|
|
||||||
**After Brainy** — one thing:
|
**After Brainy** — one thing:
|
||||||
Search, graph, filter, files, branches, and imports — unified in a single query.
|
Search, graph, filter, files, time travel, and imports — unified in a single library.
|
||||||
|
|
||||||
### What Each Tool Is Missing
|
### What Each Tool Is Missing
|
||||||
|
|
||||||
| Tool | Search | Graph | Filter | VFS | Branch | Import |
|
| Tool | Search | Graph | Filter | VFS | Time travel | Import |
|
||||||
|---|:---:|:---:|:---:|:---:|:---:|:---:|
|
|---|:---:|:---:|:---:|:---:|:---:|:---:|
|
||||||
| **Brainy** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
|
| **Brainy** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||||
| *— Vector databases —* | | | | | | |
|
| *— Vector databases —* | | | | | | |
|
||||||
|
|
@ -147,7 +147,7 @@ Add Cortex and you also unlock memory-mapped storage — aggregate state lives d
|
||||||
- **Searchable knowledge bases** — Build institutional memory that links documents automatically and surfaces answers across the full web of related information.
|
- **Searchable knowledge bases** — Build institutional memory that links documents automatically and surfaces answers across the full web of related information.
|
||||||
- **Semantic document search** — Index PDFs, code, or media and find them by meaning, not just keywords.
|
- **Semantic document search** — Index PDFs, code, or media and find them by meaning, not just keywords.
|
||||||
- **Relationship-aware recommendations** — Power product catalogs or content platforms where every recommendation understands what connects to what.
|
- **Relationship-aware recommendations** — Power product catalogs or content platforms where every recommendation understands what connects to what.
|
||||||
- **Safe experiments** — Let teams branch the knowledge base, experiment independently, and merge when ready — just like branching code.
|
- **Safe experiments** — Test risky changes against a scratch copy of the knowledge base, audit exactly what changed and when, and roll back to any snapshot instantly.
|
||||||
- **Unified business platforms** — Combine booking, CRM, inventory, and analytics in one queryable knowledge graph with no sync pipeline.
|
- **Unified business platforms** — Combine booking, CRM, inventory, and analytics in one queryable knowledge graph with no sync pipeline.
|
||||||
|
|
||||||
### What Brainy is good at
|
### What Brainy is good at
|
||||||
|
|
|
||||||
|
|
@ -1,693 +0,0 @@
|
||||||
# Instant Fork™
|
|
||||||
|
|
||||||
**Clone your entire Brainy database in 1-2 seconds. Test anything without fear.**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## The Problem
|
|
||||||
|
|
||||||
You need to test a risky migration. Or run an A/B experiment. Or let your team fork production data for development.
|
|
||||||
|
|
||||||
**Traditional approach:**
|
|
||||||
```javascript
|
|
||||||
// Export entire database (20 minutes)
|
|
||||||
await database.export('backup.json')
|
|
||||||
|
|
||||||
// Modify data (cross your fingers)
|
|
||||||
await database.updateAll(riskyTransformation)
|
|
||||||
|
|
||||||
// If it fails... restore from backup (another 20 minutes)
|
|
||||||
// Total downtime: 40+ minutes
|
|
||||||
```
|
|
||||||
|
|
||||||
**The pain:**
|
|
||||||
- ❌ Slow (hours for large datasets)
|
|
||||||
- ❌ Risky (one mistake = data loss)
|
|
||||||
- ❌ Expensive (full copy = 2x storage)
|
|
||||||
- ❌ Complex (manual backup/restore workflows)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## The Solution
|
|
||||||
|
|
||||||
**Brainy's Instant Fork**:
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// Clone entire database in 1-2 seconds
|
|
||||||
const experiment = await brain.fork('test-migration')
|
|
||||||
|
|
||||||
// Test your changes safely
|
|
||||||
await experiment.updateAll(riskyTransformation)
|
|
||||||
|
|
||||||
// Works? Great! Use the experimental branch.
|
|
||||||
// Failed? Just discard.
|
|
||||||
if (success) {
|
|
||||||
// Make experiment the new main branch
|
|
||||||
await brain.checkout('test-migration')
|
|
||||||
} else {
|
|
||||||
await experiment.destroy() // No harm done
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Benefits:**
|
|
||||||
- ✅ **Fast**: 1-2 seconds even with millions of entities
|
|
||||||
- ✅ **Safe**: Original data untouched
|
|
||||||
- ✅ **Cheap**: 70-90% storage savings (content-addressable blobs)
|
|
||||||
- ✅ **Simple**: One line of code
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## How It Works
|
|
||||||
|
|
||||||
Brainy uses **Snowflake-style Copy-on-Write (COW)** for instant forking:
|
|
||||||
|
|
||||||
### Architecture
|
|
||||||
|
|
||||||
1. **HNSW Index COW** (The Performance Bottleneck):
|
|
||||||
- **Shallow Copy**: O(1) Map reference copying (~10ms for 1M+ nodes)
|
|
||||||
- **Lazy Deep Copy**: Nodes copied only when modified (`ensureCOW()`)
|
|
||||||
- **Write Isolation**: Fork modifications don't affect parent
|
|
||||||
- **Implementation**: `src/hnsw/hnswIndex.ts` lines 2088-2150
|
|
||||||
|
|
||||||
2. **Metadata & Graph Indexes** (Fast Rebuild):
|
|
||||||
- **Rebuild from Storage**: < 500ms total for both indexes
|
|
||||||
- **Shared Storage**: Both indexes read from COW-enabled storage layer
|
|
||||||
- **Acceptable Overhead**: Fast enough not to need in-memory COW
|
|
||||||
|
|
||||||
3. **Storage Layer** (Shared):
|
|
||||||
- **RefManager**: Manages branch references
|
|
||||||
- **BlobStorage**: Content-addressable with deduplication
|
|
||||||
- **All Adapters**: Memory, FileSystem, S3, R2, GCS, Azure, OPFS
|
|
||||||
|
|
||||||
**Performance**:
|
|
||||||
- **Fork time**: < 100ms @ 10K entities (MEASURED in tests)
|
|
||||||
- **Storage overhead**: 10-20% (shared blobs, only changed data duplicated)
|
|
||||||
- **Memory overhead**: 10-20% (shared HNSW nodes, only modified nodes duplicated)
|
|
||||||
|
|
||||||
**Technical Details**:
|
|
||||||
```typescript
|
|
||||||
// Shallow copy HNSW (instant)
|
|
||||||
clone.index.enableCOW(this.index) // O(1) Map reference copy
|
|
||||||
|
|
||||||
// Fast rebuild small indexes from shared storage
|
|
||||||
clone.metadataIndex = new MetadataIndexManager(clone.storage) // <100ms
|
|
||||||
clone.graphIndex = new GraphAdjacencyIndex(clone.storage) // <500ms
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Basic Usage
|
|
||||||
|
|
||||||
### 1. Create a Fork
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const brain = new Brainy({ storage: { adapter: 'memory' } })
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Add some data
|
|
||||||
await brain.add({ noun: 'user', data: { name: 'Alice' } })
|
|
||||||
await brain.add({ noun: 'user', data: { name: 'Bob' } })
|
|
||||||
|
|
||||||
// Fork instantly
|
|
||||||
const fork = await brain.fork('experiment')
|
|
||||||
|
|
||||||
console.log('Fork created!', fork)
|
|
||||||
```
|
|
||||||
|
|
||||||
**What happens:**
|
|
||||||
- Original brain: unchanged
|
|
||||||
- Fork: exact copy at this moment
|
|
||||||
- Changes in fork: don't affect original
|
|
||||||
- Changes in original: don't affect fork
|
|
||||||
|
|
||||||
### 2. Work with the Fork
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// Fork is a full Brainy instance
|
|
||||||
await fork.add({ noun: 'user', data: { name: 'Charlie' } })
|
|
||||||
|
|
||||||
// All APIs work
|
|
||||||
const users = await fork.find({ noun: 'user' })
|
|
||||||
console.log(users.length) // 3 (Alice, Bob, Charlie)
|
|
||||||
|
|
||||||
// Original brain unchanged
|
|
||||||
const originalUsers = await brain.find({ noun: 'user' })
|
|
||||||
console.log(originalUsers.length) // 2 (Alice, Bob)
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Discard When Done
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// Clean up fork when finished
|
|
||||||
await fork.destroy()
|
|
||||||
|
|
||||||
// Note: fork() creates independent branches for experimentation
|
|
||||||
// Use checkout() to switch between branches or keep them separate forever
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Use Cases
|
|
||||||
|
|
||||||
### 1. Safe Migrations
|
|
||||||
|
|
||||||
**Problem**: Migrating data is risky. One mistake = data corruption.
|
|
||||||
|
|
||||||
**Solution**: Test migration in fork first.
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const brain = new Brainy({ storage: { adapter: 'filesystem', path: './data' } })
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Fork production data
|
|
||||||
const migration = await brain.fork('migration-test')
|
|
||||||
|
|
||||||
// Run migration on fork
|
|
||||||
const users = await migration.find({ noun: 'user' })
|
|
||||||
for (const user of users) {
|
|
||||||
await migration.update(user.id, {
|
|
||||||
email: user.data.email.toLowerCase(), // Normalize emails
|
|
||||||
verified: user.data.verified ?? false // Add missing field
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// Validate migration
|
|
||||||
const allValid = (await migration.find({ noun: 'user' }))
|
|
||||||
.every(u => u.data.email === u.data.email.toLowerCase())
|
|
||||||
|
|
||||||
if (allValid) {
|
|
||||||
console.log('✅ Migration safe! Apply changes to production brain')
|
|
||||||
// Apply validated migration to main brain
|
|
||||||
const users = await brain.find({ noun: 'user' })
|
|
||||||
for (const user of users) {
|
|
||||||
await brain.update(user.id, {
|
|
||||||
email: user.data.email.toLowerCase(),
|
|
||||||
verified: user.data.verified ?? false
|
|
||||||
})
|
|
||||||
}
|
|
||||||
await migration.destroy()
|
|
||||||
} else {
|
|
||||||
console.log('❌ Migration failed! Discarding fork.')
|
|
||||||
await migration.destroy()
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. A/B Testing
|
|
||||||
|
|
||||||
**Problem**: Need to test two different algorithms on the same data.
|
|
||||||
|
|
||||||
**Solution**: Create two forks, run experiments in parallel.
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const brain = new Brainy({ storage: { adapter: 's3', bucket: 'production' } })
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Create two variants
|
|
||||||
const variantA = await brain.fork('variant-a') // Control
|
|
||||||
const variantB = await brain.fork('variant-b') // Test
|
|
||||||
|
|
||||||
// Run different algorithms
|
|
||||||
await variantA.processWithAlgorithm('current')
|
|
||||||
await variantB.processWithAlgorithm('improved')
|
|
||||||
|
|
||||||
// Compare results
|
|
||||||
const metricsA = await variantA.getMetrics()
|
|
||||||
const metricsB = await variantB.getMetrics()
|
|
||||||
|
|
||||||
console.log('Variant A accuracy:', metricsA.accuracy)
|
|
||||||
console.log('Variant B accuracy:', metricsB.accuracy)
|
|
||||||
|
|
||||||
// Choose winner and update main brain
|
|
||||||
if (metricsB.accuracy > metricsA.accuracy) {
|
|
||||||
console.log('B wins! Apply algorithm B to production')
|
|
||||||
await brain.processWithAlgorithm('improved')
|
|
||||||
await variantA.destroy()
|
|
||||||
await variantB.destroy()
|
|
||||||
} else {
|
|
||||||
console.log('A wins! Keeping current algorithm.')
|
|
||||||
await variantA.destroy()
|
|
||||||
await variantB.destroy()
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Distributed Development
|
|
||||||
|
|
||||||
**Problem**: Multiple developers need to work with production data.
|
|
||||||
|
|
||||||
**Solution**: Each developer gets their own fork.
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// Main production brain
|
|
||||||
const production = new Brainy({ storage: { adapter: 's3', bucket: 'prod' } })
|
|
||||||
await production.init()
|
|
||||||
|
|
||||||
// Alice's feature branch
|
|
||||||
const aliceBranch = await production.fork('alice-feature-x')
|
|
||||||
|
|
||||||
// Bob's feature branch
|
|
||||||
const bobBranch = await production.fork('bob-feature-y')
|
|
||||||
|
|
||||||
// Both work independently (zero conflicts!)
|
|
||||||
await aliceBranch.add({ noun: 'feature', data: { name: 'X' } })
|
|
||||||
await bobBranch.add({ noun: 'feature', data: { name: 'Y' } })
|
|
||||||
|
|
||||||
// When ready, manually copy validated changes to production
|
|
||||||
await production.add({ noun: 'feature', data: { name: 'X' } })
|
|
||||||
await production.add({ noun: 'feature', data: { name: 'Y' } })
|
|
||||||
|
|
||||||
await aliceBranch.destroy()
|
|
||||||
await bobBranch.destroy()
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. Instant Backup/Restore
|
|
||||||
|
|
||||||
**Problem**: Need fast backups before risky operations.
|
|
||||||
|
|
||||||
**Solution**: Fork as backup.
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const brain = new Brainy({ storage: { adapter: 'memory' } })
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Work with production data
|
|
||||||
await brain.add({ noun: 'important', data: { value: 'critical' } })
|
|
||||||
|
|
||||||
// Instant backup (1-2 seconds)
|
|
||||||
const backup = await brain.fork('backup-before-delete')
|
|
||||||
|
|
||||||
// Do risky operation
|
|
||||||
const entities = await brain.find({ noun: 'important' })
|
|
||||||
await brain.delete(entities[0].id)
|
|
||||||
|
|
||||||
// Oops! Need to restore
|
|
||||||
// Just discard current, use backup
|
|
||||||
await brain.destroy()
|
|
||||||
|
|
||||||
// Restore from backup (or switch to backup branch)
|
|
||||||
const restored = await backup.fork('main')
|
|
||||||
|
|
||||||
console.log('✅ Data restored!')
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5. Snapshot Testing
|
|
||||||
|
|
||||||
**Problem**: Need to test code against specific data states.
|
|
||||||
|
|
||||||
**Solution**: Create fork snapshots before making changes.
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const brain = new Brainy({ storage: { adapter: 'memory' } })
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Create initial state
|
|
||||||
await brain.add({ noun: 'doc', data: { version: 1 } })
|
|
||||||
|
|
||||||
// Take snapshot before changes
|
|
||||||
const snapshot = await brain.fork('before-update')
|
|
||||||
|
|
||||||
// Make changes to main brain
|
|
||||||
const docs = await brain.find({ noun: 'doc' })
|
|
||||||
await brain.update(docs[0].id, { version: 2 })
|
|
||||||
|
|
||||||
// Test against original state
|
|
||||||
const originalData = await snapshot.find({ noun: 'doc' })
|
|
||||||
console.log(originalData[0].data.version) // 1 (original state!)
|
|
||||||
|
|
||||||
// Clean up
|
|
||||||
await snapshot.destroy()
|
|
||||||
|
|
||||||
// Note: Time-travel queries (asOf) Planned
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Advanced Features
|
|
||||||
|
|
||||||
### Fork Options
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// Custom branch name
|
|
||||||
const fork1 = await brain.fork('my-experiment')
|
|
||||||
|
|
||||||
// Auto-generated name (uses timestamp)
|
|
||||||
const fork2 = await brain.fork() // 'fork-1635789012345'
|
|
||||||
|
|
||||||
// Fork with metadata (for tracking)
|
|
||||||
const fork3 = await brain.fork('test', {
|
|
||||||
author: 'Alice',
|
|
||||||
message: 'Testing new feature'
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
### Branch Management
|
|
||||||
|
|
||||||
Full branch management now available!
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// List all branches
|
|
||||||
const branches = await brain.listBranches()
|
|
||||||
console.log(branches) // ['main', 'experiment', 'test']
|
|
||||||
|
|
||||||
// Get current branch
|
|
||||||
const current = await brain.getCurrentBranch()
|
|
||||||
console.log(current) // 'main'
|
|
||||||
|
|
||||||
// Switch between branches
|
|
||||||
await brain.checkout('experiment')
|
|
||||||
|
|
||||||
// Delete a branch
|
|
||||||
await brain.deleteBranch('old-experiment')
|
|
||||||
```
|
|
||||||
|
|
||||||
### Commit Tracking
|
|
||||||
|
|
||||||
Git-style commit tracking!
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// Create a commit (snapshot of current state)
|
|
||||||
await brain.add({ type: 'user', data: { name: 'Alice' } })
|
|
||||||
|
|
||||||
const commitHash = await brain.commit({
|
|
||||||
message: 'Add Alice user',
|
|
||||||
author: 'dev@example.com'
|
|
||||||
})
|
|
||||||
|
|
||||||
console.log(commitHash) // 'a3f2c1b9...'
|
|
||||||
```
|
|
||||||
|
|
||||||
### Commit History
|
|
||||||
|
|
||||||
View commit history!
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// Get commit history for current branch
|
|
||||||
const history = await brain.getHistory({ limit: 10 })
|
|
||||||
|
|
||||||
history.forEach(commit => {
|
|
||||||
console.log(`${commit.hash}: ${commit.message}`)
|
|
||||||
console.log(` By: ${commit.author} at ${new Date(commit.timestamp)}`)
|
|
||||||
})
|
|
||||||
|
|
||||||
## Performance Characteristics
|
|
||||||
|
|
||||||
### Fork Speed (Measured)
|
|
||||||
|
|
||||||
| Entities | Traditional Copy | Brainy Fork | Speedup |
|
|
||||||
|----------|------------------|-------------|---------|
|
|
||||||
| 1,000 | 2-5 seconds | 0.5 seconds | 4-10x |
|
|
||||||
| 10,000 | 20-40 seconds | 0.8 seconds | 25-50x |
|
|
||||||
| 100,000 | 3-5 minutes | 1.2 seconds | 150-250x |
|
|
||||||
| 1,000,000 | 30-60 minutes | 1.8 seconds | 1000-2000x |
|
|
||||||
|
|
||||||
### Storage Overhead
|
|
||||||
|
|
||||||
```
|
|
||||||
Scenario: 1M entities, 10 forks
|
|
||||||
|
|
||||||
Traditional: 10 full copies = 80GB × 10 = 800GB
|
|
||||||
Brainy: 1 base + 10% changes = 80GB + 8GB = 88GB
|
|
||||||
|
|
||||||
Savings: 89% less storage
|
|
||||||
```
|
|
||||||
|
|
||||||
### Memory Overhead
|
|
||||||
|
|
||||||
```
|
|
||||||
Scenario: 1M entities in memory
|
|
||||||
|
|
||||||
Traditional fork: 2x memory (10GB → 20GB)
|
|
||||||
Brainy fork: 1.2x memory (10GB → 12GB)
|
|
||||||
|
|
||||||
Savings: 40% less memory
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Zero Configuration
|
|
||||||
|
|
||||||
**Fork is enabled by default. No setup required.**
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// This is all you need:
|
|
||||||
const brain = new Brainy({ storage: { adapter: 'memory' } })
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Fork is ready to use:
|
|
||||||
const fork = await brain.fork()
|
|
||||||
|
|
||||||
// That's it!
|
|
||||||
```
|
|
||||||
|
|
||||||
**Automatic optimizations:**
|
|
||||||
- ✅ Compression: zstd for metadata, none for vectors (automatic)
|
|
||||||
- ✅ Deduplication: content-addressable (automatic)
|
|
||||||
- ✅ Caching: LRU with memory limits (automatic)
|
|
||||||
- ✅ Garbage collection: cleanup unused blobs (automatic)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Integration with Brainy Features
|
|
||||||
|
|
||||||
### Works with All Storage Adapters
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// Memory
|
|
||||||
await new Brainy({ storage: { adapter: 'memory' } }).fork()
|
|
||||||
|
|
||||||
// FileSystem
|
|
||||||
await new Brainy({ storage: { adapter: 'filesystem', path: './data' } }).fork()
|
|
||||||
|
|
||||||
// S3
|
|
||||||
await new Brainy({ storage: { adapter: 's3', bucket: 'my-data' } }).fork()
|
|
||||||
|
|
||||||
// All adapters supported: Memory, OPFS, FileSystem, S3, R2, GCS, Azure, TypeAware
|
|
||||||
```
|
|
||||||
|
|
||||||
### Works with find(), VFS, Triple Intelligence
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const brain = new Brainy({
|
|
||||||
storage: { adapter: 'memory' },
|
|
||||||
vfs: { enabled: true },
|
|
||||||
intelligence: { enabled: true }
|
|
||||||
})
|
|
||||||
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Create VFS files
|
|
||||||
await brain.vfs.writeFile('/project/README.md', '# My Project')
|
|
||||||
|
|
||||||
// Add entities
|
|
||||||
await brain.add({ noun: 'user', data: { name: 'Alice' } })
|
|
||||||
|
|
||||||
// Fork everything
|
|
||||||
const fork = await brain.fork('test')
|
|
||||||
|
|
||||||
// All features work on fork:
|
|
||||||
await fork.vfs.readFile('/project/README.md') // ✅ VFS
|
|
||||||
await fork.find({ noun: 'user' }) // ✅ find()
|
|
||||||
await fork.query('users named Alice') // ✅ Triple Intelligence
|
|
||||||
```
|
|
||||||
|
|
||||||
### Works at Billion Scale
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// Tested at 1M entities, extrapolates to 1B
|
|
||||||
const brain = new Brainy({
|
|
||||||
storage: { adapter: 'gcs', bucket: 'billion-scale' },
|
|
||||||
hnsw: { typeAware: true } // 87% memory reduction
|
|
||||||
})
|
|
||||||
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Fork 1B entities: still < 2 seconds
|
|
||||||
const fork = await brain.fork()
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## FAQ
|
|
||||||
|
|
||||||
### Q: Does fork() copy all data?
|
|
||||||
|
|
||||||
**A: No.** Fork uses copy-on-write (COW). Unchanged data is shared between parent and fork via content-addressable blobs. Only modified data creates new blobs.
|
|
||||||
|
|
||||||
### Q: Is fork() safe for production?
|
|
||||||
|
|
||||||
**A: Yes.** Fork is battle-tested at scale. Uses proven Git-like COW technology. Zero risk to original data.
|
|
||||||
|
|
||||||
### Q: Does it work with all storage adapters?
|
|
||||||
|
|
||||||
**A: Yes.** Fork works with Memory, OPFS, FileSystem, S3, R2, GCS, Azure, and TypeAware adapters.
|
|
||||||
|
|
||||||
### Q: What happens to the fork if I modify the original?
|
|
||||||
|
|
||||||
**A: Nothing.** Fork is isolated. Changes in parent don't affect fork. Changes in fork don't affect parent.
|
|
||||||
|
|
||||||
### Q: Can I merge forks back to main?
|
|
||||||
|
|
||||||
**A: Use the "experimental branching" paradigm.** Instead of merging, either (1) make your experimental branch the new main with `checkout()`, or (2) manually copy specific entities you want. See CHANGELOG v6.0.0 for migration patterns.
|
|
||||||
|
|
||||||
### Q: How long are forks kept?
|
|
||||||
|
|
||||||
**A: Forever (or until you delete them).** Forks persist like branches. Delete with `fork.destroy()` or set retention policy (Enterprise).
|
|
||||||
|
|
||||||
### Q: What's the performance impact?
|
|
||||||
|
|
||||||
**A: Minimal.** Fork time: 1-2 seconds @ 1M entities. Storage: 10-20% overhead. Memory: 20-40% overhead.
|
|
||||||
|
|
||||||
### Q: Can I fork a fork?
|
|
||||||
|
|
||||||
**A: Yes.** Fork anything, anytime. Create branch trees as deep as needed.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Comparison to Other Databases
|
|
||||||
|
|
||||||
### vs PostgreSQL
|
|
||||||
|
|
||||||
**PostgreSQL:**
|
|
||||||
```sql
|
|
||||||
-- Create copy (full table scan, minutes)
|
|
||||||
CREATE TABLE users_backup AS SELECT * FROM users;
|
|
||||||
|
|
||||||
-- Modify (risky!)
|
|
||||||
UPDATE users SET email = LOWER(email);
|
|
||||||
|
|
||||||
-- Restore (if failed)
|
|
||||||
DROP TABLE users;
|
|
||||||
ALTER TABLE users_backup RENAME TO users;
|
|
||||||
```
|
|
||||||
|
|
||||||
**Brainy:**
|
|
||||||
```javascript
|
|
||||||
const fork = await brain.fork('test')
|
|
||||||
await fork.updateAll({ email: (u) => u.email.toLowerCase() })
|
|
||||||
if (success) await brain.checkout('test') // Make test branch active
|
|
||||||
else await fork.destroy()
|
|
||||||
```
|
|
||||||
|
|
||||||
**Winner: Brainy** (1000x faster, safer)
|
|
||||||
|
|
||||||
### vs MongoDB
|
|
||||||
|
|
||||||
**MongoDB:**
|
|
||||||
```javascript
|
|
||||||
// No native fork/clone
|
|
||||||
// Must manually export/import
|
|
||||||
|
|
||||||
// Export (slow)
|
|
||||||
mongoexport --db mydb --collection users --out users.json
|
|
||||||
|
|
||||||
// Import to new collection (slow)
|
|
||||||
mongoimport --db mydb --collection users_backup --file users.json
|
|
||||||
```
|
|
||||||
|
|
||||||
**Brainy:**
|
|
||||||
```javascript
|
|
||||||
const fork = await brain.fork() // Done!
|
|
||||||
```
|
|
||||||
|
|
||||||
**Winner: Brainy** (100x faster, built-in)
|
|
||||||
|
|
||||||
### vs Pinecone/Weaviate
|
|
||||||
|
|
||||||
**Pinecone/Weaviate:**
|
|
||||||
```
|
|
||||||
❌ No fork/clone feature at all
|
|
||||||
❌ Manual backup/restore only
|
|
||||||
❌ Downtime required for testing
|
|
||||||
```
|
|
||||||
|
|
||||||
**Brainy:**
|
|
||||||
```javascript
|
|
||||||
✅ Fork in 1-2 seconds
|
|
||||||
✅ Zero downtime
|
|
||||||
✅ Zero risk
|
|
||||||
```
|
|
||||||
|
|
||||||
**Winner: Brainy** (only vector DB with instant fork)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## What's Implemented vs. What's Next
|
|
||||||
|
|
||||||
### ✅ Available in - ✅ `fork()` - Instant clone in <100ms
|
|
||||||
- ✅ `listBranches()` - List all forks
|
|
||||||
- ✅ `getCurrentBranch()` - Get active branch
|
|
||||||
- ✅ `checkout()` - Switch between branches
|
|
||||||
- ✅ `deleteBranch()` - Delete branches
|
|
||||||
- ✅ `commit()` - Create state snapshots
|
|
||||||
- ✅ `getHistory()` - View commit history
|
|
||||||
|
|
||||||
### 🔮 Planned for:
|
|
||||||
|
|
||||||
**Temporal Features:**
|
|
||||||
- `asOf(timestamp)` - Query data at specific time (✅ available)
|
|
||||||
- `rollback(commitHash)` - Restore to previous state
|
|
||||||
- Full audit trail for all changes
|
|
||||||
|
|
||||||
These features require additional temporal infrastructure and are being carefully designed
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## CLI Support
|
|
||||||
|
|
||||||
All fork/merge/commit features are available via CLI:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Fork (instant clone)
|
|
||||||
brainy fork feature-x --message "Testing new feature" --author "dev@example.com"
|
|
||||||
|
|
||||||
# List branches
|
|
||||||
brainy branch list
|
|
||||||
|
|
||||||
# Switch branches
|
|
||||||
brainy checkout feature-x
|
|
||||||
|
|
||||||
# Create commit
|
|
||||||
brainy commit --message "Add new feature" --author "dev@example.com"
|
|
||||||
|
|
||||||
# View history
|
|
||||||
brainy history --limit 10
|
|
||||||
|
|
||||||
# Merge branches
|
|
||||||
brainy merge feature-x main --strategy last-write-wins
|
|
||||||
|
|
||||||
# Delete branch
|
|
||||||
brainy branch delete old-feature --force
|
|
||||||
```
|
|
||||||
|
|
||||||
## Try It Now
|
|
||||||
|
|
||||||
```bash
|
|
||||||
npm install @soulcraft/brainy
|
|
||||||
```
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
|
||||||
|
|
||||||
const brain = new Brainy()
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
await brain.add({ noun: 'test', data: { value: 1 } })
|
|
||||||
|
|
||||||
const fork = await brain.fork('experiment')
|
|
||||||
|
|
||||||
console.log('Fork created in < 2 seconds! 🚀')
|
|
||||||
```
|
|
||||||
|
|
||||||
**Zero config. Zero complexity. Pure power.**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Learn More
|
|
||||||
|
|
||||||
- [COW Architecture](../architecture/copy-on-write.md)
|
|
||||||
- [Performance Benchmarks](../benchmarks/fork-performance.md)
|
|
||||||
- [Enterprise Features](../enterprise/temporal-cloning.md)
|
|
||||||
- [API Reference](../api/fork.md)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Brainy** | [GitHub](https://github.com/soulcraftlabs/brainy) | [npm](https://npmjs.com/package/@soulcraft/brainy)
|
|
||||||
|
|
@ -134,8 +134,8 @@ console.log(health.overall)
|
||||||
await reader.close()
|
await reader.close()
|
||||||
```
|
```
|
||||||
|
|
||||||
Every mutation method (`add`, `update`, `delete`, `relate`, `commit`,
|
Every mutation method (`add`, `update`, `delete`, `relate`, `transact`,
|
||||||
`fork`, `branch`, ...) throws on a read-only instance with a clear message.
|
`restore`, ...) throws on a read-only instance with a clear message.
|
||||||
|
|
||||||
## Backups
|
## Backups
|
||||||
|
|
||||||
|
|
@ -148,9 +148,10 @@ brainy inspect backup /data/brain /backups/brain-2026-05-15.tar
|
||||||
```
|
```
|
||||||
|
|
||||||
For periodic backups (hourly, daily), schedule this via cron or your
|
For periodic backups (hourly, daily), schedule this via cron or your
|
||||||
container scheduler. For point-in-time recovery, use Brainy's COW
|
container scheduler. For point-in-time recovery, use the Db API's
|
||||||
`commit()` API — the snapshots there are content-addressed and never
|
`db.persist(path)` — a self-contained hard-link snapshot that later writes
|
||||||
overwritten.
|
can never alter, restorable with `brain.restore(path, { confirm: true })`.
|
||||||
|
See [Snapshots & Time Travel](./snapshots-and-time-travel.md).
|
||||||
|
|
||||||
## Comparing two stores
|
## Comparing two stores
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -7,8 +7,8 @@ template: guide
|
||||||
order: 8
|
order: 8
|
||||||
description: Use the per-entity `_rev` counter and `update({ ifRev })` to coordinate concurrent writes safely. Covers the lock pattern, idempotent inserts with `ifAbsent`, and recovery on conflict.
|
description: Use the per-entity `_rev` counter and `update({ ifRev })` to coordinate concurrent writes safely. Covers the lock pattern, idempotent inserts with `ifAbsent`, and recovery on conflict.
|
||||||
next:
|
next:
|
||||||
|
- concepts/consistency-model
|
||||||
- guides/find-limits
|
- guides/find-limits
|
||||||
- api/README
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Optimistic concurrency with `_rev`
|
# Optimistic concurrency with `_rev`
|
||||||
|
|
@ -25,7 +25,7 @@ Brainy 7.31.0 adds a per-entity revision counter so multiple writers can coordin
|
||||||
| `add({ id, ifAbsent: true })` | By-ID idempotent insert. Returns the existing `id` if one is already present; no throw, no overwrite. |
|
| `add({ id, ifAbsent: true })` | By-ID idempotent insert. Returns the existing `id` if one is already present; no throw, no overwrite. |
|
||||||
| `addMany({ items, ifAbsent: true })` | Applies `ifAbsent` to every item. Per-item `ifAbsent` overrides the batch flag. |
|
| `addMany({ items, ifAbsent: true })` | Applies `ifAbsent` to every item. Per-item `ifAbsent` overrides the batch flag. |
|
||||||
|
|
||||||
`_rev` is independent of `brain.versions.*` (named snapshots), of `brain.fork()` / branches (COW), and of VFS file versioning. They all coexist; `_rev` is the per-write counter used for CAS.
|
`_rev` is the **per-entity** counter. Its store-wide counterpart is the generation counter behind the [Db API](../concepts/consistency-model.md): `brain.transact(ops, { ifAtGeneration })` is CAS over the whole store, `update({ ifRev })` (and `ifRev` on `transact()` update operations) is CAS over one entity.
|
||||||
|
|
||||||
## The lock pattern
|
## The lock pattern
|
||||||
|
|
||||||
|
|
@ -134,42 +134,49 @@ await brain.addIfMissing({ // ← not a real API
|
||||||
})
|
})
|
||||||
```
|
```
|
||||||
|
|
||||||
It's race-prone outside a transaction: two concurrent imports both see "not found," both insert, you get duplicates. Without a unique-index primitive (which Brainy doesn't have today), this pattern needs to live inside a transaction:
|
It's race-prone as a plain read-then-write: two concurrent imports both see "not found," both insert, you get duplicates. Without a unique-index primitive (which Brainy doesn't have today), close the race with whole-store CAS — read at a pinned generation, then commit only if nothing moved:
|
||||||
|
|
||||||
```ts
|
```ts
|
||||||
// The race-safe shape, available once brain.transact() ships in 8.0.
|
import { GenerationConflictError } from '@soulcraft/brainy'
|
||||||
await brain.transact(async tx => {
|
|
||||||
const existing = await tx.find({
|
async function addIfMissingByEmail(email: string, data: string) {
|
||||||
type: 'Person',
|
for (let attempt = 0; attempt < 5; attempt++) {
|
||||||
where: { email: 'x@y.com' },
|
const db = brain.now()
|
||||||
|
try {
|
||||||
|
const existing = await db.find({
|
||||||
|
type: NounType.Person,
|
||||||
|
where: { email },
|
||||||
limit: 1
|
limit: 1
|
||||||
})
|
})
|
||||||
if (existing.length === 0) {
|
if (existing.length > 0) return existing[0].id
|
||||||
await tx.add({ type: 'Person', data: '...', metadata: { email: 'x@y.com' } })
|
|
||||||
|
const committed = await brain.transact(
|
||||||
|
[{ op: 'add', type: NounType.Person, subtype: 'customer', data, metadata: { email } }],
|
||||||
|
{ ifAtGeneration: db.generation } // rejects if ANYTHING committed since the read
|
||||||
|
)
|
||||||
|
return committed.receipt!.ids[0]
|
||||||
|
} catch (err) {
|
||||||
|
if (err instanceof GenerationConflictError) continue // world moved — re-read + retry
|
||||||
|
throw err
|
||||||
|
} finally {
|
||||||
|
await db.release()
|
||||||
}
|
}
|
||||||
})
|
}
|
||||||
|
throw new Error('addIfMissingByEmail conflict after 5 attempts')
|
||||||
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
For 7.31.0, lean on `ifAbsent` when you control the ID, and accept the inherent dedup race when you don't. The 8.0 `brain.transact()` is the natural home for the attribute-based variant.
|
`ifAtGeneration` is deliberately coarse — *any* committed write invalidates it — so keep the retry bound. When you control the ID, `ifAbsent` stays the cheaper tool.
|
||||||
|
|
||||||
## How `_rev` interacts with the other versioning systems
|
## How `_rev` relates to generations
|
||||||
|
|
||||||
Brainy has several persistence primitives that all touch the word "version" in different ways. `_rev` is independent of every one of them:
|
Brainy 8.0 has exactly two write-coordination counters, at two granularities:
|
||||||
|
|
||||||
| System | What it tracks | When it advances |
|
| Counter | Scope | What it tracks | CAS surface | Conflict error |
|
||||||
|---|---|---|
|
|---|---|---|---|---|
|
||||||
| **`_rev`** (new in 7.31.0) | Per-entity write counter | Auto, on every successful `update()` |
|
| **`_rev`** | One entity | Per-entity write count, bumped on every successful update | `update({ ifRev })`, `{ op: 'update', ifRev }` in `transact()` | `RevisionConflictError` |
|
||||||
| **`brain.versions.save()`** | Named snapshots per entity | Explicit — you call `save()` with a tag |
|
| **Generation** | Whole store | One tick per committed `transact()` batch or single-operation write | `transact(ops, { ifAtGeneration })` | `GenerationConflictError` |
|
||||||
| **`brain.fork()` / branches** | Whole-brain copy-on-write | Explicit — you call `fork(name)` |
|
|
||||||
| **VFS file versioning** | Per-VFS-file snapshots (Document entity) | Same as `brain.versions.save()` |
|
|
||||||
|
|
||||||
If you're on a branch, each branch has its own copy of every entity (that's what COW means), so each branch has its own `_rev` per entity — same as every other field. Snapshots taken via `brain.versions.save()` capture the entity at that moment including its `_rev` at that time; the snapshot's own `version: number` is the snapshot index, distinct from `_rev`.
|
They compose: a `transact()` batch can carry per-entity `ifRev` checks *and* a whole-store `ifAtGeneration`; any failed check rejects the entire batch before anything is staged. Generations also power snapshots and time travel (`brain.now()`, `brain.asOf()`, `db.persist()`) — see the [consistency model](../concepts/consistency-model.md) and [Snapshots & Time Travel](./snapshots-and-time-travel.md).
|
||||||
|
|
||||||
## What's coming in 8.0
|
A snapshot or historical view captures each entity *including* its `_rev` at that moment, so reading the past and writing back with `ifRev` against the live state works exactly as you'd hope: the write fails if the entity moved since the state you copied from.
|
||||||
|
|
||||||
Brainy 8.0 ships a Datomic-style immutable `Db` API where `brain.transact(tx)` is the public multi-write atomicity primitive. Two things change for code written against 7.31.0's surface:
|
|
||||||
|
|
||||||
- `_rev` survives unchanged. It's part of the 8.0 entity shape; the auto-bump moves to the new `transact()` write path. Code that uses `update({ ifRev })` keeps working.
|
|
||||||
- A new `brain.transact(tx, { ifAtGeneration: prev.generation })` adds generation-based CAS for whole-transaction atomicity, layered on top of `_rev`. Per-entity for single-record patterns (the lock above), generation-based for "did the world move under me."
|
|
||||||
|
|
||||||
If you adopt `_rev` + `ifRev` in 7.31.0, no migration work is needed for 8.0.
|
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# Schema Migrations
|
# Schema Migrations
|
||||||
|
|
||||||
Brainy includes a built-in migration system for transforming entity and verb metadata across storage versions. Migrations are pure functions that run once per storage instance, with automatic backup, resume support, and error tracking.
|
Brainy includes a built-in migration system for transforming entity and verb metadata across storage versions. Migrations are pure functions that run once per storage instance, with optional snapshot backup (`backupTo`), resume support, and error tracking.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -48,15 +48,13 @@ When `brain.init()` runs:
|
||||||
|
|
||||||
When `brain.migrate()` runs:
|
When `brain.migrate()` runs:
|
||||||
|
|
||||||
1. **Backup** — creates an instant COW branch (`pre-migration-7.17.0`) tagged with `system:backup` metadata. Rollback is possible by switching to this branch.
|
1. **Backup (optional)** — with `backupTo`, a hard-link snapshot of the current generation is persisted before any transform runs. Rollback is `brain.restore(backupPath, { confirm: true })`.
|
||||||
|
|
||||||
2. **Transform main branch** — iterates all nouns/verbs in paginated batches. For each entity, calls the `transform` function. If it returns a new object, saves it. If it returns `null`, skips. Vectors are never touched.
|
2. **Transform** — iterates all nouns/verbs in paginated batches. For each entity, calls the `transform` function. If it returns a new object, saves it. If it returns `null`, skips. Vectors are never touched.
|
||||||
|
|
||||||
3. **Transform other branches** — switches to each user branch, runs the same transforms. Inherited (already-migrated) entities return `null` and are skipped automatically.
|
3. **Save state** — records each completed migration ID so it never re-runs.
|
||||||
|
|
||||||
4. **Save state** — records each completed migration ID so it never re-runs.
|
4. **Rebuild indexes** — if any entities were modified, rebuilds the MetadataIndex.
|
||||||
|
|
||||||
5. **Rebuild indexes** — if any entities were modified, rebuilds the MetadataIndex.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -78,7 +76,7 @@ interface Migration {
|
||||||
|
|
||||||
- **Return a new object** to modify the entity's metadata.
|
- **Return a new object** to modify the entity's metadata.
|
||||||
- **Return `null`** to skip (no change needed).
|
- **Return `null`** to skip (no change needed).
|
||||||
- **Must be idempotent** — running the same transform twice on the same data should produce the same result (or return `null` the second time). This is required because branch iterations may re-encounter inherited entities.
|
- **Must be idempotent** — running the same transform twice on the same data should produce the same result (or return `null` the second time). This is required because interrupted runs resume and re-encounter already-migrated entities.
|
||||||
- **Must be pure** — no side effects, no async, no external state.
|
- **Must be pure** — no side effects, no async, no external state.
|
||||||
- Transforms only see metadata. Vectors, embeddings, and the `data` field stored inside metadata are available as properties on the metadata object.
|
- Transforms only see metadata. Vectors, embeddings, and the `data` field stored inside metadata are available as properties on the metadata object.
|
||||||
|
|
||||||
|
|
@ -110,9 +108,9 @@ const preview = await brain.migrate({ dryRun: true })
|
||||||
// preview.sampleChanges — up to 5 before/after samples
|
// preview.sampleChanges — up to 5 before/after samples
|
||||||
// preview.estimatedTime — rough time estimate string
|
// preview.estimatedTime — rough time estimate string
|
||||||
|
|
||||||
// Apply migrations
|
// Apply migrations (optionally with a pre-migration snapshot)
|
||||||
const result = await brain.migrate()
|
const result = await brain.migrate({ backupTo: '/backups/pre-migration' })
|
||||||
// result.backupBranch — name of COW backup branch, or null
|
// result.backupPath — snapshot path, or null when no backupTo was supplied
|
||||||
// result.migrationsApplied — array of migration IDs that ran
|
// result.migrationsApplied — array of migration IDs that ran
|
||||||
// result.entitiesProcessed — total entities scanned
|
// result.entitiesProcessed — total entities scanned
|
||||||
// result.entitiesModified — entities actually changed
|
// result.entitiesModified — entities actually changed
|
||||||
|
|
@ -164,20 +162,20 @@ const result = await brain.migrate({ maxErrors: 10000 })
|
||||||
|
|
||||||
## Backup and Rollback
|
## Backup and Rollback
|
||||||
|
|
||||||
Before modifying any data, `brain.migrate()` calls `brain.fork()` to create a COW snapshot. This is instant regardless of dataset size — it's a pointer copy, not a data copy.
|
Pass `backupTo` and `brain.migrate()` persists a snapshot of the current generation **before any transform runs**. On filesystem storage the snapshot is a hard-link farm — created without copying entity data, and immune to later writes (see [Snapshots & Time Travel](./snapshots-and-time-travel.md)):
|
||||||
|
|
||||||
The backup branch is named `pre-migration-{version}` and tagged with metadata:
|
|
||||||
- `type: 'system:backup'`
|
|
||||||
- `migrationVersion: '7.17.0'`
|
|
||||||
- `author: 'brainy-migration'`
|
|
||||||
|
|
||||||
To roll back, switch to the backup branch:
|
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
await brain.checkout('pre-migration-7.17.0')
|
const result = await brain.migrate({ backupTo: '/backups/pre-migration-8.0' })
|
||||||
|
console.log(result.backupPath) // '/backups/pre-migration-8.0' (null when no backupTo)
|
||||||
```
|
```
|
||||||
|
|
||||||
Old backup branches from previous migrations are cleaned up automatically before each new migration run.
|
To roll back, restore the snapshot wholesale:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
await brain.restore('/backups/pre-migration-8.0', { confirm: true })
|
||||||
|
```
|
||||||
|
|
||||||
|
Without `backupTo`, no backup is taken — transforms are idempotent (they return `null` when already applied), but a pre-migration snapshot is the cheap insurance for anything destructive.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
277
docs/guides/snapshots-and-time-travel.md
Normal file
277
docs/guides/snapshots-and-time-travel.md
Normal file
|
|
@ -0,0 +1,277 @@
|
||||||
|
---
|
||||||
|
title: Snapshots & Time Travel
|
||||||
|
slug: guides/snapshots-and-time-travel
|
||||||
|
public: true
|
||||||
|
category: guides
|
||||||
|
template: guide
|
||||||
|
order: 9
|
||||||
|
description: Recipes for the Db API — instant backups with persist(), restore, time-travel debugging with asOf(), persist-before-migrate, what-if analysis with with(), and audit trails via transaction metadata.
|
||||||
|
next:
|
||||||
|
- concepts/consistency-model
|
||||||
|
- guides/optimistic-concurrency
|
||||||
|
---
|
||||||
|
|
||||||
|
# Snapshots & Time Travel
|
||||||
|
|
||||||
|
Brainy 8.0 treats the database as a **value**: `brain.now()` pins the
|
||||||
|
current state as an immutable `Db`, `brain.transact()` commits an atomic
|
||||||
|
batch and hands you the resulting value, `brain.asOf()` opens past state,
|
||||||
|
and `db.persist()` cuts a self-contained snapshot. This guide is the recipe
|
||||||
|
book. The precise guarantees behind every recipe live in the
|
||||||
|
[consistency model](../concepts/consistency-model.md).
|
||||||
|
|
||||||
|
## Instant backup
|
||||||
|
|
||||||
|
Pin the current state, persist it, release:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const db = brain.now()
|
||||||
|
try {
|
||||||
|
await db.persist('/backups/2026-06-11')
|
||||||
|
} finally {
|
||||||
|
await db.release()
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
On filesystem storage the snapshot is built from **hard links**: every data
|
||||||
|
file in Brainy is immutable-by-rename, so the snapshot is created without
|
||||||
|
copying entity data and shares disk space with the live store. Later writes
|
||||||
|
can never alter it — a rewrite swaps the inode, the snapshot keeps the old
|
||||||
|
bytes. Cross-device targets fall back to per-file byte copies, and
|
||||||
|
persisting an in-memory brain serializes it to the same directory layout —
|
||||||
|
a real, durable store.
|
||||||
|
|
||||||
|
Two things to know:
|
||||||
|
|
||||||
|
- `persist()` requires the view to still be the store's **latest**
|
||||||
|
generation. If something committed after your pin, it throws
|
||||||
|
`GenerationConflictError` instead of snapshotting the wrong state — pin
|
||||||
|
and persist before further writes, or retry with a fresh `brain.now()`.
|
||||||
|
- The target directory must be empty or absent.
|
||||||
|
|
||||||
|
For scheduled backups, this loop is the whole job:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const db = brain.now()
|
||||||
|
try {
|
||||||
|
await db.persist(`/backups/${new Date().toISOString().slice(0, 10)}`)
|
||||||
|
} finally {
|
||||||
|
await db.release()
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Restore
|
||||||
|
|
||||||
|
`restore()` replaces the store's **entire** current state from a snapshot —
|
||||||
|
entities, relationships, indexes, history. It is deliberately loud about it:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
await brain.restore('/backups/2026-06-11', { confirm: true })
|
||||||
|
```
|
||||||
|
|
||||||
|
- `{ confirm: true }` is mandatory — current state is destroyed.
|
||||||
|
- The snapshot is copied in (never linked), so it stays independent and can
|
||||||
|
be restored again later.
|
||||||
|
- All indexes are rebuilt from the restored records.
|
||||||
|
- The generation counter is floored at its pre-restore value, so generation
|
||||||
|
numbers you observed before the restore are never reissued.
|
||||||
|
- Live `Db` pins do not survive a restore — release them first.
|
||||||
|
|
||||||
|
## Open a snapshot read-only
|
||||||
|
|
||||||
|
You do not have to restore to look inside a snapshot. `Brainy.load()` opens
|
||||||
|
it as a self-contained read-only store with the **full query surface**,
|
||||||
|
including vector search:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const db = await Brainy.load('/backups/2026-06-11')
|
||||||
|
|
||||||
|
const hits = await db.search('unpaid invoices from the spring campaign')
|
||||||
|
const orders = await db.find({ type: NounType.Document, subtype: 'order' })
|
||||||
|
|
||||||
|
await db.release() // closes the underlying read-only instance
|
||||||
|
```
|
||||||
|
|
||||||
|
`brain.asOf('/backups/2026-06-11')` does the same from an existing brain.
|
||||||
|
This is also the 8.0 answer to "named branches": a branch is a name → path
|
||||||
|
mapping your application keeps, where each path is a persisted snapshot.
|
||||||
|
Need a writable copy? Restore the snapshot into a fresh data directory and
|
||||||
|
open a writer on it — instead of switching a shared store between branches
|
||||||
|
in place, every line of code always sees exactly the store it opened.
|
||||||
|
|
||||||
|
## Time-travel debugging
|
||||||
|
|
||||||
|
When production data looks wrong, query the past directly — by wall-clock
|
||||||
|
time or by generation:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// What did this order look like yesterday?
|
||||||
|
const yesterday = await brain.asOf(new Date(Date.now() - 86_400_000))
|
||||||
|
const before = await yesterday.get(orderId)
|
||||||
|
|
||||||
|
// Full queries work at any reachable generation — search, graph, filters:
|
||||||
|
const thenActive = await yesterday.find({
|
||||||
|
type: NounType.Document,
|
||||||
|
subtype: 'order',
|
||||||
|
where: { status: 'active' }
|
||||||
|
})
|
||||||
|
|
||||||
|
await yesterday.release()
|
||||||
|
```
|
||||||
|
|
||||||
|
Pin two points in time and diff them:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const before = await brain.asOf(1041)
|
||||||
|
const after = brain.now()
|
||||||
|
|
||||||
|
const changed = await after.since(before)
|
||||||
|
changed.nouns // entity ids touched by transactions in between
|
||||||
|
changed.verbs // relationship ids touched in between
|
||||||
|
|
||||||
|
await before.release()
|
||||||
|
await after.release()
|
||||||
|
```
|
||||||
|
|
||||||
|
Three things to remember:
|
||||||
|
|
||||||
|
- History granularity is `transact()` commits — single-operation writes
|
||||||
|
advance the clock but do not produce historical records (see the
|
||||||
|
[consistency model](../concepts/consistency-model.md)). Use `transact()`
|
||||||
|
for writes you want to travel back through.
|
||||||
|
- The first index-accelerated query (semantic search, traversal, cursors,
|
||||||
|
aggregation) at a historical generation builds an in-memory index
|
||||||
|
materialization — O(n at that generation), once per `Db`, freed on
|
||||||
|
`release()`. Metadata-level reads are free.
|
||||||
|
- Generations reclaimed by `compactHistory()` throw
|
||||||
|
`GenerationCompactedError` — persist anything you need to keep forever.
|
||||||
|
|
||||||
|
## Safe schema migration
|
||||||
|
|
||||||
|
`brain.migrate()` integrates with snapshots directly: pass `backupTo` and a
|
||||||
|
hard-link snapshot of the current generation is persisted **before any
|
||||||
|
transform runs**:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const result = await brain.migrate({ backupTo: '/backups/pre-migration-8.0' })
|
||||||
|
console.log(result.migrationsApplied, result.backupPath)
|
||||||
|
|
||||||
|
// If the migration went wrong, roll the whole store back:
|
||||||
|
await brain.restore('/backups/pre-migration-8.0', { confirm: true })
|
||||||
|
```
|
||||||
|
|
||||||
|
The same persist-before-mutate pattern works for any risky bulk operation,
|
||||||
|
not just migrations:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const pin = brain.now()
|
||||||
|
try {
|
||||||
|
await pin.persist('/backups/pre-bulk-edit')
|
||||||
|
} finally {
|
||||||
|
await pin.release()
|
||||||
|
}
|
||||||
|
await runRiskyBulkEdit(brain)
|
||||||
|
```
|
||||||
|
|
||||||
|
## What-if analysis
|
||||||
|
|
||||||
|
`db.with(ops)` applies a transaction **speculatively, in memory** — nothing
|
||||||
|
touches disk, the generation counter, or the indexes. Ask "what would the
|
||||||
|
store look like if…", then commit the same operations for real:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const ops = [
|
||||||
|
{ op: 'update', id: employeeId, metadata: { team: 'platform' } },
|
||||||
|
{ op: 'relate', from: employeeId, to: milestoneId, type: VerbType.ParticipatesIn, subtype: 'assignment' }
|
||||||
|
]
|
||||||
|
|
||||||
|
const base = brain.now()
|
||||||
|
const whatIf = await base.with(ops)
|
||||||
|
|
||||||
|
await whatIf.get(employeeId) // sees the change
|
||||||
|
await whatIf.find({ where: { team: 'platform' } }) // metadata finds work
|
||||||
|
await whatIf.related(employeeId) // overlay relations included
|
||||||
|
|
||||||
|
await whatIf.release()
|
||||||
|
await base.release()
|
||||||
|
|
||||||
|
// Looks right — make it real, atomically:
|
||||||
|
await brain.transact(ops)
|
||||||
|
```
|
||||||
|
|
||||||
|
**The boundary:** speculative entities carry no embeddings (`with()` never
|
||||||
|
invokes the embedder), so semantic search, traversal, cursors, aggregation,
|
||||||
|
and `persist()` throw `SpeculativeOverlayError` on overlay views instead of
|
||||||
|
returning silently incomplete results. `get()`, metadata-filter `find()`,
|
||||||
|
and filter-based `related()` are fully supported. Overlays chain — calling
|
||||||
|
`with()` on an overlay stacks another layer.
|
||||||
|
|
||||||
|
## Audit trails
|
||||||
|
|
||||||
|
`transact()` reifies transaction metadata: whatever you pass as `meta` is
|
||||||
|
recorded durably alongside the committed generation and timestamp, readable
|
||||||
|
via `brain.transactionLog()`:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
await brain.transact(
|
||||||
|
[{ op: 'update', id: invoiceId, metadata: { status: 'approved' } }],
|
||||||
|
{ meta: { author: 'approvals-service', actor: 'jane@example.com', reason: 'PO-7741' } }
|
||||||
|
)
|
||||||
|
|
||||||
|
const log = await brain.transactionLog({ limit: 20 }) // newest first
|
||||||
|
// [{ generation: 1042, timestamp: 1765432100000, meta: { author: 'approvals-service', ... } }]
|
||||||
|
```
|
||||||
|
|
||||||
|
Combine the log with `asOf()` to reconstruct exactly what any transaction
|
||||||
|
did:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const [entry] = await brain.transactionLog({ limit: 1 })
|
||||||
|
|
||||||
|
const after = await brain.asOf(entry.generation)
|
||||||
|
const before = await brain.asOf(entry.generation - 1)
|
||||||
|
|
||||||
|
const touched = await after.since(before)
|
||||||
|
for (const id of touched.nouns) {
|
||||||
|
console.log(id, await before.get(id), '→', await after.get(id))
|
||||||
|
}
|
||||||
|
|
||||||
|
await before.release()
|
||||||
|
await after.release()
|
||||||
|
```
|
||||||
|
|
||||||
|
For per-entity write coordination (rather than whole-store history), the
|
||||||
|
`_rev` counter and `ifRev` CAS remain the right tool — see
|
||||||
|
[optimistic concurrency](./optimistic-concurrency.md).
|
||||||
|
|
||||||
|
## Keeping history bounded
|
||||||
|
|
||||||
|
Historical records cost disk space. Reclaim what no live pin protects:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
await brain.compactHistory({
|
||||||
|
retainGenerations: 100, // keep the 100 most recent commits
|
||||||
|
retainMs: 7 * 24 * 60 * 60 * 1000 // and everything from the last 7 days
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
Compaction never breaks a pinned read — record-sets are reclaimed only when
|
||||||
|
no live `Db` could need them. Release views you are done with (including the
|
||||||
|
ones `transact()` returns), and `persist()` any generation you want to keep
|
||||||
|
beyond the retention window: snapshots are self-contained and unaffected by
|
||||||
|
compaction.
|
||||||
|
|
||||||
|
## From branches to values
|
||||||
|
|
||||||
|
If you used the pre-8.0 `fork`/`checkout`/`commit`/`versions` surface, every
|
||||||
|
use case maps to a sharper tool:
|
||||||
|
|
||||||
|
| Pre-8.0 habit | 8.0 recipe |
|
||||||
|
|---|---|
|
||||||
|
| `fork()` to experiment safely | `db.with(ops)` for speculation in memory; a restored snapshot in a fresh directory for a long-lived writable copy |
|
||||||
|
| `commit()` checkpoints | `transact(ops, { meta })` — every batch is an atomic, logged, time-travelable commit |
|
||||||
|
| `checkout()` to switch branches | Open the snapshot you want — `Brainy.load(path)` read-only, or restore into its own directory. No in-place switching: every handle always sees one unambiguous store. |
|
||||||
|
| `getHistory()` | `brain.transactionLog()` + `db.since(priorDb)` |
|
||||||
|
| `versions.save()` per-entity snapshots | A pinned `Db` or persisted snapshot captures *every* entity at that moment; `asOf()` reads any entity's past state |
|
||||||
|
| `versions.restore()` | `brain.restore(snapshot, { confirm: true })` for the whole store, or read the old entity via `asOf()` and write it back with `transact()` |
|
||||||
|
| Backup branches | `db.persist(path)` — instant, hard-link-shared, self-contained |
|
||||||
|
|
@ -5,7 +5,7 @@ public: true
|
||||||
category: guides
|
category: guides
|
||||||
template: guide
|
template: guide
|
||||||
order: 2
|
order: 2
|
||||||
description: "Two adapters cover every deployment: in-memory for tests + ephemeral workloads, filesystem for everything that needs to persist. Both support copy-on-write branching. Cloud backup is operator tooling, not a built-in adapter."
|
description: "Two adapters cover every deployment: in-memory for tests + ephemeral workloads, filesystem for everything that needs to persist. Both share one on-disk contract, including generational history and snapshots. Cloud backup is operator tooling, not a built-in adapter."
|
||||||
next:
|
next:
|
||||||
- guides/plugins
|
- guides/plugins
|
||||||
- concepts/zero-config
|
- concepts/zero-config
|
||||||
|
|
@ -20,8 +20,10 @@ Brainy 8.0 ships **two storage adapters**:
|
||||||
- **`MemoryStorage`** — in-memory only. The right choice for tests, ephemeral
|
- **`MemoryStorage`** — in-memory only. The right choice for tests, ephemeral
|
||||||
workloads, and short-lived demos.
|
workloads, and short-lived demos.
|
||||||
|
|
||||||
Both implement the same `StorageAdapter` interface, support copy-on-write
|
Both implement the same `StorageAdapter` interface, support the full Db API
|
||||||
branching, and use the same on-disk layout (memory's "disk" is a JS Map).
|
(generational history, snapshots, restore — see the
|
||||||
|
[consistency model](../concepts/consistency-model.md)), and use the same
|
||||||
|
on-disk layout (memory's "disk" is a JS Map).
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
|
|
@ -44,7 +46,7 @@ const brainAuto = new Brainy({ storage: { type: 'auto' } })
|
||||||
|
|
||||||
| Use case | Adapter | Why |
|
| Use case | Adapter | Why |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Production app | `filesystem` | Durable, branchable, mmap-able |
|
| Production app | `filesystem` | Durable, snapshot-able, mmap-able |
|
||||||
| Tests, CI | `memory` | No disk teardown; fast |
|
| Tests, CI | `memory` | No disk teardown; fast |
|
||||||
| Short-lived data pipeline | `memory` | No persistence needed |
|
| Short-lived data pipeline | `memory` | No persistence needed |
|
||||||
| In-browser demo | `memory` | Filesystem unavailable in browsers |
|
| In-browser demo | `memory` | Filesystem unavailable in browsers |
|
||||||
|
|
@ -70,20 +72,20 @@ azcopy sync /var/lib/brainy "https://account.blob.core.windows.net/brainy?sv=...
|
||||||
Brainy's filesystem layout is sync-friendly:
|
Brainy's filesystem layout is sync-friendly:
|
||||||
- Atomic writes (temp + rename) — readers never see torn files
|
- Atomic writes (temp + rename) — readers never see torn files
|
||||||
- Per-shard files — `rsync`-style incremental sync works well
|
- Per-shard files — `rsync`-style incremental sync works well
|
||||||
- Content-addressed blobs (`_blobs/`) — immutable, cache-friendly
|
- Immutable generation records (`_generations/`) — append-only, cache-friendly
|
||||||
- Branch refs live under `_cow/` — pick up branches automatically
|
|
||||||
|
|
||||||
For point-in-time backups, take a filesystem snapshot (ZFS, btrfs, LVM, EBS,
|
For point-in-time backups, take a filesystem snapshot (ZFS, btrfs, LVM, EBS,
|
||||||
etc.) or use `brain.persist(path)` to write a self-contained snapshot you
|
etc.) or use `brain.now().persist(path)` to write a self-contained snapshot
|
||||||
can sync independently of the live brain.
|
you can sync independently of the live brain — see
|
||||||
|
[Snapshots & Time Travel](./snapshots-and-time-travel.md).
|
||||||
|
|
||||||
## Why no cloud adapters in 8.0?
|
## Why no cloud adapters in 8.0?
|
||||||
|
|
||||||
Cloud storage adapters lived in Brainy 4.x-7.x. They were dropped in 8.0
|
Cloud storage adapters lived in Brainy 4.x-7.x. They were dropped in 8.0
|
||||||
per **BR-BRAINY-80-STORAGE-SIMPLIFY** because:
|
because:
|
||||||
|
|
||||||
- Zero production consumers used them at scale. Every Soulcraft consumer
|
- Zero production consumers used them at scale — every known production
|
||||||
ran on local filesystem.
|
deployment ran on local filesystem.
|
||||||
- Cloud-storage HNSW / DiskANN doesn't perform — vector indexes need
|
- Cloud-storage HNSW / DiskANN doesn't perform — vector indexes need
|
||||||
low-latency random reads that S3 / GCS / R2 / Azure can't provide
|
low-latency random reads that S3 / GCS / R2 / Azure can't provide
|
||||||
consistently.
|
consistently.
|
||||||
|
|
@ -97,23 +99,21 @@ Brainy 8.0 is smaller, faster to install, and clearer about what it does.
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
```ts
|
```ts
|
||||||
interface StorageOptions {
|
// BrainyConfig['storage'] — either a config object or a pre-constructed adapter:
|
||||||
// The adapter type. Defaults to 'auto'.
|
storage?:
|
||||||
type?: 'auto' | 'memory' | 'filesystem'
|
| {
|
||||||
|
// The adapter type. Defaults to 'auto'
|
||||||
|
// (filesystem on Node-like runtimes, memory otherwise).
|
||||||
|
type: 'auto' | 'memory' | 'filesystem'
|
||||||
|
|
||||||
// Force a specific adapter regardless of type.
|
// Root directory for filesystem storage. Passed through to storage
|
||||||
forceMemoryStorage?: boolean
|
// factories, including plugin-provided ones.
|
||||||
forceFileSystemStorage?: boolean
|
|
||||||
|
|
||||||
// Filesystem only.
|
|
||||||
rootDirectory?: string
|
rootDirectory?: string
|
||||||
|
|
||||||
// COW branch to open. Defaults to 'main'.
|
// Adapter-specific options.
|
||||||
branch?: string
|
options?: any
|
||||||
|
}
|
||||||
// COW compression toggle. Defaults to true.
|
| StorageAdapter // e.g. storage: new MemoryStorage()
|
||||||
enableCompression?: boolean
|
|
||||||
}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Direct construction
|
## Direct construction
|
||||||
|
|
@ -143,6 +143,7 @@ backup tooling. The recipe:
|
||||||
4. Set up an operator backup job using `gsutil` / `aws s3` / `rclone` /
|
4. Set up an operator backup job using `gsutil` / `aws s3` / `rclone` /
|
||||||
`azcopy` on a cron — hourly or whatever your RPO requires. Point it at
|
`azcopy` on a cron — hourly or whatever your RPO requires. Point it at
|
||||||
the brainy data dir.
|
the brainy data dir.
|
||||||
5. For point-in-time backups, use filesystem snapshots or `brain.persist()`.
|
5. For point-in-time backups, use filesystem snapshots or
|
||||||
|
`brain.now().persist(path)`.
|
||||||
|
|
||||||
Same data, same APIs, no library-side cloud code.
|
Same data, same APIs, no library-side cloud code.
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,16 @@
|
||||||
|
---
|
||||||
|
title: Transactions & Atomicity
|
||||||
|
slug: guides/transactions
|
||||||
|
public: true
|
||||||
|
category: guides
|
||||||
|
template: guide
|
||||||
|
order: 10
|
||||||
|
description: How Brainy keeps every write atomic — automatic per-operation transactions with rollback, and brain.transact() for atomic multi-write batches with compare-and-swap.
|
||||||
|
next:
|
||||||
|
- concepts/consistency-model
|
||||||
|
- guides/optimistic-concurrency
|
||||||
|
---
|
||||||
|
|
||||||
# Transaction System
|
# Transaction System
|
||||||
|
|
||||||
**Status:** ✅ Production Ready
|
**Status:** ✅ Production Ready
|
||||||
|
|
@ -11,18 +24,18 @@ Brainy's transaction system provides **atomic operations** with automatic rollba
|
||||||
- **Atomicity**: All operations succeed or all rollback
|
- **Atomicity**: All operations succeed or all rollback
|
||||||
- **Consistency**: Indexes and storage remain consistent
|
- **Consistency**: Indexes and storage remain consistent
|
||||||
- **Automatic**: Transparently used by all `brain.add()`, `brain.update()`, `brain.delete()`, and `brain.relate()` operations
|
- **Automatic**: Transparently used by all `brain.add()`, `brain.update()`, `brain.delete()`, and `brain.relate()` operations
|
||||||
- **Compatible**: Works seamlessly with COW, sharding, and type-aware storage
|
- **Composable**: `brain.transact()` runs a multi-write batch through the same machinery as exactly one atomic commit
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
```
|
```
|
||||||
User Code (brain.add(), brain.update(), etc.)
|
User Code (brain.add(), brain.update(), brain.transact(), etc.)
|
||||||
↓
|
↓
|
||||||
Transaction Manager (orchestration)
|
Transaction Manager (orchestration)
|
||||||
↓
|
↓
|
||||||
Operations (SaveNounMetadataOperation, SaveNounOperation, etc.)
|
Operations (SaveNounMetadataOperation, SaveNounOperation, etc.)
|
||||||
↓
|
↓
|
||||||
Storage Adapter (COW, sharding, type-aware routing)
|
Storage Adapter (sharding, ID-first routing)
|
||||||
```
|
```
|
||||||
|
|
||||||
### How It Works
|
### How It Works
|
||||||
|
|
@ -74,31 +87,41 @@ class SaveNounMetadataOperation {
|
||||||
|
|
||||||
## Compatibility with Advanced Features
|
## Compatibility with Advanced Features
|
||||||
|
|
||||||
### COW (Copy-on-Write)
|
### Multi-Write Batches: `brain.transact()`
|
||||||
|
|
||||||
✅ **Fully Compatible**
|
✅ **The 8.0 path for atomic multi-entity writes**
|
||||||
|
|
||||||
Transactions work transparently with COW branches:
|
Single-operation methods each commit their own transaction. When several
|
||||||
|
writes must succeed or fail **together**, use `brain.transact()` — a
|
||||||
|
declarative batch that commits as exactly one generation, with optional
|
||||||
|
whole-store compare-and-swap and durable transaction metadata:
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// Create branch
|
const db = await brain.transact([
|
||||||
await brain.cow.createBranch('feature-branch')
|
{ op: 'add', id: orderId, type: NounType.Document, subtype: 'order', data: 'Order #1042' },
|
||||||
await brain.cow.checkout('feature-branch')
|
{ op: 'update', id: customerId, metadata: { lastOrderAt: Date.now() }, ifRev: customer._rev },
|
||||||
|
{ op: 'relate', from: customerId, to: orderId, type: VerbType.Creates, subtype: 'purchase' }
|
||||||
|
], { meta: { author: 'order-service' } })
|
||||||
|
|
||||||
// Add entity (uses transaction on this branch)
|
db.receipt.ids // resolved id per operation, in input order
|
||||||
const id = await brain.add({
|
|
||||||
data: { name: 'Feature Entity' },
|
|
||||||
type: NounType.Thing
|
|
||||||
})
|
|
||||||
|
|
||||||
// On rollback: Branch remains clean, no partial commits
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**How It Works:**
|
**How It Works:**
|
||||||
- Transactions use `StorageAdapter` interface
|
- The batch executes through the same TransactionManager as single
|
||||||
- COW operates at storage layer (refManager, blobStorage, commitLog)
|
operations, wrapped in the generational commit protocol: before-images are
|
||||||
- Branch isolation prevents cross-branch contamination
|
staged and fsynced first, and the atomic manifest rename is the commit
|
||||||
- Rollback = discard uncommitted changes (COW makes this trivial)
|
point — a crash anywhere before it rolls back to the exact
|
||||||
|
pre-transaction bytes.
|
||||||
|
- Per-entity `ifRev` and whole-store `ifAtGeneration` provide
|
||||||
|
compare-and-swap at two granularities; any conflict rejects the entire
|
||||||
|
batch before anything is staged.
|
||||||
|
- The returned `Db` is a pinned, snapshot-isolated view of the committed
|
||||||
|
state.
|
||||||
|
|
||||||
|
See the **[consistency model](concepts/consistency-model.md)** for the
|
||||||
|
full guarantees (snapshot isolation, time travel, snapshots) and
|
||||||
|
**[Snapshots & Time Travel](guides/snapshots-and-time-travel.md)** for
|
||||||
|
recipes.
|
||||||
|
|
||||||
### Sharding
|
### Sharding
|
||||||
|
|
||||||
|
|
@ -306,37 +329,31 @@ try {
|
||||||
|
|
||||||
### Transaction Overhead
|
### Transaction Overhead
|
||||||
|
|
||||||
**MEASURED Performance Impact:**
|
**What a transaction costs:**
|
||||||
- Average overhead: ~2-5ms per transaction (measured: `tests/transaction/transaction.bench.ts`)
|
- A typical single-operation write wraps 2-8 operations (metadata + data + indexes) in one transaction
|
||||||
- Operations per transaction: 2-8 (metadata + data + indexes)
|
- The overhead is bookkeeping (operation objects + undo state), not extra I/O on the success path
|
||||||
- Rollback cost: ~1-3ms (restore previous state)
|
- Rollback cost is proportional to the operations already applied (each is undone in reverse order)
|
||||||
|
|
||||||
**Optimization:**
|
**Optimization:**
|
||||||
- Operations executed sequentially (not parallel) for consistency
|
- Operations executed sequentially (not parallel) for consistency
|
||||||
- Rollback only happens on failure (success path is fast)
|
- Rollback only happens on failure (success path is fast)
|
||||||
- Index updates batched within transaction
|
- Index updates batched within transaction
|
||||||
|
|
||||||
### Statistics and Monitoring
|
### Auditing Committed Batches
|
||||||
|
|
||||||
|
Every committed `brain.transact()` batch is recorded in the transaction
|
||||||
|
log, newest first:
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// Get transaction statistics
|
await brain.transact(ops, { meta: { author: 'import-job' } })
|
||||||
const stats = brain.transactionManager?.getStats()
|
|
||||||
console.log(stats)
|
const entries = await brain.transactionLog({ limit: 10 })
|
||||||
// {
|
// [{ generation: 1042, timestamp: 1765432100000, meta: { author: 'import-job' } }]
|
||||||
// totalTransactions: 1234,
|
|
||||||
// successfulTransactions: 1200,
|
|
||||||
// failedTransactions: 34,
|
|
||||||
// rollbacks: 34,
|
|
||||||
// averageOperationsPerTransaction: 4.2
|
|
||||||
// }
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Metrics Available:**
|
Single-operation writes advance the generation counter but do not append
|
||||||
- `totalTransactions`: Total number of transactions executed
|
log entries — see the [consistency model](concepts/consistency-model.md)
|
||||||
- `successfulTransactions`: Number of successful commits
|
for the history-granularity contract.
|
||||||
- `failedTransactions`: Number of rollbacks
|
|
||||||
- `rollbacks`: Total rollback count
|
|
||||||
- `averageOperationsPerTransaction`: Average operations per transaction
|
|
||||||
|
|
||||||
## Best Practices
|
## Best Practices
|
||||||
|
|
||||||
|
|
@ -376,19 +393,18 @@ if (!isValidVector(vector, brain.dimension)) {
|
||||||
await brain.add({ data, type, vector })
|
await brain.add({ data, type, vector })
|
||||||
```
|
```
|
||||||
|
|
||||||
### 4. Monitor Transaction Statistics
|
### 4. Batch Related Writes with `transact()`
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// ✅ Recommended: Monitor in production
|
// ✅ Recommended: writes that must land together go in one batch
|
||||||
setInterval(() => {
|
await brain.transact([
|
||||||
const stats = brain.transactionManager?.getStats()
|
{ op: 'add', id: orderId, type: NounType.Document, subtype: 'order', data: 'Order #1042' },
|
||||||
if (stats) {
|
{ op: 'relate', from: customerId, to: orderId, type: VerbType.Creates, subtype: 'purchase' }
|
||||||
const failureRate = stats.failedTransactions / stats.totalTransactions
|
])
|
||||||
if (failureRate > 0.05) { // > 5% failure rate
|
|
||||||
console.warn('High transaction failure rate:', failureRate)
|
// ❌ Avoid: sequential single operations when partial application is unacceptable
|
||||||
}
|
const id = await brain.add({ ... }) // commits alone
|
||||||
}
|
await brain.relate({ ... }) // a crash here leaves the entity unlinked
|
||||||
}, 60000) // Check every minute
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### 5. Understand Atomicity Guarantees
|
### 5. Understand Atomicity Guarantees
|
||||||
|
|
@ -440,15 +456,19 @@ describe('Transaction Tests', () => {
|
||||||
### Integration Tests
|
### Integration Tests
|
||||||
|
|
||||||
See `tests/transaction/integration/` for comprehensive integration tests covering:
|
See `tests/transaction/integration/` for comprehensive integration tests covering:
|
||||||
- COW integration (`cow-transactions.test.ts`)
|
|
||||||
- Sharding integration (`sharding-transactions.test.ts`)
|
- Sharding integration (`sharding-transactions.test.ts`)
|
||||||
- TypeAware integration (`typeaware-transactions.test.ts`)
|
- Type-aware integration (`typeaware-transactions.test.ts`)
|
||||||
|
- Distributed scenarios (`distributed-transactions.test.ts`)
|
||||||
|
|
||||||
|
The atomicity guarantees of `brain.transact()` — including crash recovery
|
||||||
|
through the real recovery path — are proven in
|
||||||
|
`tests/integration/db-mvcc.test.ts`.
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
### High Rollback Rate
|
### High Rollback Rate
|
||||||
|
|
||||||
**Symptom:** `failedTransactions` / `totalTransactions` > 5%
|
**Symptom:** a high share of writes throw and roll back
|
||||||
|
|
||||||
**Possible Causes:**
|
**Possible Causes:**
|
||||||
1. Invalid vector dimensions
|
1. Invalid vector dimensions
|
||||||
|
|
@ -478,21 +498,6 @@ See `tests/transaction/integration/` for comprehensive integration tests coverin
|
||||||
- Disable unused indexes
|
- Disable unused indexes
|
||||||
- Use SSD storage
|
- Use SSD storage
|
||||||
|
|
||||||
### Transaction Statistics Missing
|
|
||||||
|
|
||||||
**Symptom:** `brain.transactionManager?.getStats()` returns `undefined`
|
|
||||||
|
|
||||||
**Cause:** TransactionManager not initialized
|
|
||||||
|
|
||||||
**Solution:**
|
|
||||||
```typescript
|
|
||||||
// Ensure Brainy is initialized
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Then access stats
|
|
||||||
const stats = brain.transactionManager?.getStats()
|
|
||||||
```
|
|
||||||
|
|
||||||
## Architecture Details
|
## Architecture Details
|
||||||
|
|
||||||
### Transaction Lifecycle
|
### Transaction Lifecycle
|
||||||
|
|
@ -545,28 +550,19 @@ interface StorageAdapter {
|
||||||
|
|
||||||
## Additional Resources
|
## Additional Resources
|
||||||
|
|
||||||
- **Unit Tests:** `tests/transaction/transaction.test.ts` (36 passing tests)
|
- **Unit Tests:** `tests/transaction/Transaction.test.ts`, `tests/transaction/TransactionManager.test.ts`
|
||||||
- **Integration Tests:** `tests/transaction/integration/` (35 test scenarios)
|
- **Integration Tests:** `tests/transaction/integration/`
|
||||||
- **Compatibility Analysis:** `.strategy/TRANSACTION_COMPATIBILITY_ANALYSIS.md` (internal)
|
- **MVCC Proofs:** `tests/integration/db-mvcc.test.ts` (atomicity, CAS, crash recovery for `brain.transact()`)
|
||||||
- **Performance Benchmarks:** `tests/transaction/transaction.bench.ts`
|
- **Consistency Model:** [docs/concepts/consistency-model.md](concepts/consistency-model.md)
|
||||||
|
|
||||||
## Version History
|
|
||||||
|
|
||||||
- Initial transaction system release
|
|
||||||
- Atomic operations with rollback
|
|
||||||
- Compatible with COW, sharding, type-aware storage
|
|
||||||
- 36/36 unit tests passing
|
|
||||||
- 35 integration test scenarios
|
|
||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
Brainy's transaction system provides **production-ready atomic operations** with automatic rollback. It works transparently with all Brainy APIs and is fully compatible with advanced features like COW, sharding, and type-aware storage.
|
Brainy's transaction system provides **production-ready atomic operations** with automatic rollback. Every single-operation write is transactional out of the box, and `brain.transact()` extends the same guarantee to multi-write batches — one atomic commit, with compare-and-swap and durable transaction metadata.
|
||||||
|
|
||||||
**Key Takeaways:**
|
**Key Takeaways:**
|
||||||
- ✅ **Automatic**: No manual transaction management needed
|
- ✅ **Automatic**: No manual transaction management needed for single operations
|
||||||
- ✅ **Atomic**: All operations succeed or all rollback
|
- ✅ **Atomic**: All operations succeed or all rollback — per operation and per `transact()` batch
|
||||||
- ✅ **Compatible**: Works with all storage adapters and features
|
- ✅ **Compatible**: Works with all storage adapters and features
|
||||||
- ✅ **Production-Ready**: Tested with 71 test scenarios (36 unit + 35 integration)
|
- ✅ **Coordinated**: Per-entity `ifRev` and whole-store `ifAtGeneration` CAS reject conflicting batches before anything is staged
|
||||||
- ✅ **Performant**: ~2-5ms overhead per transaction (measured)
|
|
||||||
|
|
||||||
Start using transactions today - they're already built into `brain.add()`, `brain.update()`, `brain.delete()`, and `brain.relate()`!
|
Start using transactions today - they're already built into `brain.add()`, `brain.update()`, `brain.delete()`, and `brain.relate()` — and reach for `brain.transact()` whenever several writes must land together.
|
||||||
|
|
|
||||||
|
|
@ -267,8 +267,10 @@ async function gameDevProject() {
|
||||||
where: { 'attributes.location': 'Village Square' }
|
where: { 'attributes.location': 'Village Square' }
|
||||||
})
|
})
|
||||||
|
|
||||||
// Track game balance changes ✅
|
// Track game balance changes — read the file as it was a week ago ✅
|
||||||
const balanceHistory = await vfs.getHistory('/game/data/balance.json')
|
const lastWeek = await brain.asOf(new Date(Date.now() - 7 * 86_400_000))
|
||||||
|
const oldBalance = await lastWeek.find({ where: { path: '/game/data/balance.json' }, limit: 1 })
|
||||||
|
await lastWeek.release()
|
||||||
|
|
||||||
// Collaborative development tracking ✅
|
// Collaborative development tracking ✅
|
||||||
await vfs.addTodo('/game/quests/main_quest.json', {
|
await vfs.addTodo('/game/quests/main_quest.json', {
|
||||||
|
|
|
||||||
|
|
@ -142,26 +142,30 @@ await brain.init() // Required!
|
||||||
await brain.vfs.writeFile(...) // Now this works
|
await brain.vfs.writeFile(...) // Now this works
|
||||||
```
|
```
|
||||||
|
|
||||||
## Fork Support
|
## Snapshot Support
|
||||||
|
|
||||||
VFS works seamlessly with Brainy's Copy-on-Write (COW) fork feature:
|
VFS files are ordinary entities, so they participate fully in the 8.0 Db
|
||||||
|
API: a persisted snapshot contains every VFS file, and a snapshot opened
|
||||||
|
with `Brainy.load()` serves VFS reads at the snapshot's state:
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
const brain = new Brainy({ storage: { type: 'memory' } })
|
const brain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './data' } })
|
||||||
await brain.init()
|
await brain.init()
|
||||||
|
|
||||||
// Create files in parent
|
// Create files
|
||||||
await brain.vfs.writeFile('/config.json', '{"version": 1}')
|
await brain.vfs.writeFile('/config.json', '{"version": 1}')
|
||||||
|
|
||||||
// Fork inherits parent's files
|
// Snapshot the whole store (VFS files included)
|
||||||
const fork = await brain.fork('experiment')
|
const pin = brain.now()
|
||||||
const files = await fork.vfs.readdir('/') // Sees parent's config.json!
|
await pin.persist('/backups/with-vfs')
|
||||||
|
await pin.release()
|
||||||
|
|
||||||
// Fork modifications are isolated
|
// Later changes never touch the snapshot
|
||||||
await fork.vfs.writeFile('/test.txt', 'Fork only')
|
await brain.vfs.writeFile('/config.json', '{"version": 2}')
|
||||||
await brain.vfs.readdir('/') // Parent doesn't see test.txt
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
See [Snapshots & Time Travel](../guides/snapshots-and-time-travel.md).
|
||||||
|
|
||||||
## FAQ
|
## FAQ
|
||||||
|
|
||||||
### Q: Do I need to call `vfs.init()` anymore?
|
### Q: Do I need to call `vfs.init()` anymore?
|
||||||
|
|
@ -177,7 +181,7 @@ await brain.vfs.readdir('/') // Parent doesn't see test.txt
|
||||||
**A:** Yes, VFS configuration is passed through `brain.init()` config. The VFS-specific options are applied during auto-initialization.
|
**A:** Yes, VFS configuration is passed through `brain.init()` config. The VFS-specific options are applied during auto-initialization.
|
||||||
|
|
||||||
### Q: Does this work with all storage adapters?
|
### Q: Does this work with all storage adapters?
|
||||||
**A:** Yes! VFS auto-initialization works with all storage adapters: Memory, FileSystem, OPFS, S3, R2, Azure Blob, and Google Cloud Storage.
|
**A:** Yes! VFS auto-initialization works with both shipped adapters (FileSystem and Memory) and any plugin-provided storage adapter.
|
||||||
|
|
||||||
### Q: What if I need multiple VFS instances?
|
### Q: What if I need multiple VFS instances?
|
||||||
**A:** Each Brainy instance has its own VFS. Create multiple Brainy instances if you need multiple VFS instances.
|
**A:** Each Brainy instance has its own VFS. Create multiple Brainy instances if you need multiple VFS instances.
|
||||||
|
|
@ -187,4 +191,4 @@ await brain.vfs.readdir('/') // Parent doesn't see test.txt
|
||||||
- [VFS Quick Start](./QUICK_START.md) - 5-minute setup guide
|
- [VFS Quick Start](./QUICK_START.md) - 5-minute setup guide
|
||||||
- [VFS API Guide](./VFS_API_GUIDE.md) - Complete API reference
|
- [VFS API Guide](./VFS_API_GUIDE.md) - Complete API reference
|
||||||
- [Common Patterns](./COMMON_PATTERNS.md) - Best practices and patterns
|
- [Common Patterns](./COMMON_PATTERNS.md) - Best practices and patterns
|
||||||
- [Instant Fork](../features/instant-fork.md) - VFS + Copy-on-Write
|
- [Snapshots & Time Travel](../guides/snapshots-and-time-travel.md) - Backups and point-in-time reads
|
||||||
|
|
|
||||||
|
|
@ -1,274 +0,0 @@
|
||||||
/**
|
|
||||||
* Instant Fork Usage Examples (v5.0.0)
|
|
||||||
*
|
|
||||||
* COW is ZERO-CONFIG in v5.0.0:
|
|
||||||
* - No setup needed
|
|
||||||
* - No configuration
|
|
||||||
* - Just works automatically
|
|
||||||
*
|
|
||||||
* This example shows how EASY and ELEGANT fork() is for developers.
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
|
||||||
|
|
||||||
// ========== Example 1: Basic Fork (Zero Config) ==========
|
|
||||||
|
|
||||||
async function basicFork() {
|
|
||||||
// Create Brainy (COW automatic!)
|
|
||||||
const brain = new Brainy({
|
|
||||||
storage: { adapter: 'memory' }
|
|
||||||
})
|
|
||||||
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Add some data
|
|
||||||
await brain.add({ noun: 'user', data: { name: 'Alice' } })
|
|
||||||
await brain.add({ noun: 'user', data: { name: 'Bob' } })
|
|
||||||
|
|
||||||
// Fork instantly (1-2 seconds, even with millions of entities!)
|
|
||||||
const experiment = await brain.fork('experiment')
|
|
||||||
|
|
||||||
// Make changes in fork (doesn't affect main)
|
|
||||||
await experiment.add({ noun: 'user', data: { name: 'Charlie' } })
|
|
||||||
|
|
||||||
// Main brain unchanged
|
|
||||||
console.log(await brain.find({ noun: 'user' })) // Alice, Bob
|
|
||||||
console.log(await experiment.find({ noun: 'user' })) // Alice, Bob, Charlie
|
|
||||||
|
|
||||||
// That's it! Zero config, pure elegance.
|
|
||||||
}
|
|
||||||
|
|
||||||
// ========== Example 2: Safe Experimentation ==========
|
|
||||||
|
|
||||||
async function safeExperimentation() {
|
|
||||||
const brain = new Brainy({
|
|
||||||
storage: { adapter: 'filesystem', path: './data' }
|
|
||||||
})
|
|
||||||
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Production data
|
|
||||||
const users = await brain.find({ noun: 'user' })
|
|
||||||
console.log(`Production: ${users.length} users`)
|
|
||||||
|
|
||||||
// Test a risky operation in fork
|
|
||||||
const test = await brain.fork('test-migration')
|
|
||||||
|
|
||||||
// Run migration on fork (safe!)
|
|
||||||
for (const user of await test.find({ noun: 'user' })) {
|
|
||||||
await test.update(user.id, {
|
|
||||||
email: user.data.email.toLowerCase() // Risky transformation
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// Test it
|
|
||||||
const results = await test.find({ noun: 'user' })
|
|
||||||
|
|
||||||
if (results.every(r => r.data.email === r.data.email.toLowerCase())) {
|
|
||||||
console.log('✅ Migration safe, apply to production')
|
|
||||||
// Apply to production...
|
|
||||||
} else {
|
|
||||||
console.log('❌ Migration failed, discard fork')
|
|
||||||
await test.destroy() // Discard failed experiment
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ========== Example 3: A/B Testing ==========
|
|
||||||
|
|
||||||
async function abTesting() {
|
|
||||||
const brain = new Brainy({
|
|
||||||
storage: { adapter: 's3', bucket: 'my-data' }
|
|
||||||
})
|
|
||||||
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Create variant A (control)
|
|
||||||
const variantA = await brain.fork('variant-a')
|
|
||||||
|
|
||||||
// Create variant B (test)
|
|
||||||
const variantB = await brain.fork('variant-b')
|
|
||||||
|
|
||||||
// Run different algorithms
|
|
||||||
await variantA.processWithAlgorithm('current')
|
|
||||||
await variantB.processWithAlgorithm('improved')
|
|
||||||
|
|
||||||
// Compare results
|
|
||||||
const metricsA = await variantA.getMetrics()
|
|
||||||
const metricsB = await variantB.getMetrics()
|
|
||||||
|
|
||||||
console.log('A:', metricsA.accuracy)
|
|
||||||
console.log('B:', metricsB.accuracy)
|
|
||||||
|
|
||||||
// Choose winner, apply to main
|
|
||||||
if (metricsB.accuracy > metricsA.accuracy) {
|
|
||||||
console.log('B wins! Deploying...')
|
|
||||||
// Merge B to main
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ========== Example 4: Time Travel (Enterprise) ==========
|
|
||||||
|
|
||||||
async function timeTravel() {
|
|
||||||
const brain = new Brainy({
|
|
||||||
storage: { adapter: 'memory' }
|
|
||||||
})
|
|
||||||
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Add data over time
|
|
||||||
await brain.add({ noun: 'doc', data: { version: 1 } })
|
|
||||||
await brain.commit({ message: 'Version 1' })
|
|
||||||
|
|
||||||
const yesterday = Date.now() - 86400000 // 24 hours ago
|
|
||||||
|
|
||||||
await brain.add({ noun: 'doc', data: { version: 2 } })
|
|
||||||
await brain.commit({ message: 'Version 2' })
|
|
||||||
|
|
||||||
// Query as of yesterday (time travel!)
|
|
||||||
const snapshot = await brain.asOf(yesterday)
|
|
||||||
|
|
||||||
const docs = await snapshot.find({ noun: 'doc' })
|
|
||||||
console.log(docs[0].data.version) // 1 (from yesterday!)
|
|
||||||
|
|
||||||
// Zero config, pure magic ✨
|
|
||||||
}
|
|
||||||
|
|
||||||
// ========== Example 5: Backup & Restore (Instant) ==========
|
|
||||||
|
|
||||||
async function instantBackup() {
|
|
||||||
const brain = new Brainy({
|
|
||||||
storage: { adapter: 'memory' }
|
|
||||||
})
|
|
||||||
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Work with production data
|
|
||||||
await brain.add({ noun: 'important', data: { value: 'critical' } })
|
|
||||||
|
|
||||||
// Instant backup (1-2 seconds!)
|
|
||||||
const backup = await brain.fork('backup-2024-01-01')
|
|
||||||
|
|
||||||
// Continue working
|
|
||||||
await brain.delete((await brain.find({ noun: 'important' }))[0].id)
|
|
||||||
|
|
||||||
// Oops! Need to restore
|
|
||||||
const restored = await brain.rollback(backup.getCurrentCommit())
|
|
||||||
|
|
||||||
// Data restored instantly!
|
|
||||||
console.log(await brain.find({ noun: 'important' })) // Back!
|
|
||||||
}
|
|
||||||
|
|
||||||
// ========== Example 6: Distributed Teams (Fork per Developer) ==========
|
|
||||||
|
|
||||||
async function distributedDevelopment() {
|
|
||||||
// Main production brain
|
|
||||||
const production = new Brainy({
|
|
||||||
storage: { adapter: 's3', bucket: 'production-data' }
|
|
||||||
})
|
|
||||||
|
|
||||||
await production.init()
|
|
||||||
|
|
||||||
// Alice's fork
|
|
||||||
const alice = await production.fork('alice-feature-x')
|
|
||||||
|
|
||||||
// Bob's fork
|
|
||||||
const bob = await production.fork('bob-feature-y')
|
|
||||||
|
|
||||||
// Both work independently (zero conflicts!)
|
|
||||||
await alice.add({ noun: 'feature', data: { name: 'X' } })
|
|
||||||
await bob.add({ noun: 'feature', data: { name: 'Y' } })
|
|
||||||
|
|
||||||
// Merge when ready
|
|
||||||
await production.merge(alice, { author: 'Alice' })
|
|
||||||
await production.merge(bob, { author: 'Bob' })
|
|
||||||
|
|
||||||
// Production has both features!
|
|
||||||
}
|
|
||||||
|
|
||||||
// ========== Example 7: VFS Snapshots ==========
|
|
||||||
|
|
||||||
async function vfsSnapshots() {
|
|
||||||
const brain = new Brainy({
|
|
||||||
storage: { adapter: 'memory' },
|
|
||||||
vfs: { enabled: true }
|
|
||||||
})
|
|
||||||
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Create file structure
|
|
||||||
await brain.vfs.writeFile('/project/README.md', '# My Project')
|
|
||||||
await brain.vfs.writeFile('/project/src/index.ts', 'console.log("v1")')
|
|
||||||
|
|
||||||
await brain.commit({ message: 'Initial project' })
|
|
||||||
|
|
||||||
// Fork for refactoring
|
|
||||||
const refactor = await brain.fork('refactor')
|
|
||||||
|
|
||||||
// Refactor code in fork
|
|
||||||
await refactor.vfs.writeFile('/project/src/index.ts', 'console.log("v2")')
|
|
||||||
await refactor.vfs.mkdir('/project/src/utils')
|
|
||||||
|
|
||||||
// Test refactor
|
|
||||||
// ...
|
|
||||||
|
|
||||||
// Switch to refactor branch if successful
|
|
||||||
// await brain.checkout('refactor')
|
|
||||||
}
|
|
||||||
|
|
||||||
// ========== The Key: It's All Zero Config! ==========
|
|
||||||
|
|
||||||
async function zeroConfigDemo() {
|
|
||||||
// This is ALL you need:
|
|
||||||
const brain = new Brainy({ storage: { adapter: 'memory' } })
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// COW is automatic:
|
|
||||||
// ✅ Compression: automatic (based on data type)
|
|
||||||
// ✅ Deduplication: automatic (content-addressable)
|
|
||||||
// ✅ Caching: automatic (LRU with memory limits)
|
|
||||||
// ✅ Reference counting: automatic (safe deletion)
|
|
||||||
// ✅ Garbage collection: automatic (optional manual trigger)
|
|
||||||
|
|
||||||
// Fork is instant:
|
|
||||||
const fork = await brain.fork() // < 2 seconds even at 1M entities
|
|
||||||
|
|
||||||
// That's it! No configuration, no complexity, pure elegance.
|
|
||||||
}
|
|
||||||
|
|
||||||
// ========== API Summary ==========
|
|
||||||
|
|
||||||
/*
|
|
||||||
DEVELOPER API (v5.0.0):
|
|
||||||
|
|
||||||
// Fork operations
|
|
||||||
brain.fork(branch?) → Create instant clone
|
|
||||||
brain.asOf(timestamp) → Time-travel query (Enterprise)
|
|
||||||
brain.rollback(commitHash) → Restore to commit (Enterprise)
|
|
||||||
brain.commit(options?) → Create commit (automatic)
|
|
||||||
|
|
||||||
// Branch operations
|
|
||||||
brain.listBranches() → List all branches
|
|
||||||
brain.checkout(branch) → Switch to branch
|
|
||||||
brain.deleteBranch(branch) → Delete a branch
|
|
||||||
|
|
||||||
// Time queries
|
|
||||||
brain.getHistory(limit?) → Get commit history
|
|
||||||
brain.findAtTime(timestamp) → Find commit at time
|
|
||||||
brain.getStats() → Get storage stats
|
|
||||||
|
|
||||||
// All existing Brainy APIs work the same:
|
|
||||||
brain.add()
|
|
||||||
brain.find()
|
|
||||||
brain.search()
|
|
||||||
brain.vfs.*
|
|
||||||
brain.query() // Triple Intelligence
|
|
||||||
|
|
||||||
ZERO CONFIG REQUIRED!
|
|
||||||
*/
|
|
||||||
|
|
||||||
// Run examples
|
|
||||||
if (require.main === module) {
|
|
||||||
basicFork()
|
|
||||||
.then(() => console.log('✅ All examples complete'))
|
|
||||||
.catch(console.error)
|
|
||||||
}
|
|
||||||
|
|
@ -4592,7 +4592,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
* const db = await brain.transact([
|
* const db = await brain.transact([
|
||||||
* { op: 'add', id: aId, type: NounType.Person, subtype: 'employee', data: 'Ada' },
|
* { op: 'add', id: aId, type: NounType.Person, subtype: 'employee', data: 'Ada' },
|
||||||
* { op: 'add', id: bId, type: NounType.Project, subtype: 'milestone', data: 'Apollo' },
|
* { op: 'add', id: bId, type: NounType.Project, subtype: 'milestone', data: 'Apollo' },
|
||||||
* { op: 'relate', from: aId, to: bId, type: VerbType.WorksOn, subtype: 'assignment' }
|
* { op: 'relate', from: aId, to: bId, type: VerbType.ParticipatesIn, subtype: 'assignment' }
|
||||||
* ], { meta: { author: 'import-job-42' } })
|
* ], { meta: { author: 'import-job-42' } })
|
||||||
* db.receipt.ids // [aId, bId, relationshipId]
|
* db.receipt.ids // [aId, bId, relationshipId]
|
||||||
*/
|
*/
|
||||||
|
|
|
||||||
|
|
@ -8,7 +8,8 @@
|
||||||
* Root cause: clear() deleted the VFS root entity but didn't reset/reinitialize
|
* Root cause: clear() deleted the VFS root entity but didn't reset/reinitialize
|
||||||
* the VFS instance. The VFS remained in memory pointing to the deleted root.
|
* the VFS instance. The VFS remained in memory pointing to the deleted root.
|
||||||
*
|
*
|
||||||
* Fix: Reset VFS state in clear() following the checkout() pattern.
|
* Fix: clear() resets the in-memory VFS instance so the next VFS operation
|
||||||
|
* lazily reinitializes against a fresh root entity.
|
||||||
*
|
*
|
||||||
* These tests verify:
|
* These tests verify:
|
||||||
* 1. VFS operations work after clear() without instance recreation
|
* 1. VFS operations work after clear() without instance recreation
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue