David Snelling
a3467e1f9b
The temporal model had a hole exactly where files were concerned: every
entity write is an immutable generation with before-images, but VFS content
BYTES lived under an eager refCount GC left over from the pre-8.0 design —
unlink could physically destroy bytes that in-window history still
referenced, and overwrite never released the old hash at all (an unbounded
silent leak whose accidental byproduct was the only thing "preserving"
history). Reading the past could therefore return a stale field, a dangling
hash, or nothing, depending on luck.
Fix: blob reclamation becomes a HISTORY decision instead of a LIVENESS
decision. Each blob's metadata now carries historyRefCount alongside the
live refCount:
- The commit seam counts one history reference per persisted before-image
record carrying a content hash (commitTransaction staging and the
group-commit flush), recorded BEFORE the record-set persists and carried
in the generation delta (blobHashes — always present on new deltas, so
compaction only falls back to reading records for pre-contract
generations). An aborted transaction compensates best-effort.
- unlink/rmdir/overwrite drop ONLY the live reference (BlobStorage.delete →
release; overwrite finally releases the superseded hash — cancelling the
dedup increment on same-content rewrites and closing the leak), and only
AFTER the canonical mutation commits, so a failed delete can never leave a
live file whose bytes compaction might reclaim.
- History compaction is the ONE reclamation point: after deleting a
generation's record-set it releases that set's references and physically
reclaims any hash at zero live AND zero history references. Pins are
exempt automatically. Crash ordering is over-count-only in every path
(record before persist, release after delete), so a crash can leak until
the scrub recounts but can never reclaim bytes a retained generation
needs. scrubBlobHistoryRefCounts() restores exactness; existing stores get
a one-time marker-gated backfill on open, failing into leak-safe mode
(reclamation disabled) rather than guessing.
On top of the protected history, the temporal API the generational model
always implied:
- vfs.readFile(path, { asOf }) — the exact bytes as of a generation or Date,
materialized from the history (pinned view released so compaction is
never blocked by a read).
- vfs.history(path) — FileVersion[] ascending ({ generation, timestamp,
hash, size, mimeType? }), the newest entry being the live state.
- Overwrites now refresh the file entity's data/embedding text — semantic
search and the data field previously served the FIRST version's text
forever (the stale-field defect a consumer's incident recovery depended
on by luck).
Integration suite (temporal-vfs.test.ts): per-version exact reads +
history listing, leak-fix + history protection on overwrite, rm keeps bytes
readable, compaction reclaims past-window bytes and preserves in-window
(including the cross-file dedup case where an old file's history and a
newer file's removal share one hash), data freshness, and scrub exactness.
|
||
|---|---|---|
| .. | ||
| api | ||
| architecture | ||
| concepts | ||
| guides | ||
| operations | ||
| vfs | ||
| ADR-001-generational-mvcc.md | ||
| BATCHING.md | ||
| DATA_MODEL.md | ||
| DEVELOPER_LEARNING_PATH.md | ||
| eli5.md | ||
| FIND_SYSTEM.md | ||
| MIGRATION-V3-TO-V4.md | ||
| neural-extraction.md | ||
| PERFORMANCE.md | ||
| PLUGINS.md | ||
| PRODUCTION_SERVICE_ARCHITECTURE.md | ||
| QUERY_OPERATORS.md | ||
| README.md | ||
| RELEASE-GUIDE.md | ||
| SCALING.md | ||
| STAGE3-CANONICAL-TAXONOMY.md | ||
| transactions.md | ||
| troubleshooting.md | ||
| universal-display-augmentation.md | ||
Brainy Documentation
The multi-dimensional AI database with Triple Intelligence — vector search, graph traversal, and metadata filtering in one unified API.
Quick Start
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
const brain = new Brainy()
await brain.init()
// Add entities — data is embedded for semantic search, metadata is indexed for filtering
const id = await brain.add({
data: 'Revolutionary AI Breakthrough',
type: NounType.Document,
metadata: { category: 'technology', rating: 4.8 }
})
// Search with Triple Intelligence
const results = await brain.find({
query: 'artificial intelligence', // Semantic search (on data)
where: { rating: { greaterThan: 4.0 } }, // Metadata filter
connected: { from: authorId, depth: 2 } // Graph traversal
})
Core Documentation
| Document | Description |
|---|---|
| API Reference | Complete API documentation — start here |
| Data Model | Entity structure, data vs metadata, storage fields |
| Query Operators | All BFO operators with examples and indexed/in-memory matrix |
| Find System | Natural language find() and hybrid search details |
| Consistency Model | The Db API guarantees — snapshot isolation, atomic transactions, time travel |
Architecture
| Document | Description |
|---|---|
| Architecture Overview | High-level system design |
| Triple Intelligence | Vector + Graph + Metadata unified query |
| Noun-Verb Taxonomy | 42 nouns + 127 verbs type system |
| Stage 3 Canonical Taxonomy | Complete type reference |
| Storage Architecture | Storage adapters and optimization |
| Index Architecture | Vector, Graph, and Metadata indexing |
| Zero Configuration | Auto-adapts to any environment |
Virtual Filesystem (VFS)
| Document | Description |
|---|---|
| VFS Quick Start | Get started in 30 seconds |
| VFS Core | Core concepts and architecture |
| VFS API Guide | Complete VFS API reference |
| Common Patterns | VFS usage patterns |
See vfs/ for the complete VFS documentation set.
Guides
| Document | Description |
|---|---|
| Import Anything | CSV, Excel, PDF, URL imports |
| Snapshots & Time Travel | Backups, restore, what-if analysis, audit trails |
| Natural Language | Query in plain English |
| Neural API | AI-powered features |
| Enterprise for Everyone | No limits, no tiers |
| Framework Integration | React, Vue, Angular, Svelte |
Storage & Deployment
| Document | Description |
|---|---|
| Storage Architecture | Filesystem and memory adapters, on-disk artifact layout, operator-layer backup |
| Capacity Planning | Scale to millions of entities |
Plugins
| Document | Description |
|---|---|
| Plugins | Plugin system overview — providers, plugins config, brain.use() |
Performance & Scaling
| Document | Description |
|---|---|
| Performance | Optimization techniques |
| Scaling | Scale to billions of entities |
| Batching | Batch operations guide |
Migration & Reference
| Document | Description |
|---|---|
| v3 to v4 Migration | Upgrade guide |
| Release Guide | How to release new versions |
| Production Architecture | Ops reference |
Internal
| Document | Description |
|---|---|
| Audit Report | Feature audit |
| Honest Status | Actual implementation status |
License
Brainy is MIT licensed. See LICENSE for details.