# Brainy Data Storage Architecture (8.0) **Complete on-disk reference for the 8.0 layout.** This document describes what a Brainy 8.0 data directory actually contains: the canonical entity records, the system area, the generational-MVCC bookkeeping, the column store, the blob area, and the lock files — plus how the in-memory indexes rebuild from them. The authoritative design records are [ADR-001 (generational MVCC)](../ADR-001-generational-mvcc.md) and [index-architecture.md](./index-architecture.md); this document is the on-disk map that ties them together. 8.0 removed the 7.x copy-on-write subsystem (`_cow/`, `branches/{branch}/…` paths) and the cloud/OPFS storage adapters. The two storage backends are **filesystem** and **memory**; both speak the same path vocabulary (memory storage keys its internal map by the identical path strings). --- ## 1. Directory Tree A real 8.0 filesystem store (`path`, default `./brainy-data`): ``` brainy-data/ │ ├── entities/ # Canonical records (current state) │ ├── nouns/ │ │ └── {shard}/ # 256 shards: first 2 hex chars of the UUID │ │ └── {id}/ # One directory per entity UUID │ │ ├── vectors.json.gz # Embedding + HNSW node state │ │ └── metadata.json.gz # Everything else (type, subtype, data, fields, _rev) │ └── verbs/ │ └── {shard}/ │ └── {id}/ │ ├── vectors.json.gz # Relationship embedding (when present) │ └── metadata.json.gz # sourceId, targetId, verb, subtype, weight, data… │ ├── _system/ # System singletons + bucketed system keys │ ├── generation.json(.gz) # { generation, updatedAt } — the write watermark │ ├── manifest.json # { version, generation, … } — MVCC commit point │ ├── tx-log.jsonl # One line per committed transact() batch (append-only) │ ├── counts.json # Entity/verb totals │ ├── type-statistics.json.gz # Per-NounType counts │ ├── subtype-statistics.json.gz # Per-(NounType, subtype) counts │ ├── verb-subtype-statistics.json.gz # Per-(VerbType, subtype) counts │ ├── statistics.json # Aggregate statistics blob (counts, index sizes) │ ├── hnsw-system.json # Vector-index entry point + max level │ ├── __metadata_field_registry__.json.gz # Which metadata fields are indexed │ ├── brainy:entityIdMapper.json.gz # UUID ↔ u64 mapping for native index providers │ └── idx/ │ └── {bucket}/ # 256 buckets: FNV-1a hash of the key │ ├── __metadata_field_index__field_{name}.json.gz # Sparse field indexes │ ├── __chunk__*.json.gz # Metadata-index roaring-bitmap chunks │ ├── __sparse_index__*.json.gz# Zone maps + bloom filters │ └── graph-lsm-verbs-{source|target}-*.json.gz # Graph LSM SSTables + manifest │ ├── _generations/ # MVCC history (written ONLY by transact()) │ └── {N}/ # One directory per committed generation N │ ├── tx.json # The generation-N delta (immutable) │ └── prev/ │ └── {id}.json # Before-image of each touched record (immutable) │ ├── _column_index/ # Column store manifests (one dir per field) │ └── {field}/ │ └── MANIFEST.json.gz # Run list + zone metadata for that column │ ├── _blobs/ # Binary blob area (`.bin` convention) │ ├── _column_index/ │ │ └── {field}/ │ │ └── L0-000001.bin # Column-store runs (level-0 segments) │ └── … # VFS file content and other binary blobs │ └── locks/ # Process coordination (NEVER snapshotted) ├── _writer.lock # Single-writer lock: pid, hostname, heartbeat ├── _flush_requests/ # Reader→writer flush RPC (.req files) └── _flush_responses/ # Writer acks (.ack files) ``` Most JSON objects are gzip-compressed (`.json.gz`) — compression is on by default for filesystem storage (`storage.options.compression`, zlib level 6). A few hot singletons (`manifest.json`, `counts.json`, `hnsw-system.json`, `tx-log.jsonl`) are written uncompressed for cheap partial reads and appends. --- ## 2. Canonical Entity Records Each entity (noun) and relationship (verb) is **two files** under one ID-first directory. The split keeps vector I/O (large, append-mostly) separate from metadata I/O (small, read-heavy). ### Noun vector file — `entities/nouns/{shard}/{id}/vectors.json.gz` ```json { "id": "421d92e7-4241-470a-80f4-4b39414e7a83", "vector": [-0.139, -0.056, 0.028, "…384 dims…"], "connections": { "0": ["neighbor-uuid…"] }, "level": 0 } ``` The HNSW node state (`connections`, `level`) is persisted with the vector so the vector index can rebuild without recomputing the graph. ### Noun metadata file — `entities/nouns/{shard}/{id}/metadata.json.gz` ```json { "data": "React is a JavaScript library for building user interfaces", "noun": "concept", "subtype": "cli-add", "createdAt": 1781198053726, "updatedAt": 1781198053726, "_rev": 1 } ``` - `noun` is the NounType. **Type lives in metadata, not in the path** — lookup by ID is a single path construction, no type needed. - `subtype` is the per-product sub-classification (required on write by default in 8.0). - `_rev` increments on every write and backs `update({ ifRev })` CAS. - Consumer metadata fields sit alongside the standard ones. ### Verb files — `entities/verbs/{shard}/{id}/…` Same two-file split. The verb metadata record carries the graph edge: `sourceId`, `targetId`, `verb` (VerbType), `subtype`, `weight`, `data`, `metadata`, timestamps, `_rev`. Verb IDs are Brainy-generated UUIDs by contract (8.0 rejects caller-supplied verb ids) because native graph providers intern the raw UUID bytes as u64 handles. ### Path construction ```typescript const shard = id.substring(0, 2) // '42' const metadataPath = `entities/nouns/${shard}/${id}/metadata.json` const vectorsPath = `entities/nouns/${shard}/${id}/vectors.json` // Verbs: same shape under entities/verbs/ ``` Implemented in `src/storage/baseStorage.ts` (path generators) and `src/storage/sharding.ts` (`getShardIdFromUuid`). --- ## 3. The `_system/` Area Two kinds of keys live here, resolved by `BaseStorage.parsePath()`: 1. **Singletons** — well-known keys written at `_system/.json`: `counts`, `statistics`, `type-statistics`, `hnsw-system`, `__metadata_field_registry__`, `brainy:entityIdMapper`, plus the MVCC trio (`generation.json`, `manifest.json`, `tx-log.jsonl`). 2. **Bucketed system keys** — everything else (field indexes, bitmap chunks, sparse-index segments, graph LSM SSTables) hashes into one of 256 `_system/idx/{bucket}/` directories via FNV-1a, so no single directory accumulates unbounded entries. Notable singletons: | File | Contents | |------|----------| | `generation.json` | `{ generation, updatedAt }` — monotonic watermark, bumped by **every** write batch | | `manifest.json` | MVCC commit point: highest *committed* generation (see §4) | | `tx-log.jsonl` | One JSON line per committed `transact()` batch: generation, timestamp, `meta` | | `type-statistics.json.gz` | Per-NounType counts (backs `brain.counts.byType`) | | `subtype-statistics.json.gz` | `{ counts: { [type]: { [subtype]: n } }, updatedAt }` (contract-bound shape) | | `verb-subtype-statistics.json.gz` | Same shape for VerbTypes | | `hnsw-system.json` | `{ entryPointId, maxLevel }` — per-node state lives in each entity's `vectors.json` | | `brainy:entityIdMapper.json.gz` | UUID ↔ u64 interning table for the BigInt provider contract | | `__metadata_field_registry__.json.gz` | Registry of indexed metadata field names | --- ## 4. Generational MVCC (`_generations/` + the `_system` trio) Full design in [ADR-001](../ADR-001-generational-mvcc.md). The on-disk shape: ``` _system/generation.json { generation, updatedAt } atomic tmp+rename _system/manifest.json { version, generation, … } atomic tmp+rename — THE commit point _system/tx-log.jsonl one line per committed transact() append-only _generations/{N}/tx.json the generation-N delta immutable once written _generations/{N}/prev/{id}.json before-image of {id} immutable once written ``` Commit protocol (writer side): stage before-images and the delta under `_generations/N/`, fsync, apply the delta to the canonical `entities/…` records, then atomically rename `manifest.json` to publish generation N. The tx-log line is appended last (advisory). Crash recovery on open discards any `_generations/{N}` newer than the manifest. Two write classes share the generation clock: - **Single-operation writes** (`add`/`update`/`remove`/`relate` outside `transact()`) bump `generation.json` so watermarks and `_rev` CAS stay sound, but write **no** history — they are not visible to `db.since()` and remain visible through earlier pins. - **`transact()` batches** write the full `_generations/{N}` record and a tx-log line, and are the unit of time travel (`brain.asOf()`). Snapshots (`db.persist(path)`) hard-link the entire store **except `locks/`** into a self-contained directory openable via `Brainy.load(path)`. --- ## 5. Column Store (`_column_index/` + `_blobs/_column_index/`) The metadata index persists per-field columnar runs for O(log n) range and membership queries at scale: - `_column_index/{field}/MANIFEST.json.gz` — the run list and zone metadata for one field (`createdAt`, `subtype`, `noun`, `_rev`, consumer fields, `__words__` for tokenized text…). - `_blobs/_column_index/{field}/L0-NNNNNN.bin` — the actual level-0 run segments, stored through the shared `_blobs/.bin` binary convention. Sparse per-field indexes, roaring-bitmap chunks, and zone-map/bloom segments additionally live as bucketed keys under `_system/idx/` (see §3). Which path serves a given `where` clause is the query planner's decision — inspect it with `brainy inspect explain --where '…'`. --- ## 6. Blob Area (`_blobs/`) `_blobs/.bin` is the flat binary-blob convention shared by every storage adapter (`saveBinaryBlob`/`getBinaryBlob` in the storage contract): - **VFS file content** — VFS entities are regular nouns (path, ownership, and timestamps in entity metadata); the file *bytes* are blobs. - **Column-store runs** (under the `_column_index/` key prefix, §5). - Any other binary payload an index provider persists. Writes use unique temp names + rename, so concurrent writers of the same key cannot tear each other's blobs. --- ## 7. Locks (`locks/`) ``` locks/_writer.lock # single-writer lock: { pid, hostname, startedAt, heartbeat, version } locks/_flush_requests/ # readers drop .req to ask the writer to flush locks/_flush_responses/ # writer answers with .ack ``` - One **writer** per data directory, enforced at `init()`; stale locks (dead PID / stale heartbeat) are reclaimed automatically. - Read-only processes (`Brainy.openReadOnly()`, the `brainy inspect` CLI family) can ask the live writer to flush via the request/response files, so out-of-process diagnostics see fresh state. - `locks/` is excluded from snapshots (`SNAPSHOT_EXCLUDED_TOP_DIRS` in `src/storage/adapters/fileSystemStorage.ts`). --- ## 8. In-Memory Indexes and What Rebuilds From What | Index | In memory | Persisted state | Rebuild source | |-------|-----------|-----------------|----------------| | **Vector (HNSW)** | Graph of vector connections | `_system/hnsw-system.json` + per-entity `vectors.json` | Walk entity vector files; lazy mode loads structure only and pages vectors on demand | | **Metadata index** | Field → value bitmaps + column-store readers | `_system/idx/` chunks + `_column_index/` manifests + `_blobs/_column_index/` runs | Loaded directly; full rebuild re-scans entity metadata | | **Graph adjacency** | sourceId/targetId → verb-id LSM trees | `graph-lsm-verbs-{source,target}-*` SSTables under `_system/idx/` | Loaded from SSTables; full rebuild re-scans verb metadata | | **Counts/statistics** | Per-type and per-subtype maps | `_system/{type,subtype,verb-subtype}-statistics.json.gz`, `counts.json` | Recomputable by scanning entities (`brainy inspect repair`) | A pluggable index provider (the 8.0 plugin contract in `@soulcraft/brainy/plugin`) may replace any of the JS implementations; the persisted formats above are contract-bound so JS and native implementations can interleave on the same directory. --- ## 9. Sharding Strategy **Entities:** first 2 hex characters of the UUID → 256 uniform shards. Deterministic, configuration-free, and keeps per-directory entry counts low (at 1M entities: ~3,900 directories per shard). Paginated whole-store walks (`getNouns`/`getVerbs`) iterate shards `00`–`ff` in order. **System keys:** FNV-1a hash of the key → 256 `_system/idx/` buckets. Same motivation, different keyspace (system keys are not UUIDs). **What is never sharded:** the `_system/` singletons, `_generations/{N}` directories (keyed by generation number), `_column_index/{field}` manifests (keyed by field name), and `locks/`. --- ## 10. Durability and Atomicity - **Per-object atomicity:** every JSON object and blob is written to a unique temp file then `rename()`d — readers never observe torn objects. - **Transaction atomicity:** the `manifest.json` rename is the single commit point for `transact()` batches (§4); everything staged before it is discarded by crash recovery if the rename never lands. - **Compression:** gzip per object (`.json.gz`), transparent to all readers. Native index providers that mmap binary formats use the uncompressed `_blobs/` area instead. --- ## 11. `clear()` Semantics `brain.clear()` removes all entities, relationships, indexes, statistics, and MVCC history, then re-resolves every index exactly as `init()` does — including plugin-provided vector/metadata/id-mapper factories and VFS root re-creation. The data directory afterwards contains a fresh, empty store (the writer lock remains held by the running process). --- ## 12. Common Scenarios ### Adding an entity ``` brain.add({ data, type, subtype }) 1. Generate UUID → shard = first 2 hex chars 2. Embed data → 384-dim vector 3. Write entities/nouns/{shard}/{id}/vectors.json.gz (vector + HNSW node state) 4. Write entities/nouns/{shard}/{id}/metadata.json.gz (type/subtype/data/fields, _rev: 1) 5. Update in-memory indexes (HNSW insert, metadata index, statistics) 6. Bump _system/generation.json (no _generations/ entry — single-op write) ``` ### Committing a transaction ``` await brain.transact(tx => { tx.add(…); tx.update(…) }) 1. Stage _generations/{N}/prev/{id}.json before-images + tx.json delta; fsync 2. Apply the delta to canonical entities/… records 3. Atomic-rename _system/manifest.json → generation N is committed 4. Append one line to _system/tx-log.jsonl ``` ### Cold start ``` await brain.init() 1. Acquire locks/_writer.lock (or open read-only) 2. Crash recovery: drop _generations/{N} newer than manifest.json 3. Load _system singletons (counts, statistics, field registry, id mapper) 4. Vector index: hnsw-system.json + entity vectors.json (lazy mode if large) 5. Graph adjacency: load LSM SSTables from _system/idx/ 6. Metadata index: column-store manifests + bitmap chunks on demand ``` ### Snapshot and restore ``` const db = brain.now(); await db.persist('/backups/today'); await db.release() → hard-links everything except locks/ into a self-contained directory await Brainy.load('/backups/today') // open snapshot read-only as a Db await brain.restore('/backups/today', { confirm: true }) // replace store state ``` --- ## 13. Summary - **Two backends** (filesystem, memory), one path vocabulary. - **Two files per entity** under ID-first `entities/{kind}/{shard}/{id}/`. - **Type and subtype are metadata**, not directory structure; type queries go through the metadata index, not the filesystem. - **`_system/`** holds singletons plus 256 hash buckets of index state. - **`_generations/` + `manifest.json` + `tx-log.jsonl`** implement generational MVCC; `transact()` is the unit of history. - **`_column_index/` + `_blobs/`** hold the columnar metadata runs and binary blobs (VFS content included). - **`locks/`** coordinates the single writer and reader flush requests, and never travels with snapshots. --- ## Next Steps - [ADR-001 — Generational MVCC](../ADR-001-generational-mvcc.md) - [Index Architecture](./index-architecture.md) - [Consistency Model](../concepts/consistency-model.md) - [VFS Guide](../vfs/README.md)