17 KiB
Brainy Data Storage Architecture (8.0)
Complete on-disk reference for the 8.0 layout.
This document describes what a Brainy 8.0 data directory actually contains: the canonical entity records, the system area, the generational-MVCC bookkeeping, the column store, the blob area, and the lock files — plus how the in-memory indexes rebuild from them. The authoritative design records are ADR-001 (generational MVCC) and index-architecture.md; this document is the on-disk map that ties them together.
8.0 removed the 7.x copy-on-write subsystem (_cow/, branches/{branch}/…
paths) and the cloud/OPFS storage adapters. The two storage backends are
filesystem and memory; both speak the same path vocabulary (memory
storage keys its internal map by the identical path strings).
1. Directory Tree
A real 8.0 filesystem store (rootDirectory, default ./brainy-data):
brainy-data/
│
├── entities/ # Canonical records (current state)
│ ├── nouns/
│ │ └── {shard}/ # 256 shards: first 2 hex chars of the UUID
│ │ └── {id}/ # One directory per entity UUID
│ │ ├── vectors.json.gz # Embedding + HNSW node state
│ │ └── metadata.json.gz # Everything else (type, subtype, data, fields, _rev)
│ └── verbs/
│ └── {shard}/
│ └── {id}/
│ ├── vectors.json.gz # Relationship embedding (when present)
│ └── metadata.json.gz # sourceId, targetId, verb, subtype, weight, data…
│
├── _system/ # System singletons + bucketed system keys
│ ├── generation.json(.gz) # { generation, updatedAt } — the write watermark
│ ├── manifest.json # { version, generation, … } — MVCC commit point
│ ├── tx-log.jsonl # One line per committed transact() batch (append-only)
│ ├── counts.json # Entity/verb totals
│ ├── type-statistics.json.gz # Per-NounType counts
│ ├── subtype-statistics.json.gz # Per-(NounType, subtype) counts
│ ├── verb-subtype-statistics.json.gz # Per-(VerbType, subtype) counts
│ ├── statistics.json # Aggregate statistics blob (counts, index sizes)
│ ├── hnsw-system.json # Vector-index entry point + max level
│ ├── __metadata_field_registry__.json.gz # Which metadata fields are indexed
│ ├── brainy:entityIdMapper.json.gz # UUID ↔ u64 mapping for native index providers
│ └── idx/
│ └── {bucket}/ # 256 buckets: FNV-1a hash of the key
│ ├── __metadata_field_index__field_{name}.json.gz # Sparse field indexes
│ ├── __chunk__*.json.gz # Metadata-index roaring-bitmap chunks
│ ├── __sparse_index__*.json.gz# Zone maps + bloom filters
│ └── graph-lsm-verbs-{source|target}-*.json.gz # Graph LSM SSTables + manifest
│
├── _generations/ # MVCC history (written ONLY by transact())
│ └── {N}/ # One directory per committed generation N
│ ├── tx.json # The generation-N delta (immutable)
│ └── prev/
│ └── {id}.json # Before-image of each touched record (immutable)
│
├── _column_index/ # Column store manifests (one dir per field)
│ └── {field}/
│ └── MANIFEST.json.gz # Run list + zone metadata for that column
│
├── _blobs/ # Binary blob area (`<key>.bin` convention)
│ ├── _column_index/
│ │ └── {field}/
│ │ └── L0-000001.bin # Column-store runs (level-0 segments)
│ └── … # VFS file content and other binary blobs
│
└── locks/ # Process coordination (NEVER snapshotted)
├── _writer.lock # Single-writer lock: pid, hostname, heartbeat
├── _flush_requests/ # Reader→writer flush RPC (.req files)
└── _flush_responses/ # Writer acks (.ack files)
Most JSON objects are gzip-compressed (.json.gz) — compression is on by
default for filesystem storage (storage.options.compression, zlib level 6).
A few hot singletons (manifest.json, counts.json, hnsw-system.json,
tx-log.jsonl) are written uncompressed for cheap partial reads and appends.
2. Canonical Entity Records
Each entity (noun) and relationship (verb) is two files under one ID-first directory. The split keeps vector I/O (large, append-mostly) separate from metadata I/O (small, read-heavy).
Noun vector file — entities/nouns/{shard}/{id}/vectors.json.gz
{
"id": "421d92e7-4241-470a-80f4-4b39414e7a83",
"vector": [-0.139, -0.056, 0.028, "…384 dims…"],
"connections": { "0": ["neighbor-uuid…"] },
"level": 0
}
The HNSW node state (connections, level) is persisted with the vector so
the vector index can rebuild without recomputing the graph.
Noun metadata file — entities/nouns/{shard}/{id}/metadata.json.gz
{
"data": "React is a JavaScript library for building user interfaces",
"noun": "concept",
"subtype": "cli-add",
"createdAt": 1781198053726,
"updatedAt": 1781198053726,
"_rev": 1
}
nounis the NounType. Type lives in metadata, not in the path — lookup by ID is a single path construction, no type needed.subtypeis the per-product sub-classification (required on write by default in 8.0)._revincrements on every write and backsupdate({ ifRev })CAS.- Consumer metadata fields sit alongside the standard ones.
Verb files — entities/verbs/{shard}/{id}/…
Same two-file split. The verb metadata record carries the graph edge:
sourceId, targetId, verb (VerbType), subtype, weight, data,
metadata, timestamps, _rev. Verb IDs are Brainy-generated UUIDs by
contract (8.0 rejects caller-supplied verb ids) because native graph providers
intern the raw UUID bytes as u64 handles.
Path construction
const shard = id.substring(0, 2) // '42'
const metadataPath = `entities/nouns/${shard}/${id}/metadata.json`
const vectorsPath = `entities/nouns/${shard}/${id}/vectors.json`
// Verbs: same shape under entities/verbs/
Implemented in src/storage/baseStorage.ts (path generators) and
src/storage/sharding.ts (getShardIdFromUuid).
3. The _system/ Area
Two kinds of keys live here, resolved by BaseStorage.parsePath():
- Singletons — well-known keys written at
_system/<key>.json:counts,statistics,type-statistics,hnsw-system,__metadata_field_registry__,brainy:entityIdMapper, plus the MVCC trio (generation.json,manifest.json,tx-log.jsonl). - Bucketed system keys — everything else (field indexes, bitmap chunks,
sparse-index segments, graph LSM SSTables) hashes into one of 256
_system/idx/{bucket}/directories via FNV-1a, so no single directory accumulates unbounded entries.
Notable singletons:
| File | Contents |
|---|---|
generation.json |
{ generation, updatedAt } — monotonic watermark, bumped by every write batch |
manifest.json |
MVCC commit point: highest committed generation (see §4) |
tx-log.jsonl |
One JSON line per committed transact() batch: generation, timestamp, meta |
type-statistics.json.gz |
Per-NounType counts (backs brain.counts.byType) |
subtype-statistics.json.gz |
{ counts: { [type]: { [subtype]: n } }, updatedAt } (contract-bound shape) |
verb-subtype-statistics.json.gz |
Same shape for VerbTypes |
hnsw-system.json |
{ entryPointId, maxLevel } — per-node state lives in each entity's vectors.json |
brainy:entityIdMapper.json.gz |
UUID ↔ u64 interning table for the BigInt provider contract |
__metadata_field_registry__.json.gz |
Registry of indexed metadata field names |
4. Generational MVCC (_generations/ + the _system trio)
Full design in ADR-001. The on-disk shape:
_system/generation.json { generation, updatedAt } atomic tmp+rename
_system/manifest.json { version, generation, … } atomic tmp+rename — THE commit point
_system/tx-log.jsonl one line per committed transact() append-only
_generations/{N}/tx.json the generation-N delta immutable once written
_generations/{N}/prev/{id}.json before-image of {id} immutable once written
Commit protocol (writer side): stage before-images and the delta under
_generations/N/, fsync, apply the delta to the canonical entities/…
records, then atomically rename manifest.json to publish generation N. The
tx-log line is appended last (advisory). Crash recovery on open discards any
_generations/{N} newer than the manifest.
Two write classes share the generation clock:
- Single-operation writes (
add/update/remove/relateoutsidetransact()) bumpgeneration.jsonso watermarks and_revCAS stay sound, but write no history — they are not visible todb.since()and remain visible through earlier pins. transact()batches write the full_generations/{N}record and a tx-log line, and are the unit of time travel (brain.asOf()).
Snapshots (db.persist(path)) hard-link the entire store except locks/
into a self-contained directory openable via Brainy.load(path).
5. Column Store (_column_index/ + _blobs/_column_index/)
The metadata index persists per-field columnar runs for O(log n) range and membership queries at scale:
_column_index/{field}/MANIFEST.json.gz— the run list and zone metadata for one field (createdAt,subtype,noun,_rev, consumer fields,__words__for tokenized text…)._blobs/_column_index/{field}/L0-NNNNNN.bin— the actual level-0 run segments, stored through the shared_blobs/<key>.binbinary convention.
Sparse per-field indexes, roaring-bitmap chunks, and zone-map/bloom segments
additionally live as bucketed keys under _system/idx/ (see §3). Which path
serves a given where clause is the query planner's decision — inspect it
with brainy inspect explain <dir> --where '…'.
6. Blob Area (_blobs/)
_blobs/<key>.bin is the flat binary-blob convention shared by every storage
adapter (saveBinaryBlob/getBinaryBlob in the storage contract):
- VFS file content — VFS entities are regular nouns (path, ownership, and timestamps in entity metadata); the file bytes are blobs.
- Column-store runs (under the
_column_index/key prefix, §5). - Any other binary payload an index provider persists.
Writes use unique temp names + rename, so concurrent writers of the same key cannot tear each other's blobs.
7. Locks (locks/)
locks/_writer.lock # single-writer lock: { pid, hostname, startedAt, heartbeat, version }
locks/_flush_requests/ # readers drop <uuid>.req to ask the writer to flush
locks/_flush_responses/ # writer answers with <uuid>.ack
- One writer per data directory, enforced at
init(); stale locks (dead PID / stale heartbeat) are reclaimed automatically. - Read-only processes (
Brainy.openReadOnly(), thebrainy inspectCLI family) can ask the live writer to flush via the request/response files, so out-of-process diagnostics see fresh state. locks/is excluded from snapshots (SNAPSHOT_EXCLUDED_TOP_DIRSinsrc/storage/adapters/fileSystemStorage.ts).
8. In-Memory Indexes and What Rebuilds From What
| Index | In memory | Persisted state | Rebuild source |
|---|---|---|---|
| Vector (HNSW) | Graph of vector connections | _system/hnsw-system.json + per-entity vectors.json |
Walk entity vector files; lazy mode loads structure only and pages vectors on demand |
| Metadata index | Field → value bitmaps + column-store readers | _system/idx/ chunks + _column_index/ manifests + _blobs/_column_index/ runs |
Loaded directly; full rebuild re-scans entity metadata |
| Graph adjacency | sourceId/targetId → verb-id LSM trees | graph-lsm-verbs-{source,target}-* SSTables under _system/idx/ |
Loaded from SSTables; full rebuild re-scans verb metadata |
| Counts/statistics | Per-type and per-subtype maps | _system/{type,subtype,verb-subtype}-statistics.json.gz, counts.json |
Recomputable by scanning entities (brainy inspect repair) |
A pluggable index provider (the 8.0 plugin contract in
@soulcraft/brainy/plugin) may replace any of the JS implementations; the
persisted formats above are contract-bound so JS and native implementations
can interleave on the same directory.
9. Sharding Strategy
Entities: first 2 hex characters of the UUID → 256 uniform shards.
Deterministic, configuration-free, and keeps per-directory entry counts low
(at 1M entities: ~3,900 directories per shard). Paginated whole-store walks
(getNouns/getVerbs) iterate shards 00–ff in order.
System keys: FNV-1a hash of the key → 256 _system/idx/ buckets. Same
motivation, different keyspace (system keys are not UUIDs).
What is never sharded: the _system/ singletons, _generations/{N}
directories (keyed by generation number), _column_index/{field} manifests
(keyed by field name), and locks/.
10. Durability and Atomicity
- Per-object atomicity: every JSON object and blob is written to a unique
temp file then
rename()d — readers never observe torn objects. - Transaction atomicity: the
manifest.jsonrename is the single commit point fortransact()batches (§4); everything staged before it is discarded by crash recovery if the rename never lands. - Compression: gzip per object (
.json.gz), transparent to all readers. Native index providers that mmap binary formats use the uncompressed_blobs/area instead.
11. clear() Semantics
brain.clear() removes all entities, relationships, indexes, statistics, and
MVCC history, then re-resolves every index exactly as init() does —
including plugin-provided vector/metadata/id-mapper factories and VFS root
re-creation. The data directory afterwards contains a fresh, empty store (the
writer lock remains held by the running process).
12. Common Scenarios
Adding an entity
brain.add({ data, type, subtype })
1. Generate UUID → shard = first 2 hex chars
2. Embed data → 384-dim vector
3. Write entities/nouns/{shard}/{id}/vectors.json.gz (vector + HNSW node state)
4. Write entities/nouns/{shard}/{id}/metadata.json.gz (type/subtype/data/fields, _rev: 1)
5. Update in-memory indexes (HNSW insert, metadata index, statistics)
6. Bump _system/generation.json (no _generations/ entry — single-op write)
Committing a transaction
await brain.transact(tx => { tx.add(…); tx.update(…) })
1. Stage _generations/{N}/prev/{id}.json before-images + tx.json delta; fsync
2. Apply the delta to canonical entities/… records
3. Atomic-rename _system/manifest.json → generation N is committed
4. Append one line to _system/tx-log.jsonl
Cold start
await brain.init()
1. Acquire locks/_writer.lock (or open read-only)
2. Crash recovery: drop _generations/{N} newer than manifest.json
3. Load _system singletons (counts, statistics, field registry, id mapper)
4. Vector index: hnsw-system.json + entity vectors.json (lazy mode if large)
5. Graph adjacency: load LSM SSTables from _system/idx/
6. Metadata index: column-store manifests + bitmap chunks on demand
Snapshot and restore
const db = brain.now(); await db.persist('/backups/today'); await db.release()
→ hard-links everything except locks/ into a self-contained directory
await Brainy.load('/backups/today') // open snapshot read-only as a Db
await brain.restore('/backups/today', { confirm: true }) // replace store state
13. Summary
- Two backends (filesystem, memory), one path vocabulary.
- Two files per entity under ID-first
entities/{kind}/{shard}/{id}/. - Type and subtype are metadata, not directory structure; type queries go through the metadata index, not the filesystem.
_system/holds singletons plus 256 hash buckets of index state._generations/+manifest.json+tx-log.jsonlimplement generational MVCC;transact()is the unit of history._column_index/+_blobs/hold the columnar metadata runs and binary blobs (VFS content included).locks/coordinates the single writer and reader flush requests, and never travels with snapshots.