open-brainy/docs/architecture/data-storage-architecture.md

Brainy Data Storage Architecture (8.0)

Complete on-disk reference for the 8.0 layout.

This document describes what a Brainy 8.0 data directory actually contains: the canonical entity records, the system area, the generational-MVCC bookkeeping, the column store, the blob area, and the lock files — plus how the in-memory indexes rebuild from them. The authoritative design records are ADR-001 (generational MVCC) and index-architecture.md; this document is the on-disk map that ties them together.

8.0 removed the 7.x copy-on-write subsystem (_cow/, branches/{branch}/… paths) and the cloud/OPFS storage adapters. The two storage backends are filesystem and memory; both speak the same path vocabulary (memory storage keys its internal map by the identical path strings).

---

1. Directory Tree

A real 8.0 filesystem store (path, default ./brainy-data):

brainy-data/
│
├── entities/                            # Canonical records (current state)
│   ├── nouns/
│   │   └── {shard}/                     # 256 shards: first 2 hex chars of the UUID
│   │       └── {id}/                    # One directory per entity UUID
│   │           ├── vectors.json.gz      # Embedding + HNSW node state
│   │           └── metadata.json.gz     # Everything else (type, subtype, data, fields, _rev)
│   └── verbs/
│       └── {shard}/
│           └── {id}/
│               ├── vectors.json.gz      # Relationship embedding (when present)
│               └── metadata.json.gz     # sourceId, targetId, verb, subtype, weight, data…
│
├── _system/                             # System singletons + bucketed system keys
│   ├── generation.json(.gz)             # { generation, updatedAt } — the write watermark
│   ├── manifest.json                    # { version, generation, … } — MVCC commit point
│   ├── tx-log.jsonl                     # One line per committed transact() batch (append-only)
│   ├── counts.json                      # Entity/verb totals
│   ├── type-statistics.json.gz          # Per-NounType counts
│   ├── subtype-statistics.json.gz       # Per-(NounType, subtype) counts
│   ├── verb-subtype-statistics.json.gz  # Per-(VerbType, subtype) counts
│   ├── statistics.json                  # Aggregate statistics blob (counts, index sizes)
│   ├── hnsw-system.json                 # Vector-index entry point + max level
│   ├── __metadata_field_registry__.json.gz   # Which metadata fields are indexed
│   ├── brainy:entityIdMapper.json.gz    # UUID ↔ u64 mapping for native index providers
│   └── idx/
│       └── {bucket}/                    # 256 buckets: FNV-1a hash of the key
│           ├── __metadata_field_index__field_{name}.json.gz   # Sparse field indexes
│           ├── __chunk__*.json.gz       # Metadata-index roaring-bitmap chunks
│           ├── __sparse_index__*.json.gz# Zone maps + bloom filters
│           └── graph-lsm-verbs-{source|target}-*.json.gz      # Graph LSM SSTables + manifest
│
├── _generations/                        # MVCC history (written ONLY by transact())
│   └── {N}/                             # One directory per committed generation N
│       ├── tx.json                      # The generation-N delta (immutable)
│       └── prev/
│           └── {id}.json                # Before-image of each touched record (immutable)
│
├── _column_index/                       # Column store manifests (one dir per field)
│   └── {field}/
│       └── MANIFEST.json.gz             # Run list + zone metadata for that column
│
├── _blobs/                              # Binary blob area (`<key>.bin` convention)
│   ├── _column_index/
│   │   └── {field}/
│   │       └── L0-000001.bin            # Column-store runs (level-0 segments)
│   └── …                                # VFS file content and other binary blobs
│
└── locks/                               # Process coordination (NEVER snapshotted)
    ├── _writer.lock                     # Single-writer lock: pid, hostname, heartbeat
    ├── _flush_requests/                 # Reader→writer flush RPC (.req files)
    └── _flush_responses/                # Writer acks (.ack files)

Most JSON objects are gzip-compressed (.json.gz) — compression is on by default for filesystem storage (storage.options.compression, zlib level 6). A few hot singletons (manifest.json, counts.json, hnsw-system.json, tx-log.jsonl) are written uncompressed for cheap partial reads and appends.

---

2. Canonical Entity Records

Each entity (noun) and relationship (verb) is two files under one ID-first directory. The split keeps vector I/O (large, append-mostly) separate from metadata I/O (small, read-heavy).

Noun vector file — entities/nouns/{shard}/{id}/vectors.json.gz

{
  "id": "421d92e7-4241-470a-80f4-4b39414e7a83",
  "vector": [-0.139, -0.056, 0.028, "…384 dims…"],
  "connections": { "0": ["neighbor-uuid…"] },
  "level": 0
}

The HNSW node state (connections, level) is persisted with the vector so the vector index can rebuild without recomputing the graph.

Noun metadata file — entities/nouns/{shard}/{id}/metadata.json.gz

{
  "data": "React is a JavaScript library for building user interfaces",
  "noun": "concept",
  "subtype": "cli-add",
  "createdAt": 1781198053726,
  "updatedAt": 1781198053726,
  "_rev": 1
}

• noun is the NounType. Type lives in metadata, not in the path — lookup by ID is a single path construction, no type needed.
• subtype is the per-product sub-classification (required on write by default in 8.0).
• _rev increments on every write and backs update({ ifRev }) CAS.
• Consumer metadata fields sit alongside the standard ones.

Verb files — entities/verbs/{shard}/{id}/…

Same two-file split. The verb metadata record carries the graph edge: sourceId, targetId, verb (VerbType), subtype, weight, data, metadata, timestamps, _rev. Verb IDs are Brainy-generated UUIDs by contract (8.0 rejects caller-supplied verb ids) because native graph providers intern the raw UUID bytes as u64 handles.

Path construction

const shard = id.substring(0, 2)                                  // '42'
const metadataPath = `entities/nouns/${shard}/${id}/metadata.json`
const vectorsPath  = `entities/nouns/${shard}/${id}/vectors.json`
// Verbs: same shape under entities/verbs/

Implemented in src/storage/baseStorage.ts (path generators) and src/storage/sharding.ts (getShardIdFromUuid).

---

3. The _system/ Area

Two kinds of keys live here, resolved by BaseStorage.parsePath():

1. Singletons — well-known keys written at _system/<key>.json: counts, statistics, type-statistics, hnsw-system, __metadata_field_registry__, brainy:entityIdMapper, plus the MVCC trio (generation.json, manifest.json, tx-log.jsonl).
2. Bucketed system keys — everything else (field indexes, bitmap chunks, sparse-index segments, graph LSM SSTables) hashes into one of 256 _system/idx/{bucket}/ directories via FNV-1a, so no single directory accumulates unbounded entries.

Notable singletons:

File	Contents
generation.json	{ generation, updatedAt } — monotonic watermark, bumped by every write batch
manifest.json	MVCC commit point: highest committed generation (see §4)
tx-log.jsonl	One JSON line per committed transact() batch: generation, timestamp, meta
type-statistics.json.gz	Per-NounType counts (backs brain.counts.byType)
subtype-statistics.json.gz	{ counts: { [type]: { [subtype]: n } }, updatedAt } (contract-bound shape)
verb-subtype-statistics.json.gz	Same shape for VerbTypes
hnsw-system.json	{ entryPointId, maxLevel } — per-node state lives in each entity's vectors.json
brainy:entityIdMapper.json.gz	UUID ↔ u64 interning table for the BigInt provider contract
__metadata_field_registry__.json.gz	Registry of indexed metadata field names

---

4. Generational MVCC (_generations/ + the _system trio)

Full design in ADR-001. The on-disk shape:

_system/generation.json      { generation, updatedAt }          atomic tmp+rename
_system/manifest.json        { version, generation, … }         atomic tmp+rename — THE commit point
_system/tx-log.jsonl         one line per committed transact()  append-only
_generations/{N}/tx.json     the generation-N delta             immutable once written
_generations/{N}/prev/{id}.json   before-image of {id}          immutable once written

Commit protocol (writer side): stage before-images and the delta under _generations/N/, fsync, apply the delta to the canonical entities/… records, then atomically rename manifest.json to publish generation N. The tx-log line is appended last (advisory). Crash recovery on open discards any _generations/{N} newer than the manifest.

Two write classes share the generation clock:

• Single-operation writes (add/update/remove/relate outside transact()) bump generation.json so watermarks and _rev CAS stay sound, but write no history — they are not visible to db.since() and remain visible through earlier pins.
• transact() batches write the full _generations/{N} record and a tx-log line, and are the unit of time travel (brain.asOf()).

Snapshots (db.persist(path)) hard-link the entire store except locks/ into a self-contained directory openable via Brainy.load(path).

---

5. Column Store (_column_index/ + _blobs/_column_index/)

The metadata index persists per-field columnar runs for O(log n) range and membership queries at scale:

• _column_index/{field}/MANIFEST.json.gz — the run list and zone metadata for one field (createdAt, subtype, noun, _rev, consumer fields, __words__ for tokenized text…).
• _blobs/_column_index/{field}/L0-NNNNNN.bin — the actual level-0 run segments, stored through the shared _blobs/<key>.bin binary convention.
• _column_index/{field}/k/{kind}/… — the same two files again, for a second value kind on the same field (see below). Absent for a field that holds one kind, which is nearly all of them.

One posting column per (field, kind)

A field is not obliged to hold one type of value. category may carry 'electronics' on some rows and 5 on others, and both are real values of that field. A segment, though, has one encoding — i64, f64, UTF-8, or boolean — so a field that holds several kinds gets one column per kind:

• The first kind a field ever sees owns the plain _column_index/{field}/ layout above. A single-kind field is therefore byte-identical to what earlier versions wrote, and an index written before typed postings opens unchanged.
• Every later kind gets its own column beside it at _column_index/{field}/k/{kind}/, where {kind} is number, string or boolean.

What that buys at query time:

Equality	Answered from the column matching the query value's own kind. where {category: 5} reads the number postings; where {category: '5'} reads the string postings. Neither borrows the other's rows — a row written with the number 5 is not a row whose category is the text '5'.
A kind the field never held	Matches nothing. That is the true answer, not a coerced one.
Ranges	Routed by the kind of the bounds: numeric bounds read the numeric postings and ignore the field's strings. An unbounded range is the "has any value here" probe behind exists, and reads every kind.
orderBy	A number and a string have no order between them, so a mixed field orders by kind first (number, string, boolean) and by value within a kind. A single-kind field sorts exactly as it always did.
Numbers	One kind, one column: an integer column is written as i64 and widens to f64 the first time a non-integer arrives, so 4.5 is stored as itself rather than rounded.

null and undefined are not kinds and are never posted; their absence is what the exists / missing operators read.

Older readers are unaffected by the additional columns: they see the field's primary column exactly where it has always been, and a k/{kind} directory is simply a name they never query.

Sparse per-field indexes, roaring-bitmap chunks, and zone-map/bloom segments additionally live as bucketed keys under _system/idx/ (see §3). Which path serves a given where clause is the query planner's decision — inspect it with brainy inspect explain <dir> --where '…'.

---

6. Blob Area (_blobs/)

_blobs/<key>.bin is the flat binary-blob convention shared by every storage adapter (saveBinaryBlob/getBinaryBlob in the storage contract):

• VFS file content — VFS entities are regular nouns (path, ownership, and timestamps in entity metadata); the file bytes are blobs.
• Column-store runs (under the _column_index/ key prefix, §5).
• Any other binary payload an index provider persists.

Writes use unique temp names + rename, so concurrent writers of the same key cannot tear each other's blobs.

---

7. Locks (locks/)

locks/_writer.lock          # single-writer lock: { pid, hostname, startedAt, heartbeat, version }
locks/_flush_requests/      # readers drop <uuid>.req to ask the writer to flush
locks/_flush_responses/     # writer answers with <uuid>.ack

• One writer per data directory, enforced at init(); stale locks (dead PID / stale heartbeat) are reclaimed automatically.
• Read-only processes (Brainy.openReadOnly(), the brainy inspect CLI family) can ask the live writer to flush via the request/response files, so out-of-process diagnostics see fresh state.
• locks/ is excluded from snapshots (SNAPSHOT_EXCLUDED_TOP_DIRS in src/storage/adapters/fileSystemStorage.ts).

---

8. In-Memory Indexes and What Rebuilds From What

Index	In memory	Persisted state	Rebuild source
Vector (HNSW)	Graph of vector connections	_system/hnsw-system.json + per-entity vectors.json	Walk entity vector files; lazy mode loads structure only and pages vectors on demand
Metadata index	Field → value bitmaps + column-store readers	_system/idx/ chunks + _column_index/ manifests + _blobs/_column_index/ runs	Loaded directly; full rebuild re-scans entity metadata
Graph adjacency	sourceId/targetId → verb-id LSM trees	graph-lsm-verbs-{source,target}-* SSTables under _system/idx/	Loaded from SSTables; full rebuild re-scans verb metadata
Counts/statistics	Per-type and per-subtype maps	_system/{type,subtype,verb-subtype}-statistics.json.gz, counts.json	Recomputable by scanning entities (brainy inspect repair)

A pluggable index provider (the 8.0 plugin contract in @soulcraftlabs/brainy/plugin) may replace any of the JS implementations; the persisted formats above are contract-bound so JS and native implementations can interleave on the same directory.

---

9. Sharding Strategy

Entities: first 2 hex characters of the UUID → 256 uniform shards. Deterministic, configuration-free, and keeps per-directory entry counts low (at 1M entities: ~3,900 directories per shard). Paginated whole-store walks (getNouns/getVerbs) iterate shards 00–ff in order.

System keys: FNV-1a hash of the key → 256 _system/idx/ buckets. Same motivation, different keyspace (system keys are not UUIDs).

What is never sharded: the _system/ singletons, _generations/{N} directories (keyed by generation number), _column_index/{field} manifests (keyed by field name), and locks/.

---

10. Durability and Atomicity

• Per-object atomicity: every JSON object and blob is written to a unique temp file then rename()d — readers never observe torn objects.
• Transaction atomicity: the manifest.json rename is the single commit point for transact() batches (§4); everything staged before it is discarded by crash recovery if the rename never lands.
• Compression: gzip per object (.json.gz), transparent to all readers. Native index providers that mmap binary formats use the uncompressed _blobs/ area instead.

---

11. clear() Semantics

brain.clear() removes all entities, relationships, indexes, statistics, and MVCC history, then re-resolves every index exactly as init() does — including plugin-provided vector/metadata/id-mapper factories and VFS root re-creation. The data directory afterwards contains a fresh, empty store (the writer lock remains held by the running process).

---

12. Common Scenarios

Adding an entity

brain.add({ data, type, subtype })
1. Generate UUID → shard = first 2 hex chars
2. Embed data → 384-dim vector
3. Write entities/nouns/{shard}/{id}/vectors.json.gz   (vector + HNSW node state)
4. Write entities/nouns/{shard}/{id}/metadata.json.gz  (type/subtype/data/fields, _rev: 1)
5. Update in-memory indexes (HNSW insert, metadata index, statistics)
6. Bump _system/generation.json (no _generations/ entry — single-op write)

Committing a transaction

await brain.transact(tx => { tx.add(…); tx.update(…) })
1. Stage _generations/{N}/prev/{id}.json before-images + tx.json delta; fsync
2. Apply the delta to canonical entities/… records
3. Atomic-rename _system/manifest.json → generation N is committed
4. Append one line to _system/tx-log.jsonl

Cold start

await brain.init()
1. Acquire locks/_writer.lock (or open read-only)
2. Crash recovery: drop _generations/{N} newer than manifest.json
3. Load _system singletons (counts, statistics, field registry, id mapper)
4. Vector index: hnsw-system.json + entity vectors.json (lazy mode if large)
5. Graph adjacency: load LSM SSTables from _system/idx/
6. Metadata index: column-store manifests + bitmap chunks on demand

Snapshot and restore

const db = brain.now(); await db.persist('/backups/today'); await db.release()
→ hard-links everything except locks/ into a self-contained directory

await Brainy.load('/backups/today')        // open snapshot read-only as a Db
await brain.restore('/backups/today', { confirm: true })   // replace store state

---

13. Summary

• Two backends (filesystem, memory), one path vocabulary.
• Two files per entity under ID-first entities/{kind}/{shard}/{id}/.
• Type and subtype are metadata, not directory structure; type queries go through the metadata index, not the filesystem.
• _system/ holds singletons plus 256 hash buckets of index state.
• _generations/ + manifest.json + tx-log.jsonl implement generational MVCC. History is per-write: every add()/update()/remove()/ relate() gets its own generation, and transact() groups several ops into one atomic generation. (Single-op retention has been the model since 8.0; you never need to route a write through transact() just to keep its history.)
• _column_index/ + _blobs/ hold the columnar metadata runs and binary blobs (VFS content included).
• locks/ coordinates the single writer and reader flush requests, and never travels with snapshots.

---

Next Steps

• ADR-001 — Generational MVCC
• Index Architecture
• Consistency Model
• VFS Guide