open-brainy/docs/architecture/data-storage-architecture.md

# Brainy Data Storage Architecture (8.0)

**Complete on-disk reference for the 8.0 layout.**

This document describes what a Brainy 8.0 data directory actually contains: the
canonical entity records, the system area, the generational-MVCC bookkeeping,
the column store, the blob area, and the lock files — plus how the in-memory
indexes rebuild from them. The authoritative design records are
[ADR-001 (generational MVCC)](../ADR-001-generational-mvcc.md) and
[index-architecture.md](./index-architecture.md); this document is the on-disk
map that ties them together.

8.0 removed the 7.x copy-on-write subsystem (`_cow/`, `branches/{branch}/…`
paths) and the cloud/OPFS storage adapters. The two storage backends are
**filesystem** and **memory**; both speak the same path vocabulary (memory
storage keys its internal map by the identical path strings).

---

## 1. Directory Tree

A real 8.0 filesystem store (`path`, default `./brainy-data`):

```
brainy-data/
│
├── entities/                            # Canonical records (current state)
│   ├── nouns/
│   │   └── {shard}/                     # 256 shards: first 2 hex chars of the UUID
│   │       └── {id}/                    # One directory per entity UUID
│   │           ├── vectors.json.gz      # Embedding + HNSW node state
│   │           └── metadata.json.gz     # Everything else (type, subtype, data, fields, _rev)
│   └── verbs/
│       └── {shard}/
│           └── {id}/
│               ├── vectors.json.gz      # Relationship embedding (when present)
│               └── metadata.json.gz     # sourceId, targetId, verb, subtype, weight, data…
│
├── _system/                             # System singletons + bucketed system keys
│   ├── generation.json(.gz)             # { generation, updatedAt } — the write watermark
│   ├── manifest.json                    # { version, generation, … } — MVCC commit point
│   ├── tx-log.jsonl                     # One line per committed transact() batch (append-only)
│   ├── counts.json                      # Entity/verb totals
│   ├── type-statistics.json.gz          # Per-NounType counts
│   ├── subtype-statistics.json.gz       # Per-(NounType, subtype) counts
│   ├── verb-subtype-statistics.json.gz  # Per-(VerbType, subtype) counts
│   ├── statistics.json                  # Aggregate statistics blob (counts, index sizes)
│   ├── hnsw-system.json                 # Vector-index entry point + max level
│   ├── __metadata_field_registry__.json.gz   # Which metadata fields are indexed
│   ├── brainy:entityIdMapper.json.gz    # UUID ↔ u64 mapping for native index providers
│   └── idx/
│       └── {bucket}/                    # 256 buckets: FNV-1a hash of the key
│           ├── __metadata_field_index__field_{name}.json.gz   # Sparse field indexes
│           ├── __chunk__*.json.gz       # Metadata-index roaring-bitmap chunks
│           ├── __sparse_index__*.json.gz# Zone maps + bloom filters
│           └── graph-lsm-verbs-{source|target}-*.json.gz      # Graph LSM SSTables + manifest
│
├── _generations/                        # MVCC history (written ONLY by transact())
│   └── {N}/                             # One directory per committed generation N
│       ├── tx.json                      # The generation-N delta (immutable)
│       └── prev/
│           └── {id}.json                # Before-image of each touched record (immutable)
│
├── _column_index/                       # Column store manifests (one dir per field)
│   └── {field}/
│       └── MANIFEST.json.gz             # Run list + zone metadata for that column
│
├── _blobs/                              # Binary blob area (`<key>.bin` convention)
│   ├── _column_index/
│   │   └── {field}/
│   │       └── L0-000001.bin            # Column-store runs (level-0 segments)
│   └── …                                # VFS file content and other binary blobs
│
└── locks/                               # Process coordination (NEVER snapshotted)
    ├── _writer.lock                     # Single-writer lock: pid, hostname, heartbeat
    ├── _flush_requests/                 # Reader→writer flush RPC (.req files)
    └── _flush_responses/                # Writer acks (.ack files)
```

Most JSON objects are gzip-compressed (`.json.gz`) — compression is on by
default for filesystem storage (`storage.options.compression`, zlib level 6).
A few hot singletons (`manifest.json`, `counts.json`, `hnsw-system.json`,
`tx-log.jsonl`) are written uncompressed for cheap partial reads and appends.

---

## 2. Canonical Entity Records

Each entity (noun) and relationship (verb) is **two files** under one
ID-first directory. The split keeps vector I/O (large, append-mostly) separate
from metadata I/O (small, read-heavy).

### Noun vector file — `entities/nouns/{shard}/{id}/vectors.json.gz`

```json
{
  "id": "421d92e7-4241-470a-80f4-4b39414e7a83",
  "vector": [-0.139, -0.056, 0.028, "…384 dims…"],
  "connections": { "0": ["neighbor-uuid…"] },
  "level": 0
}
```

The HNSW node state (`connections`, `level`) is persisted with the vector so
the vector index can rebuild without recomputing the graph.

### Noun metadata file — `entities/nouns/{shard}/{id}/metadata.json.gz`

```json
{
  "data": "React is a JavaScript library for building user interfaces",
  "noun": "concept",
  "subtype": "cli-add",
  "createdAt": 1781198053726,
  "updatedAt": 1781198053726,
  "_rev": 1
}
```

- `noun` is the NounType. **Type lives in metadata, not in the path** — lookup
  by ID is a single path construction, no type needed.
- `subtype` is the per-product sub-classification (required on write by
  default in 8.0).
- `_rev` increments on every write and backs `update({ ifRev })` CAS.
- Consumer metadata fields sit alongside the standard ones.

### Verb files — `entities/verbs/{shard}/{id}/…`

Same two-file split. The verb metadata record carries the graph edge:
`sourceId`, `targetId`, `verb` (VerbType), `subtype`, `weight`, `data`,
`metadata`, timestamps, `_rev`. Verb IDs are Brainy-generated UUIDs by
contract (8.0 rejects caller-supplied verb ids) because native graph providers
intern the raw UUID bytes as u64 handles.

### Path construction

```typescript
const shard = id.substring(0, 2)                                  // '42'
const metadataPath = `entities/nouns/${shard}/${id}/metadata.json`
const vectorsPath  = `entities/nouns/${shard}/${id}/vectors.json`
// Verbs: same shape under entities/verbs/
```

Implemented in `src/storage/baseStorage.ts` (path generators) and
`src/storage/sharding.ts` (`getShardIdFromUuid`).

---

## 3. The `_system/` Area

Two kinds of keys live here, resolved by `BaseStorage.parsePath()`:

1. **Singletons** — well-known keys written at `_system/<key>.json`:
   `counts`, `statistics`, `type-statistics`, `hnsw-system`,
   `__metadata_field_registry__`, `brainy:entityIdMapper`, plus the MVCC
   trio (`generation.json`, `manifest.json`, `tx-log.jsonl`).
2. **Bucketed system keys** — everything else (field indexes, bitmap chunks,
   sparse-index segments, graph LSM SSTables) hashes into one of 256
   `_system/idx/{bucket}/` directories via FNV-1a, so no single directory
   accumulates unbounded entries.

Notable singletons:

| File | Contents |
|------|----------|
| `generation.json` | `{ generation, updatedAt }` — monotonic watermark, bumped by **every** write batch |
| `manifest.json` | MVCC commit point: highest *committed* generation (see §4) |
| `tx-log.jsonl` | One JSON line per committed `transact()` batch: generation, timestamp, `meta` |
| `type-statistics.json.gz` | Per-NounType counts (backs `brain.counts.byType`) |
| `subtype-statistics.json.gz` | `{ counts: { [type]: { [subtype]: n } }, updatedAt }` (contract-bound shape) |
| `verb-subtype-statistics.json.gz` | Same shape for VerbTypes |
| `hnsw-system.json` | `{ entryPointId, maxLevel }` — per-node state lives in each entity's `vectors.json` |
| `brainy:entityIdMapper.json.gz` | UUID ↔ u64 interning table for the BigInt provider contract |
| `__metadata_field_registry__.json.gz` | Registry of indexed metadata field names |

---

## 4. Generational MVCC (`_generations/` + the `_system` trio)

Full design in [ADR-001](../ADR-001-generational-mvcc.md). The on-disk shape:

```
_system/generation.json      { generation, updatedAt }          atomic tmp+rename
_system/manifest.json        { version, generation, … }         atomic tmp+rename — THE commit point
_system/tx-log.jsonl         one line per committed transact()  append-only
_generations/{N}/tx.json     the generation-N delta             immutable once written
_generations/{N}/prev/{id}.json   before-image of {id}          immutable once written
```

Commit protocol (writer side): stage before-images and the delta under
`_generations/N/`, fsync, apply the delta to the canonical `entities/…`
records, then atomically rename `manifest.json` to publish generation N. The
tx-log line is appended last (advisory). Crash recovery on open discards any
`_generations/{N}` newer than the manifest.

Two write classes share the generation clock:

- **Single-operation writes** (`add`/`update`/`remove`/`relate` outside
  `transact()`) bump `generation.json` so watermarks and `_rev` CAS stay sound,
  but write **no** history — they are not visible to `db.since()` and remain
  visible through earlier pins.
- **`transact()` batches** write the full `_generations/{N}` record and a
  tx-log line, and are the unit of time travel (`brain.asOf()`).

Snapshots (`db.persist(path)`) hard-link the entire store **except `locks/`**
into a self-contained directory openable via `Brainy.load(path)`.

---

## 5. Column Store (`_column_index/` + `_blobs/_column_index/`)

The metadata index persists per-field columnar runs for O(log n) range and
membership queries at scale:

- `_column_index/{field}/MANIFEST.json.gz` — the run list and zone metadata
  for one field (`createdAt`, `subtype`, `noun`, `_rev`, consumer fields,
  `__words__` for tokenized text…).
- `_blobs/_column_index/{field}/L0-NNNNNN.bin` — the actual level-0 run
  segments, stored through the shared `_blobs/<key>.bin` binary convention.

Sparse per-field indexes, roaring-bitmap chunks, and zone-map/bloom segments
additionally live as bucketed keys under `_system/idx/` (see §3). Which path
serves a given `where` clause is the query planner's decision — inspect it
with `brainy inspect explain <dir> --where '…'`.

---

## 6. Blob Area (`_blobs/`)

`_blobs/<key>.bin` is the flat binary-blob convention shared by every storage
adapter (`saveBinaryBlob`/`getBinaryBlob` in the storage contract):

- **VFS file content** — VFS entities are regular nouns (path, ownership, and
  timestamps in entity metadata); the file *bytes* are blobs.
- **Column-store runs** (under the `_column_index/` key prefix, §5).
- Any other binary payload an index provider persists.

Writes use unique temp names + rename, so concurrent writers of the same key
cannot tear each other's blobs.

---

## 7. Locks (`locks/`)

```
locks/_writer.lock          # single-writer lock: { pid, hostname, startedAt, heartbeat, version }
locks/_flush_requests/      # readers drop <uuid>.req to ask the writer to flush
locks/_flush_responses/     # writer answers with <uuid>.ack
```

- One **writer** per data directory, enforced at `init()`; stale locks (dead
  PID / stale heartbeat) are reclaimed automatically.
- Read-only processes (`Brainy.openReadOnly()`, the `brainy inspect` CLI
  family) can ask the live writer to flush via the request/response files, so
  out-of-process diagnostics see fresh state.
- `locks/` is excluded from snapshots (`SNAPSHOT_EXCLUDED_TOP_DIRS` in
  `src/storage/adapters/fileSystemStorage.ts`).

---

## 8. In-Memory Indexes and What Rebuilds From What

| Index | In memory | Persisted state | Rebuild source |
|-------|-----------|-----------------|----------------|
| **Vector (HNSW)** | Graph of vector connections | `_system/hnsw-system.json` + per-entity `vectors.json` | Walk entity vector files; lazy mode loads structure only and pages vectors on demand |
| **Metadata index** | Field → value bitmaps + column-store readers | `_system/idx/` chunks + `_column_index/` manifests + `_blobs/_column_index/` runs | Loaded directly; full rebuild re-scans entity metadata |
| **Graph adjacency** | sourceId/targetId → verb-id LSM trees | `graph-lsm-verbs-{source,target}-*` SSTables under `_system/idx/` | Loaded from SSTables; full rebuild re-scans verb metadata |
| **Counts/statistics** | Per-type and per-subtype maps | `_system/{type,subtype,verb-subtype}-statistics.json.gz`, `counts.json` | Recomputable by scanning entities (`brainy inspect repair`) |

A pluggable index provider (the 8.0 plugin contract in
`@soulcraft/brainy/plugin`) may replace any of the JS implementations; the
persisted formats above are contract-bound so JS and native implementations
can interleave on the same directory.

---

## 9. Sharding Strategy

**Entities:** first 2 hex characters of the UUID → 256 uniform shards.
Deterministic, configuration-free, and keeps per-directory entry counts low
(at 1M entities: ~3,900 directories per shard). Paginated whole-store walks
(`getNouns`/`getVerbs`) iterate shards `00`–`ff` in order.

**System keys:** FNV-1a hash of the key → 256 `_system/idx/` buckets. Same
motivation, different keyspace (system keys are not UUIDs).

**What is never sharded:** the `_system/` singletons, `_generations/{N}`
directories (keyed by generation number), `_column_index/{field}` manifests
(keyed by field name), and `locks/`.

---

## 10. Durability and Atomicity

- **Per-object atomicity:** every JSON object and blob is written to a unique
  temp file then `rename()`d — readers never observe torn objects.
- **Transaction atomicity:** the `manifest.json` rename is the single commit
  point for `transact()` batches (§4); everything staged before it is
  discarded by crash recovery if the rename never lands.
- **Compression:** gzip per object (`.json.gz`), transparent to all readers.
  Native index providers that mmap binary formats use the uncompressed
  `_blobs/` area instead.

---

## 11. `clear()` Semantics

`brain.clear()` removes all entities, relationships, indexes, statistics, and
MVCC history, then re-resolves every index exactly as `init()` does —
including plugin-provided vector/metadata/id-mapper factories and VFS root
re-creation. The data directory afterwards contains a fresh, empty store (the
writer lock remains held by the running process).

---

## 12. Common Scenarios

### Adding an entity

```
brain.add({ data, type, subtype })
1. Generate UUID → shard = first 2 hex chars
2. Embed data → 384-dim vector
3. Write entities/nouns/{shard}/{id}/vectors.json.gz   (vector + HNSW node state)
4. Write entities/nouns/{shard}/{id}/metadata.json.gz  (type/subtype/data/fields, _rev: 1)
5. Update in-memory indexes (HNSW insert, metadata index, statistics)
6. Bump _system/generation.json (no _generations/ entry — single-op write)
```

### Committing a transaction

```
await brain.transact(tx => { tx.add(…); tx.update(…) })
1. Stage _generations/{N}/prev/{id}.json before-images + tx.json delta; fsync
2. Apply the delta to canonical entities/… records
3. Atomic-rename _system/manifest.json → generation N is committed
4. Append one line to _system/tx-log.jsonl
```

### Cold start

```
await brain.init()
1. Acquire locks/_writer.lock (or open read-only)
2. Crash recovery: drop _generations/{N} newer than manifest.json
3. Load _system singletons (counts, statistics, field registry, id mapper)
4. Vector index: hnsw-system.json + entity vectors.json (lazy mode if large)
5. Graph adjacency: load LSM SSTables from _system/idx/
6. Metadata index: column-store manifests + bitmap chunks on demand
```

### Snapshot and restore

```
const db = brain.now(); await db.persist('/backups/today'); await db.release()
→ hard-links everything except locks/ into a self-contained directory

await Brainy.load('/backups/today')        // open snapshot read-only as a Db
await brain.restore('/backups/today', { confirm: true })   // replace store state
```

---

## 13. Summary

- **Two backends** (filesystem, memory), one path vocabulary.
- **Two files per entity** under ID-first `entities/{kind}/{shard}/{id}/`.
- **Type and subtype are metadata**, not directory structure; type queries go
  through the metadata index, not the filesystem.
- **`_system/`** holds singletons plus 256 hash buckets of index state.
- **`_generations/` + `manifest.json` + `tx-log.jsonl`** implement
  generational MVCC. History is per-write: every `add()`/`update()`/`remove()`/
  `relate()` gets its own generation, and `transact()` groups several ops into
  one atomic generation. (Single-op retention has been the model since 8.0;
  you never need to route a write through `transact()` just to keep its history.)
- **`_column_index/` + `_blobs/`** hold the columnar metadata runs and binary
  blobs (VFS content included).
- **`locks/`** coordinates the single writer and reader flush requests, and
  never travels with snapshots.

---

## Next Steps

- [ADR-001 — Generational MVCC](../ADR-001-generational-mvcc.md)
- [Index Architecture](./index-architecture.md)
- [Consistency Model](../concepts/consistency-model.md)
- [VFS Guide](../vfs/README.md)