David Snelling
c44678390e
- brain.fillSubtypes(rules): idempotent subtype back-fill for pre-8.0 data.
One rule per NounType/VerbType (literal default or per-entry function);
fills only entries still missing a subtype through the public update()/
updateRelation() paths; returns { scanned, filled, skipped, errors, byType }.
Full unit suite in tests/unit/brainy/fill-subtypes.test.ts.
- Fix getNouns/getVerbs pagination hasMore (peek one past the window) —
was permanently false, silently truncating every multi-page walk.
- find({ near }) without near.id now throws a teaching error instead of an
opaque storage sharding failure; CLI --threshold without --near applies a
plain score floor.
- CLI init/close audit: every one-shot command init()s, close()s, and exits
explicitly; delete the unmaintained interactive REPL; replace the cloud-era
storage subcommands with status/batch-delete; new types/validate commands.
- requireSubtype JSDoc now documents the 8.0 default-on contract; audit()
recommendation points at fillSubtypes.
- Docs: data-storage-architecture rewritten to the real 8.0 on-disk layout;
README storage section reflects filesystem+memory and snapshots; eli5 and
SEMANTIC_VFS /as-of/ semantics corrected; internal tracker IDs and
.strategy references scrubbed from published files.
385 lines
17 KiB
Markdown
385 lines
17 KiB
Markdown
# Brainy Data Storage Architecture (8.0)
|
||
|
||
**Complete on-disk reference for the 8.0 layout.**
|
||
|
||
This document describes what a Brainy 8.0 data directory actually contains: the
|
||
canonical entity records, the system area, the generational-MVCC bookkeeping,
|
||
the column store, the blob area, and the lock files — plus how the in-memory
|
||
indexes rebuild from them. The authoritative design records are
|
||
[ADR-001 (generational MVCC)](../ADR-001-generational-mvcc.md) and
|
||
[index-architecture.md](./index-architecture.md); this document is the on-disk
|
||
map that ties them together.
|
||
|
||
8.0 removed the 7.x copy-on-write subsystem (`_cow/`, `branches/{branch}/…`
|
||
paths) and the cloud/OPFS storage adapters. The two storage backends are
|
||
**filesystem** and **memory**; both speak the same path vocabulary (memory
|
||
storage keys its internal map by the identical path strings).
|
||
|
||
---
|
||
|
||
## 1. Directory Tree
|
||
|
||
A real 8.0 filesystem store (`rootDirectory`, default `./brainy-data`):
|
||
|
||
```
|
||
brainy-data/
|
||
│
|
||
├── entities/ # Canonical records (current state)
|
||
│ ├── nouns/
|
||
│ │ └── {shard}/ # 256 shards: first 2 hex chars of the UUID
|
||
│ │ └── {id}/ # One directory per entity UUID
|
||
│ │ ├── vectors.json.gz # Embedding + HNSW node state
|
||
│ │ └── metadata.json.gz # Everything else (type, subtype, data, fields, _rev)
|
||
│ └── verbs/
|
||
│ └── {shard}/
|
||
│ └── {id}/
|
||
│ ├── vectors.json.gz # Relationship embedding (when present)
|
||
│ └── metadata.json.gz # sourceId, targetId, verb, subtype, weight, data…
|
||
│
|
||
├── _system/ # System singletons + bucketed system keys
|
||
│ ├── generation.json(.gz) # { generation, updatedAt } — the write watermark
|
||
│ ├── manifest.json # { version, generation, … } — MVCC commit point
|
||
│ ├── tx-log.jsonl # One line per committed transact() batch (append-only)
|
||
│ ├── counts.json # Entity/verb totals
|
||
│ ├── type-statistics.json.gz # Per-NounType counts
|
||
│ ├── subtype-statistics.json.gz # Per-(NounType, subtype) counts
|
||
│ ├── verb-subtype-statistics.json.gz # Per-(VerbType, subtype) counts
|
||
│ ├── statistics.json # Aggregate statistics blob (counts, index sizes)
|
||
│ ├── hnsw-system.json # Vector-index entry point + max level
|
||
│ ├── __metadata_field_registry__.json.gz # Which metadata fields are indexed
|
||
│ ├── brainy:entityIdMapper.json.gz # UUID ↔ u64 mapping for native index providers
|
||
│ └── idx/
|
||
│ └── {bucket}/ # 256 buckets: FNV-1a hash of the key
|
||
│ ├── __metadata_field_index__field_{name}.json.gz # Sparse field indexes
|
||
│ ├── __chunk__*.json.gz # Metadata-index roaring-bitmap chunks
|
||
│ ├── __sparse_index__*.json.gz# Zone maps + bloom filters
|
||
│ └── graph-lsm-verbs-{source|target}-*.json.gz # Graph LSM SSTables + manifest
|
||
│
|
||
├── _generations/ # MVCC history (written ONLY by transact())
|
||
│ └── {N}/ # One directory per committed generation N
|
||
│ ├── tx.json # The generation-N delta (immutable)
|
||
│ └── prev/
|
||
│ └── {id}.json # Before-image of each touched record (immutable)
|
||
│
|
||
├── _column_index/ # Column store manifests (one dir per field)
|
||
│ └── {field}/
|
||
│ └── MANIFEST.json.gz # Run list + zone metadata for that column
|
||
│
|
||
├── _blobs/ # Binary blob area (`<key>.bin` convention)
|
||
│ ├── _column_index/
|
||
│ │ └── {field}/
|
||
│ │ └── L0-000001.bin # Column-store runs (level-0 segments)
|
||
│ └── … # VFS file content and other binary blobs
|
||
│
|
||
└── locks/ # Process coordination (NEVER snapshotted)
|
||
├── _writer.lock # Single-writer lock: pid, hostname, heartbeat
|
||
├── _flush_requests/ # Reader→writer flush RPC (.req files)
|
||
└── _flush_responses/ # Writer acks (.ack files)
|
||
```
|
||
|
||
Most JSON objects are gzip-compressed (`.json.gz`) — compression is on by
|
||
default for filesystem storage (`storage.options.compression`, zlib level 6).
|
||
A few hot singletons (`manifest.json`, `counts.json`, `hnsw-system.json`,
|
||
`tx-log.jsonl`) are written uncompressed for cheap partial reads and appends.
|
||
|
||
---
|
||
|
||
## 2. Canonical Entity Records
|
||
|
||
Each entity (noun) and relationship (verb) is **two files** under one
|
||
ID-first directory. The split keeps vector I/O (large, append-mostly) separate
|
||
from metadata I/O (small, read-heavy).
|
||
|
||
### Noun vector file — `entities/nouns/{shard}/{id}/vectors.json.gz`
|
||
|
||
```json
|
||
{
|
||
"id": "421d92e7-4241-470a-80f4-4b39414e7a83",
|
||
"vector": [-0.139, -0.056, 0.028, "…384 dims…"],
|
||
"connections": { "0": ["neighbor-uuid…"] },
|
||
"level": 0
|
||
}
|
||
```
|
||
|
||
The HNSW node state (`connections`, `level`) is persisted with the vector so
|
||
the vector index can rebuild without recomputing the graph.
|
||
|
||
### Noun metadata file — `entities/nouns/{shard}/{id}/metadata.json.gz`
|
||
|
||
```json
|
||
{
|
||
"data": "React is a JavaScript library for building user interfaces",
|
||
"noun": "concept",
|
||
"subtype": "cli-add",
|
||
"createdAt": 1781198053726,
|
||
"updatedAt": 1781198053726,
|
||
"_rev": 1
|
||
}
|
||
```
|
||
|
||
- `noun` is the NounType. **Type lives in metadata, not in the path** — lookup
|
||
by ID is a single path construction, no type needed.
|
||
- `subtype` is the per-product sub-classification (required on write by
|
||
default in 8.0).
|
||
- `_rev` increments on every write and backs `update({ ifRev })` CAS.
|
||
- Consumer metadata fields sit alongside the standard ones.
|
||
|
||
### Verb files — `entities/verbs/{shard}/{id}/…`
|
||
|
||
Same two-file split. The verb metadata record carries the graph edge:
|
||
`sourceId`, `targetId`, `verb` (VerbType), `subtype`, `weight`, `data`,
|
||
`metadata`, timestamps, `_rev`. Verb IDs are Brainy-generated UUIDs by
|
||
contract (8.0 rejects caller-supplied verb ids) because native graph providers
|
||
intern the raw UUID bytes as u64 handles.
|
||
|
||
### Path construction
|
||
|
||
```typescript
|
||
const shard = id.substring(0, 2) // '42'
|
||
const metadataPath = `entities/nouns/${shard}/${id}/metadata.json`
|
||
const vectorsPath = `entities/nouns/${shard}/${id}/vectors.json`
|
||
// Verbs: same shape under entities/verbs/
|
||
```
|
||
|
||
Implemented in `src/storage/baseStorage.ts` (path generators) and
|
||
`src/storage/sharding.ts` (`getShardIdFromUuid`).
|
||
|
||
---
|
||
|
||
## 3. The `_system/` Area
|
||
|
||
Two kinds of keys live here, resolved by `BaseStorage.parsePath()`:
|
||
|
||
1. **Singletons** — well-known keys written at `_system/<key>.json`:
|
||
`counts`, `statistics`, `type-statistics`, `hnsw-system`,
|
||
`__metadata_field_registry__`, `brainy:entityIdMapper`, plus the MVCC
|
||
trio (`generation.json`, `manifest.json`, `tx-log.jsonl`).
|
||
2. **Bucketed system keys** — everything else (field indexes, bitmap chunks,
|
||
sparse-index segments, graph LSM SSTables) hashes into one of 256
|
||
`_system/idx/{bucket}/` directories via FNV-1a, so no single directory
|
||
accumulates unbounded entries.
|
||
|
||
Notable singletons:
|
||
|
||
| File | Contents |
|
||
|------|----------|
|
||
| `generation.json` | `{ generation, updatedAt }` — monotonic watermark, bumped by **every** write batch |
|
||
| `manifest.json` | MVCC commit point: highest *committed* generation (see §4) |
|
||
| `tx-log.jsonl` | One JSON line per committed `transact()` batch: generation, timestamp, `meta` |
|
||
| `type-statistics.json.gz` | Per-NounType counts (backs `brain.counts.byType`) |
|
||
| `subtype-statistics.json.gz` | `{ counts: { [type]: { [subtype]: n } }, updatedAt }` (contract-bound shape) |
|
||
| `verb-subtype-statistics.json.gz` | Same shape for VerbTypes |
|
||
| `hnsw-system.json` | `{ entryPointId, maxLevel }` — per-node state lives in each entity's `vectors.json` |
|
||
| `brainy:entityIdMapper.json.gz` | UUID ↔ u64 interning table for the BigInt provider contract |
|
||
| `__metadata_field_registry__.json.gz` | Registry of indexed metadata field names |
|
||
|
||
---
|
||
|
||
## 4. Generational MVCC (`_generations/` + the `_system` trio)
|
||
|
||
Full design in [ADR-001](../ADR-001-generational-mvcc.md). The on-disk shape:
|
||
|
||
```
|
||
_system/generation.json { generation, updatedAt } atomic tmp+rename
|
||
_system/manifest.json { version, generation, … } atomic tmp+rename — THE commit point
|
||
_system/tx-log.jsonl one line per committed transact() append-only
|
||
_generations/{N}/tx.json the generation-N delta immutable once written
|
||
_generations/{N}/prev/{id}.json before-image of {id} immutable once written
|
||
```
|
||
|
||
Commit protocol (writer side): stage before-images and the delta under
|
||
`_generations/N/`, fsync, apply the delta to the canonical `entities/…`
|
||
records, then atomically rename `manifest.json` to publish generation N. The
|
||
tx-log line is appended last (advisory). Crash recovery on open discards any
|
||
`_generations/{N}` newer than the manifest.
|
||
|
||
Two write classes share the generation clock:
|
||
|
||
- **Single-operation writes** (`add`/`update`/`delete`/`relate` outside
|
||
`transact()`) bump `generation.json` so watermarks and `_rev` CAS stay sound,
|
||
but write **no** history — they are not visible to `db.since()` and remain
|
||
visible through earlier pins.
|
||
- **`transact()` batches** write the full `_generations/{N}` record and a
|
||
tx-log line, and are the unit of time travel (`brain.asOf()`).
|
||
|
||
Snapshots (`db.persist(path)`) hard-link the entire store **except `locks/`**
|
||
into a self-contained directory openable via `Brainy.load(path)`.
|
||
|
||
---
|
||
|
||
## 5. Column Store (`_column_index/` + `_blobs/_column_index/`)
|
||
|
||
The metadata index persists per-field columnar runs for O(log n) range and
|
||
membership queries at scale:
|
||
|
||
- `_column_index/{field}/MANIFEST.json.gz` — the run list and zone metadata
|
||
for one field (`createdAt`, `subtype`, `noun`, `_rev`, consumer fields,
|
||
`__words__` for tokenized text…).
|
||
- `_blobs/_column_index/{field}/L0-NNNNNN.bin` — the actual level-0 run
|
||
segments, stored through the shared `_blobs/<key>.bin` binary convention.
|
||
|
||
Sparse per-field indexes, roaring-bitmap chunks, and zone-map/bloom segments
|
||
additionally live as bucketed keys under `_system/idx/` (see §3). Which path
|
||
serves a given `where` clause is the query planner's decision — inspect it
|
||
with `brainy inspect explain <dir> --where '…'`.
|
||
|
||
---
|
||
|
||
## 6. Blob Area (`_blobs/`)
|
||
|
||
`_blobs/<key>.bin` is the flat binary-blob convention shared by every storage
|
||
adapter (`saveBinaryBlob`/`getBinaryBlob` in the storage contract):
|
||
|
||
- **VFS file content** — VFS entities are regular nouns (path, ownership, and
|
||
timestamps in entity metadata); the file *bytes* are blobs.
|
||
- **Column-store runs** (under the `_column_index/` key prefix, §5).
|
||
- Any other binary payload an index provider persists.
|
||
|
||
Writes use unique temp names + rename, so concurrent writers of the same key
|
||
cannot tear each other's blobs.
|
||
|
||
---
|
||
|
||
## 7. Locks (`locks/`)
|
||
|
||
```
|
||
locks/_writer.lock # single-writer lock: { pid, hostname, startedAt, heartbeat, version }
|
||
locks/_flush_requests/ # readers drop <uuid>.req to ask the writer to flush
|
||
locks/_flush_responses/ # writer answers with <uuid>.ack
|
||
```
|
||
|
||
- One **writer** per data directory, enforced at `init()`; stale locks (dead
|
||
PID / stale heartbeat) are reclaimed automatically.
|
||
- Read-only processes (`Brainy.openReadOnly()`, the `brainy inspect` CLI
|
||
family) can ask the live writer to flush via the request/response files, so
|
||
out-of-process diagnostics see fresh state.
|
||
- `locks/` is excluded from snapshots (`SNAPSHOT_EXCLUDED_TOP_DIRS` in
|
||
`src/storage/adapters/fileSystemStorage.ts`).
|
||
|
||
---
|
||
|
||
## 8. In-Memory Indexes and What Rebuilds From What
|
||
|
||
| Index | In memory | Persisted state | Rebuild source |
|
||
|-------|-----------|-----------------|----------------|
|
||
| **Vector (HNSW)** | Graph of vector connections | `_system/hnsw-system.json` + per-entity `vectors.json` | Walk entity vector files; lazy mode loads structure only and pages vectors on demand |
|
||
| **Metadata index** | Field → value bitmaps + column-store readers | `_system/idx/` chunks + `_column_index/` manifests + `_blobs/_column_index/` runs | Loaded directly; full rebuild re-scans entity metadata |
|
||
| **Graph adjacency** | sourceId/targetId → verb-id LSM trees | `graph-lsm-verbs-{source,target}-*` SSTables under `_system/idx/` | Loaded from SSTables; full rebuild re-scans verb metadata |
|
||
| **Counts/statistics** | Per-type and per-subtype maps | `_system/{type,subtype,verb-subtype}-statistics.json.gz`, `counts.json` | Recomputable by scanning entities (`brainy inspect repair`) |
|
||
|
||
A pluggable index provider (the 8.0 plugin contract in
|
||
`@soulcraft/brainy/plugin`) may replace any of the JS implementations; the
|
||
persisted formats above are contract-bound so JS and native implementations
|
||
can interleave on the same directory.
|
||
|
||
---
|
||
|
||
## 9. Sharding Strategy
|
||
|
||
**Entities:** first 2 hex characters of the UUID → 256 uniform shards.
|
||
Deterministic, configuration-free, and keeps per-directory entry counts low
|
||
(at 1M entities: ~3,900 directories per shard). Paginated whole-store walks
|
||
(`getNouns`/`getVerbs`) iterate shards `00`–`ff` in order.
|
||
|
||
**System keys:** FNV-1a hash of the key → 256 `_system/idx/` buckets. Same
|
||
motivation, different keyspace (system keys are not UUIDs).
|
||
|
||
**What is never sharded:** the `_system/` singletons, `_generations/{N}`
|
||
directories (keyed by generation number), `_column_index/{field}` manifests
|
||
(keyed by field name), and `locks/`.
|
||
|
||
---
|
||
|
||
## 10. Durability and Atomicity
|
||
|
||
- **Per-object atomicity:** every JSON object and blob is written to a unique
|
||
temp file then `rename()`d — readers never observe torn objects.
|
||
- **Transaction atomicity:** the `manifest.json` rename is the single commit
|
||
point for `transact()` batches (§4); everything staged before it is
|
||
discarded by crash recovery if the rename never lands.
|
||
- **Compression:** gzip per object (`.json.gz`), transparent to all readers.
|
||
Native index providers that mmap binary formats use the uncompressed
|
||
`_blobs/` area instead.
|
||
|
||
---
|
||
|
||
## 11. `clear()` Semantics
|
||
|
||
`brain.clear()` removes all entities, relationships, indexes, statistics, and
|
||
MVCC history, then re-resolves every index exactly as `init()` does —
|
||
including plugin-provided vector/metadata/id-mapper factories and VFS root
|
||
re-creation. The data directory afterwards contains a fresh, empty store (the
|
||
writer lock remains held by the running process).
|
||
|
||
---
|
||
|
||
## 12. Common Scenarios
|
||
|
||
### Adding an entity
|
||
|
||
```
|
||
brain.add({ data, type, subtype })
|
||
1. Generate UUID → shard = first 2 hex chars
|
||
2. Embed data → 384-dim vector
|
||
3. Write entities/nouns/{shard}/{id}/vectors.json.gz (vector + HNSW node state)
|
||
4. Write entities/nouns/{shard}/{id}/metadata.json.gz (type/subtype/data/fields, _rev: 1)
|
||
5. Update in-memory indexes (HNSW insert, metadata index, statistics)
|
||
6. Bump _system/generation.json (no _generations/ entry — single-op write)
|
||
```
|
||
|
||
### Committing a transaction
|
||
|
||
```
|
||
await brain.transact(tx => { tx.add(…); tx.update(…) })
|
||
1. Stage _generations/{N}/prev/{id}.json before-images + tx.json delta; fsync
|
||
2. Apply the delta to canonical entities/… records
|
||
3. Atomic-rename _system/manifest.json → generation N is committed
|
||
4. Append one line to _system/tx-log.jsonl
|
||
```
|
||
|
||
### Cold start
|
||
|
||
```
|
||
await brain.init()
|
||
1. Acquire locks/_writer.lock (or open read-only)
|
||
2. Crash recovery: drop _generations/{N} newer than manifest.json
|
||
3. Load _system singletons (counts, statistics, field registry, id mapper)
|
||
4. Vector index: hnsw-system.json + entity vectors.json (lazy mode if large)
|
||
5. Graph adjacency: load LSM SSTables from _system/idx/
|
||
6. Metadata index: column-store manifests + bitmap chunks on demand
|
||
```
|
||
|
||
### Snapshot and restore
|
||
|
||
```
|
||
const db = brain.now(); await db.persist('/backups/today'); await db.release()
|
||
→ hard-links everything except locks/ into a self-contained directory
|
||
|
||
await Brainy.load('/backups/today') // open snapshot read-only as a Db
|
||
await brain.restore('/backups/today', { confirm: true }) // replace store state
|
||
```
|
||
|
||
---
|
||
|
||
## 13. Summary
|
||
|
||
- **Two backends** (filesystem, memory), one path vocabulary.
|
||
- **Two files per entity** under ID-first `entities/{kind}/{shard}/{id}/`.
|
||
- **Type and subtype are metadata**, not directory structure; type queries go
|
||
through the metadata index, not the filesystem.
|
||
- **`_system/`** holds singletons plus 256 hash buckets of index state.
|
||
- **`_generations/` + `manifest.json` + `tx-log.jsonl`** implement
|
||
generational MVCC; `transact()` is the unit of history.
|
||
- **`_column_index/` + `_blobs/`** hold the columnar metadata runs and binary
|
||
blobs (VFS content included).
|
||
- **`locks/`** coordinates the single writer and reader flush requests, and
|
||
never travels with snapshots.
|
||
|
||
---
|
||
|
||
## Next Steps
|
||
|
||
- [ADR-001 — Generational MVCC](../ADR-001-generational-mvcc.md)
|
||
- [Index Architecture](./index-architecture.md)
|
||
- [Consistency Model](../concepts/consistency-model.md)
|
||
- [VFS Guide](../vfs/README.md)
|