David Snelling
4e058720b4
Some checks failed
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
Delta Gate / Delta gate — candidate vs control (push) Failing after 7s
The metadata index fixed a field's value type from the first value it saw.
Every later value of another kind was coerced to that type, and when coercion
failed — `Number('electronics')` is NaN — the value was dropped from the index
with no error at all. The row stayed readable by id and by vector search and
vanished only from equality filters on that one field, which is what made it so
quiet: writing `category: 'electronics'` rows and then `category: 5` rows left
`where { category: 5 }` returning nothing, while the same rows in a
numbers-only corpus answered correctly.
The column store now keeps one posting column per (field, kind), where a kind
is a JavaScript typeof class. The first kind a field sees keeps the historical
`_column_index/<field>/` layout, so a single-kind field is byte-identical to
what earlier versions wrote and an index written before this opens unchanged;
each later kind takes its own column at `_column_index/<field>/k/<kind>/`.
Equality reads the column matching the query value's own kind, so `{c: 5}` and
`{c: '5'}` match different rows and neither is coerced into the other. Ranges
route by the kind of their bounds, and an unbounded range — the "has any value"
probe behind `exists` — reads every kind. A mixed field orders by kind first,
then by value, because a number and a string have no order between them. A
value that cannot be encoded for the column its own kind selected now raises
instead of being skipped: that path is unreachable by construction, and if it
is ever reached it is the silent drop this change exists to end.
Two neighbours fell out of the same routing. A boolean query value is now
encoded to the 1/0 the column stores, so boolean equality matches at all. And
an integer column widens to f64 the first time a non-integer arrives, so 4.5 is
stored as itself rather than rounded to 5 and answering the wrong query.
Field type inference reports every kind a field holds beside its dominant
reading, rather than leaving callers to treat one type as the whole answer.
Pins: mixed-kind equality in both write orders, `5` vs `'5'`, booleans mixed in,
a numeric range over a mixed field's numbers, close/reopen keeping every typed
posting, and an index in the pre-existing on-disk shape still reading.
`tests/critical-neural-validation.test.ts` — which writes `category` as strings
in one test and as numbers in another against one shared brain — passes whole
for the first time.
(cherry picked from commit a128f0eda5)
422 lines
20 KiB
Markdown
422 lines
20 KiB
Markdown
# Brainy Data Storage Architecture (8.0)
|
||
|
||
**Complete on-disk reference for the 8.0 layout.**
|
||
|
||
This document describes what a Brainy 8.0 data directory actually contains: the
|
||
canonical entity records, the system area, the generational-MVCC bookkeeping,
|
||
the column store, the blob area, and the lock files — plus how the in-memory
|
||
indexes rebuild from them. The authoritative design records are
|
||
[ADR-001 (generational MVCC)](../ADR-001-generational-mvcc.md) and
|
||
[index-architecture.md](./index-architecture.md); this document is the on-disk
|
||
map that ties them together.
|
||
|
||
8.0 removed the 7.x copy-on-write subsystem (`_cow/`, `branches/{branch}/…`
|
||
paths) and the cloud/OPFS storage adapters. The two storage backends are
|
||
**filesystem** and **memory**; both speak the same path vocabulary (memory
|
||
storage keys its internal map by the identical path strings).
|
||
|
||
---
|
||
|
||
## 1. Directory Tree
|
||
|
||
A real 8.0 filesystem store (`path`, default `./brainy-data`):
|
||
|
||
```
|
||
brainy-data/
|
||
│
|
||
├── entities/ # Canonical records (current state)
|
||
│ ├── nouns/
|
||
│ │ └── {shard}/ # 256 shards: first 2 hex chars of the UUID
|
||
│ │ └── {id}/ # One directory per entity UUID
|
||
│ │ ├── vectors.json.gz # Embedding + HNSW node state
|
||
│ │ └── metadata.json.gz # Everything else (type, subtype, data, fields, _rev)
|
||
│ └── verbs/
|
||
│ └── {shard}/
|
||
│ └── {id}/
|
||
│ ├── vectors.json.gz # Relationship embedding (when present)
|
||
│ └── metadata.json.gz # sourceId, targetId, verb, subtype, weight, data…
|
||
│
|
||
├── _system/ # System singletons + bucketed system keys
|
||
│ ├── generation.json(.gz) # { generation, updatedAt } — the write watermark
|
||
│ ├── manifest.json # { version, generation, … } — MVCC commit point
|
||
│ ├── tx-log.jsonl # One line per committed transact() batch (append-only)
|
||
│ ├── counts.json # Entity/verb totals
|
||
│ ├── type-statistics.json.gz # Per-NounType counts
|
||
│ ├── subtype-statistics.json.gz # Per-(NounType, subtype) counts
|
||
│ ├── verb-subtype-statistics.json.gz # Per-(VerbType, subtype) counts
|
||
│ ├── statistics.json # Aggregate statistics blob (counts, index sizes)
|
||
│ ├── hnsw-system.json # Vector-index entry point + max level
|
||
│ ├── __metadata_field_registry__.json.gz # Which metadata fields are indexed
|
||
│ ├── brainy:entityIdMapper.json.gz # UUID ↔ u64 mapping for native index providers
|
||
│ └── idx/
|
||
│ └── {bucket}/ # 256 buckets: FNV-1a hash of the key
|
||
│ ├── __metadata_field_index__field_{name}.json.gz # Sparse field indexes
|
||
│ ├── __chunk__*.json.gz # Metadata-index roaring-bitmap chunks
|
||
│ ├── __sparse_index__*.json.gz# Zone maps + bloom filters
|
||
│ └── graph-lsm-verbs-{source|target}-*.json.gz # Graph LSM SSTables + manifest
|
||
│
|
||
├── _generations/ # MVCC history (written ONLY by transact())
|
||
│ └── {N}/ # One directory per committed generation N
|
||
│ ├── tx.json # The generation-N delta (immutable)
|
||
│ └── prev/
|
||
│ └── {id}.json # Before-image of each touched record (immutable)
|
||
│
|
||
├── _column_index/ # Column store manifests (one dir per field)
|
||
│ └── {field}/
|
||
│ └── MANIFEST.json.gz # Run list + zone metadata for that column
|
||
│
|
||
├── _blobs/ # Binary blob area (`<key>.bin` convention)
|
||
│ ├── _column_index/
|
||
│ │ └── {field}/
|
||
│ │ └── L0-000001.bin # Column-store runs (level-0 segments)
|
||
│ └── … # VFS file content and other binary blobs
|
||
│
|
||
└── locks/ # Process coordination (NEVER snapshotted)
|
||
├── _writer.lock # Single-writer lock: pid, hostname, heartbeat
|
||
├── _flush_requests/ # Reader→writer flush RPC (.req files)
|
||
└── _flush_responses/ # Writer acks (.ack files)
|
||
```
|
||
|
||
Most JSON objects are gzip-compressed (`.json.gz`) — compression is on by
|
||
default for filesystem storage (`storage.options.compression`, zlib level 6).
|
||
A few hot singletons (`manifest.json`, `counts.json`, `hnsw-system.json`,
|
||
`tx-log.jsonl`) are written uncompressed for cheap partial reads and appends.
|
||
|
||
---
|
||
|
||
## 2. Canonical Entity Records
|
||
|
||
Each entity (noun) and relationship (verb) is **two files** under one
|
||
ID-first directory. The split keeps vector I/O (large, append-mostly) separate
|
||
from metadata I/O (small, read-heavy).
|
||
|
||
### Noun vector file — `entities/nouns/{shard}/{id}/vectors.json.gz`
|
||
|
||
```json
|
||
{
|
||
"id": "421d92e7-4241-470a-80f4-4b39414e7a83",
|
||
"vector": [-0.139, -0.056, 0.028, "…384 dims…"],
|
||
"connections": { "0": ["neighbor-uuid…"] },
|
||
"level": 0
|
||
}
|
||
```
|
||
|
||
The HNSW node state (`connections`, `level`) is persisted with the vector so
|
||
the vector index can rebuild without recomputing the graph.
|
||
|
||
### Noun metadata file — `entities/nouns/{shard}/{id}/metadata.json.gz`
|
||
|
||
```json
|
||
{
|
||
"data": "React is a JavaScript library for building user interfaces",
|
||
"noun": "concept",
|
||
"subtype": "cli-add",
|
||
"createdAt": 1781198053726,
|
||
"updatedAt": 1781198053726,
|
||
"_rev": 1
|
||
}
|
||
```
|
||
|
||
- `noun` is the NounType. **Type lives in metadata, not in the path** — lookup
|
||
by ID is a single path construction, no type needed.
|
||
- `subtype` is the per-product sub-classification (required on write by
|
||
default in 8.0).
|
||
- `_rev` increments on every write and backs `update({ ifRev })` CAS.
|
||
- Consumer metadata fields sit alongside the standard ones.
|
||
|
||
### Verb files — `entities/verbs/{shard}/{id}/…`
|
||
|
||
Same two-file split. The verb metadata record carries the graph edge:
|
||
`sourceId`, `targetId`, `verb` (VerbType), `subtype`, `weight`, `data`,
|
||
`metadata`, timestamps, `_rev`. Verb IDs are Brainy-generated UUIDs by
|
||
contract (8.0 rejects caller-supplied verb ids) because native graph providers
|
||
intern the raw UUID bytes as u64 handles.
|
||
|
||
### Path construction
|
||
|
||
```typescript
|
||
const shard = id.substring(0, 2) // '42'
|
||
const metadataPath = `entities/nouns/${shard}/${id}/metadata.json`
|
||
const vectorsPath = `entities/nouns/${shard}/${id}/vectors.json`
|
||
// Verbs: same shape under entities/verbs/
|
||
```
|
||
|
||
Implemented in `src/storage/baseStorage.ts` (path generators) and
|
||
`src/storage/sharding.ts` (`getShardIdFromUuid`).
|
||
|
||
---
|
||
|
||
## 3. The `_system/` Area
|
||
|
||
Two kinds of keys live here, resolved by `BaseStorage.parsePath()`:
|
||
|
||
1. **Singletons** — well-known keys written at `_system/<key>.json`:
|
||
`counts`, `statistics`, `type-statistics`, `hnsw-system`,
|
||
`__metadata_field_registry__`, `brainy:entityIdMapper`, plus the MVCC
|
||
trio (`generation.json`, `manifest.json`, `tx-log.jsonl`).
|
||
2. **Bucketed system keys** — everything else (field indexes, bitmap chunks,
|
||
sparse-index segments, graph LSM SSTables) hashes into one of 256
|
||
`_system/idx/{bucket}/` directories via FNV-1a, so no single directory
|
||
accumulates unbounded entries.
|
||
|
||
Notable singletons:
|
||
|
||
| File | Contents |
|
||
|------|----------|
|
||
| `generation.json` | `{ generation, updatedAt }` — monotonic watermark, bumped by **every** write batch |
|
||
| `manifest.json` | MVCC commit point: highest *committed* generation (see §4) |
|
||
| `tx-log.jsonl` | One JSON line per committed `transact()` batch: generation, timestamp, `meta` |
|
||
| `type-statistics.json.gz` | Per-NounType counts (backs `brain.counts.byType`) |
|
||
| `subtype-statistics.json.gz` | `{ counts: { [type]: { [subtype]: n } }, updatedAt }` (contract-bound shape) |
|
||
| `verb-subtype-statistics.json.gz` | Same shape for VerbTypes |
|
||
| `hnsw-system.json` | `{ entryPointId, maxLevel }` — per-node state lives in each entity's `vectors.json` |
|
||
| `brainy:entityIdMapper.json.gz` | UUID ↔ u64 interning table for the BigInt provider contract |
|
||
| `__metadata_field_registry__.json.gz` | Registry of indexed metadata field names |
|
||
|
||
---
|
||
|
||
## 4. Generational MVCC (`_generations/` + the `_system` trio)
|
||
|
||
Full design in [ADR-001](../ADR-001-generational-mvcc.md). The on-disk shape:
|
||
|
||
```
|
||
_system/generation.json { generation, updatedAt } atomic tmp+rename
|
||
_system/manifest.json { version, generation, … } atomic tmp+rename — THE commit point
|
||
_system/tx-log.jsonl one line per committed transact() append-only
|
||
_generations/{N}/tx.json the generation-N delta immutable once written
|
||
_generations/{N}/prev/{id}.json before-image of {id} immutable once written
|
||
```
|
||
|
||
Commit protocol (writer side): stage before-images and the delta under
|
||
`_generations/N/`, fsync, apply the delta to the canonical `entities/…`
|
||
records, then atomically rename `manifest.json` to publish generation N. The
|
||
tx-log line is appended last (advisory). Crash recovery on open discards any
|
||
`_generations/{N}` newer than the manifest.
|
||
|
||
Two write classes share the generation clock:
|
||
|
||
- **Single-operation writes** (`add`/`update`/`remove`/`relate` outside
|
||
`transact()`) bump `generation.json` so watermarks and `_rev` CAS stay sound,
|
||
but write **no** history — they are not visible to `db.since()` and remain
|
||
visible through earlier pins.
|
||
- **`transact()` batches** write the full `_generations/{N}` record and a
|
||
tx-log line, and are the unit of time travel (`brain.asOf()`).
|
||
|
||
Snapshots (`db.persist(path)`) hard-link the entire store **except `locks/`**
|
||
into a self-contained directory openable via `Brainy.load(path)`.
|
||
|
||
---
|
||
|
||
## 5. Column Store (`_column_index/` + `_blobs/_column_index/`)
|
||
|
||
The metadata index persists per-field columnar runs for O(log n) range and
|
||
membership queries at scale:
|
||
|
||
- `_column_index/{field}/MANIFEST.json.gz` — the run list and zone metadata
|
||
for one field (`createdAt`, `subtype`, `noun`, `_rev`, consumer fields,
|
||
`__words__` for tokenized text…).
|
||
- `_blobs/_column_index/{field}/L0-NNNNNN.bin` — the actual level-0 run
|
||
segments, stored through the shared `_blobs/<key>.bin` binary convention.
|
||
- `_column_index/{field}/k/{kind}/…` — the same two files again, for a
|
||
**second value kind** on the same field (see below). Absent for a field that
|
||
holds one kind, which is nearly all of them.
|
||
|
||
### One posting column per (field, kind)
|
||
|
||
A field is not obliged to hold one type of value. `category` may carry
|
||
`'electronics'` on some rows and `5` on others, and both are real values of
|
||
that field. A segment, though, has one encoding — i64, f64, UTF-8, or boolean
|
||
— so a field that holds several kinds gets **one column per kind**:
|
||
|
||
- The first kind a field ever sees owns the plain `_column_index/{field}/`
|
||
layout above. A single-kind field is therefore byte-identical to what earlier
|
||
versions wrote, and an index written before typed postings opens unchanged.
|
||
- Every later kind gets its own column beside it at
|
||
`_column_index/{field}/k/{kind}/`, where `{kind}` is `number`, `string` or
|
||
`boolean`.
|
||
|
||
What that buys at query time:
|
||
|
||
| | |
|
||
|---|---|
|
||
| **Equality** | Answered from the column matching the **query value's own kind**. `where {category: 5}` reads the number postings; `where {category: '5'}` reads the string postings. Neither borrows the other's rows — a row written with the number `5` is not a row whose category is the text `'5'`. |
|
||
| **A kind the field never held** | Matches nothing. That is the true answer, not a coerced one. |
|
||
| **Ranges** | Routed by the kind of the bounds: numeric bounds read the numeric postings and ignore the field's strings. An **unbounded** range is the "has any value here" probe behind `exists`, and reads every kind. |
|
||
| **`orderBy`** | A number and a string have no order between them, so a mixed field orders by kind first (number, string, boolean) and by value within a kind. A single-kind field sorts exactly as it always did. |
|
||
| **Numbers** | One kind, one column: an integer column is written as i64 and widens to f64 the first time a non-integer arrives, so `4.5` is stored as itself rather than rounded. |
|
||
|
||
`null` and `undefined` are not kinds and are never posted; their absence is
|
||
what the `exists` / `missing` operators read.
|
||
|
||
Older readers are unaffected by the additional columns: they see the field's
|
||
primary column exactly where it has always been, and a `k/{kind}` directory is
|
||
simply a name they never query.
|
||
|
||
Sparse per-field indexes, roaring-bitmap chunks, and zone-map/bloom segments
|
||
additionally live as bucketed keys under `_system/idx/` (see §3). Which path
|
||
serves a given `where` clause is the query planner's decision — inspect it
|
||
with `brainy inspect explain <dir> --where '…'`.
|
||
|
||
---
|
||
|
||
## 6. Blob Area (`_blobs/`)
|
||
|
||
`_blobs/<key>.bin` is the flat binary-blob convention shared by every storage
|
||
adapter (`saveBinaryBlob`/`getBinaryBlob` in the storage contract):
|
||
|
||
- **VFS file content** — VFS entities are regular nouns (path, ownership, and
|
||
timestamps in entity metadata); the file *bytes* are blobs.
|
||
- **Column-store runs** (under the `_column_index/` key prefix, §5).
|
||
- Any other binary payload an index provider persists.
|
||
|
||
Writes use unique temp names + rename, so concurrent writers of the same key
|
||
cannot tear each other's blobs.
|
||
|
||
---
|
||
|
||
## 7. Locks (`locks/`)
|
||
|
||
```
|
||
locks/_writer.lock # single-writer lock: { pid, hostname, startedAt, heartbeat, version }
|
||
locks/_flush_requests/ # readers drop <uuid>.req to ask the writer to flush
|
||
locks/_flush_responses/ # writer answers with <uuid>.ack
|
||
```
|
||
|
||
- One **writer** per data directory, enforced at `init()`; stale locks (dead
|
||
PID / stale heartbeat) are reclaimed automatically.
|
||
- Read-only processes (`Brainy.openReadOnly()`, the `brainy inspect` CLI
|
||
family) can ask the live writer to flush via the request/response files, so
|
||
out-of-process diagnostics see fresh state.
|
||
- `locks/` is excluded from snapshots (`SNAPSHOT_EXCLUDED_TOP_DIRS` in
|
||
`src/storage/adapters/fileSystemStorage.ts`).
|
||
|
||
---
|
||
|
||
## 8. In-Memory Indexes and What Rebuilds From What
|
||
|
||
| Index | In memory | Persisted state | Rebuild source |
|
||
|-------|-----------|-----------------|----------------|
|
||
| **Vector (HNSW)** | Graph of vector connections | `_system/hnsw-system.json` + per-entity `vectors.json` | Walk entity vector files; lazy mode loads structure only and pages vectors on demand |
|
||
| **Metadata index** | Field → value bitmaps + column-store readers | `_system/idx/` chunks + `_column_index/` manifests + `_blobs/_column_index/` runs | Loaded directly; full rebuild re-scans entity metadata |
|
||
| **Graph adjacency** | sourceId/targetId → verb-id LSM trees | `graph-lsm-verbs-{source,target}-*` SSTables under `_system/idx/` | Loaded from SSTables; full rebuild re-scans verb metadata |
|
||
| **Counts/statistics** | Per-type and per-subtype maps | `_system/{type,subtype,verb-subtype}-statistics.json.gz`, `counts.json` | Recomputable by scanning entities (`brainy inspect repair`) |
|
||
|
||
A pluggable index provider (the 8.0 plugin contract in
|
||
`@soulcraftlabs/brainy/plugin`) may replace any of the JS implementations; the
|
||
persisted formats above are contract-bound so JS and native implementations
|
||
can interleave on the same directory.
|
||
|
||
---
|
||
|
||
## 9. Sharding Strategy
|
||
|
||
**Entities:** first 2 hex characters of the UUID → 256 uniform shards.
|
||
Deterministic, configuration-free, and keeps per-directory entry counts low
|
||
(at 1M entities: ~3,900 directories per shard). Paginated whole-store walks
|
||
(`getNouns`/`getVerbs`) iterate shards `00`–`ff` in order.
|
||
|
||
**System keys:** FNV-1a hash of the key → 256 `_system/idx/` buckets. Same
|
||
motivation, different keyspace (system keys are not UUIDs).
|
||
|
||
**What is never sharded:** the `_system/` singletons, `_generations/{N}`
|
||
directories (keyed by generation number), `_column_index/{field}` manifests
|
||
(keyed by field name), and `locks/`.
|
||
|
||
---
|
||
|
||
## 10. Durability and Atomicity
|
||
|
||
- **Per-object atomicity:** every JSON object and blob is written to a unique
|
||
temp file then `rename()`d — readers never observe torn objects.
|
||
- **Transaction atomicity:** the `manifest.json` rename is the single commit
|
||
point for `transact()` batches (§4); everything staged before it is
|
||
discarded by crash recovery if the rename never lands.
|
||
- **Compression:** gzip per object (`.json.gz`), transparent to all readers.
|
||
Native index providers that mmap binary formats use the uncompressed
|
||
`_blobs/` area instead.
|
||
|
||
---
|
||
|
||
## 11. `clear()` Semantics
|
||
|
||
`brain.clear()` removes all entities, relationships, indexes, statistics, and
|
||
MVCC history, then re-resolves every index exactly as `init()` does —
|
||
including plugin-provided vector/metadata/id-mapper factories and VFS root
|
||
re-creation. The data directory afterwards contains a fresh, empty store (the
|
||
writer lock remains held by the running process).
|
||
|
||
---
|
||
|
||
## 12. Common Scenarios
|
||
|
||
### Adding an entity
|
||
|
||
```
|
||
brain.add({ data, type, subtype })
|
||
1. Generate UUID → shard = first 2 hex chars
|
||
2. Embed data → 384-dim vector
|
||
3. Write entities/nouns/{shard}/{id}/vectors.json.gz (vector + HNSW node state)
|
||
4. Write entities/nouns/{shard}/{id}/metadata.json.gz (type/subtype/data/fields, _rev: 1)
|
||
5. Update in-memory indexes (HNSW insert, metadata index, statistics)
|
||
6. Bump _system/generation.json (no _generations/ entry — single-op write)
|
||
```
|
||
|
||
### Committing a transaction
|
||
|
||
```
|
||
await brain.transact(tx => { tx.add(…); tx.update(…) })
|
||
1. Stage _generations/{N}/prev/{id}.json before-images + tx.json delta; fsync
|
||
2. Apply the delta to canonical entities/… records
|
||
3. Atomic-rename _system/manifest.json → generation N is committed
|
||
4. Append one line to _system/tx-log.jsonl
|
||
```
|
||
|
||
### Cold start
|
||
|
||
```
|
||
await brain.init()
|
||
1. Acquire locks/_writer.lock (or open read-only)
|
||
2. Crash recovery: drop _generations/{N} newer than manifest.json
|
||
3. Load _system singletons (counts, statistics, field registry, id mapper)
|
||
4. Vector index: hnsw-system.json + entity vectors.json (lazy mode if large)
|
||
5. Graph adjacency: load LSM SSTables from _system/idx/
|
||
6. Metadata index: column-store manifests + bitmap chunks on demand
|
||
```
|
||
|
||
### Snapshot and restore
|
||
|
||
```
|
||
const db = brain.now(); await db.persist('/backups/today'); await db.release()
|
||
→ hard-links everything except locks/ into a self-contained directory
|
||
|
||
await Brainy.load('/backups/today') // open snapshot read-only as a Db
|
||
await brain.restore('/backups/today', { confirm: true }) // replace store state
|
||
```
|
||
|
||
---
|
||
|
||
## 13. Summary
|
||
|
||
- **Two backends** (filesystem, memory), one path vocabulary.
|
||
- **Two files per entity** under ID-first `entities/{kind}/{shard}/{id}/`.
|
||
- **Type and subtype are metadata**, not directory structure; type queries go
|
||
through the metadata index, not the filesystem.
|
||
- **`_system/`** holds singletons plus 256 hash buckets of index state.
|
||
- **`_generations/` + `manifest.json` + `tx-log.jsonl`** implement
|
||
generational MVCC. History is per-write: every `add()`/`update()`/`remove()`/
|
||
`relate()` gets its own generation, and `transact()` groups several ops into
|
||
one atomic generation. (Single-op retention has been the model since 8.0;
|
||
you never need to route a write through `transact()` just to keep its history.)
|
||
- **`_column_index/` + `_blobs/`** hold the columnar metadata runs and binary
|
||
blobs (VFS content included).
|
||
- **`locks/`** coordinates the single writer and reader flush requests, and
|
||
never travels with snapshots.
|
||
|
||
---
|
||
|
||
## Next Steps
|
||
|
||
- [ADR-001 — Generational MVCC](../ADR-001-generational-mvcc.md)
|
||
- [Index Architecture](./index-architecture.md)
|
||
- [Consistency Model](../concepts/consistency-model.md)
|
||
- [VFS Guide](../vfs/README.md)
|