feat(8.0): generational MVCC storage + Datomic-style Db API (now/transact/asOf/with/persist)
One mechanism replaces the COW and versioning subsystems: immutable
generation-stamped records behind a Datomic-style database value.
Record layer (src/db/generationStore.ts):
- Monotonic generation counter in _system/generation.json; bumped once per
transact() commit and once per single-operation write (storage hook), so
brain.generation() is always a meaningful watermark.
- Commit protocol: stage before-images + tx.json delta -> fsync -> execute
batch via TransactionManager -> atomic tmp+rename of _system/manifest.json
(the rename IS the commit point) -> append _system/tx-log.jsonl.
- Crash recovery on open rolls uncommitted generations back byte-identically
and forces an index rebuild; refcounted pins gate compactHistory(), which
records a horizon (asOf below it throws GenerationCompactedError).
Db API:
- Db: get/find/search/related pinned at a generation, with() speculative
overlays, since() diffs, persist() hard-link-farm snapshots, timestamp,
generation, release() + FinalizationRegistry backstop.
- Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch
(GenerationConflictError CAS), asOf(generation|Date|path), restore(path,
{confirm}), compactHistory(), generation(), static open() + load().
- get()/metadata find()/related() are fully correct at any reachable pinned
generation; index-accelerated queries at historical generations throw
NotYetSupportedAtHistoricalGenerationError - never silently-wrong results.
- VersionedIndexProvider (generation/isGenerationVisible/pin/release) in
plugin.ts: feature-detected, balanced pin/release in lockstep with Db
lifecycle, post-commit applier + replay-gap model documented.
Storage primitives (BaseStorage + filesystem/memory adapters): raw-object
read/write/list/remove, fsync barrier, noun/verb raw before-image capture,
tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and
append-in-place exceptions), restoreFromDirectory + derived-state reload.
Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation
across 200 mutations, batch atomicity under injected execution failure,
ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety
under pins, with() overlay isolation, generation monotonicity across
reopen, crash consistency through the real recovery path, and versioned
provider pin/release balance. Design record in
docs/ADR-001-generational-mvcc.md.
This commit is contained in:
parent
49e49483d1
commit
431cd64406
16 changed files with 6244 additions and 5 deletions
|
|
@ -954,6 +954,254 @@ export class FileSystemStorage extends BaseStorage {
|
|||
}
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
// Generational record layer (8.0 MVCC) — durability + snapshot primitives
|
||||
// ===========================================================================
|
||||
|
||||
/**
|
||||
* Storage-root-relative paths that are mutated **in place** (appended to)
|
||||
* rather than replaced via atomic tmp+rename. `snapshotToDirectory()` must
|
||||
* byte-copy these instead of hard-linking them: a hard link shares the
|
||||
* inode, so a post-snapshot append to the live file would silently mutate
|
||||
* the snapshot. Every other persisted file in this adapter is written via
|
||||
* tmp+rename (objects, blobs, counts, locks are excluded entirely), which
|
||||
* makes hard links safe — a rewrite swaps in a new inode and the snapshot
|
||||
* keeps the old one.
|
||||
*/
|
||||
private static readonly SNAPSHOT_BYTE_COPY_PATHS = new Set<string>([
|
||||
`${SYSTEM_DIR}/tx-log.jsonl`
|
||||
])
|
||||
|
||||
/**
|
||||
* Top-level directories excluded from snapshots: process-local lock state
|
||||
* (writer lock, flush-request RPC files) must never travel with the data.
|
||||
*/
|
||||
private static readonly SNAPSHOT_EXCLUDED_TOP_DIRS = new Set<string>(['locks'])
|
||||
|
||||
/**
|
||||
* Remove every object under a storage-root-relative prefix — one recursive
|
||||
* directory removal instead of the base class's list+delete loop.
|
||||
*
|
||||
* @param prefix - Storage-root-relative directory prefix to remove.
|
||||
*/
|
||||
public async removeRawPrefix(prefix: string): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
await fs.promises.rm(path.join(this.rootDir, prefix), { recursive: true, force: true })
|
||||
}
|
||||
|
||||
/**
|
||||
* Durability barrier: `fsync` each listed object file (resolving the
|
||||
* compressed `.gz` variant when present) and then the set of parent
|
||||
* directories, so both the file contents and the rename directory entries
|
||||
* are durable before the commit protocol proceeds. Paths whose file no
|
||||
* longer exists are skipped (a later step may have replaced them).
|
||||
*
|
||||
* @param paths - Storage-root-relative object paths previously written.
|
||||
*/
|
||||
public async syncRawObjects(paths: string[]): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
const parentDirs = new Set<string>()
|
||||
|
||||
for (const objectPath of paths) {
|
||||
const fullPath = path.join(this.rootDir, objectPath)
|
||||
for (const candidate of [`${fullPath}.gz`, fullPath]) {
|
||||
let handle: any
|
||||
try {
|
||||
handle = await fs.promises.open(candidate, 'r')
|
||||
} catch (error: any) {
|
||||
if (error.code === 'ENOENT') continue
|
||||
throw error
|
||||
}
|
||||
try {
|
||||
await handle.sync()
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
parentDirs.add(path.dirname(fullPath))
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
for (const dir of parentDirs) {
|
||||
let handle: any
|
||||
try {
|
||||
handle = await fs.promises.open(dir, 'r')
|
||||
} catch {
|
||||
continue // directory vanished or platform disallows opening dirs
|
||||
}
|
||||
try {
|
||||
await handle.sync()
|
||||
} catch {
|
||||
// Some platforms (and some filesystems) reject directory fsync —
|
||||
// file-level fsync above already covers the data itself.
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Append one line to `_system/tx-log.jsonl`. Plain `appendFile` — the
|
||||
* tx-log is the one append-in-place file in the store (and is byte-copied,
|
||||
* never hard-linked, by {@link FileSystemStorage.snapshotToDirectory}).
|
||||
*
|
||||
* @param line - One complete JSON document, without trailing newline.
|
||||
*/
|
||||
public async appendTxLogLine(line: string): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
const logPath = path.join(this.systemDir, 'tx-log.jsonl')
|
||||
await fs.promises.appendFile(logPath, `${line}\n`, 'utf-8')
|
||||
}
|
||||
|
||||
/**
|
||||
* Read all tx-log lines, oldest first (empty array when no log exists).
|
||||
* Torn trailing lines from a crashed append are returned as-is — callers
|
||||
* tolerate unparseable lines.
|
||||
*/
|
||||
public async readTxLogLines(): Promise<string[]> {
|
||||
await this.ensureInitialized()
|
||||
const logPath = path.join(this.systemDir, 'tx-log.jsonl')
|
||||
try {
|
||||
const content: string = await fs.promises.readFile(logPath, 'utf-8')
|
||||
return content.split('\n').filter((l: string) => l.length > 0)
|
||||
} catch (error: any) {
|
||||
if (error.code === 'ENOENT') return []
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Snapshot the entire store into `targetPath` as a hard-link farm
|
||||
* (Cassandra-style: instant, space-shared). Safe because every data file
|
||||
* is immutable-by-rename — rewrites swap in new inodes, leaving the
|
||||
* snapshot's links pointing at the old bytes. The two exceptions are
|
||||
* handled explicitly: append-in-place files
|
||||
* ({@link FileSystemStorage.SNAPSHOT_BYTE_COPY_PATHS}) are byte-copied, and
|
||||
* process-local lock state
|
||||
* ({@link FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS}) is excluded.
|
||||
* Cross-device targets (where `link(2)` fails with `EXDEV`) fall back to
|
||||
* byte copies per file.
|
||||
*
|
||||
* @param targetPath - Absolute directory for the snapshot. Created if
|
||||
* missing; must be empty or absent (refuses to overwrite).
|
||||
*/
|
||||
public async snapshotToDirectory(targetPath: string): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
try {
|
||||
const existing = await fs.promises.readdir(targetPath)
|
||||
if (existing.length > 0) {
|
||||
throw new Error(
|
||||
`snapshotToDirectory: target ${targetPath} already exists and is not empty. ` +
|
||||
`Choose a fresh directory per snapshot.`
|
||||
)
|
||||
}
|
||||
} catch (error: any) {
|
||||
if (error.code !== 'ENOENT') throw error
|
||||
}
|
||||
await fs.promises.mkdir(targetPath, { recursive: true })
|
||||
|
||||
const files: string[] = []
|
||||
await this.collectSnapshotFiles(this.rootDir, '', files)
|
||||
|
||||
for (const relPath of files) {
|
||||
const sourceFile = path.join(this.rootDir, relPath)
|
||||
const targetFile = path.join(targetPath, relPath)
|
||||
await fs.promises.mkdir(path.dirname(targetFile), { recursive: true })
|
||||
|
||||
// Byte-copy list: compare against the normalized (extension-preserving)
|
||||
// relative path with separators unified, so `_system/tx-log.jsonl`
|
||||
// matches on every platform.
|
||||
const normalized = relPath.split(path.sep).join('/')
|
||||
if (FileSystemStorage.SNAPSHOT_BYTE_COPY_PATHS.has(normalized)) {
|
||||
await fs.promises.copyFile(sourceFile, targetFile)
|
||||
continue
|
||||
}
|
||||
|
||||
try {
|
||||
await fs.promises.link(sourceFile, targetFile)
|
||||
} catch (error: any) {
|
||||
// EXDEV: cross-device; EPERM/ENOTSUP: filesystem forbids links.
|
||||
// ENOENT: the live file was atomically replaced mid-walk — retry as
|
||||
// a copy of whatever is current (single-writer discipline means this
|
||||
// only happens for derived files being flushed concurrently).
|
||||
if (['EXDEV', 'EPERM', 'ENOTSUP', 'ENOENT'].includes(error.code)) {
|
||||
try {
|
||||
await fs.promises.copyFile(sourceFile, targetFile)
|
||||
} catch (copyError: any) {
|
||||
if (copyError.code !== 'ENOENT') throw copyError
|
||||
}
|
||||
} else {
|
||||
throw error
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Recursively collect snapshot-eligible files under `dirAbs`, excluding the
|
||||
* lock directory and in-flight `*.tmp.*` write files.
|
||||
*/
|
||||
private async collectSnapshotFiles(dirAbs: string, relPrefix: string, out: string[]): Promise<void> {
|
||||
let entries: any[]
|
||||
try {
|
||||
entries = await fs.promises.readdir(dirAbs, { withFileTypes: true })
|
||||
} catch (error: any) {
|
||||
if (error.code === 'ENOENT') return
|
||||
throw error
|
||||
}
|
||||
|
||||
for (const entry of entries) {
|
||||
const rel = relPrefix ? path.join(relPrefix, entry.name) : entry.name
|
||||
if (entry.isDirectory()) {
|
||||
if (relPrefix === '' && FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS.has(entry.name)) {
|
||||
continue
|
||||
}
|
||||
await this.collectSnapshotFiles(path.join(dirAbs, entry.name), rel, out)
|
||||
} else if (entry.isFile()) {
|
||||
if (entry.name.includes('.tmp.')) continue // in-flight atomic write
|
||||
out.push(rel)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Replace the store's contents from a snapshot directory: every current
|
||||
* top-level entry except `locks/` (the live writer lock must survive) is
|
||||
* removed, the snapshot is byte-copied in (`fs.cp` — never hard-linked, so
|
||||
* the snapshot stays independent of the restored store), and all
|
||||
* adapter-internal derived state is reloaded.
|
||||
*
|
||||
* @param sourcePath - Absolute path of a directory produced by
|
||||
* {@link FileSystemStorage.snapshotToDirectory}.
|
||||
*/
|
||||
public async restoreFromDirectory(sourcePath: string): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
const sourceStat = await fs.promises.stat(sourcePath).catch(() => null)
|
||||
if (!sourceStat || !sourceStat.isDirectory()) {
|
||||
throw new Error(`restoreFromDirectory: ${sourcePath} is not a directory`)
|
||||
}
|
||||
|
||||
// Clear current contents, preserving live lock state.
|
||||
const currentEntries = await fs.promises.readdir(this.rootDir)
|
||||
for (const entry of currentEntries) {
|
||||
if (FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS.has(entry)) continue
|
||||
await fs.promises.rm(path.join(this.rootDir, entry), { recursive: true, force: true })
|
||||
}
|
||||
|
||||
// Byte-copy the snapshot in (defensively skipping lock dirs in old snapshots).
|
||||
const sourceEntries = await fs.promises.readdir(sourcePath)
|
||||
for (const entry of sourceEntries) {
|
||||
if (FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS.has(entry)) continue
|
||||
await fs.promises.cp(path.join(sourcePath, entry), path.join(this.rootDir, entry), {
|
||||
recursive: true
|
||||
})
|
||||
}
|
||||
|
||||
await this.reloadDerivedState()
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
// Raw binary-blob primitive (mmap-friendly)
|
||||
// ===========================================================================
|
||||
|
|
|
|||
|
|
@ -525,4 +525,47 @@ export class HistoricalStorageAdapter extends BaseStorage {
|
|||
protected async persistCounts(): Promise<void> {
|
||||
// No-op: Historical storage is read-only
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
// Generational record layer (8.0 MVCC) — not applicable to this adapter
|
||||
//
|
||||
// HistoricalStorageAdapter is the pre-8.0 commit-based time-travel view and
|
||||
// is read-only by construction. The generation store never registers its
|
||||
// write hooks on reader-mode instances, so these methods are unreachable in
|
||||
// normal operation; they throw explicitly rather than fake behavior.
|
||||
// ===========================================================================
|
||||
|
||||
/**
|
||||
* WRITE BLOCKED: the tx-log belongs to the live store, not a historical view.
|
||||
*/
|
||||
public async appendTxLogLine(_line: string): Promise<void> {
|
||||
throw new Error('Historical storage is read-only. Cannot append to the transaction log.')
|
||||
}
|
||||
|
||||
/**
|
||||
* Not supported: historical commit views predate the generational tx-log.
|
||||
*/
|
||||
public async readTxLogLines(): Promise<string[]> {
|
||||
throw new Error(
|
||||
'Transaction-log reads are not supported on a historical commit view. ' +
|
||||
'Use the live store (or a Db from brain.now()/brain.asOf()).'
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Not supported: snapshot a live store via db.persist(path) instead.
|
||||
*/
|
||||
public async snapshotToDirectory(_targetPath: string): Promise<void> {
|
||||
throw new Error(
|
||||
'snapshotToDirectory is not supported on a historical commit view. ' +
|
||||
'Call db.persist(path) on the live store instead.'
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* WRITE BLOCKED: cannot restore into a read-only historical view.
|
||||
*/
|
||||
public async restoreFromDirectory(_sourcePath: string): Promise<void> {
|
||||
throw new Error('Historical storage is read-only. Cannot restore from a snapshot.')
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -3,6 +3,9 @@
|
|||
* In-memory storage adapter for environments where persistent storage is not available or needed
|
||||
*/
|
||||
|
||||
import * as fs from 'node:fs'
|
||||
import * as nodePath from 'node:path'
|
||||
import * as zlib from 'node:zlib'
|
||||
import {
|
||||
GraphVerb,
|
||||
HNSWNoun,
|
||||
|
|
@ -35,6 +38,11 @@ export class MemoryStorage extends BaseStorage {
|
|||
// storage has no local file, so getBinaryBlobPath() returns null.
|
||||
private blobStore: Map<string, Buffer> = new Map()
|
||||
|
||||
// Transaction log (8.0 MVCC) — the in-memory equivalent of the filesystem
|
||||
// adapter's `_system/tx-log.jsonl`. One JSON document per committed
|
||||
// transact(); serialized to a real JSONL file by snapshotToDirectory().
|
||||
private txLogLines: string[] = []
|
||||
|
||||
// Backward compatibility aliases
|
||||
private get metadata(): Map<string, any> {
|
||||
return this.objectStore
|
||||
|
|
@ -189,6 +197,132 @@ export class MemoryStorage extends BaseStorage {
|
|||
return null
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
// Generational record layer (8.0 MVCC) — tx-log + snapshot primitives
|
||||
// ===========================================================================
|
||||
|
||||
/**
|
||||
* Append one line to the in-memory transaction log (mirror of the
|
||||
* filesystem adapter's `_system/tx-log.jsonl` append).
|
||||
*
|
||||
* @param line - One complete JSON document, without trailing newline.
|
||||
*/
|
||||
public async appendTxLogLine(line: string): Promise<void> {
|
||||
this.txLogLines.push(line)
|
||||
}
|
||||
|
||||
/**
|
||||
* Read all transaction-log lines, oldest first (a copy — callers cannot
|
||||
* mutate the log).
|
||||
*/
|
||||
public async readTxLogLines(): Promise<string[]> {
|
||||
return [...this.txLogLines]
|
||||
}
|
||||
|
||||
/**
|
||||
* Serialize the entire in-memory store to a directory in the exact layout
|
||||
* the filesystem adapter uses (uncompressed JSON objects, `_blobs/*.bin`
|
||||
* payloads, `_system/tx-log.jsonl`). The resulting directory is a
|
||||
* self-contained store openable via `Brainy.load(path)` — persisting an
|
||||
* in-memory brain produces a real, durable snapshot.
|
||||
*
|
||||
* @param targetPath - Absolute directory for the snapshot. Created if
|
||||
* missing; must be empty or absent (refuses to overwrite).
|
||||
*/
|
||||
public async snapshotToDirectory(targetPath: string): Promise<void> {
|
||||
try {
|
||||
const existing = await fs.promises.readdir(targetPath)
|
||||
if (existing.length > 0) {
|
||||
throw new Error(
|
||||
`snapshotToDirectory: target ${targetPath} already exists and is not empty. ` +
|
||||
`Choose a fresh directory per snapshot.`
|
||||
)
|
||||
}
|
||||
} catch (error: any) {
|
||||
if (error.code !== 'ENOENT') throw error
|
||||
}
|
||||
await fs.promises.mkdir(targetPath, { recursive: true })
|
||||
|
||||
for (const [key, value] of this.objectStore.entries()) {
|
||||
const filePath = nodePath.join(targetPath, ...key.split('/'))
|
||||
await fs.promises.mkdir(nodePath.dirname(filePath), { recursive: true })
|
||||
await fs.promises.writeFile(filePath, JSON.stringify(value, null, 2), 'utf-8')
|
||||
}
|
||||
|
||||
for (const [key, data] of this.blobStore.entries()) {
|
||||
const blobPath = nodePath.join(targetPath, '_blobs', ...key.split('/')) + '.bin'
|
||||
await fs.promises.mkdir(nodePath.dirname(blobPath), { recursive: true })
|
||||
await fs.promises.writeFile(blobPath, data)
|
||||
}
|
||||
|
||||
if (this.txLogLines.length > 0) {
|
||||
const logPath = nodePath.join(targetPath, '_system', 'tx-log.jsonl')
|
||||
await fs.promises.mkdir(nodePath.dirname(logPath), { recursive: true })
|
||||
await fs.promises.writeFile(logPath, this.txLogLines.map((l) => `${l}\n`).join(''), 'utf-8')
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Replace the in-memory store's contents from a snapshot directory —
|
||||
* accepts snapshots produced by either adapter (handles the filesystem
|
||||
* adapter's gzip-compressed `.json.gz` objects transparently), then reloads
|
||||
* all derived state.
|
||||
*
|
||||
* @param sourcePath - Absolute path of a snapshot directory.
|
||||
*/
|
||||
public async restoreFromDirectory(sourcePath: string): Promise<void> {
|
||||
const sourceStat = await fs.promises.stat(sourcePath).catch(() => null)
|
||||
if (!sourceStat || !sourceStat.isDirectory()) {
|
||||
throw new Error(`restoreFromDirectory: ${sourcePath} is not a directory`)
|
||||
}
|
||||
|
||||
this.objectStore.clear()
|
||||
this.blobStore.clear()
|
||||
this.txLogLines = []
|
||||
this.statistics = null
|
||||
|
||||
await this.loadDirectoryIntoStore(sourcePath, '')
|
||||
await this.reloadDerivedState()
|
||||
}
|
||||
|
||||
/**
|
||||
* Recursively load a snapshot directory into the object/blob/tx-log stores.
|
||||
* Lock directories from filesystem-adapter snapshots are skipped (process-
|
||||
* local state, meaningless in memory).
|
||||
*/
|
||||
private async loadDirectoryIntoStore(dirAbs: string, relPrefix: string): Promise<void> {
|
||||
const entries = await fs.promises.readdir(dirAbs, { withFileTypes: true })
|
||||
|
||||
for (const entry of entries) {
|
||||
const rel = relPrefix ? `${relPrefix}/${entry.name}` : entry.name
|
||||
const abs = nodePath.join(dirAbs, entry.name)
|
||||
|
||||
if (entry.isDirectory()) {
|
||||
if (relPrefix === '' && entry.name === 'locks') continue
|
||||
await this.loadDirectoryIntoStore(abs, rel)
|
||||
continue
|
||||
}
|
||||
if (!entry.isFile()) continue
|
||||
|
||||
if (rel === '_system/tx-log.jsonl') {
|
||||
const content = await fs.promises.readFile(abs, 'utf-8')
|
||||
this.txLogLines = content.split('\n').filter((l) => l.length > 0)
|
||||
} else if (relPrefix.startsWith('_blobs') && rel.endsWith('.bin')) {
|
||||
const key = rel.slice('_blobs/'.length, -'.bin'.length)
|
||||
this.blobStore.set(key, await fs.promises.readFile(abs))
|
||||
} else if (rel.endsWith('.json.gz') || rel.endsWith('.gz')) {
|
||||
const raw = await fs.promises.readFile(abs)
|
||||
const key = rel.slice(0, -'.gz'.length)
|
||||
this.objectStore.set(key, JSON.parse(zlib.gunzipSync(raw).toString('utf-8')))
|
||||
} else if (entry.name.includes('.tmp.')) {
|
||||
// In-flight atomic-write remnant from a crashed filesystem store — skip.
|
||||
} else {
|
||||
const content = await fs.promises.readFile(abs, 'utf-8')
|
||||
this.objectStore.set(rel, JSON.parse(content))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get multiple metadata objects in batches (CRITICAL: Prevents socket exhaustion)
|
||||
* Memory storage implementation is simple since all data is already in memory
|
||||
|
|
@ -217,6 +351,7 @@ export class MemoryStorage extends BaseStorage {
|
|||
public async clear(): Promise<void> {
|
||||
this.objectStore.clear()
|
||||
this.blobStore.clear()
|
||||
this.txLogLines = []
|
||||
this.statistics = null
|
||||
this.totalNounCount = 0
|
||||
this.totalVerbCount = 0
|
||||
|
|
|
|||
|
|
@ -881,6 +881,271 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
return Array.from(pathsSet)
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// GENERATIONAL RECORD LAYER PRIMITIVES (8.0 MVCC)
|
||||
//
|
||||
// The narrow surface `GenerationStore` (src/db/generationStore.ts) needs from
|
||||
// an adapter — see the `GenerationStorage` contract in src/db/types.ts.
|
||||
//
|
||||
// Raw-object methods operate on storage-root-relative paths and deliberately
|
||||
// bypass branch scoping and the write cache: the record layer owns the
|
||||
// `_system/` + `_generations/` areas outright and its files must never be
|
||||
// shadowed by branch resolution. Entity-raw methods, by contrast, go through
|
||||
// the branch-scoped, write-cache-coherent helpers so before-images capture
|
||||
// exactly the bytes the live read paths see.
|
||||
// ============================================================================
|
||||
|
||||
/**
|
||||
* Hook invoked after every entity-visible single-operation write (noun/verb
|
||||
* metadata save or delete). Registered by the generation store so
|
||||
* `brain.generation()` advances on writes performed outside `transact()`.
|
||||
* See {@link BaseStorage.setGenerationBumpHook}.
|
||||
*/
|
||||
protected generationBumpHook?: () => void
|
||||
|
||||
/**
|
||||
* Register (or detach, with `undefined`) the generation-bump hook.
|
||||
*
|
||||
* The hook fires once per entity-visible metadata mutation — noun/verb
|
||||
* metadata saves and deletes, the one storage write every logical Brainy
|
||||
* mutation (`add`, `update`, `delete`, `relate`, `updateRelation`,
|
||||
* `unrelate`) performs exactly once per entity it touches. It does NOT fire
|
||||
* for derived-index writes (HNSW node data, metadata-index chunks, LSM
|
||||
* segments, statistics), so the generation counter tracks *data* mutations,
|
||||
* not index maintenance. The counter is a monotonic watermark, not an
|
||||
* operation count: a cascade delete bumps once per removed record.
|
||||
*
|
||||
* @param hook - Callback invoked synchronously after each qualifying write,
|
||||
* or `undefined` to detach.
|
||||
*/
|
||||
public setGenerationBumpHook(hook: (() => void) | undefined): void {
|
||||
this.generationBumpHook = hook
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a raw object at a storage-root-relative path. Bypasses branch
|
||||
* scoping and the write cache (record-layer files are written through
|
||||
* {@link BaseStorage.writeRawObject} only).
|
||||
*
|
||||
* @param path - Storage-root-relative object path (e.g. `_system/manifest.json`).
|
||||
* @returns The parsed object, or `null` if absent.
|
||||
*/
|
||||
public async readRawObject(path: string): Promise<any | null> {
|
||||
await this.ensureInitialized()
|
||||
return this.readObjectFromPath(path)
|
||||
}
|
||||
|
||||
/**
|
||||
* Write a raw object at a storage-root-relative path. On disk this is an
|
||||
* atomic tmp+rename (the filesystem adapter's primitive), which is what
|
||||
* makes the manifest rename a valid commit point.
|
||||
*
|
||||
* @param path - Storage-root-relative object path.
|
||||
* @param data - JSON-serializable object to persist.
|
||||
*/
|
||||
public async writeRawObject(path: string, data: any): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
await this.writeObjectToPath(path, data)
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a raw object at a storage-root-relative path (no-op if absent).
|
||||
*
|
||||
* @param path - Storage-root-relative object path.
|
||||
*/
|
||||
public async deleteRawObject(path: string): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
await this.deleteObjectFromPath(path)
|
||||
}
|
||||
|
||||
/**
|
||||
* List raw object paths under a storage-root-relative prefix (normalized,
|
||||
* `.gz`-stripped — the adapter primitives already normalize).
|
||||
*
|
||||
* @param prefix - Storage-root-relative directory prefix.
|
||||
* @returns Normalized object paths under the prefix (empty when none).
|
||||
*/
|
||||
public async listRawObjects(prefix: string): Promise<string[]> {
|
||||
await this.ensureInitialized()
|
||||
return this.listObjectsUnderPath(prefix)
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove every object under a storage-root-relative prefix. The filesystem
|
||||
* adapter overrides this with a recursive directory removal; this default
|
||||
* lists and deletes individually (exactly what the in-memory adapter needs).
|
||||
*
|
||||
* @param prefix - Storage-root-relative directory prefix to remove.
|
||||
*/
|
||||
public async removeRawPrefix(prefix: string): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
const paths = await this.listObjectsUnderPath(prefix)
|
||||
for (const p of paths) {
|
||||
await this.deleteObjectFromPath(p)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Durability barrier for the commit protocol: ensure the listed raw-object
|
||||
* paths are durable before the caller proceeds. The base implementation is
|
||||
* a no-op (in-memory writes are durable-by-definition within the process);
|
||||
* the filesystem adapter overrides it with real `fsync` of the files and
|
||||
* their parent directories.
|
||||
*
|
||||
* @param paths - Storage-root-relative object paths previously written via
|
||||
* {@link BaseStorage.writeRawObject}.
|
||||
*/
|
||||
public async syncRawObjects(paths: string[]): Promise<void> {
|
||||
void paths
|
||||
}
|
||||
|
||||
/**
|
||||
* Read an entity's raw stored objects — the exact bytes at its canonical
|
||||
* metadata + vector paths (branch-scoped, write-cache coherent). Used by
|
||||
* the generation store to capture before-images.
|
||||
*
|
||||
* @param id - The entity id.
|
||||
* @returns The raw stored metadata and vector objects (`null` per part when
|
||||
* the corresponding file is absent).
|
||||
*/
|
||||
public async readNounRaw(id: string): Promise<{ metadata: any | null; vector: any | null }> {
|
||||
await this.ensureInitialized()
|
||||
const [metadata, vector] = await Promise.all([
|
||||
this.readWithInheritance(getNounMetadataPath(id)),
|
||||
this.readWithInheritance(getNounVectorPath(id))
|
||||
])
|
||||
return { metadata: metadata ?? null, vector: vector ?? null }
|
||||
}
|
||||
|
||||
/**
|
||||
* Restore an entity's raw stored objects byte-for-byte (a `null` part
|
||||
* deletes that file). Used by crash recovery and transaction aborts to
|
||||
* restore before-images.
|
||||
*
|
||||
* Bypasses the statistics/count bookkeeping of the normal save paths on
|
||||
* purpose: restores must reproduce the exact prior bytes, and the count
|
||||
* rollups are derived state with their own rebuild paths
|
||||
* (`rebuildTypeCounts()` / `rebuildSubtypeCounts()`).
|
||||
*
|
||||
* @param id - The entity id.
|
||||
* @param record - Raw stored objects as returned by {@link BaseStorage.readNounRaw}.
|
||||
*/
|
||||
public async writeNounRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
if (record.metadata === null) {
|
||||
await this.deleteObjectFromBranch(getNounMetadataPath(id))
|
||||
} else {
|
||||
await this.writeObjectToBranch(getNounMetadataPath(id), record.metadata)
|
||||
}
|
||||
if (record.vector === null) {
|
||||
await this.deleteObjectFromBranch(getNounVectorPath(id))
|
||||
} else {
|
||||
await this.writeObjectToBranch(getNounVectorPath(id), record.vector)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a relationship's raw stored objects (verb-side mirror of
|
||||
* {@link BaseStorage.readNounRaw}).
|
||||
*
|
||||
* @param id - The relationship id.
|
||||
* @returns The raw stored metadata and vector objects (`null` per part when
|
||||
* the corresponding file is absent).
|
||||
*/
|
||||
public async readVerbRaw(id: string): Promise<{ metadata: any | null; vector: any | null }> {
|
||||
await this.ensureInitialized()
|
||||
const [metadata, vector] = await Promise.all([
|
||||
this.readWithInheritance(getVerbMetadataPath(id)),
|
||||
this.readWithInheritance(getVerbVectorPath(id))
|
||||
])
|
||||
return { metadata: metadata ?? null, vector: vector ?? null }
|
||||
}
|
||||
|
||||
/**
|
||||
* Restore a relationship's raw stored objects byte-for-byte (verb-side
|
||||
* mirror of {@link BaseStorage.writeNounRaw}; same bookkeeping caveats).
|
||||
*
|
||||
* @param id - The relationship id.
|
||||
* @param record - Raw stored objects as returned by {@link BaseStorage.readVerbRaw}.
|
||||
*/
|
||||
public async writeVerbRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
if (record.metadata === null) {
|
||||
await this.deleteObjectFromBranch(getVerbMetadataPath(id))
|
||||
} else {
|
||||
await this.writeObjectToBranch(getVerbMetadataPath(id), record.metadata)
|
||||
}
|
||||
if (record.vector === null) {
|
||||
await this.deleteObjectFromBranch(getVerbVectorPath(id))
|
||||
} else {
|
||||
await this.writeObjectToBranch(getVerbVectorPath(id), record.vector)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Append one line to the transaction log (`_system/tx-log.jsonl`). The
|
||||
* filesystem adapter appends to a real JSONL file; the in-memory adapter
|
||||
* keeps a line array (serialized by `snapshotToDirectory`).
|
||||
*
|
||||
* @param line - One complete JSON document, without trailing newline.
|
||||
*/
|
||||
public abstract appendTxLogLine(line: string): Promise<void>
|
||||
|
||||
/**
|
||||
* Read all transaction-log lines, oldest first (empty array when no log
|
||||
* exists). A torn trailing line from a crashed append is returned as-is —
|
||||
* callers tolerate unparseable lines.
|
||||
*/
|
||||
public abstract readTxLogLines(): Promise<string[]>
|
||||
|
||||
/**
|
||||
* Snapshot the entire store into `targetPath`. The filesystem adapter
|
||||
* builds a hard-link farm (instant, space-shared — safe because data files
|
||||
* are immutable-by-rename); the in-memory adapter serializes its object
|
||||
* store to a filesystem-storage-compatible directory. The result is a
|
||||
* self-contained store openable via `Brainy.load(path)`.
|
||||
*
|
||||
* @param targetPath - Absolute directory path for the snapshot (created if
|
||||
* missing; must be empty or absent).
|
||||
*/
|
||||
public abstract snapshotToDirectory(targetPath: string): Promise<void>
|
||||
|
||||
/**
|
||||
* Replace the entire store's contents from a snapshot directory previously
|
||||
* produced by {@link BaseStorage.snapshotToDirectory}. Implementations
|
||||
* clear current contents (preserving live lock files), copy the snapshot
|
||||
* in (byte copy — never hard links, so the snapshot stays independent),
|
||||
* and then call {@link BaseStorage.reloadDerivedState}.
|
||||
*
|
||||
* @param sourcePath - Absolute path of the snapshot directory.
|
||||
*/
|
||||
public abstract restoreFromDirectory(sourcePath: string): Promise<void>
|
||||
|
||||
/**
|
||||
* Reset and reload every piece of adapter-internal derived state after the
|
||||
* underlying objects changed wholesale (restore-from-snapshot): the
|
||||
* write-through cache, the id→type/subtype caches, type/subtype statistics,
|
||||
* total counts, and the graph-index singleton (invalidated so the next
|
||||
* accessor rebuilds from the restored verbs).
|
||||
*/
|
||||
protected async reloadDerivedState(): Promise<void> {
|
||||
this.clearWriteCache()
|
||||
this.nounTypeByIdCache.clear()
|
||||
this.nounSubtypeByIdCache.clear()
|
||||
this.verbSubtypeByIdCache.clear()
|
||||
this.nounCountsByType.fill(0)
|
||||
this.verbCountsByType.fill(0)
|
||||
this.subtypeCountsByType.clear()
|
||||
this.verbSubtypeCountsByType.clear()
|
||||
this.statisticsCache = null
|
||||
this.statisticsModified = false
|
||||
this.invalidateGraphIndex()
|
||||
await this.loadTypeStatistics()
|
||||
await this.loadSubtypeStatistics()
|
||||
await this.loadVerbSubtypeStatistics()
|
||||
await this.initializeCounts()
|
||||
}
|
||||
|
||||
/**
|
||||
* Save a noun to storage (vector only, metadata saved separately)
|
||||
* @param noun Pure HNSW vector data (no metadata)
|
||||
|
|
@ -2315,6 +2580,10 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
// Ignore persist errors - will retry on next operation
|
||||
})
|
||||
}
|
||||
|
||||
// 8.0 MVCC: entity-visible write — advance the generation watermark
|
||||
// (suppressed inside transact batches by the generation store).
|
||||
this.generationBumpHook?.()
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -2717,6 +2986,10 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
this.decrementSubtypeCount(priorType, priorSubtype)
|
||||
}
|
||||
}
|
||||
|
||||
// 8.0 MVCC: entity-visible write — advance the generation watermark
|
||||
// (suppressed inside transact batches by the generation store).
|
||||
this.generationBumpHook?.()
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -2750,6 +3023,8 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
// Backward compatibility: fallback to old path if no verb type
|
||||
const keyInfo = this.analyzeKey(id, 'verb-metadata')
|
||||
await this.writeObjectToBranch(keyInfo.fullPath, metadata)
|
||||
// 8.0 MVCC: still an entity-visible write — advance the watermark.
|
||||
this.generationBumpHook?.()
|
||||
return
|
||||
}
|
||||
|
||||
|
|
@ -2804,6 +3079,10 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
// Ignore persist errors - will retry on next operation
|
||||
})
|
||||
}
|
||||
|
||||
// 8.0 MVCC: entity-visible write — advance the generation watermark
|
||||
// (suppressed inside transact batches by the generation store).
|
||||
this.generationBumpHook?.()
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -2841,6 +3120,10 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
this.verbSubtypeByIdCache.delete(id)
|
||||
this.decrementVerbSubtypeCount(priorEntry.verb, priorEntry.subtype)
|
||||
}
|
||||
|
||||
// 8.0 MVCC: entity-visible write — advance the generation watermark
|
||||
// (suppressed inside transact batches by the generation store).
|
||||
this.generationBumpHook?.()
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue