soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat: temporal VFS — file content joins the Model-B immutability model

Browse source 
The temporal model had a hole exactly where files were concerned: every
entity write is an immutable generation with before-images, but VFS content
BYTES lived under an eager refCount GC left over from the pre-8.0 design —
unlink could physically destroy bytes that in-window history still
referenced, and overwrite never released the old hash at all (an unbounded
silent leak whose accidental byproduct was the only thing "preserving"
history). Reading the past could therefore return a stale field, a dangling
hash, or nothing, depending on luck.

Fix: blob reclamation becomes a HISTORY decision instead of a LIVENESS
decision. Each blob's metadata now carries historyRefCount alongside the
live refCount:

- The commit seam counts one history reference per persisted before-image
  record carrying a content hash (commitTransaction staging and the
  group-commit flush), recorded BEFORE the record-set persists and carried
  in the generation delta (blobHashes — always present on new deltas, so
  compaction only falls back to reading records for pre-contract
  generations). An aborted transaction compensates best-effort.
- unlink/rmdir/overwrite drop ONLY the live reference (BlobStorage.delete →
  release; overwrite finally releases the superseded hash — cancelling the
  dedup increment on same-content rewrites and closing the leak), and only
  AFTER the canonical mutation commits, so a failed delete can never leave a
  live file whose bytes compaction might reclaim.
- History compaction is the ONE reclamation point: after deleting a
  generation's record-set it releases that set's references and physically
  reclaims any hash at zero live AND zero history references. Pins are
  exempt automatically. Crash ordering is over-count-only in every path
  (record before persist, release after delete), so a crash can leak until
  the scrub recounts but can never reclaim bytes a retained generation
  needs. scrubBlobHistoryRefCounts() restores exactness; existing stores get
  a one-time marker-gated backfill on open, failing into leak-safe mode
  (reclamation disabled) rather than guessing.

On top of the protected history, the temporal API the generational model
always implied:

- vfs.readFile(path, { asOf }) — the exact bytes as of a generation or Date,
  materialized from the history (pinned view released so compaction is
  never blocked by a read).
- vfs.history(path) — FileVersion[] ascending ({ generation, timestamp,
  hash, size, mimeType? }), the newest entry being the live state.
- Overwrites now refresh the file entity's data/embedding text — semantic
  search and the data field previously served the FIRST version's text
  forever (the stale-field defect a consumer's incident recovery depended
  on by luck).

Integration suite (temporal-vfs.test.ts): per-version exact reads +
history listing, leak-fix + history protection on overwrite, rm keeps bytes
readable, compaction reclaims past-window bytes and preserves in-window
(including the cross-file dedup case where an old file's history and a
newer file's removal share one hash), data freshness, and scrub exactness.
This commit is contained in:
David Snelling 2026-07-10 16:43:48 -07:00
parent 4af8fb31e2
commit a3467e1f9b
13 changed files with 890 additions and 57 deletions
Download patch file Download diff file Expand all files Collapse all files

194
src/vfs/VirtualFileSystem.ts
View file

@ -36,6 +36,7 @@ import {
VFSErrorCode,
WriteOptions,
ReadOptions,
FileVersion,
MkdirOptions,
ReaddirOptions,
CopyOptions,
@ -406,6 +407,14 @@ export class VirtualFileSystem implements IVirtualFileSystem {
async readFile(path: string, options?: ReadOptions): Promise<Buffer> {
await this.ensureInitialized()
// Temporal read: the file's exact bytes as of a past generation or
// instant. Materializes the entity from the generation history — content
// blobs referenced by any in-window generation are retention-protected,
// so the bytes are guaranteed present. Bypasses the content cache.
if (options?.asOf !== undefined) {
return this.readFileAt(path, options.asOf, options)
}
// Check cache first
if (options?.cache !== false && this.contentCache.has(path)) {
const cached = this.contentCache.get(path)!
@ -472,6 +481,122 @@ export class VirtualFileSystem implements IVirtualFileSystem {
}
}
/**
* @description The temporal read behind `readFile(path, { asOf })`: resolve
* the path's CURRENT entity, materialize its state at the target
* generation/instant from the Model-B history, and read that version's
* content blob (retention-protected, so present for every in-window
* generation). The pinned view is always released so compaction is never
* blocked by a read.
* @param path - The file path (resolved against the live tree).
* @param asOf - A generation number or wall-clock `Date`.
* @param options - `encoding` applies; caching does not (never cached).
* @returns The exact bytes the file held at that generation.
* @throws VFSError ENOENT when the file did not exist at that generation;
* the generation store's compacted-generation error when `asOf` is past
* the retention window's horizon.
*/
private async readFileAt(
path: string,
asOf: number | Date,
options?: ReadOptions
): Promise<Buffer> {
const entityId = await this.pathResolver.resolve(path)
const db = await this.brain.asOf(asOf)
try {
const entity = await db.get(entityId)
if (!entity) {
throw new VFSError(
VFSErrorCode.ENOENT,
`File did not exist as of ${String(asOf)}: ${path}`,
path,
'readFile'
)
}
const storage = (entity.metadata as Record<string, any> | undefined)?.storage
if (storage?.type !== 'blob' || typeof storage.hash !== 'string') {
throw new VFSError(
VFSErrorCode.EIO,
`File had no blob storage as of ${String(asOf)}: ${path}`,
path,
'readFile'
)
}
const content = await this.blobStorage.read(storage.hash)
if (options?.encoding) {
return Buffer.from(content.toString(options.encoding))
}
return content
} finally {
await db.release()
}
}
/**
* @description A file's version history within the retention window: one
* entry per generation that wrote the file, ascending — the newest entry is
* the current state. Pair with `readFile(path, { asOf: entry.generation })`
* to fetch any version's exact bytes, and restore by writing those bytes
* back (a NEW write — history is never rewritten). Bounded by the retention
* window: versions whose generations were compacted are not listed.
* @param path - The file path (resolved against the live tree).
* @returns The versions, oldest first.
* @example
* const versions = await brain.vfs.history('/pages/home.json')
* const before = await brain.vfs.readFile('/pages/home.json', {
* asOf: versions[versions.length - 2].generation
* })
*/
async history(path: string): Promise<FileVersion[]> {
await this.ensureInitialized()
const entityId = await this.pathResolver.resolve(path)
// Boundary note: reaches Brainy's internal generation store (the same
// bracket-access precedent as the blobStorage getter) — the per-id
// generation chain and horizon are not on the public Brainy surface.
const generationStore = (this.brain as unknown as {
generationStore: {
horizon(): number
generation(): number
generationsTouching(
kind: 'noun' | 'verb',
id: string,
fromGen: number,
toGen: number
): Promise<number[]>
}
}).generationStore
const gens = await generationStore.generationsTouching(
'noun',
entityId,
generationStore.horizon(),
generationStore.generation()
)
const versions: FileVersion[] = []
for (const gen of gens) {
const db = await this.brain.asOf(gen)
try {
const entity = await db.get(entityId)
const meta = entity?.metadata as Record<string, any> | undefined
const storage = meta?.storage
if (storage?.type === 'blob' && typeof storage.hash === 'string') {
versions.push({
generation: gen,
timestamp: db.timestamp,
hash: storage.hash,
size: typeof meta?.size === 'number' ? meta.size : (storage.size ?? 0),
...(typeof meta?.mimeType === 'string' && { mimeType: meta.mimeType })
})
}
} finally {
await db.release()
}
}
return versions
}
/**
* Write a file
*/
@ -493,14 +618,18 @@ export class VirtualFileSystem implements IVirtualFileSystem {
// Ensure parent directory exists
const parentId = await this.ensureDirectory(parentPath)
// Check if file already exists
// Check if file already exists (and capture its current content hash so
// the overwrite can release the superseded live reference afterwards).
let existingId: string | null = null
let previousHash: string | undefined
try {
existingId = await this.pathResolver.resolve(path, { cache: false })
// Verify the entity still exists in the brain
const existing = await this.brain.get(existingId)
if (!existing) {
existingId = null // Entity was deleted but cache wasn't cleared
} else if (existing.metadata?.storage?.type === 'blob') {
previousHash = existing.metadata.storage.hash as string | undefined
}
} catch (err) {
// File doesn't exist, which is fine
@ -551,14 +680,32 @@ export class VirtualFileSystem implements IVirtualFileSystem {
Object.assign(metadata, await this.extractMetadata(buffer, mimeType))
}
// For embedding: use text content, for storage: use raw data. Computed
// for BOTH branches — an overwrite must refresh the entity's `data` (and
// therefore its embedding), or semantic search and any `data` read keep
// serving the FIRST version's text forever.
const embeddingData = mimeDetector.isTextFile(mimeType)
? buffer.toString('utf-8')
: `File: ${name} (${mimeType}, ${buffer.length} bytes)`
if (existingId) {
// Update existing file
// No entity.data - content is in BlobStorage
// Update existing file — content bytes live in BlobStorage; `data`
// carries the embedding text and MUST track the new content.
await this.brain.update({
id: existingId,
data: embeddingData,
metadata
})
// The update committed: drop the superseded live reference. When the
// content is unchanged this cancels the dedup increment the write()
// above just made (net: one live reference per referencing file); when
// it changed, the old bytes stay retention-protected for asOf reads
// via the update's own generation record.
if (previousHash) {
await this.blobStorage.release(previousHash)
}
// Ensure Contains relationship exists (fix for missing relationships)
const existingRelations = await this.brain.related({
from: parentId,
@ -578,9 +725,6 @@ export class VirtualFileSystem implements IVirtualFileSystem {
}
} else {
// Create new file entity
// For embedding: use text content, for storage: use raw data
const embeddingData = mimeDetector.isTextFile(mimeType) ? buffer.toString('utf-8') : `File: ${name} (${mimeType}, ${buffer.length} bytes)`
const entity = await this.brain.add({
data: embeddingData, // Always provide string for embeddings
type: this.getFileNounType(mimeType),
@ -649,14 +793,21 @@ export class VirtualFileSystem implements IVirtualFileSystem {
throw new VFSError(VFSErrorCode.EISDIR, `Is a directory: ${path}`, path, 'unlink')
}
// Delete blob from BlobStorage (decrements ref count)
if (entity.metadata.storage?.type === 'blob') {
await this.blobStorage.delete(entity.metadata.storage.hash)
}
// Delete the entity
// Delete the entity FIRST, then drop its live blob reference — never
// release a reference while the referencing entity might survive (a
// failed remove must not leave a live file whose bytes compaction could
// later reclaim).
await this.brain.remove(entityId)
// Drop the file's LIVE reference to its content blob. The bytes are NOT
// deleted — the remove's own generation record references this hash, so
// `readFile(path, { asOf })` keeps serving it inside the retention
// window; history compaction reclaims the bytes when the last referencing
// generation is reclaimed.
if (entity.metadata.storage?.type === 'blob') {
await this.blobStorage.release(entity.metadata.storage.hash)
}
// Invalidate caches
this.pathResolver.invalidatePath(path)
this.invalidateCaches(path)
@ -1032,23 +1183,26 @@ export class VirtualFileSystem implements IVirtualFileSystem {
// Phase 1: Gather all descendants in ONE batch fetch
const descendants = await this.gatherDescendants(entityId, Infinity)
// Phase 2: Parallel blob cleanup (chunked to avoid overwhelming storage)
// Blob deletion is reference-counted, so safe to call for all files
// Phase 2: Batch delete all entities (including root directory) FIRST —
// never release a blob reference while its referencing entity might
// survive a failed delete (see unlink's ordering note).
const allIds = [...descendants.map(d => d.id), entityId]
await this.brain.removeMany({ ids: allIds, continueOnError: false })
// Phase 3: Drop the deleted files' LIVE blob references (chunked).
// Live-reference drops only — the bytes stay for in-window asOf reads
// and are reclaimed by history compaction (see unlink's note).
const blobFiles = descendants.filter(d =>
d.metadata.vfsType === 'file' && d.metadata.storage?.type === 'blob'
)
const BLOB_CHUNK_SIZE = 20 // Parallel delete 20 blobs at a time
const BLOB_CHUNK_SIZE = 20 // Parallel release 20 blob references at a time
for (let i = 0; i < blobFiles.length; i += BLOB_CHUNK_SIZE) {
const chunk = blobFiles.slice(i, i + BLOB_CHUNK_SIZE)
await Promise.all(chunk.map(f =>
this.blobStorage.delete(f.metadata.storage!.hash)
this.blobStorage.release(f.metadata.storage!.hash)
))
}
// Phase 3: Batch delete all entities (including root directory)
const allIds = [...descendants.map(d => d.id), entityId]
await this.brain.removeMany({ ids: allIds, continueOnError: false })
} else {
// No children or not recursive - just delete the directory entity
await this.brain.remove(entityId)

30
src/vfs/types.ts
View file

@ -195,6 +195,36 @@ export interface ReadOptions {
// VFS-specific options
cache?: boolean // Use cache if available (default: true)
decompress?: boolean // Auto-decompress (default: true)
/**
* Read the file's content as it stood at a past generation (a number) or
* wall-clock instant (a Date) — the temporal read. Serves the exact bytes
* the file held then: the historical entity is materialized from the
* generation history, and its content blob is retention-protected (bytes
* referenced by any in-window generation are never reclaimed). Bounded by
* the retention window: reading past the compaction horizon throws.
* Bypasses the content cache.
*/
asOf?: number | Date
}
/**
* One entry of a file's version history (see `vfs.history(path)`): the state
* the file held immediately after `generation` committed.
* `readFile(path, { asOf: generation })` returns these exact bytes while the
* generation remains inside the retention window.
*/
export interface FileVersion {
/** The generation whose write produced this version. */
generation: number
/** Commit timestamp of that generation (ms since epoch). */
timestamp: number
/** Content hash of this version's bytes in the content-addressed store. */
hash: string
/** Content size in bytes. */
size: number
/** Detected MIME type at that version. */
mimeType?: string
}
export interface MkdirOptions {