feat: temporal VFS — file content joins the Model-B immutability model
The temporal model had a hole exactly where files were concerned: every
entity write is an immutable generation with before-images, but VFS content
BYTES lived under an eager refCount GC left over from the pre-8.0 design —
unlink could physically destroy bytes that in-window history still
referenced, and overwrite never released the old hash at all (an unbounded
silent leak whose accidental byproduct was the only thing "preserving"
history). Reading the past could therefore return a stale field, a dangling
hash, or nothing, depending on luck.
Fix: blob reclamation becomes a HISTORY decision instead of a LIVENESS
decision. Each blob's metadata now carries historyRefCount alongside the
live refCount:
- The commit seam counts one history reference per persisted before-image
record carrying a content hash (commitTransaction staging and the
group-commit flush), recorded BEFORE the record-set persists and carried
in the generation delta (blobHashes — always present on new deltas, so
compaction only falls back to reading records for pre-contract
generations). An aborted transaction compensates best-effort.
- unlink/rmdir/overwrite drop ONLY the live reference (BlobStorage.delete →
release; overwrite finally releases the superseded hash — cancelling the
dedup increment on same-content rewrites and closing the leak), and only
AFTER the canonical mutation commits, so a failed delete can never leave a
live file whose bytes compaction might reclaim.
- History compaction is the ONE reclamation point: after deleting a
generation's record-set it releases that set's references and physically
reclaims any hash at zero live AND zero history references. Pins are
exempt automatically. Crash ordering is over-count-only in every path
(record before persist, release after delete), so a crash can leak until
the scrub recounts but can never reclaim bytes a retained generation
needs. scrubBlobHistoryRefCounts() restores exactness; existing stores get
a one-time marker-gated backfill on open, failing into leak-safe mode
(reclamation disabled) rather than guessing.
On top of the protected history, the temporal API the generational model
always implied:
- vfs.readFile(path, { asOf }) — the exact bytes as of a generation or Date,
materialized from the history (pinned view released so compaction is
never blocked by a read).
- vfs.history(path) — FileVersion[] ascending ({ generation, timestamp,
hash, size, mimeType? }), the newest entry being the live state.
- Overwrites now refresh the file entity's data/embedding text — semantic
search and the data field previously served the FIRST version's text
forever (the stale-field defect a consumer's incident recovery depended
on by luck).
Integration suite (temporal-vfs.test.ts): per-version exact reads +
history listing, leak-fix + history protection on overwrite, rm keeps bytes
readable, compaction reclaims past-window bytes and preserves in-window
(including the cross-file dedup case where an old file's history and a
newer file's removal share one hash), data freshness, and scrub exactness.
This commit is contained in:
parent
4af8fb31e2
commit
a3467e1f9b
13 changed files with 890 additions and 57 deletions
|
|
@ -869,6 +869,148 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
this.generationBumpHook = hook
|
||||
}
|
||||
|
||||
// ==========================================================================
|
||||
// Temporal-blob contract (the GenerationStorage optional methods)
|
||||
//
|
||||
// Content blobs join the Model-B immutability model through these hooks:
|
||||
// the generation store counts a history reference per before-image record
|
||||
// that carries a content hash, and compaction — the ONE reclamation point —
|
||||
// releases those references and physically deletes bytes only at zero live
|
||||
// AND zero history references. Crash ordering is over-count-only (record
|
||||
// BEFORE the record-set persists, release AFTER it is deleted), so a crash
|
||||
// can leak bytes until the scrub recounts but can never reclaim bytes a
|
||||
// retained generation still needs.
|
||||
// ==========================================================================
|
||||
|
||||
/** Set when the open-time backfill/scrub could not verify history reference
|
||||
* counts. While true, the temporal-blob hooks stop mutating counts and
|
||||
* compaction stops reclaiming blob bytes — pure leak-safe mode until a
|
||||
* successful {@link scrubBlobHistoryRefCounts} restores exactness. */
|
||||
private blobHistoryRefsUnverified = false
|
||||
|
||||
/**
|
||||
* @description Extract the content-blob hashes a generation record-set
|
||||
* references — a pure MULTISET extraction (one entry per referencing record
|
||||
* occurrence), no side effects. Only entity records can reference VFS
|
||||
* content (`metadata.storage.type === 'blob'`).
|
||||
* @param records - The record-set's before-image records.
|
||||
* @returns The referenced hashes, duplicates preserved.
|
||||
*/
|
||||
public extractBlobHashesFromRecords(
|
||||
records: Array<{ kind: string; metadata: unknown }>
|
||||
): string[] {
|
||||
const hashes: string[] = []
|
||||
for (const record of records) {
|
||||
if (record.kind !== 'noun') continue
|
||||
const storage = (record.metadata as { storage?: { type?: string; hash?: unknown } } | null)
|
||||
?.storage
|
||||
if (storage?.type === 'blob' && typeof storage.hash === 'string') {
|
||||
hashes.push(storage.hash)
|
||||
}
|
||||
}
|
||||
return hashes
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Record one history reference per hash occurrence (see the
|
||||
* contract note above — called BEFORE the referencing record-set persists).
|
||||
* No-op without a blob store or while counts are unverified.
|
||||
* @param hashes - Hash multiset from {@link extractBlobHashesFromRecords}.
|
||||
*/
|
||||
public async recordHistoryBlobReferences(hashes: string[]): Promise<void> {
|
||||
if (!this.blobStorage || this.blobHistoryRefsUnverified || hashes.length === 0) return
|
||||
for (const hash of hashes) {
|
||||
await this.blobStorage.recordHistoryReference(hash)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Release one history reference per hash occurrence and
|
||||
* physically reclaim any hash left with zero live AND zero history
|
||||
* references — compaction's blob-reclamation step (called AFTER the
|
||||
* referencing record-set is deleted). No-op without a blob store or while
|
||||
* counts are unverified (leak-safe: nothing is reclaimed on guesses).
|
||||
* @param hashes - Hash multiset recorded when the record-set was persisted.
|
||||
*/
|
||||
public async releaseHistoryBlobReferences(hashes: string[]): Promise<void> {
|
||||
if (!this.blobStorage || this.blobHistoryRefsUnverified || hashes.length === 0) return
|
||||
for (const hash of hashes) {
|
||||
await this.blobStorage.releaseHistoryReference(hash)
|
||||
}
|
||||
for (const hash of new Set(hashes)) {
|
||||
await this.blobStorage.reclaimIfUnreferenced(hash)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @description One-time (marker-gated) backfill of blob history reference
|
||||
* counts for stores whose generation history predates the temporal-blob
|
||||
* contract. Runs the scrub, then stamps `_system/blob-history-refs.json` so
|
||||
* later opens skip the walk. On scrub failure the store enters leak-safe
|
||||
* mode (counts untouched, reclamation disabled) rather than risking a
|
||||
* premature delete on wrong counts.
|
||||
*/
|
||||
public async backfillBlobHistoryRefCountsIfNeeded(): Promise<void> {
|
||||
if (!this.blobStorage) return
|
||||
const MARKER = '_system/blob-history-refs.json'
|
||||
try {
|
||||
const marker = (await this.readObjectFromPath(MARKER)) as { version?: number } | null
|
||||
if (marker?.version === 1) return
|
||||
} catch {
|
||||
// no marker — proceed to scrub
|
||||
}
|
||||
try {
|
||||
await this.scrubBlobHistoryRefCounts()
|
||||
await this.writeObjectToPath(MARKER, { version: 1, verifiedAt: new Date().toISOString() })
|
||||
} catch (err) {
|
||||
this.blobHistoryRefsUnverified = true
|
||||
console.error(
|
||||
'[Brainy] blob history-reference backfill failed — temporal-blob ' +
|
||||
'reclamation disabled for this session (leak-safe); history reads ' +
|
||||
'are unaffected. Re-open to retry.',
|
||||
err
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Recount every blob's history references from the actual
|
||||
* generation record-sets and set the counts ABSOLUTELY (uncounted blobs are
|
||||
* zeroed) — the idempotent repair that restores exactness after any crash
|
||||
* that over-counted. O(history records + stored blobs).
|
||||
* @returns Blobs counted and records walked, for observability.
|
||||
*/
|
||||
public async scrubBlobHistoryRefCounts(): Promise<{ blobs: number; records: number }> {
|
||||
if (!this.blobStorage) return { blobs: 0, records: 0 }
|
||||
const counts = new Map<string, number>()
|
||||
let records = 0
|
||||
let paths: string[] = []
|
||||
try {
|
||||
paths = await this.listObjectsUnderPath('_generations')
|
||||
} catch {
|
||||
paths = [] // no history yet
|
||||
}
|
||||
for (const p of paths) {
|
||||
if (!p.includes('/prev/')) continue
|
||||
const record = (await this.readObjectFromPath(p)) as
|
||||
| { kind?: string; metadata?: unknown }
|
||||
| null
|
||||
if (!record) continue
|
||||
records++
|
||||
for (const hash of this.extractBlobHashesFromRecords([
|
||||
{ kind: record.kind ?? '', metadata: record.metadata }
|
||||
])) {
|
||||
counts.set(hash, (counts.get(hash) ?? 0) + 1)
|
||||
}
|
||||
}
|
||||
const allHashes = await this.blobStorage.listHashes()
|
||||
for (const hash of allHashes) {
|
||||
await this.blobStorage.setHistoryRefCount(hash, counts.get(hash) ?? 0)
|
||||
}
|
||||
this.blobHistoryRefsUnverified = false
|
||||
return { blobs: allHashes.length, records }
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a raw object at a storage-root-relative path. Bypasses the write
|
||||
* cache (record-layer files are written through
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue