soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat: temporal VFS — file content joins the Model-B immutability model

Browse source 
The temporal model had a hole exactly where files were concerned: every
entity write is an immutable generation with before-images, but VFS content
BYTES lived under an eager refCount GC left over from the pre-8.0 design —
unlink could physically destroy bytes that in-window history still
referenced, and overwrite never released the old hash at all (an unbounded
silent leak whose accidental byproduct was the only thing "preserving"
history). Reading the past could therefore return a stale field, a dangling
hash, or nothing, depending on luck.

Fix: blob reclamation becomes a HISTORY decision instead of a LIVENESS
decision. Each blob's metadata now carries historyRefCount alongside the
live refCount:

- The commit seam counts one history reference per persisted before-image
  record carrying a content hash (commitTransaction staging and the
  group-commit flush), recorded BEFORE the record-set persists and carried
  in the generation delta (blobHashes — always present on new deltas, so
  compaction only falls back to reading records for pre-contract
  generations). An aborted transaction compensates best-effort.
- unlink/rmdir/overwrite drop ONLY the live reference (BlobStorage.delete →
  release; overwrite finally releases the superseded hash — cancelling the
  dedup increment on same-content rewrites and closing the leak), and only
  AFTER the canonical mutation commits, so a failed delete can never leave a
  live file whose bytes compaction might reclaim.
- History compaction is the ONE reclamation point: after deleting a
  generation's record-set it releases that set's references and physically
  reclaims any hash at zero live AND zero history references. Pins are
  exempt automatically. Crash ordering is over-count-only in every path
  (record before persist, release after delete), so a crash can leak until
  the scrub recounts but can never reclaim bytes a retained generation
  needs. scrubBlobHistoryRefCounts() restores exactness; existing stores get
  a one-time marker-gated backfill on open, failing into leak-safe mode
  (reclamation disabled) rather than guessing.

On top of the protected history, the temporal API the generational model
always implied:

- vfs.readFile(path, { asOf }) — the exact bytes as of a generation or Date,
  materialized from the history (pinned view released so compaction is
  never blocked by a read).
- vfs.history(path) — FileVersion[] ascending ({ generation, timestamp,
  hash, size, mimeType? }), the newest entry being the live state.
- Overwrites now refresh the file entity's data/embedding text — semantic
  search and the data field previously served the FIRST version's text
  forever (the stale-field defect a consumer's incident recovery depended
  on by luck).

Integration suite (temporal-vfs.test.ts): per-version exact reads +
history listing, leak-fix + history protection on overwrite, rm keeps bytes
readable, compaction reclaims past-window bytes and preserves in-window
(including the cross-file dedup case where an old file's history and a
newer file's removal share one hash), data freshness, and scrub exactness.
This commit is contained in:
David Snelling 2026-07-10 16:43:48 -07:00
parent 4af8fb31e2
commit a3467e1f9b
13 changed files with 890 additions and 57 deletions
Download patch file Download diff file Expand all files Collapse all files

113
src/db/generationStore.ts
View file

@ -437,6 +437,28 @@ export class GenerationStore {
return this.horizonGen
}
/**
* @description Read one generation's persisted before-image records — the
* compaction fallback for generations written before deltas carried
* `blobHashes`. O(that generation's records).
* @param gen - The generation whose `prev/` records to read.
* @returns The record-set (empty when absent).
*/
private async readGenerationRecords(gen: number): Promise<GenerationRecord[]> {
let paths: string[] = []
try {
paths = await this.storage.listRawObjects(`${GENERATIONS_PREFIX}/${gen}/prev`)
} catch {
return []
}
const records: GenerationRecord[] = []
for (const p of paths) {
const record = (await this.storage.readRawObject(p)) as GenerationRecord | null
if (record) records.push(record)
}
return records
}
/**
* @description Single-operation write hook (registered with the storage
* layer in {@link open}). Bumps the in-memory counter and schedules a
@ -592,6 +614,10 @@ export class GenerationStore {
}
}
// Temporal-blob contract: hashes this record-set references, recorded
// before staging; scoped outside the try so an abort can compensate.
let txBlobHashes: string[] = []
try {
// -- 3. Before-images + delta (the durable undo log) ------------------
// Read every before-image FIRST, then run the caller's CAS
@ -615,6 +641,19 @@ export class GenerationStore {
// which returns the generation reservation; nothing was applied.
args.precommit?.({ nouns: nounBefore, verbs: verbBefore })
// Temporal-blob contract: count this record-set's content-hash
// references BEFORE it is staged (over-count-only crash ordering —
// see flushPendingSingleOps' matching note).
txBlobHashes = this.storage.extractBlobHashesFromRecords
? this.storage.extractBlobHashesFromRecords([
...nounBefore.values(),
...verbBefore.values()
])
: []
if (txBlobHashes.length > 0 && this.storage.recordHistoryBlobReferences) {
await this.storage.recordHistoryBlobReferences(txBlobHashes)
}
const stagedPaths: string[] = []
let recordBytes = 0 // serialized record-set size, for retention accounting
for (const [id, record] of nounBefore) {
@ -634,7 +673,10 @@ export class GenerationStore {
timestamp,
...(args.meta && { meta: args.meta }),
nouns,
verbs
verbs,
// Always present on new deltas (empty = "no blobs"), so compaction
// only falls back to record reads for pre-contract generations.
blobHashes: txBlobHashes
}
delta.bytes = recordBytes + serializedBytes(delta)
const deltaPath = `${dir}/tx.json`
@ -697,6 +739,18 @@ export class GenerationStore {
`${(cleanupErr as Error).message} (recovery will remove it on next open)`
)
}
// Compensate the pre-staging history-reference increments. Best
// effort — a failure here only over-counts (a leak the scrub
// repairs). Reclaim inside is safe on an abort: the before-image
// hashes are the entities' still-live content, so live references
// block any physical delete.
if (txBlobHashes.length > 0 && this.storage.releaseHistoryBlobReferences) {
try {
await this.storage.releaseHistoryBlobReferences(txBlobHashes)
} catch {
// over-count-safe; the scrub restores exactness
}
}
// Return the reservation when no concurrent bump consumed a later
// number, so a failed transaction leaves generation() unchanged.
if (this.counter === gen) this.counter = gen - 1
@ -860,6 +914,21 @@ export class GenerationStore {
const buf = this.pendingBuffer.get(gen)
if (!buf) continue
const dir = `${GENERATIONS_PREFIX}/${gen}`
// Temporal-blob contract: count this record-set's content-hash
// references BEFORE persisting it (a crash between the two only
// over-counts — the scrub repairs a leak; under-counting could let
// compaction reclaim bytes a retained generation still needs).
const blobHashes = this.storage.extractBlobHashesFromRecords
? this.storage.extractBlobHashesFromRecords([
...buf.nouns.values(),
...buf.verbs.values()
])
: []
if (blobHashes.length > 0 && this.storage.recordHistoryBlobReferences) {
await this.storage.recordHistoryBlobReferences(blobHashes)
}
const nounIds: string[] = []
const verbIds: string[] = []
let recordBytes = 0
@ -882,7 +951,10 @@ export class GenerationStore {
timestamp: buf.timestamp,
groupCommit: true,
nouns: nounIds,
verbs: verbIds
verbs: verbIds,
// Always present on new deltas (empty = "no blobs"), so compaction
// only falls back to record reads for pre-contract generations.
blobHashes
}
delta.bytes = recordBytes + serializedBytes(delta)
genBytes.set(gen, delta.bytes)
@ -1722,20 +1794,45 @@ export class GenerationStore {
for (const gen of [...this.committedGensAsc()]) {
// Pins are always exempt: never reclaim a generation a live pin needs.
if (gen > minPinned) break // committedGensAsc ascending → nothing newer is eligible either
const delta = await this.getDelta(gen)
if (!noCaps) {
const violatesCount = maxGenerations !== undefined && remainingCount > maxGenerations
const violatesBytes = maxBytes !== undefined && remainingBytes > maxBytes
let violatesAge = false
if (ageCutoff !== undefined) {
const delta = await this.getDelta(gen)
violatesAge = delta.timestamp < ageCutoff
}
const violatesAge = ageCutoff !== undefined && delta.timestamp < ageCutoff
// Oldest-first: once the oldest unpinned gen trips no cap, none newer do.
if (!violatesCount && !violatesBytes && !violatesAge) break
}
const genBytes = maxBytes !== undefined ? (await this.getDelta(gen)).bytes : 0
const genBytes = maxBytes !== undefined ? delta.bytes : 0
// Temporal-blob contract: resolve the content-blob hashes this
// generation's record-set references BEFORE deleting it. New
// generations carry the multiset in their persisted delta (empty
// array when none — distinguishing "new format, no blobs" from a
// pre-contract delta); legacy generations fall back to reading the
// records themselves. Skipped entirely on non-blob-aware storage.
let blobHashes: string[] | undefined
if (this.storage.releaseHistoryBlobReferences) {
const rawDelta = (await this.storage.readRawObject(
`${GENERATIONS_PREFIX}/${gen}/tx.json`
)) as GenerationDelta | null
blobHashes = rawDelta?.blobHashes
if (blobHashes === undefined && this.storage.extractBlobHashesFromRecords) {
blobHashes = this.storage.extractBlobHashesFromRecords(
await this.readGenerationRecords(gen)
)
}
}
await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${gen}`)
this.deltaCache.delete(gen)
// AFTER the record-set is gone (over-count-only crash ordering):
// release its history references and reclaim any blob left with zero
// live AND zero history references — the system's one byte-reclaim point.
if (blobHashes && blobHashes.length > 0 && this.storage.releaseHistoryBlobReferences) {
await this.storage.releaseHistoryBlobReferences(blobHashes)
}
remainingCount--
remainingBytes -= genBytes
removed.push(gen)