feat: temporal VFS — file content joins the Model-B immutability model
The temporal model had a hole exactly where files were concerned: every
entity write is an immutable generation with before-images, but VFS content
BYTES lived under an eager refCount GC left over from the pre-8.0 design —
unlink could physically destroy bytes that in-window history still
referenced, and overwrite never released the old hash at all (an unbounded
silent leak whose accidental byproduct was the only thing "preserving"
history). Reading the past could therefore return a stale field, a dangling
hash, or nothing, depending on luck.
Fix: blob reclamation becomes a HISTORY decision instead of a LIVENESS
decision. Each blob's metadata now carries historyRefCount alongside the
live refCount:
- The commit seam counts one history reference per persisted before-image
record carrying a content hash (commitTransaction staging and the
group-commit flush), recorded BEFORE the record-set persists and carried
in the generation delta (blobHashes — always present on new deltas, so
compaction only falls back to reading records for pre-contract
generations). An aborted transaction compensates best-effort.
- unlink/rmdir/overwrite drop ONLY the live reference (BlobStorage.delete →
release; overwrite finally releases the superseded hash — cancelling the
dedup increment on same-content rewrites and closing the leak), and only
AFTER the canonical mutation commits, so a failed delete can never leave a
live file whose bytes compaction might reclaim.
- History compaction is the ONE reclamation point: after deleting a
generation's record-set it releases that set's references and physically
reclaims any hash at zero live AND zero history references. Pins are
exempt automatically. Crash ordering is over-count-only in every path
(record before persist, release after delete), so a crash can leak until
the scrub recounts but can never reclaim bytes a retained generation
needs. scrubBlobHistoryRefCounts() restores exactness; existing stores get
a one-time marker-gated backfill on open, failing into leak-safe mode
(reclamation disabled) rather than guessing.
On top of the protected history, the temporal API the generational model
always implied:
- vfs.readFile(path, { asOf }) — the exact bytes as of a generation or Date,
materialized from the history (pinned view released so compaction is
never blocked by a read).
- vfs.history(path) — FileVersion[] ascending ({ generation, timestamp,
hash, size, mimeType? }), the newest entry being the live state.
- Overwrites now refresh the file entity's data/embedding text — semantic
search and the data field previously served the FIRST version's text
forever (the stale-field defect a consumer's incident recovery depended
on by luck).
Integration suite (temporal-vfs.test.ts): per-version exact reads +
history listing, leak-fix + history protection on overwrite, rm keeps bytes
readable, compaction reclaims past-window bytes and preserves in-window
(including the cross-file dedup case where an old file's history and a
newer file's removal share one hash), data freshness, and scrub exactness.
This commit is contained in:
parent
4af8fb31e2
commit
a3467e1f9b
13 changed files with 890 additions and 57 deletions
|
|
@ -171,18 +171,23 @@ describe('BlobStorage refCount is exact under concurrency (per-hash mutex)', ()
|
|||
expect(meta?.refCount).toBe(10)
|
||||
})
|
||||
|
||||
it('the blob survives until the LAST reference drops — no premature deletion', async () => {
|
||||
it('references drop exactly; bytes survive live-zero (temporal immutability) until reclaim', async () => {
|
||||
const blobs = new BlobStorage(memAdapter() as any)
|
||||
const payload = Buffer.from('shared bytes, two referencing files')
|
||||
|
||||
const hash = await blobs.write(payload)
|
||||
await blobs.write(payload) // second reference (concurrent-equivalent path)
|
||||
|
||||
await blobs.delete(hash) // drop one reference
|
||||
await blobs.release(hash) // drop one reference
|
||||
expect((await blobs.read(hash)).toString()).toBe(payload.toString()) // still readable
|
||||
expect((await blobs.getMetadata(hash))?.refCount).toBe(1)
|
||||
|
||||
await blobs.delete(hash) // last reference
|
||||
await blobs.release(hash) // last LIVE reference — bytes still exist (history may need them)
|
||||
expect(await blobs.has(hash)).toBe(true)
|
||||
expect((await blobs.getMetadata(hash))?.refCount).toBe(0)
|
||||
|
||||
// Reclamation is compaction's job: zero-zero → physically removed.
|
||||
expect(await blobs.reclaimIfUnreferenced(hash)).toBe(true)
|
||||
expect(await blobs.has(hash)).toBe(false)
|
||||
await expect(blobs.read(hash)).rejects.toThrow()
|
||||
})
|
||||
|
|
@ -192,15 +197,15 @@ describe('BlobStorage refCount is exact under concurrency (per-hash mutex)', ()
|
|||
const payload = Buffer.from('storm payload')
|
||||
const hash = BlobStorage.hash(payload)
|
||||
|
||||
// 12 writes and 5 deletes racing: net 7 references, blob alive.
|
||||
// 12 writes and 5 releases racing: net 7 references, blob alive.
|
||||
await Promise.all([
|
||||
...Array.from({ length: 12 }, () => blobs.write(payload)),
|
||||
...Array.from({ length: 5 }, () => blobs.delete(hash))
|
||||
...Array.from({ length: 5 }, () => blobs.release(hash))
|
||||
])
|
||||
const meta = await blobs.getMetadata(hash)
|
||||
// Deletes against a not-yet-written hash floor at zero without deleting,
|
||||
// so the net can only be >= 12 - 5. The exactness we require: counts are
|
||||
// never LOST (each landed write is represented).
|
||||
// Releases against a not-yet-written hash floor at zero, so the net can
|
||||
// only be >= 12 - 5. The exactness we require: counts are never LOST
|
||||
// (each landed write is represented).
|
||||
expect(meta?.refCount).toBeGreaterThanOrEqual(7)
|
||||
expect(await blobs.has(hash)).toBe(true)
|
||||
})
|
||||
|
|
|
|||
200
tests/integration/temporal-vfs.test.ts
Normal file
200
tests/integration/temporal-vfs.test.ts
Normal file
|
|
@ -0,0 +1,200 @@
|
|||
/**
|
||||
* @module tests/integration/temporal-vfs
|
||||
* @description VFS content joins the Model-B immutability model. Previously
|
||||
* the temporal model had a hole exactly where files were concerned: entity
|
||||
* records were versioned per write (before-images), but the content BYTES
|
||||
* lived under an eager refCount GC — `rm` could physically delete bytes that
|
||||
* in-window history still referenced, and overwrite never released the old
|
||||
* hash at all (an unbounded silent leak that only accidentally preserved
|
||||
* history). Now blob bytes are retention-protected: a content blob referenced
|
||||
* by any generation inside the retention window (or live) survives, and the
|
||||
* ONE reclamation point is history compaction.
|
||||
*
|
||||
* Pins:
|
||||
* - `readFile(path, { asOf })` returns each version's exact bytes.
|
||||
* - `history(path)` lists versions ascending with generation/hash/size.
|
||||
* - overwrite releases the superseded live reference (leak fixed) while
|
||||
* history protects the bytes; rm keeps bytes readable via asOf.
|
||||
* - compaction past the last referencing generation physically reclaims
|
||||
* bytes (zero live + zero history), and never reclaims in-window content —
|
||||
* including the cross-file dedup case where an old file's history and a
|
||||
* newer file's history share one hash.
|
||||
* - overwrite refreshes the entity's `data` (embedding text) — the stale
|
||||
* `.data` defect.
|
||||
* - the backfill/scrub recounts exactly from the generation records.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import * as fs from 'node:fs'
|
||||
import * as os from 'node:os'
|
||||
import * as path from 'node:path'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
|
||||
describe('temporal VFS — blob immutability under Model B', () => {
|
||||
let dir: string
|
||||
let brain: any
|
||||
|
||||
beforeEach(async () => {
|
||||
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
|
||||
dir = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-temporal-vfs-'))
|
||||
brain = new Brainy({
|
||||
requireSubtype: false,
|
||||
storage: { type: 'filesystem', path: dir },
|
||||
dimensions: 384,
|
||||
silent: true
|
||||
})
|
||||
await brain.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
fs.rmSync(dir, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
/** The file's blob metadata via the storage layer (test introspection). */
|
||||
async function blobMeta(hash: string) {
|
||||
return brain['storage'].blobStorage.getMetadata(hash)
|
||||
}
|
||||
|
||||
it('readFile(path, {asOf}) returns each version‘s exact bytes; history(path) lists them', async () => {
|
||||
const p = '/pages/home.json'
|
||||
await brain.vfs.writeFile(p, 'v1 — original')
|
||||
const g1 = brain.generationStore.generation()
|
||||
await brain.vfs.writeFile(p, 'v2 — edited')
|
||||
await brain.vfs.writeFile(p, 'v3 — final')
|
||||
|
||||
// Live read = newest.
|
||||
expect((await brain.vfs.readFile(p)).toString()).toBe('v3 — final')
|
||||
|
||||
// History lists the file's write-generations ascending, distinct hashes.
|
||||
const versions = await brain.vfs.history(p)
|
||||
expect(versions.length).toBeGreaterThanOrEqual(3)
|
||||
const gens = versions.map((v: any) => v.generation)
|
||||
expect([...gens].sort((a: number, b: number) => a - b)).toEqual(gens)
|
||||
|
||||
// Every listed version's exact bytes are readable.
|
||||
const texts: string[] = []
|
||||
for (const v of versions) {
|
||||
texts.push((await brain.vfs.readFile(p, { asOf: v.generation })).toString())
|
||||
}
|
||||
expect(texts).toContain('v1 — original')
|
||||
expect(texts).toContain('v2 — edited')
|
||||
expect(texts).toContain('v3 — final')
|
||||
|
||||
// Direct generation read too (the incident shape: recover the past bytes).
|
||||
expect((await brain.vfs.readFile(p, { asOf: g1 })).toString()).toBe('v1 — original')
|
||||
})
|
||||
|
||||
it('overwrite releases the superseded LIVE reference (leak fixed) while history protects the bytes', async () => {
|
||||
const p = '/notes/leak.txt'
|
||||
await brain.vfs.writeFile(p, 'old content A')
|
||||
const oldHash = (await brain.vfs.history(p))[0].hash
|
||||
const g1 = brain.generationStore.generation()
|
||||
|
||||
await brain.vfs.writeFile(p, 'new content B')
|
||||
|
||||
// The old hash's LIVE reference is released (previously it leaked forever)…
|
||||
expect((await blobMeta(oldHash))?.refCount).toBe(0)
|
||||
// …but the bytes survive (history-protected) and asOf still reads them.
|
||||
expect((await brain.vfs.readFile(p, { asOf: g1 })).toString()).toBe('old content A')
|
||||
})
|
||||
|
||||
it('rm keeps the bytes readable via asOf inside the window (no premature deletion)', async () => {
|
||||
const p = '/docs/doomed.txt'
|
||||
await brain.vfs.writeFile(p, 'still recoverable after rm')
|
||||
const g = brain.generationStore.generation()
|
||||
const hash = (await brain.vfs.history(p))[0].hash
|
||||
|
||||
await brain.vfs.unlink(p)
|
||||
|
||||
// Live path is gone…
|
||||
await expect(brain.vfs.readFile(p)).rejects.toThrow()
|
||||
// …the bytes are not: zero live refs, history-protected.
|
||||
expect((await blobMeta(hash))?.refCount).toBe(0)
|
||||
expect((await brain['storage'].blobStorage.read(hash)).toString()).toBe(
|
||||
'still recoverable after rm'
|
||||
)
|
||||
// (Path-level asOf resolution of a DELETED path is future work — the
|
||||
// bytes + entity history are intact; recovery reads go through the hash
|
||||
// or a pre-delete generation view.)
|
||||
void g
|
||||
})
|
||||
|
||||
it('compaction is the ONE reclamation point: past-window bytes are reclaimed, in-window preserved', async () => {
|
||||
const p = '/pages/compact.json'
|
||||
await brain.vfs.writeFile(p, 'version ONE')
|
||||
const v1Hash = (await brain.vfs.history(p))[0].hash
|
||||
await brain.vfs.writeFile(p, 'version TWO')
|
||||
await brain.vfs.writeFile(p, 'version THREE')
|
||||
|
||||
// Sanity: v1's bytes exist (history-protected, live-released).
|
||||
expect(await blobMeta(v1Hash)).toBeDefined()
|
||||
expect((await blobMeta(v1Hash))?.refCount).toBe(0)
|
||||
|
||||
// Compact everything reclaimable (retain nothing beyond the live state).
|
||||
await brain.compactHistory({ maxGenerations: 0 })
|
||||
|
||||
// v1's bytes are physically GONE — zero live, zero history.
|
||||
expect(await blobMeta(v1Hash)).toBeUndefined()
|
||||
// The live version is untouched.
|
||||
expect((await brain.vfs.readFile(p)).toString()).toBe('version THREE')
|
||||
})
|
||||
|
||||
it('cross-file dedup: a shared hash survives while ANY in-window generation references it', async () => {
|
||||
const shared = 'identical bytes shared across files and time'
|
||||
const pA = '/a.txt'
|
||||
const pB = '/b.txt'
|
||||
|
||||
// A holds the shared content, then moves off it (A's history references it).
|
||||
await brain.vfs.writeFile(pA, shared)
|
||||
await brain.vfs.writeFile(pA, 'A moved on')
|
||||
// B now holds the shared content live, then is removed (B's remove-gen
|
||||
// before-image references it too).
|
||||
await brain.vfs.writeFile(pB, shared)
|
||||
const gBLive = brain.generationStore.generation()
|
||||
const sharedHash = (await brain.vfs.history(pA))[0].hash
|
||||
await brain.vfs.unlink(pB)
|
||||
|
||||
// Live refs are zero (A moved off; B removed) — bytes survive on history.
|
||||
expect((await blobMeta(sharedHash))?.refCount).toBe(0)
|
||||
expect((await brain['storage'].blobStorage.read(sharedHash)).toString()).toBe(shared)
|
||||
|
||||
// B's content at its live generation is still exactly readable… via asOf
|
||||
// on A's early version too (same bytes, same blob).
|
||||
const versionsA = await brain.vfs.history(pA)
|
||||
const aV1 = versionsA[0]
|
||||
expect((await brain.vfs.readFile(pA, { asOf: aV1.generation })).toString()).toBe(shared)
|
||||
void gBLive
|
||||
|
||||
// Full compaction drops the last history references → bytes reclaimed.
|
||||
await brain.compactHistory({ maxGenerations: 0 })
|
||||
expect(await blobMeta(sharedHash)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('overwrite refreshes the entity data/embedding text (the stale-.data defect)', async () => {
|
||||
const p = '/pages/data-fresh.txt'
|
||||
await brain.vfs.writeFile(p, 'first words')
|
||||
await brain.vfs.writeFile(p, 'second words entirely')
|
||||
|
||||
const entityId = await brain.vfs['pathResolver'].resolve(p)
|
||||
const entity = await brain.get(entityId)
|
||||
expect(entity.data).toBe('second words entirely')
|
||||
})
|
||||
|
||||
it('the scrub recounts history references exactly from the generation records', async () => {
|
||||
const p = '/pages/scrubbed.md'
|
||||
await brain.vfs.writeFile(p, 'scrub v1')
|
||||
await brain.vfs.writeFile(p, 'scrub v2')
|
||||
const v1Hash = (await brain.vfs.history(p))[0].hash
|
||||
await brain.flush()
|
||||
|
||||
const storage = brain['storage']
|
||||
const before = (await blobMeta(v1Hash))?.historyRefCount ?? 0
|
||||
expect(before).toBeGreaterThan(0)
|
||||
|
||||
// Corrupt the counter (simulates a legacy store / crash drift), then scrub.
|
||||
await storage.blobStorage.setHistoryRefCount(v1Hash, 0)
|
||||
const result = await storage.scrubBlobHistoryRefCounts()
|
||||
expect(result.records).toBeGreaterThan(0)
|
||||
expect((await blobMeta(v1Hash))?.historyRefCount).toBe(before)
|
||||
})
|
||||
})
|
||||
|
|
@ -258,21 +258,31 @@ describe('BlobStorage', () => {
|
|||
expect(metadata?.refCount).toBe(2)
|
||||
})
|
||||
|
||||
it('should only delete when refCount reaches 0', async () => {
|
||||
it('release() drops live references; bytes are reclaimed ONLY by reclaimIfUnreferenced at zero-zero', async () => {
|
||||
const data = Buffer.from('test')
|
||||
|
||||
// Write twice (refCount = 2)
|
||||
// Write twice (live refCount = 2)
|
||||
const hash = await blobStorage.write(data)
|
||||
await blobStorage.write(data)
|
||||
|
||||
// Delete once (refCount = 1, blob still exists)
|
||||
await blobStorage.delete(hash)
|
||||
|
||||
// Release once (refCount = 1, blob still exists)
|
||||
await blobStorage.release(hash)
|
||||
expect(await blobStorage.has(hash)).toBe(true)
|
||||
|
||||
// Delete again (refCount = 0, blob deleted)
|
||||
await blobStorage.delete(hash)
|
||||
// Release again (refCount = 0) — bytes STILL exist: content is
|
||||
// immutable under the temporal model; only history compaction reclaims.
|
||||
await blobStorage.release(hash)
|
||||
expect(await blobStorage.has(hash)).toBe(true)
|
||||
expect((await blobStorage.getMetadata(hash))?.refCount).toBe(0)
|
||||
|
||||
// A history reference blocks reclamation even at live-zero.
|
||||
await blobStorage.recordHistoryReference(hash)
|
||||
expect(await blobStorage.reclaimIfUnreferenced(hash)).toBe(false)
|
||||
expect(await blobStorage.has(hash)).toBe(true)
|
||||
|
||||
// Drop the history reference → zero-zero → reclaim succeeds.
|
||||
await blobStorage.releaseHistoryReference(hash)
|
||||
expect(await blobStorage.reclaimIfUnreferenced(hash)).toBe(true)
|
||||
expect(await blobStorage.has(hash)).toBe(false)
|
||||
})
|
||||
})
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue