diff --git a/docs/guides/upgrading-7-to-8.md b/docs/guides/upgrading-7-to-8.md new file mode 100644 index 00000000..a3c64fb9 --- /dev/null +++ b/docs/guides/upgrading-7-to-8.md @@ -0,0 +1,186 @@ +--- +title: Upgrading from 7.x to 8.0 +slug: guides/upgrading-7-to-8 +public: true +category: guides +template: guide +order: 10 +description: What the one-time 7→8 on-disk migration does, how 8.0 automatically recovers Virtual Filesystem content that older layouts stored in the removed copy-on-write area, and how to verify (or force) that recovery. +next: + - guides/storage-adapters + - guides/inspection +--- + +# Upgrading from 7.x to 8.0 + +Opening a 7.x on-disk store with Brainy 8.0 runs a **one-time, in-place layout +migration** the first time the store is opened. It is automatic (`autoMigrate` +defaults to `true`), it runs once, and it stamps a marker so every later open is +a no-op. + +Almost everything about the upgrade is transparent: your entities, relationships, +metadata, and indexes migrate and rebuild without any action on your part. This +guide covers the **one case that needs attention** — Virtual Filesystem (VFS) +content — and how 8.0 recovers it for you. + +## TL;DR + +- **Just upgrade to `@soulcraft/brainy@8.0.12` (or later) and open the store.** + If a previous upgrade left VFS content stranded, 8.0.12 **heals it on open**, + with no operator action. +- Want to force or script it? Call **`await brain.vfs.adoptOrphanedBlobs()`**. +- The recovery is **non-destructive and idempotent** — it copies, never moves, + and running it twice is a no-op. + +## What the migration does + +The 7.x on-disk layout stored entities under a per-branch path +(`branches//entities/...`). 8.0 uses a flat layout (`entities/...`). On +first open, Brainy: + +1. Collapses `branches//entities/*` into the flat `entities/*` layout. +2. Rebuilds the derived indexes (vector, graph, metadata) and count rollups from + the canonical entities. +3. Stamps `_system/migration-layout.json` so re-opening is a no-op. +4. Takes an automatic **pre-upgrade backup** of the directory before it starts, + and removes it once the upgrade is verified complete (see + [The safety net](#the-safety-net-pre-upgrade-backup) below). + +The log line to expect (once per store): + +``` +[brainy] Migrating a 7.x branch layout (branches/main) to the 8.0 flat layout + in place — 13175 entity files. This runs once; back up the directory first if + you need a rollback (8.0 does not keep the old layout). +``` + +## The one thing that needs recovery: VFS content + +If your application uses the Virtual Filesystem (VFS) to store file content (for +example, a CMS that keeps page documents at paths like +`/pages/homepage/page.json`), that content is held as **content blobs**. + +7.x kept those blobs in the branch system's **copy-on-write area** (`_cow/`). +8.0 removed the branch/copy-on-write system, and its content blobs live in the +content-addressed store (`_cas/`). The layout migration moves entities — it does +**not** move the VFS content blobs. Left alone, a 7.x store's VFS blobs would +remain in `_cow/`, and a read of one would throw: + +``` +VFS: Cannot read blob for /pages/homepage/page.json: + Blob metadata not found: 6b87cfb71ad2f04602a5c157214dc42000... +``` + +### 8.0.12 recovers them automatically + +Brainy 8.0.12 adds an **on-open recovery pass** that runs right after the layout +migration. It scans `_cow/`, and for every content blob whose `blob:` + +`blob-meta:` pair is not already in `_cas/`, it copies **both** across into the +8.0 store. It: + +- **Heals a fresh 7→8 upgrade** (where `_cow/` still holds the blobs), **and** +- **Heals a store already upgraded by an earlier 8.0.x** that stranded them — + the recovery is gated on the presence of `_cow/` and its own marker, not on + the layout-migration marker, so an already-migrated store is still healed. +- Is a **cheap no-op** on a native-8.0 or fresh store (there is no `_cow/`), and + on a store already healed (a marker records completion so later opens skip the + scan). + +So the operator action for a stranded store is simply: **upgrade to 8.0.12 and +open it.** + +```ts +import { Brainy } from '@soulcraft/brainy' + +// Opening the store is all that is required — recovery runs during init(). +const brain = new Brainy({ storage: { type: 'filesystem', path: '/data/my-store' } }) +await brain.init() + +// The previously-failing read now succeeds. +const page = await brain.vfs.readFile('/pages/homepage/page.json') +``` + +On a store that needed recovery you will see: + +``` +[brainy] Recovered 337 VFS content blob(s) stranded by a 7→8 upgrade + (adopted _cow/ → _cas/ in place). +``` + +### Forcing recovery explicitly + +If you would rather run the recovery deliberately (for example, in an upgrade +script that asserts a clean result before flipping traffic), call it directly: + +```ts +const result = await brain.vfs.adoptOrphanedBlobs() +// → { cowBlobs, adopted, alreadyPresent, incomplete } +console.log(`adopted ${result.adopted}, already present ${result.alreadyPresent}`) +if (result.incomplete > 0) { + // One or more _cow/ blobs are missing their bytes or metadata — investigate + // _cow/ before discarding your own backup. See "The safety net" below. +} +``` + +`adoptOrphanedBlobs()` self-initializes the brain, so it is safe to call +immediately after construction. It returns: + +| Field | Meaning | +| ---------------- | ---------------------------------------------------------------- | +| `cowBlobs` | Distinct content-blob hashes found in `_cow/`. | +| `adopted` | Newly copied into `_cas/` on this call. | +| `alreadyPresent` | Already in `_cas/` (a prior open or run adopted them). | +| `incomplete` | A `_cow/` blob missing its bytes *or* its metadata — **skipped** rather than half-adopted. | + +## Verifying recovery + +After opening under 8.0.12, confirm a previously-failing path reads: + +```ts +const content = await brain.vfs.readFile('/pages/homepage/page.json') +console.log(content.toString().slice(0, 80)) +``` + +If you scripted it with `adoptOrphanedBlobs()`, a clean result is +`incomplete === 0` and `adopted + alreadyPresent === cowBlobs`. + +## The safety net: pre-upgrade backup + +Brainy takes an automatic pre-upgrade backup and removes it once the upgrade is +**verified complete**. In 8.0.12 that verification includes VFS content: if the +blob recovery reports `incomplete > 0`, the completion marker is **not** stamped +(the next open retries) and the **pre-upgrade backup is retained** so you still +have a rollback while blobs remain unaccounted for. You will see: + +``` +[brainy] VFS blob recovery adopted N blob(s) but M orphaned _cow/ blob(s) are + missing their bytes or metadata and were left in place. The pre-upgrade backup + is being retained; inspect _cow/ before discarding it. +``` + +The recovery never deletes anything from `_cow/`, so the original blobs stay in +place for inspection or a manual rollback regardless. + +## Who is affected + +**Any 7.x store that used the VFS to store file content** (so it has a `_cow/` +area) is a candidate for stranded blobs on a 7→8 upgrade. Stores that never used +the VFS have no `_cow/` content blobs and are unaffected. + +Because the recovery is **gated on the presence of `_cow/`**, it is +self-selecting and safe to roll out everywhere: + +- On a store with stranded blobs → it adopts them. +- On a native-8.0, fresh, or non-VFS store → it is a no-op on the existence + check. + +There is no configuration to set and nothing to opt into. Upgrading to 8.0.12 +and opening each store is sufficient. + +## Rollback + +The recovery is copy-only, so no rollback of the recovery itself is ever needed. +If you need to roll back the **whole** 7→8 upgrade, restore the directory from +your pre-upgrade backup (retained automatically while recovery is incomplete, or +your own snapshot) and pin `@soulcraft/brainy@7.x`. 8.0 does not keep the old +branch layout in place, so a directory-level restore is the rollback path. diff --git a/src/brainy.ts b/src/brainy.ts index a1fb9eab..148b4639 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -32,7 +32,7 @@ import { getBrainyVersion } from './utils/index.js' import { embeddingManager } from './embeddings/EmbeddingManager.js' -import { matchesMetadataFilter } from './utils/metadataFilter.js' +import { matchesMetadataFilter, validateWhereFilter } from './utils/metadataFilter.js' import { NaturalLanguageProcessor } from './neural/naturalLanguageProcessor.js' import { NeuralEntityExtractor, ExtractedEntity } from './neural/entityExtractor.js' import { TripleIntelligenceSystem } from './triple/TripleIntelligenceSystem.js' @@ -453,6 +453,13 @@ export class Brainy implements BrainyInterface { * `migrationBackup: false`), or `null` if none. Removed once the 7.x→8.0 * upgrade verifies + stamps; retained on failure. See {@link createMigrationBackupIfNeeded}. */ private _migrationBackupPath: string | null = null + + /** Set when the on-open VFS-blob adoption left one or more `_cow/` blobs it + * could not fully adopt (bytes or metadata missing). While true, the + * pre-upgrade backup is NOT auto-removed — the upgrade is not verifiably + * complete for the VFS. See {@link autoAdoptLegacyVfsBlobsIfNeeded}. */ + private _vfsBlobAdoptionIncomplete = false + /** * 8.0 ⇄ native-provider version handshake — the on-disk {@link BrainFormat} * marker (`_system/brain-format.json`) as read during the store-open phase, @@ -1028,6 +1035,14 @@ export class Brainy implements BrainyInterface { } } + // Recover VFS content blobs a 7→8 upgrade left stranded in the removed + // branch system's copy-on-write area (`_cow/`). Runs BEFORE the rebuild + // (whose success stamp removes the pre-upgrade backup) so the backup still + // covers this step, and works on the raw object primitives (no blob store + // needed yet). A cheap no-op on native-8.0 / fresh brains and on a brain + // already healed. See adoptLegacyCowBlobs. + await this.autoAdoptLegacyVfsBlobsIfNeeded() + // Rebuild indexes if needed for existing data await this.rebuildIndexesIfNeeded() @@ -5228,6 +5243,15 @@ export class Brainy implements BrainyInterface { const hasFilterCriteria = params.where || params.type || params.subtype || params.service const hasGraphCriteria = params.connected + // Validate the raw where clause up front: an unrecognized operator (a typo + // like `notIn`, or any non-operator key on a field's object value) throws a + // typed BrainyError('INVALID_QUERY') instead of silently matching nothing. + // Done before the index/vector/graph paths so it fires even on an empty + // result set, and before brainy injects its own internal filter fields. + if (params.where) { + validateWhereFilter(params.where) + } + // Metadata cold-read guard: before trusting a filter result, verify the // field index actually serves a known persisted value (one-shot per brain). // A cold native index that has not loaded its `where` postings self-heals @@ -13012,6 +13036,85 @@ export class Brainy implements BrainyInterface { } } + /** + * @description On-open recovery for VFS content blobs a 7→8 upgrade left in the + * removed branch system's copy-on-write area (`_cow/`). If a `_cow/` area is + * present and a completed adoption has not already been recorded, adopt every + * orphaned blob into the 8.0 content-addressed store (`_cas/`) and stamp a + * marker so later opens skip the scan. Because it runs on every open right + * after {@link legacyLayoutMigrationPhase}, it heals BOTH a fresh 7→8 migration + * (where `_cow/` still holds the blobs) AND a brain already migrated by an + * earlier 8.0.x that stranded them — no operator action needed. + * + * Filesystem-only; a native-8.0 / fresh brain has no `_cow/` and returns on the + * cheap existence check. Non-destructive and idempotent (see + * {@link BaseStorage.adoptLegacyCowBlobs}). If any blob cannot be fully adopted + * (`incomplete > 0`), the marker is NOT stamped (the next open retries) and + * {@link _vfsBlobAdoptionIncomplete} is set so the pre-upgrade backup is + * retained — the upgrade is not verifiably complete for the VFS. + */ + private async autoAdoptLegacyVfsBlobsIfNeeded(): Promise { + if (this.getStorageType() !== 'filesystem') return + // Cheapest gate: no copy-on-write area → a native-8.0 / fresh brain, nothing + // a 7.x brain could have stranded here. + try { + const root = resolveFilesystemRoot( + (this.config.storage || {}) as StorageOptions & Record + ) + if (!fs.existsSync(`${root}/_cow`)) return + } catch { + return + } + + const storage = this.storage as unknown as { + adoptLegacyCowBlobs?: () => Promise<{ + cowBlobs: number + adopted: number + alreadyPresent: number + incomplete: number + }> + readRawObject?: (p: string) => Promise + writeRawObject?: (p: string, o: unknown) => Promise + } + if (typeof storage.adoptLegacyCowBlobs !== 'function') return + + const MARKER = '_system/vfs-blob-adoption.json' + try { + const done = (await storage.readRawObject?.(MARKER)) as { complete?: boolean } | null + if (done && done.complete) return // already fully adopted on a prior open + } catch { + // no marker yet — proceed + } + + const result = await storage.adoptLegacyCowBlobs() + + if (result.incomplete > 0) { + // Partial: retry on future opens, and hold the pre-upgrade backup. + this._vfsBlobAdoptionIncomplete = true + if (!this.config.silent) { + console.error( + `[brainy] VFS blob recovery adopted ${result.adopted} blob(s) but ${result.incomplete} ` + + `orphaned _cow/ blob(s) are missing their bytes or metadata and were left in place. ` + + `The pre-upgrade backup is being retained; inspect _cow/ before discarding it.` + ) + } + return + } + + // Clean sweep — record it so later opens skip the _cow/ scan entirely. + try { + await storage.writeRawObject?.(MARKER, { complete: true, adopted: result.adopted }) + } catch { + // marker write is best-effort; adoption already succeeded + } + if (result.adopted > 0 && !this.config.silent) { + console.warn( + `[brainy] Recovered ${result.adopted} VFS content blob(s) stranded by a 7→8 upgrade ` + + `(adopted _cow/ → _cas/ in place).` + ) + } + } + private async setupStorage(): Promise { // If the caller passed a pre-constructed storage adapter (e.g. // `storage: new MemoryStorage()`, or the historical materializer's @@ -13846,6 +13949,20 @@ export class Brainy implements BrainyInterface { */ private async removeMigrationBackupSafe(): Promise { if (!this._migrationBackupPath) return + // Backup-parity gate: the index rebuild "success" that calls this does NOT + // certify the VFS. If the on-open blob adoption could not fully bridge the + // 7.x `_cow/` blobs, the upgrade is not verifiably complete for VFS content + // — retain the backup so the operator can recover from it, and log why. + if (this._vfsBlobAdoptionIncomplete) { + if (!this.config.silent) { + console.warn( + `[brainy] Retaining the pre-upgrade backup at ${this._migrationBackupPath}: ` + + `some VFS content blobs could not be adopted from _cow/ (see the earlier warning). ` + + `Resolve those before discarding the backup.` + ) + } + return + } const backupPath = this._migrationBackupPath this._migrationBackupPath = null const storage = this.storage as StorageAdapter & { diff --git a/src/storage/baseStorage.ts b/src/storage/baseStorage.ts index 22149afb..a678a000 100644 --- a/src/storage/baseStorage.ts +++ b/src/storage/baseStorage.ts @@ -680,6 +680,91 @@ export abstract class BaseStorage extends BaseStorageAdapter { this.blobStorage = new BlobStorage(casAdapter) } + /** + * @description Adopt orphaned 7.x copy-on-write VFS content blobs into the 8.0 + * content-addressed store. 7.x kept VFS blobs (`blob:` bytes + + * `blob-meta:` JSON) under the copy-on-write area (`_cow/`) of the + * branch/versioning system that 8.0 removed. The 7→8 layout migration + * collapses entity files but historically did NOT bridge these blobs, so a VFS + * read of a still-in-`_cow/` blob throws "Blob metadata not found" and every + * page backed by it 500s. + * + * This scans `_cow/` and, for every blob whose `blob:`/`blob-meta:` pair is not + * already present in the 8.0 store (`_cas/`), copies BOTH objects across + * verbatim. Copying preserves the exact content bytes (so the content hash and + * every VFS reference still resolve) and the exact metadata (so the 7.x + * refCount is retained) — the only first-party, index-consistent recovery. + * Hand-moving files at the OS level bypasses the paired-metadata contract and + * is unsafe; this goes through the adapter's object primitives. + * + * Idempotent and non-destructive: a pair already in `_cas/` is left untouched + * (counted `alreadyPresent`); the `_cow/` originals are never deleted, so a + * re-run — or a rollback — is always possible. A no-op on memory storage and + * on any brain without a `_cow/` area (returns zeros). Bytes are written before + * metadata so an interruption can only leave the pre-adoption "no metadata" + * state (retryable), never a metadata-without-bytes blob. + * + * @returns `cowBlobs` (distinct blob hashes found in `_cow/`), `adopted` + * (newly copied into `_cas/`), `alreadyPresent` (already in `_cas/`), + * `incomplete` (a `_cow/` blob missing its bytes or its metadata — skipped + * and reported rather than half-adopted). + */ + public async adoptLegacyCowBlobs(): Promise<{ + cowBlobs: number + adopted: number + alreadyPresent: number + incomplete: number + }> { + let cowPaths: string[] + try { + cowPaths = await this.listObjectsUnderPath('_cow/') + } catch { + // No `_cow/` area (fresh/native-8.0 brain, or a non-listing backend). + return { cowBlobs: 0, adopted: 0, alreadyPresent: 0, incomplete: 0 } + } + + // Distinct content-blob hashes from `_cow/blob:` keys. Non-blob `_cow/` + // entries (7.x branch COW state) are ignored — only VFS content is adopted. + const hashes = new Set() + for (const p of cowPaths) { + const key = p.replace(/^_cow\//, '') + const m = /^blob:(.+)$/.exec(key) + if (m) hashes.add(m[1]) + } + + let adopted = 0 + let alreadyPresent = 0 + let incomplete = 0 + for (const hash of hashes) { + // A blob counts as present only when BOTH its bytes and its metadata + // already live in `_cas/`. A half-adopted blob (bytes without meta — the + // exact "Blob metadata not found" state) is re-adopted. + const casBlob = await this.readObjectFromPath(`_cas/blob:${hash}`) + const casMeta = await this.readObjectFromPath(`_cas/blob-meta:${hash}`) + if (casBlob !== null && casMeta !== null) { + alreadyPresent++ + continue + } + + const cowBlob = await this.readObjectFromPath(`_cow/blob:${hash}`) + const cowMeta = await this.readObjectFromPath(`_cow/blob-meta:${hash}`) + if (cowBlob === null || cowMeta === null) { + // Can't register a blob the store can't fully describe — report it so an + // operator investigates rather than silently half-adopting. + incomplete++ + continue + } + + // Bytes first, then metadata: a VFS read checks metadata before bytes, so + // metadata's presence must imply the bytes are already there. + await this.writeObjectToPath(`_cas/blob:${hash}`, cowBlob) + await this.writeObjectToPath(`_cas/blob-meta:${hash}`, cowMeta) + adopted++ + } + + return { cowBlobs: hashes.size, adopted, alreadyPresent, incomplete } + } + /** * @description Write a canonical object (write-cache coherent). The object * lands in the write-through cache before the asynchronous write starts, so diff --git a/src/vfs/VirtualFileSystem.ts b/src/vfs/VirtualFileSystem.ts index 9dd5d39b..bbf8328e 100644 --- a/src/vfs/VirtualFileSystem.ts +++ b/src/vfs/VirtualFileSystem.ts @@ -352,6 +352,52 @@ export class VirtualFileSystem implements IVirtualFileSystem { } } + /** + * @description Recover VFS content blobs orphaned by a 7→8 upgrade. 7.x stored + * VFS blobs under the copy-on-write area (`_cow/`) of the branch/versioning + * system that 8.0 removed; a brain upgraded before the migration adopted them + * reads a stranded blob and throws "Blob metadata not found" (every page + * backed by it 500s). This adopts every orphaned `_cow/` blob into the 8.0 + * content-addressed store (`_cas/`) IN PLACE — no snapshot restore — then + * clears the VFS caches so the recovered content reads live immediately. + * + * Idempotent and non-destructive: blobs already adopted are skipped and the + * `_cow/` originals are never deleted (a re-run or rollback stays possible). + * Safe to run on a healthy or native-8.0 brain (returns zeros). Delegates to + * {@link BaseStorage.adoptLegacyCowBlobs}. + * + * @returns `{ cowBlobs, adopted, alreadyPresent, incomplete }` counts. + * @example + * const brain = new Brainy({ storage: { type: 'filesystem', path: '/data/brain' } }) + * await brain.init() + * const r = await brain.vfs.adoptOrphanedBlobs() + * console.log(`Adopted ${r.adopted} stranded VFS blobs; ${r.alreadyPresent} already present.`) + */ + async adoptOrphanedBlobs(): Promise<{ + cowBlobs: number + adopted: number + alreadyPresent: number + incomplete: number + }> { + // The recovery scan works on raw storage objects, so it only needs the + // brain's storage wired up — not the VFS's own init. brain.init() is + // idempotent (a no-op on an already-open brain) and, on a cold brain, + // sets up storage AND runs the on-open auto-adoption itself; the explicit + // scan below is then the idempotent "force re-scan" equivalent. + await this.brain.init() + const storage = this.brain['storage'] as unknown as + | { adoptLegacyCowBlobs?: () => Promise<{ cowBlobs: number; adopted: number; alreadyPresent: number; incomplete: number }> } + | undefined + if (!storage || typeof storage.adoptLegacyCowBlobs !== 'function') { + return { cowBlobs: 0, adopted: 0, alreadyPresent: 0, incomplete: 0 } + } + const result = await storage.adoptLegacyCowBlobs() + // Drop any cached content/stat so a re-read hits the freshly adopted blobs. + this.contentCache.clear() + this.statCache.clear() + return result + } + // ============= File Operations ============= /** diff --git a/tests/integration/vfs-cow-blob-adoption.test.ts b/tests/integration/vfs-cow-blob-adoption.test.ts new file mode 100644 index 00000000..b7dec300 --- /dev/null +++ b/tests/integration/vfs-cow-blob-adoption.test.ts @@ -0,0 +1,149 @@ +/** + * @module tests/integration/vfs-cow-blob-adoption + * @description P0 recovery: a 7→8 upgrade left VFS content blobs in the removed + * branch system's copy-on-write area (`_cow/`) instead of the 8.0 + * content-addressed store (`_cas/`), so a VFS read of a stranded blob throws + * "Blob metadata not found" and every page backed by it 500s. + * + * Proves the cure end to end: + * - `storage.adoptLegacyCowBlobs()` copies each orphaned `_cow/` blob (bytes + + * metadata) into `_cas/` verbatim, idempotently, non-destructively, and + * reports an `incomplete` blob (bytes without metadata) rather than + * half-adopting it. + * - a real brain that upgraded into this state SELF-HEALS on its next open + * (auto-adoption) — the stranded VFS file reads correctly again with no + * operator action. + * - `brain.vfs.adoptOrphanedBlobs()` is the explicit equivalent. + */ +import { describe, it, expect, afterEach } from 'vitest' +import * as fs from 'node:fs' +import * as os from 'node:os' +import * as path from 'node:path' +import { Brainy } from '../../src/brainy.js' +import { FileSystemStorage } from '../../src/storage/adapters/fileSystemStorage.js' + +const tmpDirs: string[] = [] +function mkTmp(): string { + const d = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-cow-adopt-')) + tmpDirs.push(d) + return d +} + +/** Release a raw FileSystemStorage: flush, stop its timer, drop the writer lock + * so a subsequent open of the same directory is not blocked. */ +async function teardownRawStorage(s: any): Promise { + try { await s.flush?.() } catch { /* best effort */ } + try { s.stopFlushRequestWatcher?.() } catch { /* best effort */ } + try { await s.releaseWriterLock?.() } catch { /* best effort */ } +} +afterEach(() => { + for (const d of tmpDirs.splice(0)) fs.rmSync(d, { recursive: true, force: true }) +}) + +/** Move every `_cas/blob*` object into `_cow/` and delete the `_cas/` copy — + * reproducing the exact on-disk state a 7.x brain leaves after the 8.0 layout + * migration collapses entities but does not adopt the copy-on-write blobs. */ +async function strandCasBlobsIntoCow(dir: string): Promise { + const s: any = new FileSystemStorage(dir) + await s.init() + const casKeys = (await s.listRawObjects('_cas/')).filter((k: string) => + /(^|\/)(blob:|blob-meta:)/.test(k) + ) + let moved = 0 + for (const full of casKeys) { + const key = full.replace(/^_cas\//, '') + const obj = await s.readRawObject(`_cas/${key}`) + if (obj === null) continue + await s.writeRawObject(`_cow/${key}`, obj) + await s.deleteRawObject(`_cas/${key}`) + moved++ + } + await teardownRawStorage(s) + return moved +} + +describe('VFS _cow/ blob adoption (P0 7→8 recovery)', () => { + it('adoptLegacyCowBlobs copies bytes + metadata into _cas/, verbatim, idempotent, non-destructive', async () => { + const dir = mkTmp() + const storage: any = new FileSystemStorage(dir) + await storage.init() + + const hash = 'a'.repeat(64) + const bytes = Buffer.from('the homepage payload') + // 7.x layout: the pair lives ONLY under _cow/ (blob-meta carries the refCount). + await storage.writeRawObject(`_cow/blob:${hash}`, { _binary: true, data: bytes.toString('base64') }) + await storage.writeRawObject(`_cow/blob-meta:${hash}`, { hash, size: bytes.length, refCount: 3, mimeType: 'application/json' }) + + // Pre-adoption: _cas/ has neither → this is the "Blob metadata not found" state. + expect(await storage.readRawObject(`_cas/blob-meta:${hash}`)).toBeNull() + + const r = await storage.adoptLegacyCowBlobs() + expect(r).toMatchObject({ cowBlobs: 1, adopted: 1, alreadyPresent: 0, incomplete: 0 }) + + // Post: _cas/ has both, metadata verbatim (refCount preserved), bytes intact. + const casMeta = await storage.readRawObject(`_cas/blob-meta:${hash}`) + expect(casMeta).toMatchObject({ hash, refCount: 3, size: bytes.length }) + const casBlob = await storage.readRawObject(`_cas/blob:${hash}`) + expect(Buffer.from(casBlob.data, 'base64').toString()).toBe('the homepage payload') + + // Idempotent: a second run adopts nothing, counts the pair present. + expect(await storage.adoptLegacyCowBlobs()).toMatchObject({ adopted: 0, alreadyPresent: 1 }) + // Non-destructive: the _cow/ original is still there (rollback stays possible). + expect(await storage.readRawObject(`_cow/blob:${hash}`)).not.toBeNull() + + // Incomplete: bytes without metadata are reported, not half-adopted. + await storage.writeRawObject(`_cow/blob:${'b'.repeat(64)}`, { _binary: true, data: 'x' }) + const r3 = await storage.adoptLegacyCowBlobs() + expect(r3.incomplete).toBe(1) + expect(await storage.readRawObject(`_cas/blob:${'b'.repeat(64)}`)).toBeNull() // not adopted + await teardownRawStorage(storage) + }) + + it('a brain that upgraded into the stranded state self-heals on next open — the VFS page reads again', async () => { + const dir = mkTmp() + + // 1. Author a VFS page (its content blob lands in _cas/). + let brain: any = new Brainy({ storage: { type: 'filesystem', path: dir }, plugins: [], silent: true }) + await brain.init() + await brain.vfs.writeFile('/pages/homepage/page.json', '{"title":"Homepage"}') + expect((await brain.vfs.readFile('/pages/homepage/page.json')).toString()).toContain('Homepage') + await brain.close() + + // 2. Reproduce the post-7→8 stranding: the blob is now only under _cow/. + const moved = await strandCasBlobsIntoCow(dir) + expect(moved).toBeGreaterThan(0) + + // 3. Reopen: the on-open auto-adoption heals it with no operator action. + brain = new Brainy({ storage: { type: 'filesystem', path: dir }, plugins: [], silent: true }) + await brain.init() + const healed = await brain.vfs.readFile('/pages/homepage/page.json') + expect(healed.toString()).toContain('Homepage') + + // A subsequent open records the adoption marker and does not re-scan-heal. + await brain.close() + brain = new Brainy({ storage: { type: 'filesystem', path: dir }, plugins: [], silent: true }) + await brain.init() + expect((await brain.vfs.readFile('/pages/homepage/page.json')).toString()).toContain('Homepage') + await brain.close() + }) + + it('brain.vfs.adoptOrphanedBlobs() is the explicit recovery equivalent', async () => { + const dir = mkTmp() + const body = JSON.stringify({ page: 'about', body: 'x'.repeat(400) }) + let brain: any = new Brainy({ storage: { type: 'filesystem', path: dir }, plugins: [], silent: true }) + await brain.init() + await brain.vfs.writeFile('/about.json', body) + await brain.close() + await strandCasBlobsIntoCow(dir) + + // The explicit API self-initializes (like other VFS methods); init's + // auto-heal adopts, so the explicit call sees the pair present. Either way + // it reports the _cow/ blobs it accounted for, and the read is healed. + brain = new Brainy({ storage: { type: 'filesystem', path: dir }, plugins: [], silent: true }) + const r = await brain.vfs.adoptOrphanedBlobs() + expect(r.cowBlobs).toBeGreaterThan(0) + expect(r.adopted + r.alreadyPresent).toBe(r.cowBlobs) + expect((await brain.vfs.readFile('/about.json')).toString()).toContain('about') + await brain.close() + }) +})