David Snelling c0f6ccd958 fix: recover VFS content blobs stranded by a 7→8 upgrade, in place on open

The 7.x branch system stored Virtual Filesystem content as blobs in its
copy-on-write area (`_cow/`). 8.0 removed that system and stores content blobs
in the content-addressed store (`_cas/`), but the one-time layout migration only
moves entities — it never adopted the `_cow/` content blobs. A store that used
the VFS was left with every VFS-backed read throwing "Blob metadata not found"
and its pages 500ing, with no first-party recovery and the pre-upgrade backup
already removed on entity-migration "success".

Add an on-open recovery pass (`autoAdoptLegacyVfsBlobsIfNeeded`) that runs right
after the layout migration and adopts every orphaned `_cow/` blob into `_cas/` —
copying both the bytes and the metadata via the raw object primitives,
idempotently and non-destructively (the `_cow/` originals are never deleted). It
is a separate phase, not folded into the migration, so it also heals a store
already upgraded by an earlier 8.0.x that stranded the blobs (whose layout-
migration marker is stamped): it is gated on the presence of `_cow/` and its own
`_system/vfs-blob-adoption.json` marker. Filesystem-only; native-8.0 and non-VFS
stores no-op on a cheap existence check.

- `BaseStorage.adoptLegacyCowBlobs()` — the scan/copy primitive, returning
  `{ cowBlobs, adopted, alreadyPresent, incomplete }`; skips (does not half-adopt)
  a blob missing its bytes or metadata.
- `brain.vfs.adoptOrphanedBlobs()` — the explicit force path; self-initializes.
- Backup retention: the automatic pre-upgrade backup is now kept if any blob
  can't be fully adopted (`incomplete > 0`), instead of being removed on
  entity-migration success alone — a blob-parity gate the migration signal
  cannot provide.

Adds an end-to-end integration test (storage-level adopt with idempotency and
incomplete handling; self-heal on reopen; the explicit API) and an upgrade guide.

2026-07-07 12:23:36 -07:00

7.7 KiB

Raw Blame History

title

slug

public

Upgrading from 7.x to 8.0

Opening a 7.x on-disk store with Brainy 8.0 runs a one-time, in-place layout migration the first time the store is opened. It is automatic (autoMigrate defaults to true), it runs once, and it stamps a marker so every later open is a no-op.

Almost everything about the upgrade is transparent: your entities, relationships, metadata, and indexes migrate and rebuild without any action on your part. This guide covers the one case that needs attention — Virtual Filesystem (VFS) content — and how 8.0 recovers it for you.

TL;DR

Just upgrade to @soulcraft/brainy@8.0.12 (or later) and open the store. If a previous upgrade left VFS content stranded, 8.0.12 heals it on open, with no operator action.
Want to force or script it? Call await brain.vfs.adoptOrphanedBlobs().
The recovery is non-destructive and idempotent — it copies, never moves, and running it twice is a no-op.

What the migration does

The 7.x on-disk layout stored entities under a per-branch path (branches/<head>/entities/...). 8.0 uses a flat layout (entities/...). On first open, Brainy:

Collapses branches/<head>/entities/* into the flat entities/* layout.
Rebuilds the derived indexes (vector, graph, metadata) and count rollups from the canonical entities.
Stamps _system/migration-layout.json so re-opening is a no-op.
Takes an automatic pre-upgrade backup of the directory before it starts, and removes it once the upgrade is verified complete (see The safety net below).

The log line to expect (once per store):

[brainy] Migrating a 7.x branch layout (branches/main) to the 8.0 flat layout
  in place — 13175 entity files. This runs once; back up the directory first if
  you need a rollback (8.0 does not keep the old layout).

The one thing that needs recovery: VFS content

If your application uses the Virtual Filesystem (VFS) to store file content (for example, a CMS that keeps page documents at paths like /pages/homepage/page.json), that content is held as content blobs.

7.x kept those blobs in the branch system's copy-on-write area (_cow/). 8.0 removed the branch/copy-on-write system, and its content blobs live in the content-addressed store (_cas/). The layout migration moves entities — it does not move the VFS content blobs. Left alone, a 7.x store's VFS blobs would remain in _cow/, and a read of one would throw:

VFS: Cannot read blob for /pages/homepage/page.json:
  Blob metadata not found: 6b87cfb71ad2f04602a5c157214dc42000...

8.0.12 recovers them automatically

Brainy 8.0.12 adds an on-open recovery pass that runs right after the layout migration. It scans _cow/, and for every content blob whose blob: + blob-meta: pair is not already in _cas/, it copies both across into the 8.0 store. It:

Heals a fresh 7→8 upgrade (where _cow/ still holds the blobs), and
Heals a store already upgraded by an earlier 8.0.x that stranded them — the recovery is gated on the presence of _cow/ and its own marker, not on the layout-migration marker, so an already-migrated store is still healed.
Is a cheap no-op on a native-8.0 or fresh store (there is no _cow/), and on a store already healed (a marker records completion so later opens skip the scan).

So the operator action for a stranded store is simply: upgrade to 8.0.12 and open it.

import { Brainy } from '@soulcraft/brainy'

// Opening the store is all that is required — recovery runs during init().
const brain = new Brainy({ storage: { type: 'filesystem', path: '/data/my-store' } })
await brain.init()

// The previously-failing read now succeeds.
const page = await brain.vfs.readFile('/pages/homepage/page.json')

On a store that needed recovery you will see:

[brainy] Recovered 337 VFS content blob(s) stranded by a 7→8 upgrade
  (adopted _cow/ → _cas/ in place).

Forcing recovery explicitly

If you would rather run the recovery deliberately (for example, in an upgrade script that asserts a clean result before flipping traffic), call it directly:

const result = await brain.vfs.adoptOrphanedBlobs()
// → { cowBlobs, adopted, alreadyPresent, incomplete }
console.log(`adopted ${result.adopted}, already present ${result.alreadyPresent}`)
if (result.incomplete > 0) {
  // One or more _cow/ blobs are missing their bytes or metadata — investigate
  // _cow/ before discarding your own backup. See "The safety net" below.
}

adoptOrphanedBlobs() self-initializes the brain, so it is safe to call immediately after construction. It returns:

Field	Meaning
`cowBlobs`	Distinct content-blob hashes found in `_cow/`.
`adopted`	Newly copied into `_cas/` on this call.
`alreadyPresent`	Already in `_cas/` (a prior open or run adopted them).
`incomplete`	A `_cow/` blob missing its bytes or its metadata — skipped rather than half-adopted.

Verifying recovery

After opening under 8.0.12, confirm a previously-failing path reads:

const content = await brain.vfs.readFile('/pages/homepage/page.json')
console.log(content.toString().slice(0, 80))

If you scripted it with adoptOrphanedBlobs(), a clean result is incomplete === 0 and adopted + alreadyPresent === cowBlobs.

The safety net: pre-upgrade backup

Brainy takes an automatic pre-upgrade backup and removes it once the upgrade is verified complete. In 8.0.12 that verification includes VFS content: if the blob recovery reports incomplete > 0, the completion marker is not stamped (the next open retries) and the pre-upgrade backup is retained so you still have a rollback while blobs remain unaccounted for. You will see:

[brainy] VFS blob recovery adopted N blob(s) but M orphaned _cow/ blob(s) are
  missing their bytes or metadata and were left in place. The pre-upgrade backup
  is being retained; inspect _cow/ before discarding it.

The recovery never deletes anything from _cow/, so the original blobs stay in place for inspection or a manual rollback regardless.

Who is affected

Any 7.x store that used the VFS to store file content (so it has a _cow/ area) is a candidate for stranded blobs on a 7→8 upgrade. Stores that never used the VFS have no _cow/ content blobs and are unaffected.

Because the recovery is gated on the presence of _cow/, it is self-selecting and safe to roll out everywhere:

On a store with stranded blobs → it adopts them.
On a native-8.0, fresh, or non-VFS store → it is a no-op on the existence check.

There is no configuration to set and nothing to opt into. Upgrading to 8.0.12 and opening each store is sufficient.

Rollback

The recovery is copy-only, so no rollback of the recovery itself is ever needed. If you need to roll back the whole 7→8 upgrade, restore the directory from your pre-upgrade backup (retained automatically while recovery is incomplete, or your own snapshot) and pin @soulcraft/brainy@7.x. 8.0 does not keep the old branch layout in place, so a directory-level restore is the rollback path.

7.7 KiB Raw Blame History