David Snelling
c0f6ccd958
The 7.x branch system stored Virtual Filesystem content as blobs in its
copy-on-write area (`_cow/`). 8.0 removed that system and stores content blobs
in the content-addressed store (`_cas/`), but the one-time layout migration only
moves entities — it never adopted the `_cow/` content blobs. A store that used
the VFS was left with every VFS-backed read throwing "Blob metadata not found"
and its pages 500ing, with no first-party recovery and the pre-upgrade backup
already removed on entity-migration "success".
Add an on-open recovery pass (`autoAdoptLegacyVfsBlobsIfNeeded`) that runs right
after the layout migration and adopts every orphaned `_cow/` blob into `_cas/` —
copying both the bytes and the metadata via the raw object primitives,
idempotently and non-destructively (the `_cow/` originals are never deleted). It
is a separate phase, not folded into the migration, so it also heals a store
already upgraded by an earlier 8.0.x that stranded the blobs (whose layout-
migration marker is stamped): it is gated on the presence of `_cow/` and its own
`_system/vfs-blob-adoption.json` marker. Filesystem-only; native-8.0 and non-VFS
stores no-op on a cheap existence check.
- `BaseStorage.adoptLegacyCowBlobs()` — the scan/copy primitive, returning
`{ cowBlobs, adopted, alreadyPresent, incomplete }`; skips (does not half-adopt)
a blob missing its bytes or metadata.
- `brain.vfs.adoptOrphanedBlobs()` — the explicit force path; self-initializes.
- Backup retention: the automatic pre-upgrade backup is now kept if any blob
can't be fully adopted (`incomplete > 0`), instead of being removed on
entity-migration success alone — a blob-parity gate the migration signal
cannot provide.
Adds an end-to-end integration test (storage-level adopt with idempotency and
incomplete handling; self-heal on reopen; the explicit API) and an upgrade guide.
186 lines
7.7 KiB
Markdown
186 lines
7.7 KiB
Markdown
---
|
|
title: Upgrading from 7.x to 8.0
|
|
slug: guides/upgrading-7-to-8
|
|
public: true
|
|
category: guides
|
|
template: guide
|
|
order: 10
|
|
description: What the one-time 7→8 on-disk migration does, how 8.0 automatically recovers Virtual Filesystem content that older layouts stored in the removed copy-on-write area, and how to verify (or force) that recovery.
|
|
next:
|
|
- guides/storage-adapters
|
|
- guides/inspection
|
|
---
|
|
|
|
# Upgrading from 7.x to 8.0
|
|
|
|
Opening a 7.x on-disk store with Brainy 8.0 runs a **one-time, in-place layout
|
|
migration** the first time the store is opened. It is automatic (`autoMigrate`
|
|
defaults to `true`), it runs once, and it stamps a marker so every later open is
|
|
a no-op.
|
|
|
|
Almost everything about the upgrade is transparent: your entities, relationships,
|
|
metadata, and indexes migrate and rebuild without any action on your part. This
|
|
guide covers the **one case that needs attention** — Virtual Filesystem (VFS)
|
|
content — and how 8.0 recovers it for you.
|
|
|
|
## TL;DR
|
|
|
|
- **Just upgrade to `@soulcraft/brainy@8.0.12` (or later) and open the store.**
|
|
If a previous upgrade left VFS content stranded, 8.0.12 **heals it on open**,
|
|
with no operator action.
|
|
- Want to force or script it? Call **`await brain.vfs.adoptOrphanedBlobs()`**.
|
|
- The recovery is **non-destructive and idempotent** — it copies, never moves,
|
|
and running it twice is a no-op.
|
|
|
|
## What the migration does
|
|
|
|
The 7.x on-disk layout stored entities under a per-branch path
|
|
(`branches/<head>/entities/...`). 8.0 uses a flat layout (`entities/...`). On
|
|
first open, Brainy:
|
|
|
|
1. Collapses `branches/<head>/entities/*` into the flat `entities/*` layout.
|
|
2. Rebuilds the derived indexes (vector, graph, metadata) and count rollups from
|
|
the canonical entities.
|
|
3. Stamps `_system/migration-layout.json` so re-opening is a no-op.
|
|
4. Takes an automatic **pre-upgrade backup** of the directory before it starts,
|
|
and removes it once the upgrade is verified complete (see
|
|
[The safety net](#the-safety-net-pre-upgrade-backup) below).
|
|
|
|
The log line to expect (once per store):
|
|
|
|
```
|
|
[brainy] Migrating a 7.x branch layout (branches/main) to the 8.0 flat layout
|
|
in place — 13175 entity files. This runs once; back up the directory first if
|
|
you need a rollback (8.0 does not keep the old layout).
|
|
```
|
|
|
|
## The one thing that needs recovery: VFS content
|
|
|
|
If your application uses the Virtual Filesystem (VFS) to store file content (for
|
|
example, a CMS that keeps page documents at paths like
|
|
`/pages/homepage/page.json`), that content is held as **content blobs**.
|
|
|
|
7.x kept those blobs in the branch system's **copy-on-write area** (`_cow/`).
|
|
8.0 removed the branch/copy-on-write system, and its content blobs live in the
|
|
content-addressed store (`_cas/`). The layout migration moves entities — it does
|
|
**not** move the VFS content blobs. Left alone, a 7.x store's VFS blobs would
|
|
remain in `_cow/`, and a read of one would throw:
|
|
|
|
```
|
|
VFS: Cannot read blob for /pages/homepage/page.json:
|
|
Blob metadata not found: 6b87cfb71ad2f04602a5c157214dc42000...
|
|
```
|
|
|
|
### 8.0.12 recovers them automatically
|
|
|
|
Brainy 8.0.12 adds an **on-open recovery pass** that runs right after the layout
|
|
migration. It scans `_cow/`, and for every content blob whose `blob:` +
|
|
`blob-meta:` pair is not already in `_cas/`, it copies **both** across into the
|
|
8.0 store. It:
|
|
|
|
- **Heals a fresh 7→8 upgrade** (where `_cow/` still holds the blobs), **and**
|
|
- **Heals a store already upgraded by an earlier 8.0.x** that stranded them —
|
|
the recovery is gated on the presence of `_cow/` and its own marker, not on
|
|
the layout-migration marker, so an already-migrated store is still healed.
|
|
- Is a **cheap no-op** on a native-8.0 or fresh store (there is no `_cow/`), and
|
|
on a store already healed (a marker records completion so later opens skip the
|
|
scan).
|
|
|
|
So the operator action for a stranded store is simply: **upgrade to 8.0.12 and
|
|
open it.**
|
|
|
|
```ts
|
|
import { Brainy } from '@soulcraft/brainy'
|
|
|
|
// Opening the store is all that is required — recovery runs during init().
|
|
const brain = new Brainy({ storage: { type: 'filesystem', path: '/data/my-store' } })
|
|
await brain.init()
|
|
|
|
// The previously-failing read now succeeds.
|
|
const page = await brain.vfs.readFile('/pages/homepage/page.json')
|
|
```
|
|
|
|
On a store that needed recovery you will see:
|
|
|
|
```
|
|
[brainy] Recovered 337 VFS content blob(s) stranded by a 7→8 upgrade
|
|
(adopted _cow/ → _cas/ in place).
|
|
```
|
|
|
|
### Forcing recovery explicitly
|
|
|
|
If you would rather run the recovery deliberately (for example, in an upgrade
|
|
script that asserts a clean result before flipping traffic), call it directly:
|
|
|
|
```ts
|
|
const result = await brain.vfs.adoptOrphanedBlobs()
|
|
// → { cowBlobs, adopted, alreadyPresent, incomplete }
|
|
console.log(`adopted ${result.adopted}, already present ${result.alreadyPresent}`)
|
|
if (result.incomplete > 0) {
|
|
// One or more _cow/ blobs are missing their bytes or metadata — investigate
|
|
// _cow/ before discarding your own backup. See "The safety net" below.
|
|
}
|
|
```
|
|
|
|
`adoptOrphanedBlobs()` self-initializes the brain, so it is safe to call
|
|
immediately after construction. It returns:
|
|
|
|
| Field | Meaning |
|
|
| ---------------- | ---------------------------------------------------------------- |
|
|
| `cowBlobs` | Distinct content-blob hashes found in `_cow/`. |
|
|
| `adopted` | Newly copied into `_cas/` on this call. |
|
|
| `alreadyPresent` | Already in `_cas/` (a prior open or run adopted them). |
|
|
| `incomplete` | A `_cow/` blob missing its bytes *or* its metadata — **skipped** rather than half-adopted. |
|
|
|
|
## Verifying recovery
|
|
|
|
After opening under 8.0.12, confirm a previously-failing path reads:
|
|
|
|
```ts
|
|
const content = await brain.vfs.readFile('/pages/homepage/page.json')
|
|
console.log(content.toString().slice(0, 80))
|
|
```
|
|
|
|
If you scripted it with `adoptOrphanedBlobs()`, a clean result is
|
|
`incomplete === 0` and `adopted + alreadyPresent === cowBlobs`.
|
|
|
|
## The safety net: pre-upgrade backup
|
|
|
|
Brainy takes an automatic pre-upgrade backup and removes it once the upgrade is
|
|
**verified complete**. In 8.0.12 that verification includes VFS content: if the
|
|
blob recovery reports `incomplete > 0`, the completion marker is **not** stamped
|
|
(the next open retries) and the **pre-upgrade backup is retained** so you still
|
|
have a rollback while blobs remain unaccounted for. You will see:
|
|
|
|
```
|
|
[brainy] VFS blob recovery adopted N blob(s) but M orphaned _cow/ blob(s) are
|
|
missing their bytes or metadata and were left in place. The pre-upgrade backup
|
|
is being retained; inspect _cow/ before discarding it.
|
|
```
|
|
|
|
The recovery never deletes anything from `_cow/`, so the original blobs stay in
|
|
place for inspection or a manual rollback regardless.
|
|
|
|
## Who is affected
|
|
|
|
**Any 7.x store that used the VFS to store file content** (so it has a `_cow/`
|
|
area) is a candidate for stranded blobs on a 7→8 upgrade. Stores that never used
|
|
the VFS have no `_cow/` content blobs and are unaffected.
|
|
|
|
Because the recovery is **gated on the presence of `_cow/`**, it is
|
|
self-selecting and safe to roll out everywhere:
|
|
|
|
- On a store with stranded blobs → it adopts them.
|
|
- On a native-8.0, fresh, or non-VFS store → it is a no-op on the existence
|
|
check.
|
|
|
|
There is no configuration to set and nothing to opt into. Upgrading to 8.0.12
|
|
and opening each store is sufficient.
|
|
|
|
## Rollback
|
|
|
|
The recovery is copy-only, so no rollback of the recovery itself is ever needed.
|
|
If you need to roll back the **whole** 7→8 upgrade, restore the directory from
|
|
your pre-upgrade backup (retained automatically while recovery is incomplete, or
|
|
your own snapshot) and pin `@soulcraft/brainy@7.x`. 8.0 does not keep the old
|
|
branch layout in place, so a directory-level restore is the rollback path.
|