soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat(8.0): 7.x→8.0 layout migration — fix silent total data loss on first open

Browse source 
7.x stored every object branch-scoped under branches/<branch>/<basePath>; 8.0
stores the IDENTICAL entity structure at the storage ROOT plus the generational
_system/_generations layer. GenerationStore.open tolerates a missing manifest
(opens at gen 0), so a naive 8.0 open of a 7.x directory reported ZERO entities
and SILENTLY LOST ALL DATA. The fix is blocked in the normal init order — cor 3.0's
storage-factory legacy guard fires inside setupStorage(), five steps before the
old migration check — so a new phase runs BEFORE setupStorage().

legacyLayoutMigrationPhase() (init, between loadPlugins and setupStorage):
- Fast-path: returns immediately when there's no top-level branches/ dir (every
  native 8.0 brain + fresh dir pays only one fs.existsSync on the init hot path).
- Runs on a temporary BUILT-IN FileSystemStorage (never the plugin adapter, whose
  guard would throw). Memory/cloud/pre-built-adapter stores are a strict no-op.
- Detect via _system/migration-layout.json marker + branches/<head>/entities/.
- autoMigrate:false on a legacy layout THROWS explicit guidance (no silent loss).
- Acquire the writer lock, then COLLAPSE branches/<head>/entities/* → entities/*
  via the .gz-transparent raw primitives (read→write→delete, idempotent/resume-safe;
  NOT fs.rename — logical paths + memory-adapter incompatibility).
- Rebuild persisted counts (rebuildCounts → totalNounCount/counts.json;
  rebuildTypeCounts/rebuildSubtypeCounts → per-type stats) — the three indexes are
  rebuilt by the normal init's rebuildIndexesIfNeeded afterward.
- Finalize: stamp the flat-v8 marker (re-open no-op), drain the head branch
  (non-head branches = 7.x version history 8.0's MVCC does not import; left as-is).

Tests (tests/integration/migration-7x-to-8x.test.ts): the data-loss LOCK (a
built-in FileSystemStorage on a reshaped 7.x dir reports 0 nouns), a byte-equal
ROUND TRIP (find/related/counts equal the pre-reshape reference), IDEMPOTENCY,
the autoMigrate:false GUARD throw, and the native-flat NO-OP. RELEASES upgrade
checklist rewritten (the old note documented the opposite, lossy behavior).

1477 unit + migration 5 + db-mvcc 25 green; no init-path regression.

Follow-ups (hardening, not correctness for the common case): backupTo config
plumbing (currently a loud warning), a crash-mid-collapse resume test, and a
boundary-safe fake-plugin ordering test.
This commit is contained in:
David Snelling 2026-06-19 13:44:03 -07:00
parent 2c84f86815
commit 0c4a51c24e
3 changed files with 375 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

16
RELEASES.md
View file

@ -160,9 +160,19 @@ write-time `sum`/`count`/`avg`/`min`/`max`.
| `BrainyZeroConfig` type | Removed. 8.0 zero-config is automatic and internal — `new Brainy()` auto-detects storage, derives HNSW quality from `config.vector.recall`, picks `persistMode` from the adapter, and sizes caches to the detected container-memory limit. There is no config-generation type to import. |
| `brain.isFullyInitialized()` / `brain.awaitBackgroundInit()` | `await brain.ready`. The built-in filesystem/memory adapters finish initialization synchronously inside `init()`, so a single readiness promise is the whole story — these methods were no-ops once cloud adapters were removed. |
If you used 7.x COW branches, **materialize every branch you care about while
still on 7.x** (fork → export, or copy each branch's store) — 8.0 does not
read COW branch state.
**Opening a 7.x store auto-migrates it.** 8.0 stores entities at the root; 7.x
stored them branch-scoped under `branches/<branch>/`. On first open, 8.0 collapses
the **HEAD branch** (`config.storage.branch`, default `main`) to the 8.0 layout in
place and rebuilds all derived state — no action required (`autoMigrate` defaults
to `true`; set it to `false` to make 8.0 refuse a legacy layout with an explicit
error instead). Two caveats:
- **Non-HEAD branches are not imported.** 8.0 has no COW branches; only the head
branch's data is migrated. If you care about other branches, **export each one
while still on 7.x** (`fork → export`, or copy its store).
- **Back up first for rollback.** The migration mutates the directory in place and
8.0 does not keep the old layout — copy the data directory before the first 8.0
open if you need to roll back.
### Renames — find/replace pairs