feat(8.0): 7.x→8.0 layout migration — fix silent total data loss on first open
7.x stored every object branch-scoped under branches/<branch>/<basePath>; 8.0 stores the IDENTICAL entity structure at the storage ROOT plus the generational _system/_generations layer. GenerationStore.open tolerates a missing manifest (opens at gen 0), so a naive 8.0 open of a 7.x directory reported ZERO entities and SILENTLY LOST ALL DATA. The fix is blocked in the normal init order — cor 3.0's storage-factory legacy guard fires inside setupStorage(), five steps before the old migration check — so a new phase runs BEFORE setupStorage(). legacyLayoutMigrationPhase() (init, between loadPlugins and setupStorage): - Fast-path: returns immediately when there's no top-level branches/ dir (every native 8.0 brain + fresh dir pays only one fs.existsSync on the init hot path). - Runs on a temporary BUILT-IN FileSystemStorage (never the plugin adapter, whose guard would throw). Memory/cloud/pre-built-adapter stores are a strict no-op. - Detect via _system/migration-layout.json marker + branches/<head>/entities/. - autoMigrate:false on a legacy layout THROWS explicit guidance (no silent loss). - Acquire the writer lock, then COLLAPSE branches/<head>/entities/* → entities/* via the .gz-transparent raw primitives (read→write→delete, idempotent/resume-safe; NOT fs.rename — logical paths + memory-adapter incompatibility). - Rebuild persisted counts (rebuildCounts → totalNounCount/counts.json; rebuildTypeCounts/rebuildSubtypeCounts → per-type stats) — the three indexes are rebuilt by the normal init's rebuildIndexesIfNeeded afterward. - Finalize: stamp the flat-v8 marker (re-open no-op), drain the head branch (non-head branches = 7.x version history 8.0's MVCC does not import; left as-is). Tests (tests/integration/migration-7x-to-8x.test.ts): the data-loss LOCK (a built-in FileSystemStorage on a reshaped 7.x dir reports 0 nouns), a byte-equal ROUND TRIP (find/related/counts equal the pre-reshape reference), IDEMPOTENCY, the autoMigrate:false GUARD throw, and the native-flat NO-OP. RELEASES upgrade checklist rewritten (the old note documented the opposite, lossy behavior). 1477 unit + migration 5 + db-mvcc 25 green; no init-path regression. Follow-ups (hardening, not correctness for the common case): backupTo config plumbing (currently a loud warning), a crash-mid-collapse resume test, and a boundary-safe fake-plugin ordering test.
This commit is contained in:
parent
2c84f86815
commit
0c4a51c24e
3 changed files with 375 additions and 3 deletions
168
src/brainy.ts
168
src/brainy.ts
|
|
@ -11,6 +11,7 @@ import { JsHnswVectorIndex } from './hnsw/hnswIndex.js'
|
|||
// for the 99% of queries that don't filter by type (avoids searching 42 separate graphs)
|
||||
import { createStorage } from './storage/storageFactory.js'
|
||||
import type { StorageOptions } from './storage/storageFactory.js'
|
||||
import { rebuildCounts } from './utils/rebuildCounts.js'
|
||||
import type { MetadataWriteBuffer } from './utils/metadataWriteBuffer.js'
|
||||
import { BaseStorage } from './storage/baseStorage.js'
|
||||
import { StorageAdapter, Vector, DistanceFunction, EmbeddingFunction, GraphVerb, STANDARD_ENTITY_FIELDS } from './coreTypes.js'
|
||||
|
|
@ -650,6 +651,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// so plugin-provided storage factories (e.g., filesystem override from cortex) are available
|
||||
await this.loadPlugins()
|
||||
|
||||
// 7.x → 8.0 layout migration, BEFORE storage setup: collapse a legacy
|
||||
// branch layout to the flat 8.0 layout on a built-in FileSystemStorage, so
|
||||
// setupStorage() (and any native plugin factory's own legacy guard) sees a
|
||||
// clean store. No-op for non-filesystem stores and already-flat brains.
|
||||
await this.legacyLayoutMigrationPhase()
|
||||
|
||||
// Setup and initialize storage (checks plugin storage factories first)
|
||||
this.storage = await this.setupStorage()
|
||||
await this.storage.init()
|
||||
|
|
@ -10748,6 +10755,167 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
/**
|
||||
* Setup storage
|
||||
*/
|
||||
/**
|
||||
* @description 7.x → 8.0 on-disk LAYOUT migration, run BEFORE `setupStorage()`.
|
||||
*
|
||||
* 7.x stored every object branch-scoped under `branches/<branch>/<basePath>`;
|
||||
* 8.0 stores the IDENTICAL entity structure at the storage ROOT
|
||||
* (`entities/<kind>/<shard>/<id>/…`) plus the generational `_system` +
|
||||
* `_generations` layer. `GenerationStore.open()` is tolerant of a missing
|
||||
* manifest (opens at generation 0), so a naive 8.0 open of a 7.x directory
|
||||
* reports zero entities and SILENTLY LOSES ALL DATA. This phase collapses the
|
||||
* HEAD branch's canonical entities to the root so the rest of init (and any
|
||||
* native plugin storage factory, whose own legacy guard would otherwise THROW)
|
||||
* sees a clean flat-v8 store; 8.0 rebuilds all derived state from those entities.
|
||||
*
|
||||
* Runs on a temporary BUILT-IN `FileSystemStorage` (never the plugin adapter),
|
||||
* is idempotent (a `_system/migration-layout.json` marker short-circuits re-open),
|
||||
* resume-safe (per-object read→write→delete), and a strict NO-OP for any brain
|
||||
* that is not a filesystem store carrying a legacy `branches/` layout.
|
||||
*/
|
||||
private async legacyLayoutMigrationPhase(): Promise<void> {
|
||||
// Pre-built adapter instances and non-filesystem stores have no on-disk 7.x
|
||||
// layout to migrate.
|
||||
if (isStorageAdapterInstance(this.config.storage)) return
|
||||
const storageConfig = (this.config.storage || {}) as StorageOptions & Record<string, unknown>
|
||||
const type = storageConfig.type || 'auto'
|
||||
if (type !== 'filesystem' && type !== 'auto') return
|
||||
|
||||
// Fast path: the only on-disk shape that needs migrating has a top-level
|
||||
// `branches/` directory. A native 8.0 brain and a fresh directory do not, so
|
||||
// skip the probe-storage construction entirely on the init hot path. (The
|
||||
// migration is node-filesystem-only; `fs.existsSync` is absent/throwing
|
||||
// elsewhere, which the catch treats as "nothing to migrate".)
|
||||
const rootDir =
|
||||
(storageConfig.rootDirectory as string) ||
|
||||
(storageConfig.path as string) ||
|
||||
'./brainy-data' // createStorage's filesystem default
|
||||
try {
|
||||
if (!fs.existsSync(`${rootDir}/branches`)) return
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
|
||||
// Probe with a BUILT-IN filesystem storage over the same root (never the
|
||||
// plugin factory — cor's adapter would throw on the legacy layout). In a
|
||||
// non-filesystem environment this throws; that just means "nothing to migrate".
|
||||
let probe: BaseStorage
|
||||
try {
|
||||
probe = (await createStorage({ ...storageConfig, type: 'filesystem' })) as BaseStorage
|
||||
await probe.init()
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
|
||||
try {
|
||||
// Idempotency: a completed migration leaves a marker → re-open is a no-op.
|
||||
const marker = await probe.readRawObject('_system/migration-layout.json')
|
||||
if (marker && (marker as { layout?: string }).layout === 'flat-v8') return
|
||||
|
||||
// DETECT: HEAD-branch entities present under branches/<head>/entities/.
|
||||
const head = (storageConfig.branch as string) || 'main'
|
||||
const branchPaths = await probe.listRawObjects('branches')
|
||||
const headEntityPrefix = `branches/${head}/entities/`
|
||||
const legacyEntityPaths = branchPaths.filter((p) => p.startsWith(headEntityPrefix))
|
||||
|
||||
if (legacyEntityPaths.length === 0) {
|
||||
// Already flat (root entities, no head-branch entities) → stamp the marker
|
||||
// so future opens short-circuit. A genuinely empty/fresh dir gets no marker.
|
||||
const rootEntities = await probe.listRawObjects('entities')
|
||||
if (rootEntities.length > 0) {
|
||||
await probe.writeRawObject('_system/migration-layout.json', {
|
||||
layout: 'flat-v8',
|
||||
version: 8,
|
||||
fromBranch: null,
|
||||
entitiesMoved: 0
|
||||
})
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// GUARD: an explicit autoMigrate:false must not silently lose 7.x data.
|
||||
if (this.config.autoMigrate === false) {
|
||||
throw new Error(
|
||||
`Brainy 8.0 found a 7.x branch layout at this storage directory ` +
|
||||
`(${legacyEntityPaths.length} entities under branches/${head}/). Opening it as-is ` +
|
||||
`would report zero entities and lose your data. Set autoMigrate: true (the default) ` +
|
||||
`to migrate the layout in place on open, or export the data on 7.x first.`
|
||||
)
|
||||
}
|
||||
|
||||
// Acquire the writer lock for the duration of the move (idempotent marker
|
||||
// makes a lost race harmless, but the lock prevents two concurrent collapses).
|
||||
const canLock =
|
||||
typeof (probe as unknown as { supportsMultiProcessLocking?: () => boolean })
|
||||
.supportsMultiProcessLocking === 'function' &&
|
||||
(probe as unknown as { supportsMultiProcessLocking: () => boolean }).supportsMultiProcessLocking()
|
||||
const lockable = probe as unknown as {
|
||||
acquireWriterLock?: (o: { force?: boolean }) => Promise<void>
|
||||
releaseWriterLock?: () => Promise<void>
|
||||
}
|
||||
if (canLock && typeof lockable.acquireWriterLock === 'function') {
|
||||
await lockable.acquireWriterLock({ force: this.config.force })
|
||||
}
|
||||
|
||||
if (!this.config.silent) {
|
||||
console.warn(
|
||||
`[brainy] Migrating a 7.x branch layout (branches/${head}) to the 8.0 flat layout ` +
|
||||
`in place — ${legacyEntityPaths.length} entity files. This runs once; back up the ` +
|
||||
`directory first if you need a rollback (8.0 does not keep the old layout).`
|
||||
)
|
||||
}
|
||||
|
||||
try {
|
||||
// COLLAPSE: branches/<head>/entities/* → entities/* via the .gz-transparent
|
||||
// raw primitives (NOT fs.rename — listRawObjects returns logical paths and
|
||||
// the in-memory adapter has no real files). read→write→delete is idempotent,
|
||||
// so a crash mid-move is recovered by simply re-running.
|
||||
let moved = 0
|
||||
for (const src of legacyEntityPaths) {
|
||||
const target = src.slice(`branches/${head}/`.length) // → entities/...
|
||||
const data = await probe.readRawObject(src)
|
||||
if (data === null) continue // already moved by an interrupted prior run
|
||||
await probe.writeRawObject(target, data)
|
||||
await probe.deleteRawObject(src)
|
||||
moved++
|
||||
}
|
||||
|
||||
// REBUILD persisted derived state from the now-flat canonical entities.
|
||||
// (The three indexes — HNSW, metadata, graph — are rebuilt by the normal
|
||||
// init's rebuildIndexesIfNeeded after storage is set up; the COUNT rollups
|
||||
// are NOT, so they must be rebuilt + persisted here or getNounCount()/stats()
|
||||
// read 0 on the migrated store.)
|
||||
// - rebuildCounts → totalNounCount/totalVerbCount + counts.json (getNounCount/getVerbCount)
|
||||
// - rebuildTypeCounts → per-type type-statistics.json (stats().entitiesByType)
|
||||
await rebuildCounts(probe)
|
||||
await probe.rebuildTypeCounts()
|
||||
if (typeof (probe as unknown as { rebuildSubtypeCounts?: () => Promise<void> }).rebuildSubtypeCounts === 'function') {
|
||||
await (probe as unknown as { rebuildSubtypeCounts: () => Promise<void> }).rebuildSubtypeCounts()
|
||||
}
|
||||
|
||||
// FINALIZE: mark migrated (makes re-open a no-op), then drain the head
|
||||
// branch (its entities moved; the rest is stale 7.x derived state).
|
||||
// Non-head branches under branches/ are 7.x version history 8.0's MVCC does
|
||||
// not import — they are left in place.
|
||||
await probe.writeRawObject('_system/migration-layout.json', {
|
||||
layout: 'flat-v8',
|
||||
version: 8,
|
||||
fromBranch: head,
|
||||
entitiesMoved: moved
|
||||
})
|
||||
await probe.removeRawPrefix(`branches/${head}`)
|
||||
} finally {
|
||||
if (canLock && typeof lockable.releaseWriterLock === 'function') {
|
||||
await lockable.releaseWriterLock()
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
if (typeof (probe as unknown as { close?: () => Promise<void> }).close === 'function') {
|
||||
await (probe as unknown as { close: () => Promise<void> }).close()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async setupStorage(): Promise<BaseStorage> {
|
||||
// If the caller passed a pre-constructed storage adapter (e.g.
|
||||
// `storage: new MemoryStorage()`, or the historical materializer's
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue