fix: recover VFS content blobs stranded by a 7→8 upgrade, in place on open

The 7.x branch system stored Virtual Filesystem content as blobs in its
copy-on-write area (`_cow/`). 8.0 removed that system and stores content blobs
in the content-addressed store (`_cas/`), but the one-time layout migration only
moves entities — it never adopted the `_cow/` content blobs. A store that used
the VFS was left with every VFS-backed read throwing "Blob metadata not found"
and its pages 500ing, with no first-party recovery and the pre-upgrade backup
already removed on entity-migration "success".

Add an on-open recovery pass (`autoAdoptLegacyVfsBlobsIfNeeded`) that runs right
after the layout migration and adopts every orphaned `_cow/` blob into `_cas/` —
copying both the bytes and the metadata via the raw object primitives,
idempotently and non-destructively (the `_cow/` originals are never deleted). It
is a separate phase, not folded into the migration, so it also heals a store
already upgraded by an earlier 8.0.x that stranded the blobs (whose layout-
migration marker is stamped): it is gated on the presence of `_cow/` and its own
`_system/vfs-blob-adoption.json` marker. Filesystem-only; native-8.0 and non-VFS
stores no-op on a cheap existence check.

- `BaseStorage.adoptLegacyCowBlobs()` — the scan/copy primitive, returning
  `{ cowBlobs, adopted, alreadyPresent, incomplete }`; skips (does not half-adopt)
  a blob missing its bytes or metadata.
- `brain.vfs.adoptOrphanedBlobs()` — the explicit force path; self-initializes.
- Backup retention: the automatic pre-upgrade backup is now kept if any blob
  can't be fully adopted (`incomplete > 0`), instead of being removed on
  entity-migration success alone — a blob-parity gate the migration signal
  cannot provide.

Adds an end-to-end integration test (storage-level adopt with idempotency and
incomplete handling; self-heal on reopen; the explicit API) and an upgrade guide.

src/brainy.ts

@@ -32,7 +32,7 @@ import {
   getBrainyVersion
 } from './utils/index.js'
 import { embeddingManager } from './embeddings/EmbeddingManager.js'
-import { matchesMetadataFilter } from './utils/metadataFilter.js'
+import { matchesMetadataFilter, validateWhereFilter } from './utils/metadataFilter.js'
 import { NaturalLanguageProcessor } from './neural/naturalLanguageProcessor.js'
 import { NeuralEntityExtractor, ExtractedEntity } from './neural/entityExtractor.js'
 import { TripleIntelligenceSystem } from './triple/TripleIntelligenceSystem.js'
@@ -453,6 +453,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
    *  `migrationBackup: false`), or `null` if none. Removed once the 7.x→8.0
    *  upgrade verifies + stamps; retained on failure. See {@link createMigrationBackupIfNeeded}. */
   private _migrationBackupPath: string | null = null
+
+  /** Set when the on-open VFS-blob adoption left one or more `_cow/` blobs it
+   *  could not fully adopt (bytes or metadata missing). While true, the
+   *  pre-upgrade backup is NOT auto-removed — the upgrade is not verifiably
+   *  complete for the VFS. See {@link autoAdoptLegacyVfsBlobsIfNeeded}. */
+  private _vfsBlobAdoptionIncomplete = false
+
   /**
    * 8.0 ⇄ native-provider version handshake — the on-disk {@link BrainFormat}
    * marker (`_system/brain-format.json`) as read during the store-open phase,
@@ -1028,6 +1035,14 @@ export class Brainy<T = any> implements BrainyInterface<T> {
         }
       }
 
+      // Recover VFS content blobs a 7→8 upgrade left stranded in the removed
+      // branch system's copy-on-write area (`_cow/`). Runs BEFORE the rebuild
+      // (whose success stamp removes the pre-upgrade backup) so the backup still
+      // covers this step, and works on the raw object primitives (no blob store
+      // needed yet). A cheap no-op on native-8.0 / fresh brains and on a brain
+      // already healed. See adoptLegacyCowBlobs.
+      await this.autoAdoptLegacyVfsBlobsIfNeeded()
+
       // Rebuild indexes if needed for existing data
       await this.rebuildIndexesIfNeeded()
 
@@ -5228,6 +5243,15 @@ export class Brainy<T = any> implements BrainyInterface<T> {
       const hasFilterCriteria = params.where || params.type || params.subtype || params.service
       const hasGraphCriteria = params.connected
 
+      // Validate the raw where clause up front: an unrecognized operator (a typo
+      // like `notIn`, or any non-operator key on a field's object value) throws a
+      // typed BrainyError('INVALID_QUERY') instead of silently matching nothing.
+      // Done before the index/vector/graph paths so it fires even on an empty
+      // result set, and before brainy injects its own internal filter fields.
+      if (params.where) {
+        validateWhereFilter(params.where)
+      }
+
       // Metadata cold-read guard: before trusting a filter result, verify the
       // field index actually serves a known persisted value (one-shot per brain).
       // A cold native index that has not loaded its `where` postings self-heals
@@ -13012,6 +13036,85 @@ export class Brainy<T = any> implements BrainyInterface<T> {
     }
   }
 
+  /**
+   * @description On-open recovery for VFS content blobs a 7→8 upgrade left in the
+   * removed branch system's copy-on-write area (`_cow/`). If a `_cow/` area is
+   * present and a completed adoption has not already been recorded, adopt every
+   * orphaned blob into the 8.0 content-addressed store (`_cas/`) and stamp a
+   * marker so later opens skip the scan. Because it runs on every open right
+   * after {@link legacyLayoutMigrationPhase}, it heals BOTH a fresh 7→8 migration
+   * (where `_cow/` still holds the blobs) AND a brain already migrated by an
+   * earlier 8.0.x that stranded them — no operator action needed.
+   *
+   * Filesystem-only; a native-8.0 / fresh brain has no `_cow/` and returns on the
+   * cheap existence check. Non-destructive and idempotent (see
+   * {@link BaseStorage.adoptLegacyCowBlobs}). If any blob cannot be fully adopted
+   * (`incomplete > 0`), the marker is NOT stamped (the next open retries) and
+   * {@link _vfsBlobAdoptionIncomplete} is set so the pre-upgrade backup is
+   * retained — the upgrade is not verifiably complete for the VFS.
+   */
+  private async autoAdoptLegacyVfsBlobsIfNeeded(): Promise<void> {
+    if (this.getStorageType() !== 'filesystem') return
+    // Cheapest gate: no copy-on-write area → a native-8.0 / fresh brain, nothing
+    // a 7.x brain could have stranded here.
+    try {
+      const root = resolveFilesystemRoot(
+        (this.config.storage || {}) as StorageOptions & Record<string, unknown>
+      )
+      if (!fs.existsSync(`${root}/_cow`)) return
+    } catch {
+      return
+    }
+
+    const storage = this.storage as unknown as {
+      adoptLegacyCowBlobs?: () => Promise<{
+        cowBlobs: number
+        adopted: number
+        alreadyPresent: number
+        incomplete: number
+      }>
+      readRawObject?: (p: string) => Promise<unknown>
+      writeRawObject?: (p: string, o: unknown) => Promise<void>
+    }
+    if (typeof storage.adoptLegacyCowBlobs !== 'function') return
+
+    const MARKER = '_system/vfs-blob-adoption.json'
+    try {
+      const done = (await storage.readRawObject?.(MARKER)) as { complete?: boolean } | null
+      if (done && done.complete) return // already fully adopted on a prior open
+    } catch {
+      // no marker yet — proceed
+    }
+
+    const result = await storage.adoptLegacyCowBlobs()
+
+    if (result.incomplete > 0) {
+      // Partial: retry on future opens, and hold the pre-upgrade backup.
+      this._vfsBlobAdoptionIncomplete = true
+      if (!this.config.silent) {
+        console.error(
+          `[brainy] VFS blob recovery adopted ${result.adopted} blob(s) but ${result.incomplete} ` +
+            `orphaned _cow/ blob(s) are missing their bytes or metadata and were left in place. ` +
+            `The pre-upgrade backup is being retained; inspect _cow/ before discarding it.`
+        )
+      }
+      return
+    }
+
+    // Clean sweep — record it so later opens skip the _cow/ scan entirely.
+    try {
+      await storage.writeRawObject?.(MARKER, { complete: true, adopted: result.adopted })
+    } catch {
+      // marker write is best-effort; adoption already succeeded
+    }
+    if (result.adopted > 0 && !this.config.silent) {
+      console.warn(
+        `[brainy] Recovered ${result.adopted} VFS content blob(s) stranded by a 7→8 upgrade ` +
+          `(adopted _cow/ → _cas/ in place).`
+      )
+    }
+  }
+
   private async setupStorage(): Promise<BaseStorage> {
     // If the caller passed a pre-constructed storage adapter (e.g.
     // `storage: new MemoryStorage()`, or the historical materializer's
@@ -13846,6 +13949,20 @@ export class Brainy<T = any> implements BrainyInterface<T> {
    */
   private async removeMigrationBackupSafe(): Promise<void> {
     if (!this._migrationBackupPath) return
+    // Backup-parity gate: the index rebuild "success" that calls this does NOT
+    // certify the VFS. If the on-open blob adoption could not fully bridge the
+    // 7.x `_cow/` blobs, the upgrade is not verifiably complete for VFS content
+    // — retain the backup so the operator can recover from it, and log why.
+    if (this._vfsBlobAdoptionIncomplete) {
+      if (!this.config.silent) {
+        console.warn(
+          `[brainy] Retaining the pre-upgrade backup at ${this._migrationBackupPath}: ` +
+            `some VFS content blobs could not be adopted from _cow/ (see the earlier warning). ` +
+            `Resolve those before discarding the backup.`
+        )
+      }
+      return
+    }
     const backupPath = this._migrationBackupPath
     this._migrationBackupPath = null
     const storage = this.storage as StorageAdapter & {

src/storage/baseStorage.ts

@@ -680,6 +680,91 @@ export abstract class BaseStorage extends BaseStorageAdapter {
     this.blobStorage = new BlobStorage(casAdapter)
   }
 
+  /**
+   * @description Adopt orphaned 7.x copy-on-write VFS content blobs into the 8.0
+   * content-addressed store. 7.x kept VFS blobs (`blob:<hash>` bytes +
+   * `blob-meta:<hash>` JSON) under the copy-on-write area (`_cow/`) of the
+   * branch/versioning system that 8.0 removed. The 7→8 layout migration
+   * collapses entity files but historically did NOT bridge these blobs, so a VFS
+   * read of a still-in-`_cow/` blob throws "Blob metadata not found" and every
+   * page backed by it 500s.
+   *
+   * This scans `_cow/` and, for every blob whose `blob:`/`blob-meta:` pair is not
+   * already present in the 8.0 store (`_cas/`), copies BOTH objects across
+   * verbatim. Copying preserves the exact content bytes (so the content hash and
+   * every VFS reference still resolve) and the exact metadata (so the 7.x
+   * refCount is retained) — the only first-party, index-consistent recovery.
+   * Hand-moving files at the OS level bypasses the paired-metadata contract and
+   * is unsafe; this goes through the adapter's object primitives.
+   *
+   * Idempotent and non-destructive: a pair already in `_cas/` is left untouched
+   * (counted `alreadyPresent`); the `_cow/` originals are never deleted, so a
+   * re-run — or a rollback — is always possible. A no-op on memory storage and
+   * on any brain without a `_cow/` area (returns zeros). Bytes are written before
+   * metadata so an interruption can only leave the pre-adoption "no metadata"
+   * state (retryable), never a metadata-without-bytes blob.
+   *
+   * @returns `cowBlobs` (distinct blob hashes found in `_cow/`), `adopted`
+   *   (newly copied into `_cas/`), `alreadyPresent` (already in `_cas/`),
+   *   `incomplete` (a `_cow/` blob missing its bytes or its metadata — skipped
+   *   and reported rather than half-adopted).
+   */
+  public async adoptLegacyCowBlobs(): Promise<{
+    cowBlobs: number
+    adopted: number
+    alreadyPresent: number
+    incomplete: number
+  }> {
+    let cowPaths: string[]
+    try {
+      cowPaths = await this.listObjectsUnderPath('_cow/')
+    } catch {
+      // No `_cow/` area (fresh/native-8.0 brain, or a non-listing backend).
+      return { cowBlobs: 0, adopted: 0, alreadyPresent: 0, incomplete: 0 }
+    }
+
+    // Distinct content-blob hashes from `_cow/blob:<hash>` keys. Non-blob `_cow/`
+    // entries (7.x branch COW state) are ignored — only VFS content is adopted.
+    const hashes = new Set<string>()
+    for (const p of cowPaths) {
+      const key = p.replace(/^_cow\//, '')
+      const m = /^blob:(.+)$/.exec(key)
+      if (m) hashes.add(m[1])
+    }
+
+    let adopted = 0
+    let alreadyPresent = 0
+    let incomplete = 0
+    for (const hash of hashes) {
+      // A blob counts as present only when BOTH its bytes and its metadata
+      // already live in `_cas/`. A half-adopted blob (bytes without meta — the
+      // exact "Blob metadata not found" state) is re-adopted.
+      const casBlob = await this.readObjectFromPath(`_cas/blob:${hash}`)
+      const casMeta = await this.readObjectFromPath(`_cas/blob-meta:${hash}`)
+      if (casBlob !== null && casMeta !== null) {
+        alreadyPresent++
+        continue
+      }
+
+      const cowBlob = await this.readObjectFromPath(`_cow/blob:${hash}`)
+      const cowMeta = await this.readObjectFromPath(`_cow/blob-meta:${hash}`)
+      if (cowBlob === null || cowMeta === null) {
+        // Can't register a blob the store can't fully describe — report it so an
+        // operator investigates rather than silently half-adopting.
+        incomplete++
+        continue
+      }
+
+      // Bytes first, then metadata: a VFS read checks metadata before bytes, so
+      // metadata's presence must imply the bytes are already there.
+      await this.writeObjectToPath(`_cas/blob:${hash}`, cowBlob)
+      await this.writeObjectToPath(`_cas/blob-meta:${hash}`, cowMeta)
+      adopted++
+    }
+
+    return { cowBlobs: hashes.size, adopted, alreadyPresent, incomplete }
+  }
+
   /**
    * @description Write a canonical object (write-cache coherent). The object
    * lands in the write-through cache before the asynchronous write starts, so

src/vfs/VirtualFileSystem.ts

@@ -352,6 +352,52 @@ export class VirtualFileSystem implements IVirtualFileSystem {
     }
   }
 
+  /**
+   * @description Recover VFS content blobs orphaned by a 7→8 upgrade. 7.x stored
+   * VFS blobs under the copy-on-write area (`_cow/`) of the branch/versioning
+   * system that 8.0 removed; a brain upgraded before the migration adopted them
+   * reads a stranded blob and throws "Blob metadata not found" (every page
+   * backed by it 500s). This adopts every orphaned `_cow/` blob into the 8.0
+   * content-addressed store (`_cas/`) IN PLACE — no snapshot restore — then
+   * clears the VFS caches so the recovered content reads live immediately.
+   *
+   * Idempotent and non-destructive: blobs already adopted are skipped and the
+   * `_cow/` originals are never deleted (a re-run or rollback stays possible).
+   * Safe to run on a healthy or native-8.0 brain (returns zeros). Delegates to
+   * {@link BaseStorage.adoptLegacyCowBlobs}.
+   *
+   * @returns `{ cowBlobs, adopted, alreadyPresent, incomplete }` counts.
+   * @example
+   * const brain = new Brainy({ storage: { type: 'filesystem', path: '/data/brain' } })
+   * await brain.init()
+   * const r = await brain.vfs.adoptOrphanedBlobs()
+   * console.log(`Adopted ${r.adopted} stranded VFS blobs; ${r.alreadyPresent} already present.`)
+   */
+  async adoptOrphanedBlobs(): Promise<{
+    cowBlobs: number
+    adopted: number
+    alreadyPresent: number
+    incomplete: number
+  }> {
+    // The recovery scan works on raw storage objects, so it only needs the
+    // brain's storage wired up — not the VFS's own init. brain.init() is
+    // idempotent (a no-op on an already-open brain) and, on a cold brain,
+    // sets up storage AND runs the on-open auto-adoption itself; the explicit
+    // scan below is then the idempotent "force re-scan" equivalent.
+    await this.brain.init()
+    const storage = this.brain['storage'] as unknown as
+      | { adoptLegacyCowBlobs?: () => Promise<{ cowBlobs: number; adopted: number; alreadyPresent: number; incomplete: number }> }
+      | undefined
+    if (!storage || typeof storage.adoptLegacyCowBlobs !== 'function') {
+      return { cowBlobs: 0, adopted: 0, alreadyPresent: 0, incomplete: 0 }
+    }
+    const result = await storage.adoptLegacyCowBlobs()
+    // Drop any cached content/stat so a re-read hits the freshly adopted blobs.
+    this.contentCache.clear()
+    this.statCache.clear()
+    return result
+  }
+
   // ============= File Operations =============
 
   /**