feat(8.0): generational MVCC storage + Datomic-style Db API (now/transact/asOf/with/persist)

One mechanism replaces the COW and versioning subsystems: immutable
generation-stamped records behind a Datomic-style database value.

Record layer (src/db/generationStore.ts):
- Monotonic generation counter in _system/generation.json; bumped once per
  transact() commit and once per single-operation write (storage hook), so
  brain.generation() is always a meaningful watermark.
- Commit protocol: stage before-images + tx.json delta -> fsync -> execute
  batch via TransactionManager -> atomic tmp+rename of _system/manifest.json
  (the rename IS the commit point) -> append _system/tx-log.jsonl.
- Crash recovery on open rolls uncommitted generations back byte-identically
  and forces an index rebuild; refcounted pins gate compactHistory(), which
  records a horizon (asOf below it throws GenerationCompactedError).

Db API:
- Db: get/find/search/related pinned at a generation, with() speculative
  overlays, since() diffs, persist() hard-link-farm snapshots, timestamp,
  generation, release() + FinalizationRegistry backstop.
- Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch
  (GenerationConflictError CAS), asOf(generation|Date|path), restore(path,
  {confirm}), compactHistory(), generation(), static open() + load().
- get()/metadata find()/related() are fully correct at any reachable pinned
  generation; index-accelerated queries at historical generations throw
  NotYetSupportedAtHistoricalGenerationError - never silently-wrong results.
- VersionedIndexProvider (generation/isGenerationVisible/pin/release) in
  plugin.ts: feature-detected, balanced pin/release in lockstep with Db
  lifecycle, post-commit applier + replay-gap model documented.

Storage primitives (BaseStorage + filesystem/memory adapters): raw-object
read/write/list/remove, fsync barrier, noun/verb raw before-image capture,
tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and
append-in-place exceptions), restoreFromDirectory + derived-state reload.

Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation
across 200 mutations, batch atomicity under injected execution failure,
ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety
under pins, with() overlay isolation, generation monotonicity across
reopen, crash consistency through the real recovery path, and versioned
provider pin/release balance. Design record in
docs/ADR-001-generational-mvcc.md.

src/storage/baseStorage.ts

@@ -881,6 +881,271 @@ export abstract class BaseStorage extends BaseStorageAdapter {
     return Array.from(pathsSet)
   }
 
+  // ============================================================================
+  // GENERATIONAL RECORD LAYER PRIMITIVES (8.0 MVCC)
+  //
+  // The narrow surface `GenerationStore` (src/db/generationStore.ts) needs from
+  // an adapter — see the `GenerationStorage` contract in src/db/types.ts.
+  //
+  // Raw-object methods operate on storage-root-relative paths and deliberately
+  // bypass branch scoping and the write cache: the record layer owns the
+  // `_system/` + `_generations/` areas outright and its files must never be
+  // shadowed by branch resolution. Entity-raw methods, by contrast, go through
+  // the branch-scoped, write-cache-coherent helpers so before-images capture
+  // exactly the bytes the live read paths see.
+  // ============================================================================
+
+  /**
+   * Hook invoked after every entity-visible single-operation write (noun/verb
+   * metadata save or delete). Registered by the generation store so
+   * `brain.generation()` advances on writes performed outside `transact()`.
+   * See {@link BaseStorage.setGenerationBumpHook}.
+   */
+  protected generationBumpHook?: () => void
+
+  /**
+   * Register (or detach, with `undefined`) the generation-bump hook.
+   *
+   * The hook fires once per entity-visible metadata mutation — noun/verb
+   * metadata saves and deletes, the one storage write every logical Brainy
+   * mutation (`add`, `update`, `delete`, `relate`, `updateRelation`,
+   * `unrelate`) performs exactly once per entity it touches. It does NOT fire
+   * for derived-index writes (HNSW node data, metadata-index chunks, LSM
+   * segments, statistics), so the generation counter tracks *data* mutations,
+   * not index maintenance. The counter is a monotonic watermark, not an
+   * operation count: a cascade delete bumps once per removed record.
+   *
+   * @param hook - Callback invoked synchronously after each qualifying write,
+   *   or `undefined` to detach.
+   */
+  public setGenerationBumpHook(hook: (() => void) | undefined): void {
+    this.generationBumpHook = hook
+  }
+
+  /**
+   * Read a raw object at a storage-root-relative path. Bypasses branch
+   * scoping and the write cache (record-layer files are written through
+   * {@link BaseStorage.writeRawObject} only).
+   *
+   * @param path - Storage-root-relative object path (e.g. `_system/manifest.json`).
+   * @returns The parsed object, or `null` if absent.
+   */
+  public async readRawObject(path: string): Promise<any | null> {
+    await this.ensureInitialized()
+    return this.readObjectFromPath(path)
+  }
+
+  /**
+   * Write a raw object at a storage-root-relative path. On disk this is an
+   * atomic tmp+rename (the filesystem adapter's primitive), which is what
+   * makes the manifest rename a valid commit point.
+   *
+   * @param path - Storage-root-relative object path.
+   * @param data - JSON-serializable object to persist.
+   */
+  public async writeRawObject(path: string, data: any): Promise<void> {
+    await this.ensureInitialized()
+    await this.writeObjectToPath(path, data)
+  }
+
+  /**
+   * Delete a raw object at a storage-root-relative path (no-op if absent).
+   *
+   * @param path - Storage-root-relative object path.
+   */
+  public async deleteRawObject(path: string): Promise<void> {
+    await this.ensureInitialized()
+    await this.deleteObjectFromPath(path)
+  }
+
+  /**
+   * List raw object paths under a storage-root-relative prefix (normalized,
+   * `.gz`-stripped — the adapter primitives already normalize).
+   *
+   * @param prefix - Storage-root-relative directory prefix.
+   * @returns Normalized object paths under the prefix (empty when none).
+   */
+  public async listRawObjects(prefix: string): Promise<string[]> {
+    await this.ensureInitialized()
+    return this.listObjectsUnderPath(prefix)
+  }
+
+  /**
+   * Remove every object under a storage-root-relative prefix. The filesystem
+   * adapter overrides this with a recursive directory removal; this default
+   * lists and deletes individually (exactly what the in-memory adapter needs).
+   *
+   * @param prefix - Storage-root-relative directory prefix to remove.
+   */
+  public async removeRawPrefix(prefix: string): Promise<void> {
+    await this.ensureInitialized()
+    const paths = await this.listObjectsUnderPath(prefix)
+    for (const p of paths) {
+      await this.deleteObjectFromPath(p)
+    }
+  }
+
+  /**
+   * Durability barrier for the commit protocol: ensure the listed raw-object
+   * paths are durable before the caller proceeds. The base implementation is
+   * a no-op (in-memory writes are durable-by-definition within the process);
+   * the filesystem adapter overrides it with real `fsync` of the files and
+   * their parent directories.
+   *
+   * @param paths - Storage-root-relative object paths previously written via
+   *   {@link BaseStorage.writeRawObject}.
+   */
+  public async syncRawObjects(paths: string[]): Promise<void> {
+    void paths
+  }
+
+  /**
+   * Read an entity's raw stored objects — the exact bytes at its canonical
+   * metadata + vector paths (branch-scoped, write-cache coherent). Used by
+   * the generation store to capture before-images.
+   *
+   * @param id - The entity id.
+   * @returns The raw stored metadata and vector objects (`null` per part when
+   *   the corresponding file is absent).
+   */
+  public async readNounRaw(id: string): Promise<{ metadata: any | null; vector: any | null }> {
+    await this.ensureInitialized()
+    const [metadata, vector] = await Promise.all([
+      this.readWithInheritance(getNounMetadataPath(id)),
+      this.readWithInheritance(getNounVectorPath(id))
+    ])
+    return { metadata: metadata ?? null, vector: vector ?? null }
+  }
+
+  /**
+   * Restore an entity's raw stored objects byte-for-byte (a `null` part
+   * deletes that file). Used by crash recovery and transaction aborts to
+   * restore before-images.
+   *
+   * Bypasses the statistics/count bookkeeping of the normal save paths on
+   * purpose: restores must reproduce the exact prior bytes, and the count
+   * rollups are derived state with their own rebuild paths
+   * (`rebuildTypeCounts()` / `rebuildSubtypeCounts()`).
+   *
+   * @param id - The entity id.
+   * @param record - Raw stored objects as returned by {@link BaseStorage.readNounRaw}.
+   */
+  public async writeNounRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise<void> {
+    await this.ensureInitialized()
+    if (record.metadata === null) {
+      await this.deleteObjectFromBranch(getNounMetadataPath(id))
+    } else {
+      await this.writeObjectToBranch(getNounMetadataPath(id), record.metadata)
+    }
+    if (record.vector === null) {
+      await this.deleteObjectFromBranch(getNounVectorPath(id))
+    } else {
+      await this.writeObjectToBranch(getNounVectorPath(id), record.vector)
+    }
+  }
+
+  /**
+   * Read a relationship's raw stored objects (verb-side mirror of
+   * {@link BaseStorage.readNounRaw}).
+   *
+   * @param id - The relationship id.
+   * @returns The raw stored metadata and vector objects (`null` per part when
+   *   the corresponding file is absent).
+   */
+  public async readVerbRaw(id: string): Promise<{ metadata: any | null; vector: any | null }> {
+    await this.ensureInitialized()
+    const [metadata, vector] = await Promise.all([
+      this.readWithInheritance(getVerbMetadataPath(id)),
+      this.readWithInheritance(getVerbVectorPath(id))
+    ])
+    return { metadata: metadata ?? null, vector: vector ?? null }
+  }
+
+  /**
+   * Restore a relationship's raw stored objects byte-for-byte (verb-side
+   * mirror of {@link BaseStorage.writeNounRaw}; same bookkeeping caveats).
+   *
+   * @param id - The relationship id.
+   * @param record - Raw stored objects as returned by {@link BaseStorage.readVerbRaw}.
+   */
+  public async writeVerbRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise<void> {
+    await this.ensureInitialized()
+    if (record.metadata === null) {
+      await this.deleteObjectFromBranch(getVerbMetadataPath(id))
+    } else {
+      await this.writeObjectToBranch(getVerbMetadataPath(id), record.metadata)
+    }
+    if (record.vector === null) {
+      await this.deleteObjectFromBranch(getVerbVectorPath(id))
+    } else {
+      await this.writeObjectToBranch(getVerbVectorPath(id), record.vector)
+    }
+  }
+
+  /**
+   * Append one line to the transaction log (`_system/tx-log.jsonl`). The
+   * filesystem adapter appends to a real JSONL file; the in-memory adapter
+   * keeps a line array (serialized by `snapshotToDirectory`).
+   *
+   * @param line - One complete JSON document, without trailing newline.
+   */
+  public abstract appendTxLogLine(line: string): Promise<void>
+
+  /**
+   * Read all transaction-log lines, oldest first (empty array when no log
+   * exists). A torn trailing line from a crashed append is returned as-is —
+   * callers tolerate unparseable lines.
+   */
+  public abstract readTxLogLines(): Promise<string[]>
+
+  /**
+   * Snapshot the entire store into `targetPath`. The filesystem adapter
+   * builds a hard-link farm (instant, space-shared — safe because data files
+   * are immutable-by-rename); the in-memory adapter serializes its object
+   * store to a filesystem-storage-compatible directory. The result is a
+   * self-contained store openable via `Brainy.load(path)`.
+   *
+   * @param targetPath - Absolute directory path for the snapshot (created if
+   *   missing; must be empty or absent).
+   */
+  public abstract snapshotToDirectory(targetPath: string): Promise<void>
+
+  /**
+   * Replace the entire store's contents from a snapshot directory previously
+   * produced by {@link BaseStorage.snapshotToDirectory}. Implementations
+   * clear current contents (preserving live lock files), copy the snapshot
+   * in (byte copy — never hard links, so the snapshot stays independent),
+   * and then call {@link BaseStorage.reloadDerivedState}.
+   *
+   * @param sourcePath - Absolute path of the snapshot directory.
+   */
+  public abstract restoreFromDirectory(sourcePath: string): Promise<void>
+
+  /**
+   * Reset and reload every piece of adapter-internal derived state after the
+   * underlying objects changed wholesale (restore-from-snapshot): the
+   * write-through cache, the id→type/subtype caches, type/subtype statistics,
+   * total counts, and the graph-index singleton (invalidated so the next
+   * accessor rebuilds from the restored verbs).
+   */
+  protected async reloadDerivedState(): Promise<void> {
+    this.clearWriteCache()
+    this.nounTypeByIdCache.clear()
+    this.nounSubtypeByIdCache.clear()
+    this.verbSubtypeByIdCache.clear()
+    this.nounCountsByType.fill(0)
+    this.verbCountsByType.fill(0)
+    this.subtypeCountsByType.clear()
+    this.verbSubtypeCountsByType.clear()
+    this.statisticsCache = null
+    this.statisticsModified = false
+    this.invalidateGraphIndex()
+    await this.loadTypeStatistics()
+    await this.loadSubtypeStatistics()
+    await this.loadVerbSubtypeStatistics()
+    await this.initializeCounts()
+  }
+
   /**
    * Save a noun to storage (vector only, metadata saved separately)
    * @param noun Pure HNSW vector data (no metadata)
@@ -2315,6 +2580,10 @@ export abstract class BaseStorage extends BaseStorageAdapter {
         // Ignore persist errors - will retry on next operation
       })
     }
+
+    // 8.0 MVCC: entity-visible write — advance the generation watermark
+    // (suppressed inside transact batches by the generation store).
+    this.generationBumpHook?.()
   }
 
   /**
@@ -2717,6 +2986,10 @@ export abstract class BaseStorage extends BaseStorageAdapter {
         this.decrementSubtypeCount(priorType, priorSubtype)
       }
     }
+
+    // 8.0 MVCC: entity-visible write — advance the generation watermark
+    // (suppressed inside transact batches by the generation store).
+    this.generationBumpHook?.()
   }
 
   /**
@@ -2750,6 +3023,8 @@ export abstract class BaseStorage extends BaseStorageAdapter {
       // Backward compatibility: fallback to old path if no verb type
       const keyInfo = this.analyzeKey(id, 'verb-metadata')
       await this.writeObjectToBranch(keyInfo.fullPath, metadata)
+      // 8.0 MVCC: still an entity-visible write — advance the watermark.
+      this.generationBumpHook?.()
       return
     }
 
@@ -2804,6 +3079,10 @@ export abstract class BaseStorage extends BaseStorageAdapter {
         // Ignore persist errors - will retry on next operation
       })
     }
+
+    // 8.0 MVCC: entity-visible write — advance the generation watermark
+    // (suppressed inside transact batches by the generation store).
+    this.generationBumpHook?.()
   }
 
   /**
@@ -2841,6 +3120,10 @@ export abstract class BaseStorage extends BaseStorageAdapter {
       this.verbSubtypeByIdCache.delete(id)
       this.decrementVerbSubtypeCount(priorEntry.verb, priorEntry.subtype)
     }
+
+    // 8.0 MVCC: entity-visible write — advance the generation watermark
+    // (suppressed inside transact batches by the generation store).
+    this.generationBumpHook?.()
   }
 
   // ============================================================================