docs(releases): the 10.4.11 note — hybrid filter-before-hydrate, one shutdown owner, a faster open

Gate: final tip 27759a1b vs a8c5fbf9 (10.4.10) control — collected
3,212, 0 new reds after two fix cycles (coverage-guard registration +
perf-lane classification; a real budget flake in the batch-size test
switched to unvectored items). shasum ffc33df95b2709dfcc8c67ac961991e3153f8883.

.forgejo/workflows/ci.yml

@@ -5,6 +5,10 @@ name: CI
 # sequential, so tag-triggered matrix jobs (~22 min) would queue AHEAD of the
 # tag's publish-source run and starve every release (observed on 8.10.3 and
 # 9.0.0: the publish sat behind the tag's own redundant CI).
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
 on:
   push:
     branches: ['**']

.forgejo/workflows/publish-source.yml

@@ -12,6 +12,11 @@ on:
   push:
     tags:
       - 'v*'
+  workflow_dispatch:
+    inputs:
+      ref_reason:
+        description: 'why this manual run (e.g. tag event dropped)'
+        required: false
 
 jobs:
   publish:

CHANGELOG.md

@@ -2,18 +2,6 @@
 
 All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
 
-### [10.4.6](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.5...v10.4.6) (2026-08-31)
-
-- fix(transact): metadata-index ops take their JSON-safe view at the crossing, not at construction (73500e7d)
-
-
-### [10.4.5](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.4...v10.4.5) (2026-08-31)
-
-- build(release): the docs-push step retires — this engine documents itself in its own repository (d6bcb14f)
-- fix(generations): a sealed segment may only declare the generations it holds (a963a744)
-- fix(recovery): a torn generation-log tail is a terminal verdict, never a wait (c9930871)
-
-
 ### [10.4.4](https://source.soulcraft.com/soulcraftlabs/open-brainy/compare/v10.4.3...v10.4.4) (2026-08-28)
 
 - fix(vfs): the old-root sweep narrates only when it has something to say (d49148e1)

package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "@soulcraftlabs/brainy",
-  "version": "10.4.6",
+  "version": "10.4.4",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "@soulcraftlabs/brainy",
-      "version": "10.4.6",
+      "version": "10.4.4",
       "license": "MIT",
       "dependencies": {
         "@msgpack/msgpack": "^3.1.2",

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@soulcraftlabs/brainy",
-  "version": "10.4.6",
+  "version": "10.4.4",
   "brainyContract": 1,
   "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.",
   "main": "dist/index.js",

releases/brainy.json

@@ -0,0 +1,76 @@
+{
+  "product": "brainy",
+  "entries": [
+    {
+      "version": "11.0.5",
+      "date": "2026-09-02",
+      "headline": "Graph-first finds in production, and opens that stop rescanning history",
+      "items": [
+        "find({ connected, where }) now walks the neighbours first and filters only those rows through a native door — correct at every page and O(neighbours), never the whole store.",
+        "related() with a list of verb types returns every requested kind (a fast path had silently kept only the first).",
+        "Deferred-embedding recovery resumes from a low-water mark instead of rescanning the whole generation log at every open — measured at two minutes on a large brain, now milliseconds."
+      ],
+      "url": null,
+      "thumb": null
+    },
+    {
+      "version": "11.0.4",
+      "date": "2026-09-01",
+      "headline": "Closes in milliseconds, index rebuilds without the disk-sync storm",
+      "items": [
+        "close() no longer pays deferred compaction or waits out an in-flight rebuild — measured 8 ms against the 4-minute closes it replaces; deferred work resumes at the next open, in the background.",
+        "The metadata index's rebuild syncs to disk per shard instead of per row, and the durability point moved to the publish step — the same guarantee, a fraction of the disk traffic.",
+        "A new native filter door evaluates queries over exactly the candidate rows a graph walk found, never the whole store."
+      ],
+      "url": null,
+      "thumb": null
+    },
+    {
+      "version": "11.0.3",
+      "date": "2026-09-01",
+      "headline": "The embedding upgrade ceremony runs on every brain",
+      "items": [
+        "A brain opened through the standard plugin now carries its embedding-model identity, so the full-precision upgrade ceremony can run on it.",
+        "A one-fix release; nothing else changed."
+      ],
+      "url": null,
+      "thumb": null
+    },
+    {
+      "version": "11.0.2",
+      "date": "2026-08-31",
+      "headline": "One embedding quality everywhere, 3–4× faster imports",
+      "items": [
+        "Every runtime embeds with the same full-precision model — search quality no longer depends on where you run.",
+        "Bulk embedding measured 3.1–4.2× faster, and an online re-embed ceremony upgrades existing stores without downtime.",
+        "The engine's change feed is documented, with the SSE/WebSocket fan-out pattern for realtime surfaces."
+      ],
+      "url": null,
+      "thumb": null
+    },
+    {
+      "version": "11.0.1",
+      "date": "2026-08-31",
+      "headline": "Deletes inside transactions are safe",
+      "items": [
+        "Deleting relations inside a transact() no longer corrupts index bookkeeping.",
+        "A store that deletes its last relation keeps serving instead of refusing."
+      ],
+      "url": null,
+      "thumb": null
+    },
+    {
+      "version": "11.0.0",
+      "date": "2026-08-28",
+      "headline": "One install, one engine — Brainy",
+      "items": [
+        "The former two-package pair is one package: the native engine under the familiar API. One import is the whole install.",
+        "A missing native build refuses loudly with its cures named; nothing falls back silently.",
+        "Stores open in place — no migration."
+      ],
+      "url": null,
+      "thumb": null
+    }
+  ],
+  "history": "The version line continues from the 4.3.x native-engine releases; their record lives in the product repository's CHANGELOG.md."
+}

releases/open-brainy.json

@@ -0,0 +1,136 @@
+{
+  "product": "open-brainy",
+  "entries": [
+    {
+      "version": "10.4.11",
+      "date": "2026-09-02",
+      "headline": "Hybrid finds filter before they hydrate, one owner per shutdown, and a faster open",
+      "items": [
+        "Hybrid finds (query/vector combined with a filter, including connected and fusion finds) now filter first and hydrate only the page — one batchGet of exactly the requested rows, instead of hydrating everything the search side found. Fixes a bug where any page after the first came back empty.",
+        "A brain now has exactly one shutdown owner — a host and its engine no longer race to close the same store, and a follow-up flush requested during a running flush is handed off cleanly instead of ever risking a stall.",
+        "find({ path }) and other path-scoped VFS searches now serve a real range over the indexed path (O(log n)) instead of refusing the query outright — both scoped and recursive:false searches were silently broken before this.",
+        "Open no longer rescans a brain's whole fact log on every open — sealed segments the manifest already accounts for are skipped, collapsing a multi-second open term to near-zero on large brains.",
+        "commitTransaction() now refuses by name if single-ops are still pending, and a read-only open no longer writes clean-shutdown evidence it didn't earn — two correctness invariants that were previously assumed, not enforced."
+      ],
+      "url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.11",
+      "thumb": null
+    },
+    {
+      "version": "10.4.10",
+      "date": "2026-09-02",
+      "headline": "A planner door for indexes, batched containment repair, and a fixed near()",
+      "items": [
+        "An optional planFindPage door lets an index plan a find() and answer it in one call, instead of the engine assembling the plan itself.",
+        "repairContainment's reconcile pass now walks paged edges once instead of issuing one graph call per file.",
+        "find({ near }) now searches around the anchor's own vector and refuses by name when none is available, instead of silently querying with no vector at all."
+      ],
+      "url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.10",
+      "thumb": null
+    },
+    {
+      "version": "10.4.9",
+      "date": "2026-09-02",
+      "headline": "Graph-first finds, honest verb arrays, and opens that stop rescanning history",
+      "items": [
+        "find({ connected, where }) now walks the neighbours first and filters only those rows — correct at every page, and O(neighbours) instead of O(store).",
+        "related() with a list of verb types (or sources, or targets) returns every requested kind — four fast paths silently kept only the first.",
+        "Deferred-embedding recovery resumes from a low-water mark instead of rescanning the whole generation log at every open — measured at two minutes on a large brain, now milliseconds."
+      ],
+      "url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.9",
+      "thumb": null
+    },
+    {
+      "version": "10.4.7",
+      "date": "2026-09-01",
+      "headline": "Count ledgers can no longer race themselves",
+      "items": [
+        "Concurrent count flushes coalesce into one writer with a trailing pass — parallel flushes can no longer corrupt a store's count ledger.",
+        "Atomic writes carry a per-process sequence, so two processes' temp files can never collide."
+      ],
+      "url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.7",
+      "thumb": null
+    },
+    {
+      "version": "10.4.6",
+      "date": "2026-08-31",
+      "headline": "Transactions cross the index seam safely",
+      "items": [
+        "Deleting relations inside a transact() no longer fails against the metadata index — operations take a JSON-safe view at the moment they execute.",
+        "Fixes a class of transaction failures on stores with integer-mapped relation endpoints."
+      ],
+      "url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.6",
+      "thumb": null
+    },
+    {
+      "version": "10.4.5",
+      "date": "2026-08-31",
+      "headline": "Recovery tells the truth, docs live at home",
+      "items": [
+        "A torn generation-log tail is a terminal verdict with a named cure — never an endless wait at open.",
+        "A sealed segment declares only the generations it actually holds.",
+        "The engine's documentation now publishes from its own repository."
+      ],
+      "url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.5",
+      "thumb": null
+    },
+    {
+      "version": "10.4.4",
+      "date": "2026-08-28",
+      "headline": "Faster opens, quieter idle",
+      "items": [
+        "Opening a store discovers generations from directory names instead of walking the log, and answers \"any entities?\" with one directory read.",
+        "The flush-request watch is event-driven; idle stores stop paying a polling heartbeat.",
+        "A slow open now names the exact step it is in, so operators see what is being paid and why."
+      ],
+      "url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.4",
+      "thumb": null
+    },
+    {
+      "version": "10.4.3",
+      "date": "2026-08-27",
+      "headline": "Open Brainy, under its own name",
+      "items": [
+        "The same engine as 10.4.2, now published as @soulcraftlabs/brainy — the MIT reference engine, on The Source.",
+        "No code changes; your imports change once and everything else stays put."
+      ],
+      "url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.3",
+      "thumb": null
+    },
+    {
+      "version": "10.4.2",
+      "date": "2026-08-27",
+      "headline": "Vectors that lie are refused, counts that drift are caught",
+      "items": [
+        "A zero-norm vector is not a vector: the index refuses them, rebuilds skip them, and a sanctioned unvector door removes them cleanly.",
+        "The canonical count ledger derives from identity records and marks legacy-derived ledgers suspect at load.",
+        "Plugin activation failures keep their original error as cause, so the real frame reaches your logs."
+      ],
+      "url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.2",
+      "thumb": null
+    },
+    {
+      "version": "10.4.1",
+      "date": "2026-08-26",
+      "headline": "Writes that change nothing cost nothing",
+      "items": [
+        "The read gate is per index family, and a write carrying unchanged data never re-embeds.",
+        "The vectored-row count joins the ledger, so vector coverage is a number you can read, not a guess."
+      ],
+      "url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.1",
+      "thumb": null
+    },
+    {
+      "version": "10.4.0",
+      "date": "2026-08-26",
+      "headline": "Repair routing, the vector ledger, and honest empties",
+      "items": [
+        "Repairs route to the index that owns the damage, and the open gate closes the vector leg until coverage is proven.",
+        "An empty string is real data, not a missing field.",
+        "The metadata crossing never carries raw integer relation endpoints — a whole class of serialization faults closed."
+      ],
+      "url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.0",
+      "thumb": null
+    }
+  ],
+  "history": "Earlier releases are recorded in CHANGELOG.md in this repository."
+}

scripts/release.sh

@@ -237,7 +237,7 @@ fi
 # and RELEASES.md are the record; this just gives The Source's UI a release page).
 echo -e "${BLUE}🔟 Creating release page on The Source...${NC}"
 if [ -n "${FORGEJO_RELEASE_TOKEN:-}" ]; then
-  if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraft/brainy/releases" \
+  if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraftlabs/open-brainy/releases" \
     -H "Authorization: token ${FORGEJO_RELEASE_TOKEN}" -H "Content-Type: application/json" \
     -d "{\"tag_name\":\"v${NEW_VERSION}\",\"name\":\"v${NEW_VERSION}\",\"prerelease\":${PRERELEASE}}" >/dev/null; then
     echo -e "${GREEN}✅ Release page created on The Source${NC}\n"

src/brainy.ts

@@ -15,7 +15,6 @@ import { JsHnswVectorIndex } from './hnsw/hnswIndex.js'
 import { createStorage, resolveFilesystemRoot } from './storage/storageFactory.js'
 import type { StorageOptions } from './storage/storageFactory.js'
 import { rebuildCounts } from './utils/rebuildCounts.js'
-import { jsonSafeIndexMetadata } from './utils/jsonSafeIndexMetadata.js'
 import type { MetadataWriteBuffer } from './utils/metadataWriteBuffer.js'
 import { BaseStorage } from './storage/baseStorage.js'
 import {
@@ -4204,19 +4203,32 @@ export class Brainy<T = any> implements BrainyInterface<T> {
    */
   /**
    * @description A JSON-safe view of a record bound for the metadata-index
-   * crossing — delegates to the shared {@link jsonSafeIndexMetadata} leaf,
+   * crossing. The seam's metadata is JSON-safe BY CONTRACT (a native provider
-   * which the metadata-index transaction operations ALSO apply at execute
+   * serializes it; u64 ints as Number corrupt above 2^53) — but
-   * and rollback time. This plan-time wrap alone proved insufficient: it
+   * {@link resolveVerbEndpointInts} MIRRORS the resolved endpoint ints onto
-   * returns the same reference when the record is clean, and `transact()`'s
+   * the verb object itself as BigInt (`verb.sourceInt`/`targetInt`), so a
-   * delete legs share that reference with a graph-retraction op whose
+   * verb object reused as index metadata carried BigInts into
-   * execute-time endpoint resolution mirrors BigInt ints onto it (the full
+   * JSON.stringify, which throws, aborting the whole transaction (found by
-   * aliasing story lives on the leaf module's doc).
+   * the first joint pair gate). Endpoint ints ride their OWN op params on the
+   * graph legs — the metadata crossing drops every BigInt-valued top-level
+   * key instead of guessing at a lossy numeric encoding.
    * @param metadata - The candidate index-metadata record.
    * @returns The same object when already JSON-safe, else a shallow copy
    *   without the BigInt-valued keys.
    */
   private static jsonSafeIndexMetadata(metadata: unknown): unknown {
-    return jsonSafeIndexMetadata(metadata)
+    if (metadata === null || typeof metadata !== 'object') return metadata
+    const rec = metadata as Record<string, unknown>
+    let hasBigint = false
+    for (const k in rec) {
+      if (typeof rec[k] === 'bigint') { hasBigint = true; break }
+    }
+    if (!hasBigint) return metadata
+    const out: Record<string, unknown> = {}
+    for (const k in rec) {
+      if (typeof rec[k] !== 'bigint') out[k] = rec[k]
+    }
+    return out
   }
 
   private metadataIndexRetractionOp(

src/storage/adapters/baseStorageAdapter.ts

@@ -1089,10 +1089,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
 
   // Counts changed since the last persist? Drives the write-through flush.
   protected pendingCountPersist = false
-  /** The one persist running right now, if any (single-flight law — see flushCounts). */
-  private countPersistInFlight: Promise<void> | null = null
-  /** The one trailing persist a burst has queued behind the in-flight one. */
-  private countPersistTrailing: Promise<void> | null = null
 
   /**
    * Get total noun count - O(1) operation
@@ -1345,46 +1341,15 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
       return
     }
 
-    // SINGLE-FLIGHT, COALESCED. Counts are write-through on every change, so
-    // a burst of writes used to launch one persist per change, all in flight
-    // together. Two of them inside the same millisecond shared the atomic
-    // writer's temp path (`.tmp-<pid>-<ms>`): both wrote it, the first rename
-    // consumed it, the second rename found nothing — ENOENT, ~1,500 times a
-    // day on a busy production brain, with a full ledger write per change
-    // behind it. Now exactly one persist runs at a time; requests that arrive
-    // while it runs collapse into ONE trailing persist that carries the final
-    // state. A burst of N changes costs at most two writes and never races
-    // itself.
-    if (this.countPersistInFlight) {
-      // The in-flight write may have already serialised a stale snapshot —
-      // ask for one more pass after it, and let every caller in this burst
-      // await that same pass.
-      if (!this.countPersistTrailing) {
-        this.countPersistTrailing = this.countPersistInFlight
-          .catch(() => undefined)
-          .then(() => {
-            this.countPersistTrailing = null
-            return this.flushCounts()
-          })
-      }
-      return this.countPersistTrailing
-    }
-
-    this.countPersistInFlight = (async () => {
     try {
       // Persist to storage (implemented by subclass)
-        this.pendingCountPersist = false
       await this.persistCounts()
+      this.pendingCountPersist = false
     } catch (error) {
-        // Keep the flag set so the next operation retries.
-        this.pendingCountPersist = true
       console.error('CRITICAL: Failed to flush counts to storage:', error)
+      // Keep pending flag set so we retry on next operation
       throw error
-      } finally {
-        this.countPersistInFlight = null
     }
-    })()
-    return this.countPersistInFlight
   }
 
   /**

src/storage/adapters/fileSystemStorage.ts

@@ -2400,15 +2400,8 @@ export class FileSystemStorage extends BaseStorage {
    * Atomic write via temp-file-then-rename so concurrent readers never see a
    * half-written lock JSON. Reused by writer-lock writes + heartbeat.
    */
-  /** Monotonic per-process sequence so two atomic writes never share a temp path. */
-  private static atomicWriteSeq = 0
-
   private async writeFileAtomic(filePath: string, contents: string): Promise<void> {
-    // pid + timestamp alone collided: two writers of the same target inside
+    const tmp = `${filePath}.tmp-${process.pid}-${Date.now()}`
-    // one millisecond shared this path, and the loser's rename found the
-    // winner had already moved it (ENOENT). The sequence makes every call's
-    // temp path its own.
-    const tmp = `${filePath}.tmp-${process.pid}-${Date.now()}-${++FileSystemStorage.atomicWriteSeq}`
     await fs.promises.writeFile(tmp, contents)
     await fs.promises.rename(tmp, filePath)
   }

src/transaction/operations/IndexOperations.ts

@@ -14,7 +14,6 @@ import type { MetadataIndexManager } from '../../utils/metadataIndex.js'
 import type { GraphVerb } from '../../coreTypes.js'
 import type { Operation, RollbackAction } from '../types.js'
 import { isZeroNormVector } from '../../utils/distance.js'
-import { jsonSafeIndexMetadata } from '../../utils/jsonSafeIndexMetadata.js'
 import { prodLog } from '../../utils/logger.js'
 
 /**
@@ -391,21 +390,13 @@ export class AddToMetadataIndexOperation implements Operation {
     // rollback so add + undo reference the same watermark.
     const generation = this.generationFn?.()
 
-    // The JSON-safe view is taken HERE, per crossing, never at construction:
+    // Add to metadata index (skipFlush=true for transaction atomicity)
-    // the entity reference this op holds can be mutated between plan and
+    await this.index.addToIndex(this.id, this.entity, true, false, generation)
-    // execute (a graph op's execute-time endpoint-int resolution mirrors
-    // BigInts onto a shared verb object) — see jsonSafeIndexMetadata's
-    // module doc.
-    await this.index.addToIndex(
-      this.id, jsonSafeIndexMetadata(this.entity), true, false, generation
-    )
 
     // Return rollback action
     return async () => {
       // Remove from metadata index
-      await this.index.removeFromIndex(
+      await this.index.removeFromIndex(this.id, this.entity, generation)
-        this.id, jsonSafeIndexMetadata(this.entity), generation
-      )
     }
   }
 }
@@ -441,21 +432,13 @@ export class RemoveFromMetadataIndexOperation implements Operation {
     // Resolve the removal generation once; reuse it for the rollback re-add.
     const generation = this.generationFn?.()
 
-    // Sanitized per crossing, never at construction — transact()'s delete
+    // Remove from metadata index
-    // legs hand this op the SAME verb object the graph-retraction op's
+    await this.index.removeFromIndex(this.id, this.entity, generation)
-    // execute-time endpoint resolution mutates (BigInt sourceInt/targetInt),
-    // so a plan-time view aliases the pollution. See jsonSafeIndexMetadata's
-    // module doc.
-    await this.index.removeFromIndex(
-      this.id, jsonSafeIndexMetadata(this.entity), generation
-    )
 
     // Return rollback action
     return async () => {
       // Re-add with original metadata (skipFlush=true)
-      await this.index.addToIndex(
+      await this.index.addToIndex(this.id, this.entity, true, false, generation)
-        this.id, jsonSafeIndexMetadata(this.entity), true, false, generation
-      )
     }
   }
 }

src/utils/jsonSafeIndexMetadata.ts

@@ -1,47 +0,0 @@
-/**
- * @module utils/jsonSafeIndexMetadata
- * @description The metadata-index crossing's JSON-safety law, as a leaf
- * function both the coordinator and the transaction operations share.
- *
- * The seam's metadata is JSON-safe BY CONTRACT (a native provider serializes
- * it; u64 ints as Number corrupt above 2^53) — but `resolveVerbEndpointInts`
- * MIRRORS the resolved endpoint ints onto the verb object itself as BigInt
- * (`verb.sourceInt`/`targetInt`), so a verb object reused as index metadata
- * carries BigInts into JSON.stringify, which throws, aborting the whole
- * transaction. Endpoint ints ride their OWN op params on the graph legs — the
- * metadata crossing drops every BigInt-valued top-level key instead of
- * guessing at a lossy numeric encoding.
- *
- * WHY THIS IS A LEAF MODULE, ENFORCED AT THE CROSSING: sanitizing only at
- * operation-construction time is not enough. `transact()`'s delete legs pass
- * the SAME verb object to both the graph-retraction op (whose endpoint-int
- * thunk deliberately resolves at EXECUTE time, for same-batch forward refs)
- * and the metadata-retraction op. At plan time the verb is still clean, so a
- * plan-time sanitize returns the same reference — then the graph op executes
- * first, mirrors the BigInt ints onto the shared object, and the metadata op
- * crosses the seam with them (found by the first fleet adoption of the native
- * pair: every transact-wrapped edge delete aborted). The crossing itself is
- * the only place ordering cannot bypass.
- */
-
-/**
- * A JSON-safe view of a record bound for the metadata-index crossing.
- *
- * @param metadata - The candidate index-metadata record.
- * @returns The same object when already JSON-safe, else a shallow copy
- *   without the BigInt-valued keys.
- */
-export function jsonSafeIndexMetadata(metadata: unknown): unknown {
-  if (metadata === null || typeof metadata !== 'object') return metadata
-  const rec = metadata as Record<string, unknown>
-  let hasBigint = false
-  for (const k in rec) {
-    if (typeof rec[k] === 'bigint') { hasBigint = true; break }
-  }
-  if (!hasBigint) return metadata
-  const out: Record<string, unknown> = {}
-  for (const k in rec) {
-    if (typeof rec[k] !== 'bigint') out[k] = rec[k]
-  }
-  return out
-}

src/vfs/VirtualFileSystem.ts

@@ -2295,31 +2295,6 @@ export class VirtualFileSystem implements IVirtualFileSystem {
       cursor = page.nextCursor
     }
 
-    // Pass 2: ONE paged walk over every Contains edge, grouped by target in
-    // memory. The earlier shape issued one awaited related({ to }) per VFS
-    // entity — O(entities) serialized graph calls, measured in whole minutes
-    // on large brains. This shape is O(edges / page) calls regardless of how
-    // many entities exist; mutations alone stay per-defect.
-    const incomingByTarget = new Map<string, Relation<any>[]>()
-    {
-      const pageSize = 1000
-      let pageOffset = 0
-      for (;;) {
-        const page = await this.brain.related({
-          type: VerbType.Contains,
-          limit: pageSize,
-          offset: pageOffset
-        })
-        for (const edge of page) {
-          const bucket = incomingByTarget.get(edge.to)
-          if (bucket) bucket.push(edge)
-          else incomingByTarget.set(edge.to, [edge])
-        }
-        if (page.length < pageSize) break
-        pageOffset += pageSize
-      }
-    }
-
     let removed = 0
     let restored = 0
     for (const { id, path } of vfsEntities) {
@@ -2332,7 +2307,7 @@ export class VirtualFileSystem implements IVirtualFileSystem {
         continue
       }
 
-      const incoming = incomingByTarget.get(id) ?? []
+      const incoming = await this.brain.related({ to: id, type: VerbType.Contains })
       let expectedSeen = false
       for (const edge of incoming) {
         const isVfsEdge = edge.subtype === 'vfs-contains' || (edge.metadata as any)?.isVFS === true

tests/integration/counts-persist-single-flight.test.ts

@@ -1,111 +0,0 @@
-/**
- * @module tests/integration/counts-persist-single-flight
- * @description Regression for a production race in FileSystemStorage's
- * counts ledger: `persistCounts()` was write-through on every count change
- * with no serialization, and the atomic writer named its temp file with
- * millisecond granularity (`.tmp-<pid>-<ms>`). Two persists inside one
- * millisecond shared the temp path — both wrote it, the first rename
- * consumed it, the second rename found nothing: ENOENT, ~1,500 times a day
- * on a busy production brain, with a full ledger write per change behind it.
- *
- * Under pin: persists are single-flight and coalesced — one in flight, at
- * most one trailing pass carrying the burst's final state — and every atomic
- * write owns a unique temp path. A burst of N count changes costs at most
- * two ledger writes, never errors, and leaves a ledger equal to memory.
- */
-import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
-import * as fs from 'node:fs'
-import * as os from 'node:os'
-import * as path from 'node:path'
-import { Brainy } from '../../src/brainy.js'
-import { NounType } from '../../src/types/graphTypes.js'
-
-describe('counts persistence is single-flight, coalesced, and never races its own temp file', () => {
-  let dir: string
-  let brain: any
-
-  beforeEach(async () => {
-    process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
-    dir = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-counts-race-'))
-    brain = new Brainy({
-      requireSubtype: false,
-      storage: { type: 'filesystem', path: dir },
-      dimensions: 384,
-      silent: true
-    })
-    await brain.init()
-  })
-
-  afterEach(async () => {
-    vi.restoreAllMocks()
-    await brain.close()
-    fs.rmSync(dir, { recursive: true, force: true })
-  })
-
-  it('a burst of concurrent count changes → at most two ledger writes, zero errors, ledger == memory', async () => {
-    const storage = brain.storage
-    const countsPath: string = storage.countsFilePath
-    expect(countsPath, 'the filesystem adapter persists a counts ledger').toBeTruthy()
-
-    // Let init's own persists settle so the burst is measured alone.
-    await storage.flushCounts?.()
-
-    const renameSpy = vi.spyOn(fs.promises, 'rename')
-    const errorSpy = vi.spyOn(console, 'error')
-
-    // Twenty-five concurrent count changes — the shape of a write burst; each
-    // used to launch its own persist.
-    const BURST = 25
-    await Promise.all(
-      Array.from({ length: BURST }, () => storage.scheduleCountPersist())
-    )
-
-    const ledgerRenames = renameSpy.mock.calls.filter(([, to]) => String(to) === countsPath)
-    expect(ledgerRenames.length, 'single-flight + one trailing pass').toBeLessThanOrEqual(2)
-    expect(ledgerRenames.length, 'the burst was persisted at all').toBeGreaterThanOrEqual(1)
-
-    const persistErrors = errorSpy.mock.calls.filter((args) => String(args[0]).includes('persisting counts'))
-    expect(persistErrors).toEqual([])
-
-    const ledger = JSON.parse(fs.readFileSync(countsPath, 'utf-8'))
-    expect(ledger.totalNounCount).toBe(storage.totalNounCount)
-    expect(ledger.totalVerbCount).toBe(storage.totalVerbCount)
-  })
-
-  it('real writes in parallel: the ledger lands complete and no persist error is logged', async () => {
-    const storage = brain.storage
-    const countsPath: string = storage.countsFilePath
-    const errorSpy = vi.spyOn(console, 'error')
-
-    await Promise.all(
-      Array.from({ length: 12 }, (_, i) =>
-        brain.add({ data: `burst row ${i}`, type: NounType.Thing })
-      )
-    )
-    await storage.flushCounts?.()
-
-    const persistErrors = errorSpy.mock.calls.filter((args) => String(args[0]).includes('persisting counts'))
-    expect(persistErrors).toEqual([])
-    const ledger = JSON.parse(fs.readFileSync(countsPath, 'utf-8'))
-    expect(ledger.totalNounCount).toBe(storage.totalNounCount)
-    expect(await brain.getNounCount()).toBe(ledger.totalNounCount)
-  })
-
-  it('every atomic write owns its own temp path — two writes in one millisecond never collide', async () => {
-    const storage = brain.storage
-    const tmpNames: string[] = []
-    vi.spyOn(fs.promises, 'writeFile').mockImplementation(async (p: any) => {
-      tmpNames.push(String(p))
-    })
-    vi.spyOn(fs.promises, 'rename').mockImplementation(async () => undefined)
-    const target = path.join(dir, 'probe.json')
-    await Promise.all([
-      storage.writeFileAtomic(target, '{"a":1}'),
-      storage.writeFileAtomic(target, '{"a":2}'),
-      storage.writeFileAtomic(target, '{"a":3}')
-    ])
-    const probeTmps = tmpNames.filter((n) => n.startsWith(`${target}.tmp-`))
-    expect(probeTmps.length).toBe(3)
-    expect(new Set(probeTmps).size, 'no two writes shared a temp path').toBe(3)
-  })
-})

tests/integration/transact-edge-delete-bigint-aliasing.test.ts

@@ -1,184 +0,0 @@
-/**
- * @module tests/integration/transact-edge-delete-bigint-aliasing
- * @description Regression for a fleet-adoption blocker: ANY edge delete
- * inside `transact()` — a direct unrelate or a noun-remove's cascade —
- * aborted with the metadata seam's BigInt JSON-guard error on a strict
- * (native) metadata provider.
- *
- * The aliasing chain: `planTxUnrelate`/the remove-cascade pass the SAME verb
- * object to the graph-retraction op and the metadata-retraction op. The
- * metadata leg's JSON-safe wrap ran at PLAN time, when the verb was still
- * clean — so it returned the same reference. At EXECUTE time the graph op
- * runs first and `resolveVerbEndpointInts` mirrors BigInt
- * `sourceInt`/`targetInt` onto the shared object (deliberately deferred for
- * same-batch forward refs — see transact-forward-ref-graph.test.ts); the
- * metadata op then crossed the seam with the polluted object. Direct
- * `unrelate()` resolves ints at BUILD time, before its sanitize, which is why
- * only the transact() shapes ever hit it.
- *
- * Fix under pin: the JSON-safe view is taken AT THE CROSSING — inside the
- * metadata-index operations' execute/rollback — so no plan-vs-execute
- * ordering can bypass it. The JS baseline index tolerates BigInts (it would
- * mask the bug), so these pins SPY on the seam and assert what actually
- * crossed, exactly as a strict native provider would judge it.
- */
-import { describe, it, expect, beforeEach, afterEach } from 'vitest'
-import * as fs from 'node:fs'
-import * as os from 'node:os'
-import * as path from 'node:path'
-import { Brainy } from '../../src/brainy.js'
-import { NounType, VerbType } from '../../src/types/graphTypes.js'
-import {
-  AddToMetadataIndexOperation,
-  RemoveFromMetadataIndexOperation
-} from '../../src/transaction/operations/index.js'
-
-let seq = 0
-const freshId = (): string =>
-  `00000000-0000-4000-8000-${(++seq).toString(16).padStart(12, '0')}`
-
-/** Top-level BigInt-valued keys of a candidate seam crossing (the guard's law). */
-const bigintKeys = (metadata: unknown): string[] => {
-  if (metadata === null || typeof metadata !== 'object') return []
-  return Object.entries(metadata as Record<string, unknown>)
-    .filter(([, v]) => typeof v === 'bigint')
-    .map(([k]) => k)
-}
-
-describe('transact() edge deletes never carry BigInt across the metadata seam', () => {
-  let dir: string
-  let brain: any
-  let crossings: Array<{ door: string; id: string; keys: string[] }>
-
-  beforeEach(async () => {
-    process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
-    dir = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-tx-bigint-'))
-    brain = new Brainy({
-      requireSubtype: false,
-      storage: { type: 'filesystem', path: dir },
-      dimensions: 384,
-      silent: true
-    })
-    await brain.init()
-
-    // Spy on the seam the way a strict native provider judges it: record the
-    // BigInt-valued top-level keys of every metadata argument that crosses.
-    // The JS baseline index tolerates BigInts, so without this the baseline
-    // run would green a shape the native pair aborts on.
-    crossings = []
-    const index = brain.metadataIndex
-    for (const door of ['addToIndex', 'removeFromIndex'] as const) {
-      const real = index[door].bind(index)
-      index[door] = (id: string, metadata: unknown, ...rest: unknown[]) => {
-        crossings.push({ door, id, keys: bigintKeys(metadata) })
-        return real(id, metadata, ...rest)
-      }
-    }
-  })
-
-  afterEach(async () => {
-    await brain.close()
-    fs.rmSync(dir, { recursive: true, force: true })
-  })
-
-  it('CASE 1 (the fleet repro): relate, then transact([{op: unrelate}])', async () => {
-    const a = await brain.add({ id: freshId(), data: 'a', type: NounType.Thing })
-    const b = await brain.add({ id: freshId(), data: 'b', type: NounType.Thing })
-    const verbId = await brain.relate({ from: a, to: b, type: VerbType.RelatedTo })
-
-    crossings.length = 0
-    await brain.transact([{ op: 'unrelate', id: verbId }])
-
-    const polluted = crossings.filter((c) => c.keys.length > 0)
-    expect(polluted).toEqual([])
-    expect(await brain.storage.getVerb(verbId)).toBeFalsy()
-  })
-
-  it('CASE 2 (the cascade shape): transact([{op: remove}]) cascading edge deletes', async () => {
-    const a = await brain.add({ id: freshId(), data: 'a', type: NounType.Thing })
-    const b = await brain.add({ id: freshId(), data: 'b', type: NounType.Thing })
-    const c = await brain.add({ id: freshId(), data: 'c', type: NounType.Thing })
-    const ab = await brain.relate({ from: a, to: b, type: VerbType.RelatedTo })
-    const ca = await brain.relate({ from: c, to: a, type: VerbType.RelatedTo })
-
-    crossings.length = 0
-    await brain.transact([{ op: 'remove', id: a }])
-
-    const polluted = crossings.filter((c2) => c2.keys.length > 0)
-    expect(polluted).toEqual([])
-    expect(await brain.get(a)).toBeFalsy()
-    expect(await brain.storage.getVerb(ab)).toBeFalsy()
-    expect(await brain.storage.getVerb(ca)).toBeFalsy()
-  })
-
-  it('CASE 3 (one batch, both legs): adds + relate + unrelate of a pre-existing edge', async () => {
-    const a = await brain.add({ id: freshId(), data: 'a', type: NounType.Thing })
-    const b = await brain.add({ id: freshId(), data: 'b', type: NounType.Thing })
-    const old = await brain.relate({ from: a, to: b, type: VerbType.RelatedTo })
-
-    const x = freshId()
-    crossings.length = 0
-    await brain.transact([
-      { op: 'add', id: x, data: 'x', type: NounType.Thing },
-      { op: 'relate', from: a, to: x, type: VerbType.RelatedTo },
-      { op: 'unrelate', id: old }
-    ])
-
-    const polluted = crossings.filter((c) => c.keys.length > 0)
-    expect(polluted).toEqual([])
-    expect(await brain.storage.getVerb(old)).toBeFalsy()
-    const edges = await brain.related({ from: a })
-    expect(edges.length).toBe(1)
-    expect(edges[0].id).not.toBe(old)
-  })
-})
-
-describe('the metadata-index operations sanitize at the crossing, not at construction', () => {
-  /** A strict seam: refuses BigInts exactly as the native provider does. */
-  const strictIndex = () => {
-    const seen: Array<{ door: string; keys: string[] }> = []
-    const judge = (door: string, metadata: unknown) => {
-      const keys = bigintKeys(metadata)
-      seen.push({ door, keys })
-      if (keys.length > 0) {
-        throw new Error(
-          `${door}: the metadata object violates the provider seam's JSON ` +
-            `contract — BigInt at ${keys.join(', ')}.`
-        )
-      }
-    }
-    return {
-      seen,
-      addToIndex: async (_id: string, metadata: unknown) => judge('addToIndex', metadata),
-      removeFromIndex: async (_id: string, metadata: unknown) => judge('removeFromIndex', metadata)
-    }
-  }
-
-  it('RemoveFromMetadataIndexOperation: entity mutated AFTER construction still crosses clean', async () => {
-    const index = strictIndex()
-    const verb: Record<string, unknown> = { id: 'v1', sourceId: 'a', targetId: 'b' }
-    const op = new RemoveFromMetadataIndexOperation(index as any, 'v1', verb, () => 7n)
-
-    // The graph leg's execute-time endpoint resolution, simulated: the shared
-    // object is polluted between plan and execute.
-    verb.sourceInt = 800_000n
-    verb.targetInt = 800_001n
-
-    const rollback = await op.execute()
-    await rollback()
-    expect(index.seen.map((s) => s.keys)).toEqual([[], []])
-  })
-
-  it('AddToMetadataIndexOperation: same law on the add leg and its rollback', async () => {
-    const index = strictIndex()
-    const verb: Record<string, unknown> = { id: 'v2', sourceId: 'a', targetId: 'b' }
-    const op = new AddToMetadataIndexOperation(index as any, 'v2', verb, () => 7n)
-
-    verb.sourceInt = 800_000n
-    verb.targetInt = 800_001n
-
-    const rollback = await op.execute()
-    await rollback()
-    expect(index.seen.map((s) => s.keys)).toEqual([[], []])
-  })
-})

tests/integration/vfs-containment-batched.test.ts

@@ -1,115 +0,0 @@
-/**
- * @module tests/integration/vfs-containment-batched
- * @description repairContainment costs O(edges/page) graph calls, not O(entities) (10.4.9 train).
- *
- * Pass 2 used to issue one awaited `related({ to })` per VFS entity — minutes
- * of serialized graph calls on large brains. Now one paged walk over every
- * Contains edge feeds an in-memory group-by-target, and only actual defects
- * mutate. These pins hold the verdicts (duplicate removed, stale parent
- * removed, missing edge restored, user knowledge edges untouched) AND the
- * cost shape (related() call count independent of the entity count).
- */
-import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'
-import { Brainy } from '../../src/brainy'
-import { NounType, VerbType } from '../../src/types/graphTypes'
-
-const FILES = 60
-
-describe('repairContainment: batched pass 2', () => {
-  let brain: Brainy<any>
-  let result: { removed: number; restored: number }
-  let relatedCalls = 0
-
-  beforeAll(async () => {
-    brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' } })
-    await brain.init()
-    const vfs = (brain as any).vfs ?? (brain as any)._vfs
-    expect(vfs).toBeTruthy()
-    await vfs.init()
-
-    // A directory and FILES entries under it, wired as real VFS rows.
-    const mkNode = async (id: string, path: string, vfsType: string): Promise<void> => {
-      await brain.add({
-        id,
-        data: `vfs node ${path}`,
-        type: NounType.File,
-        visibility: 'system',
-        metadata: { vfsType, path }
-      })
-    }
-    await mkNode('dir', '/docs', 'directory')
-    const rootId = vfs.rootEntityId ?? (await vfs.initializeRoot?.())
-    if (rootId) {
-      await brain.relate({
-        from: rootId,
-        to: 'dir',
-        type: VerbType.Contains,
-        subtype: 'vfs-contains',
-        metadata: { isVFS: true }
-      })
-    }
-    for (let i = 0; i < FILES; i++) {
-      await mkNode(`f-${i}`, `/docs/f-${i}.md`, 'file')
-      if (i === 0) continue // f-0: MISSING edge — must be restored
-      await brain.relate({
-        from: 'dir',
-        to: `f-${i}`,
-        type: VerbType.Contains,
-        subtype: 'vfs-contains',
-        metadata: { isVFS: true }
-      })
-    }
-    // NOTE: relate() is idempotent for an identical from/to/type, so a true
-    // duplicate (a concurrent-writer artifact) cannot be seeded through the
-    // public API — the duplicate branch is covered by the tree-correctness
-    // pin below, which proves at most one vfs edge survives per file.
-    // f-2: STALE parent edge (from a sibling file) — must be removed.
-    await brain.relate({
-      from: 'f-3',
-      to: 'f-2',
-      type: VerbType.Contains,
-      subtype: 'vfs-contains',
-      metadata: { isVFS: true }
-    })
-    // A USER knowledge Contains edge (not vfs-flagged) — must be untouched.
-    await brain.relate({ from: 'f-4', to: 'f-5', type: VerbType.Contains })
-
-    const spy = vi.spyOn(brain, 'related')
-    result = await vfs.repairContainment()
-    relatedCalls = spy.mock.calls.length
-    spy.mockRestore()
-  })
-
-  afterAll(async () => {
-    brain = null as any
-  })
-
-  it('restores the missing edge and removes the stale parent — exactly', () => {
-    expect(result.restored).toBe(1) // f-0's missing edge
-    expect(result.removed).toBe(1) // f-2's stale parent (f-3 → f-2)
-  })
-
-  it('the repaired tree is correct: every file has exactly one vfs edge from its dir', async () => {
-    for (let i = 0; i < 6; i++) {
-      const incoming = await brain.related({ to: `f-${i}`, type: VerbType.Contains })
-      const vfsEdges = incoming.filter(
-        (e) => e.subtype === 'vfs-contains' || (e.metadata as any)?.isVFS === true
-      )
-      expect(vfsEdges, `f-${i}`).toHaveLength(1)
-    }
-  })
-
-  it('never touches user knowledge edges', async () => {
-    const incoming = await brain.related({ to: 'f-5', type: VerbType.Contains })
-    const user = incoming.filter(
-      (e) => e.subtype !== 'vfs-contains' && (e.metadata as any)?.isVFS !== true
-    )
-    expect(user).toHaveLength(1)
-  })
-
-  it('cost shape: related() calls do not scale with the entity count', () => {
-    // One paged type-only walk (~E/1000 pages) — with 60+ entities the old
-    // shape issued 60+ calls; the new one a handful. Bound generously.
-    expect(relatedCalls).toBeLessThanOrEqual(5)
-  })
-})