feat(open-path): init never gates on the embedding model; open goes concurrent; slow opens narrate

A production restart storm measured 90,017ms for a single brain init vs
1,117ms quiet (~80x contention multiplier), traced to performInit() eagerly
awaiting the process-global WASM embedding engine before the VFS root even
existed. Every writer's open() queued on the one throttled model compile
(90-140s on throttled CPUs).

- VirtualFileSystem.doInitializeRoot() no longer embeds '/'. The root is
  system-tier plumbing nothing ever searches; when the default WASM engine
  is active it now gets an explicit all-zero placeholder vector
  (cosineDistance returns max distance for a zero vector, so it never ranks
  ahead of real content). deferEmbedding was considered and rejected: its
  landing path kicks the embed worker synchronously right after commit,
  which would still force the cold compile within milliseconds — just off
  the awaited path, not avoided. A registered native 'embeddings' provider
  (no cold-start cost, possibly a different dimension) still embeds the
  root for real, via the new Brainy.usesDefaultWasmEmbedder() seam.

- performInit()'s eager-embedding step now only STARTS the WASM engine warm
  in the background instead of awaiting it inline. embed()/embeddingManager
  already serialize concurrent callers on one shared init promise, so the
  first real embed() converges correctly either way; a failed warm narrates
  loudly instead of surfacing as a silent latency spike or an unhandled
  rejection. eagerEmbeddings: false still means no warm at all.

- FileSystemStorage.init() batches its ~8 independent bootstrap mkdirs
  (each creates its own full subtree via recursive:true, so none depend on
  the others existing) into one Promise.all. The restore-completion step
  and initializeCounts() stay strictly sequential — they have real order
  dependencies on rootDir and systemDir respectively.

- performInit() now times five phases (storage init / generation-store
  open+fold / index init+gate / VFS bootstrap / embedding-warm-started) and
  logs one warning with the per-phase breakdown when total open exceeds
  2000ms; silent otherwise.
This commit is contained in:
David Snelling 2026-08-25 10:09:45 -07:00
parent f8f64780b1
commit 96624f408c
5 changed files with 446 additions and 53 deletions

View file

@ -239,38 +239,54 @@ export class FileSystemStorage extends BaseStorage {
// Finish any restore interrupted by a crash (resume the staged swap, or
// discard an uncommitted staging area) BEFORE counts/derived state load,
// so the rest of startup sees the completed store.
// so the rest of startup sees the completed store. ORDER-DEPENDENT:
// `swapStagedRestoreIn()` reads `fs.readdir(rootDir)` and then
// removes/renames rootDir's own TOP-LEVEL entries to place the staged
// copy — racing that against the directory-creation batch below (which
// also touches rootDir's children) could see a half-created directory
// mid-swap or a mkdir racing a concurrent rm/rename on the same path.
// Stays strictly sequential, never folded into the OPEN-PATH batch.
await this.completeInterruptedRestore()
// Create the nouns directory if it doesn't exist
await this.ensureDirectoryExists(this.nounsDir)
// OPEN-PATH FIX: the remaining bootstrap directories are mutually
// independent — each is its own subtree under rootDir, and
// `fs.mkdir(dir, { recursive: true })` creates every intermediate
// segment of ITS OWN path in one call, so it never depends on any
// sibling here existing first. Nothing between here and
// `initializeCounts()` reads any of them, so batching collapses what
// was up to 8 sequential mkdir round-trips (each a real syscall+await)
// into one wave — this is what serialized an N-writer restart storm on
// filesystem I/O it never structurally needed. `initializeCounts()`
// right after DOES depend on `systemDir` (which the batch creates), so
// it stays outside, awaited only once every directory has landed.
await Promise.all([
// Create the nouns directory if it doesn't exist
this.ensureDirectoryExists(this.nounsDir),
// Create the verbs directory if it doesn't exist
this.ensureDirectoryExists(this.verbsDir),
// Create the metadata directory if it doesn't exist
this.ensureDirectoryExists(this.metadataDir),
// Create the noun metadata directory if it doesn't exist
this.ensureDirectoryExists(this.nounMetadataDir),
// Create the verb metadata directory if it doesn't exist
this.ensureDirectoryExists(this.verbMetadataDir),
// Create both directories for backward compatibility
this.ensureDirectoryExists(this.systemDir),
// Only create legacy directory if it exists (don't create new legacy
// dirs) — a read-then-maybe-write, but on its own subtree, so it's
// still independent of every other entry in this batch.
(async () => {
if (await this.directoryExists(this.indexDir)) {
await this.ensureDirectoryExists(this.indexDir)
}
})(),
// Create the locks directory if it doesn't exist
this.ensureDirectoryExists(this.lockDir),
// Create the binary blobs directory if it doesn't exist
this.ensureDirectoryExists(this.blobsDir)
])
// Create the verbs directory if it doesn't exist
await this.ensureDirectoryExists(this.verbsDir)
// Create the metadata directory if it doesn't exist
await this.ensureDirectoryExists(this.metadataDir)
// Create the noun metadata directory if it doesn't exist
await this.ensureDirectoryExists(this.nounMetadataDir)
// Create the verb metadata directory if it doesn't exist
await this.ensureDirectoryExists(this.verbMetadataDir)
// Create both directories for backward compatibility
await this.ensureDirectoryExists(this.systemDir)
// Only create legacy directory if it exists (don't create new legacy dirs)
if (await this.directoryExists(this.indexDir)) {
await this.ensureDirectoryExists(this.indexDir)
}
// Create the locks directory if it doesn't exist
await this.ensureDirectoryExists(this.lockDir)
// Create the binary blobs directory if it doesn't exist
await this.ensureDirectoryExists(this.blobsDir)
// Initialize count management
// Initialize count management — depends on systemDir, created above.
this.countsFilePath = path.join(this.systemDir, 'counts.json')
await this.initializeCounts()