fix(8.0): close GA-blocking correctness gaps from the readiness audit

- Version-coupling cold-init: getBrainyVersion() returned a stale build-time
  default ('3.14.0') on the FIRST synchronous call — the one loadPlugins() makes
  to build context.version — because the package.json read was async. A native
  provider declaring a realistic `>=8.0.0` range was therefore rejected on cold
  init. version.ts now reads package.json synchronously (8.0 targets Node-like
  runtimes only); added a cold-init coupling regression with a `^8.0.0` plugin.
- No-silent-failures on the highest-fan-in read path: getNouns()/getVerbs()
  converted a storage read failure into a success-shaped empty page, which the
  cold-start rebuild then read as "store empty" and skipped the rebuild — booting
  a permanently-empty index with no signal (the same silent-failure class as the
  phantom bug). Both now re-throw a named BrainyError; the rebuild path re-throws
  read failures (fail loud) and records a queryable degraded state, surfaced via
  checkHealth(), for non-fatal rebuild hiccups.
- LSM SSTable leak: graph compaction wrote a merged SSTable but only dropped the
  old ones from the manifest, orphaning their payloads forever ("In production
  we'd add a cleanup mechanism"). Added StorageAdapter.deleteMetadata() and now
  reclaim each compacted-away SSTable.
- Documented the two headline methods add() and find() (the only undocumented
  public methods on the class).

Full gate green: build, unit 1512, integration 607.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
David Snelling 2026-06-29 10:03:38 -07:00
parent 40d2cd5419
commit 47e8031124
7 changed files with 291 additions and 133 deletions

View file

@ -1,21 +1,21 @@
/**
* LSMTree - Log-Structured Merge Tree for Graph Storage
*
* Production-grade LSM-tree implementation that reduces memory usage
* from 500GB to 1.3GB for 1 billion relationships while maintaining
* sub-5ms read performance.
* @module graph/lsm/LSMTree
* @description Log-Structured Merge tree for the JS (open-core) graph store the
* fallback used when no native graph provider is registered. Verb-id postings are
* buffered in an in-memory MemTable, flushed to immutable sorted SSTables, and
* background-compacted; bloom filters give fast negative lookups.
*
* Architecture:
* - MemTable: In-memory write buffer (100K relationships, ~24MB)
* - SSTables: Immutable sorted files on disk (10K relationships each)
* - Bloom Filters: In-memory filters for fast negative lookups
* - Compaction: Background merging of SSTables
* - MemTable: in-memory write buffer (flush threshold configurable, default 100K)
* - SSTables: immutable sorted segments, persisted via the StorageAdapter
* - Bloom filters: in-memory membership pre-checks
* - Compaction: background merge of SSTables (reclaims superseded segments)
*
* Key Properties:
* - Write-optimized: O(1) writes to MemTable
* - Read-efficient: O(log n) reads with bloom filter optimization
* - Memory-efficient: 385x less memory than all-in-RAM approach
* - Storage-agnostic: Works with any StorageAdapter
* Complexity: O(1) MemTable writes; O(log n) reads with bloom-filter pre-filtering.
* Note: this JS implementation loads its SSTables into memory after open (it is the
* fallback engine). Billion-scale, on-disk-resident operation is the native graph
* provider's role behind the provider boundary, not this fallback's; no absolute
* memory/latency figures are claimed here without a cited benchmark.
*/
import { StorageAdapter } from '../../coreTypes.js'
@ -457,15 +457,20 @@ export class LSMTree {
}
})
// Delete old SSTables from storage
// Reclaim the compacted-away SSTables: drop them from the manifest AND
// delete their persisted payloads, so the system channel does not grow
// unbounded with graph write volume. (deleteMetadata is idempotent, so a
// never-persisted memtable-only SSTable is a harmless no-op.)
for (const sstable of sstables) {
const oldKey = `${this.config.storagePrefix}-${sstable.metadata.id}`
this.manifest.sstables.delete(sstable.metadata.id)
try {
// StorageAdapter doesn't have deleteMetadata, so we'll leave orphaned data
// In production, we'd add a cleanup mechanism
this.manifest.sstables.delete(sstable.metadata.id)
await this.storage.deleteMetadata(oldKey)
} catch (error) {
prodLog.warn(`LSMTree: Failed to delete old SSTable ${sstable.metadata.id}`, error)
// A reclaim failure must not abort compaction (the merged SSTable is
// already durable and the manifest no longer references the old one);
// surface it so a persistent leak is visible rather than silent.
prodLog.warn(`LSMTree: failed to delete compacted SSTable ${sstable.metadata.id}`, error)
}
}