fix(8.0): close GA-blocking correctness gaps from the readiness audit
- Version-coupling cold-init: getBrainyVersion() returned a stale build-time
default ('3.14.0') on the FIRST synchronous call — the one loadPlugins() makes
to build context.version — because the package.json read was async. A native
provider declaring a realistic `>=8.0.0` range was therefore rejected on cold
init. version.ts now reads package.json synchronously (8.0 targets Node-like
runtimes only); added a cold-init coupling regression with a `^8.0.0` plugin.
- No-silent-failures on the highest-fan-in read path: getNouns()/getVerbs()
converted a storage read failure into a success-shaped empty page, which the
cold-start rebuild then read as "store empty" and skipped the rebuild — booting
a permanently-empty index with no signal (the same silent-failure class as the
phantom bug). Both now re-throw a named BrainyError; the rebuild path re-throws
read failures (fail loud) and records a queryable degraded state, surfaced via
checkHealth(), for non-fatal rebuild hiccups.
- LSM SSTable leak: graph compaction wrote a merged SSTable but only dropped the
old ones from the manifest, orphaning their payloads forever ("In production
we'd add a cleanup mechanism"). Added StorageAdapter.deleteMetadata() and now
reclaim each compacted-away SSTable.
- Documented the two headline methods add() and find() (the only undocumented
public methods on the class).
Full gate green: build, unit 1512, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
40d2cd5419
commit
47e8031124
7 changed files with 291 additions and 133 deletions
|
|
@ -1,21 +1,21 @@
|
|||
/**
|
||||
* LSMTree - Log-Structured Merge Tree for Graph Storage
|
||||
*
|
||||
* Production-grade LSM-tree implementation that reduces memory usage
|
||||
* from 500GB to 1.3GB for 1 billion relationships while maintaining
|
||||
* sub-5ms read performance.
|
||||
* @module graph/lsm/LSMTree
|
||||
* @description Log-Structured Merge tree for the JS (open-core) graph store — the
|
||||
* fallback used when no native graph provider is registered. Verb-id postings are
|
||||
* buffered in an in-memory MemTable, flushed to immutable sorted SSTables, and
|
||||
* background-compacted; bloom filters give fast negative lookups.
|
||||
*
|
||||
* Architecture:
|
||||
* - MemTable: In-memory write buffer (100K relationships, ~24MB)
|
||||
* - SSTables: Immutable sorted files on disk (10K relationships each)
|
||||
* - Bloom Filters: In-memory filters for fast negative lookups
|
||||
* - Compaction: Background merging of SSTables
|
||||
* - MemTable: in-memory write buffer (flush threshold configurable, default 100K)
|
||||
* - SSTables: immutable sorted segments, persisted via the StorageAdapter
|
||||
* - Bloom filters: in-memory membership pre-checks
|
||||
* - Compaction: background merge of SSTables (reclaims superseded segments)
|
||||
*
|
||||
* Key Properties:
|
||||
* - Write-optimized: O(1) writes to MemTable
|
||||
* - Read-efficient: O(log n) reads with bloom filter optimization
|
||||
* - Memory-efficient: 385x less memory than all-in-RAM approach
|
||||
* - Storage-agnostic: Works with any StorageAdapter
|
||||
* Complexity: O(1) MemTable writes; O(log n) reads with bloom-filter pre-filtering.
|
||||
* Note: this JS implementation loads its SSTables into memory after open (it is the
|
||||
* fallback engine). Billion-scale, on-disk-resident operation is the native graph
|
||||
* provider's role behind the provider boundary, not this fallback's; no absolute
|
||||
* memory/latency figures are claimed here without a cited benchmark.
|
||||
*/
|
||||
|
||||
import { StorageAdapter } from '../../coreTypes.js'
|
||||
|
|
@ -457,15 +457,20 @@ export class LSMTree {
|
|||
}
|
||||
})
|
||||
|
||||
// Delete old SSTables from storage
|
||||
// Reclaim the compacted-away SSTables: drop them from the manifest AND
|
||||
// delete their persisted payloads, so the system channel does not grow
|
||||
// unbounded with graph write volume. (deleteMetadata is idempotent, so a
|
||||
// never-persisted memtable-only SSTable is a harmless no-op.)
|
||||
for (const sstable of sstables) {
|
||||
const oldKey = `${this.config.storagePrefix}-${sstable.metadata.id}`
|
||||
this.manifest.sstables.delete(sstable.metadata.id)
|
||||
try {
|
||||
// StorageAdapter doesn't have deleteMetadata, so we'll leave orphaned data
|
||||
// In production, we'd add a cleanup mechanism
|
||||
this.manifest.sstables.delete(sstable.metadata.id)
|
||||
await this.storage.deleteMetadata(oldKey)
|
||||
} catch (error) {
|
||||
prodLog.warn(`LSMTree: Failed to delete old SSTable ${sstable.metadata.id}`, error)
|
||||
// A reclaim failure must not abort compaction (the merged SSTable is
|
||||
// already durable and the manifest no longer references the old one);
|
||||
// surface it so a persistent leak is visible rather than silent.
|
||||
prodLog.warn(`LSMTree: failed to delete compacted SSTable ${sstable.metadata.id}`, error)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue