docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
This commit is contained in:
parent
2626ab8d62
commit
adda1570f3
22 changed files with 570 additions and 2658 deletions
|
|
@ -1,6 +1,6 @@
|
|||
# Initialization and Rebuild Processes
|
||||
|
||||
This document explains how Brainy's four indexes (MetadataIndex, HNSWIndex, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
|
||||
This document explains how Brainy's four indexes (MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
|
||||
|
||||
## Core Principle: All Indexes Are Disk-Based
|
||||
|
||||
|
|
@ -11,7 +11,7 @@ This document explains how Brainy's four indexes (MetadataIndex, HNSWIndex, Grap
|
|||
| Index | Persisted Data | Storage Method | Since Version |
|
||||
|-------|---------------|----------------|---------------|
|
||||
| **MetadataIndex** | Field registry + chunked sparse indices with bloom filters + zone maps | `storage.saveMetadata()` | v3.42.0 (chunks), v4.2.1 (registry) |
|
||||
| **HNSWIndex** | Vector embeddings + HNSW graph connections | `storage.saveHNSWData()` + `storage.saveHNSWSystem()` | v3.35.0 |
|
||||
| **Vector Index** | Vector embeddings + graph connections | `storage.saveHNSWData()` + `storage.saveHNSWSystem()` | v3.35.0 |
|
||||
| **GraphAdjacencyIndex** | Relationships via LSM-tree SSTables | LSM-tree auto-persistence | v3.44.0 |
|
||||
| **DeletedItemsIndex** | Set of deleted IDs | `storage.saveDeletedItems()` | v3.0.0 |
|
||||
|
||||
|
|
@ -32,7 +32,7 @@ The MetadataIndex now persists two components:
|
|||
- Roaring bitmaps for compressed entity ID storage
|
||||
- Loaded on-demand based on query patterns
|
||||
|
||||
All storage operations use the **StorageAdapter** interface, which works with FileSystem, OPFS, S3, GCS, R2, and Memory backends.
|
||||
All storage operations use the **StorageAdapter** interface, which works with FileSystem and Memory backends.
|
||||
|
||||
## Initialization Process
|
||||
|
||||
|
|
@ -84,12 +84,12 @@ async init(): Promise<void> {
|
|||
|
||||
// STEP 2: Check index sizes (lazy initialization triggers here)
|
||||
const metadataStats = await this.metadataIndex.getStats()
|
||||
const hnswIndexSize = this.index.size()
|
||||
const vectorIndexSize = this.index.size()
|
||||
const graphIndexSize = await this.graphIndex.size()
|
||||
|
||||
// STEP 3: Rebuild empty indexes from storage in parallel
|
||||
if (metadataStats.totalEntries === 0 ||
|
||||
hnswIndexSize === 0 ||
|
||||
vectorIndexSize === 0 ||
|
||||
graphIndexSize === 0) {
|
||||
|
||||
const rebuildStartTime = Date.now()
|
||||
|
|
@ -97,7 +97,7 @@ async init(): Promise<void> {
|
|||
metadataStats.totalEntries === 0
|
||||
? this.metadataIndex.rebuild()
|
||||
: Promise.resolve(),
|
||||
hnswIndexSize === 0
|
||||
vectorIndexSize === 0
|
||||
? this.index.rebuild()
|
||||
: Promise.resolve(),
|
||||
graphIndexSize === 0
|
||||
|
|
@ -207,13 +207,13 @@ private async ensureIndexesLoaded(): Promise<void> {
|
|||
### What "Rebuild" Actually Means
|
||||
|
||||
**IMPORTANT**: "Rebuild" does NOT mean recomputing data. It means:
|
||||
1. **Load persisted data** from storage (HNSW connections, metadata chunks, LSM-tree SSTables)
|
||||
1. **Load persisted data** from storage (vector index connections, metadata chunks, LSM-tree SSTables)
|
||||
2. **Populate in-memory structures** (Maps, Sets, graphs)
|
||||
3. **Apply adaptive caching** (preload vectors if small dataset, lazy load if large)
|
||||
|
||||
**Complexity**: O(N) - linear scan through storage, NOT O(N log N) recomputation!
|
||||
|
||||
### 1. HNSWIndex Rebuild (Correct Pattern)
|
||||
### 1. Vector Index Rebuild (Correct Pattern)
|
||||
|
||||
```typescript
|
||||
// src/hnsw/hnswIndex.ts (lines 809-947)
|
||||
|
|
@ -236,7 +236,7 @@ public async rebuild(options: {
|
|||
const availableCache = this.unifiedCache.getRemainingCapacity()
|
||||
const shouldPreload = vectorMemory < availableCache * 0.3
|
||||
|
||||
// STEP 4: Load entities with persisted HNSW connections
|
||||
// STEP 4: Load entities with persisted vector index connections
|
||||
let hasMore = true
|
||||
let cursor: string | undefined = undefined
|
||||
|
||||
|
|
@ -246,7 +246,7 @@ public async rebuild(options: {
|
|||
})
|
||||
|
||||
for (const nounData of result.items) {
|
||||
// Load HNSW graph data from storage (NOT recomputed!)
|
||||
// Load vector graph data from storage (NOT recomputed!)
|
||||
const hnswData = await this.storage.getHNSWData(nounData.id)
|
||||
|
||||
// Create noun with restored connections
|
||||
|
|
@ -274,14 +274,14 @@ public async rebuild(options: {
|
|||
```
|
||||
|
||||
**Key Points**:
|
||||
- ✅ Loads HNSW connections from storage via `getHNSWData()`
|
||||
- ✅ Loads vector index connections from storage via `getHNSWData()`
|
||||
- ✅ Uses adaptive caching (preload vectors if < 30% of available cache)
|
||||
- ✅ O(N) complexity - just loads existing data
|
||||
- ❌ Does NOT call `addItem()` which would recompute connections (O(N log N))
|
||||
|
||||
### 2. TypeAwareHNSWIndex Rebuild (Fixed in v3.45.0)
|
||||
### 2. TypeAwareVectorIndex Rebuild (Fixed in v3.45.0)
|
||||
|
||||
**Critical Architectural Fix**: TypeAwareHNSWIndex previously had TWO major bugs:
|
||||
**Critical Architectural Fix**: The type-aware vector index previously had TWO major bugs:
|
||||
|
||||
1. **Bug #1**: Called `addItem()` during rebuild → O(N log N) recomputation instead of O(N) loading
|
||||
2. **Bug #2**: Loaded ALL nouns 31 times in parallel (once per type) → O(31*N) complexity causing timeouts
|
||||
|
|
@ -300,7 +300,7 @@ public async rebuild(options?: {
|
|||
index.clear()
|
||||
}
|
||||
|
||||
// STEP 2: Determine preloading strategy (same as HNSWIndex)
|
||||
// STEP 2: Determine preloading strategy (same as vector index)
|
||||
const totalNouns = await this.storage.getNounCount()
|
||||
const vectorMemory = totalNouns * 384 * 4
|
||||
const availableCache = this.unifiedCache.getRemainingCapacity()
|
||||
|
|
@ -319,7 +319,7 @@ public async rebuild(options?: {
|
|||
})
|
||||
|
||||
for (const nounData of result.items) {
|
||||
// CORRECT: Load persisted HNSW data (not recomputed!)
|
||||
// CORRECT: Load persisted vector index data (not recomputed!)
|
||||
const hnswData = await this.storage.getHNSWData(nounData.id)
|
||||
|
||||
const noun = {
|
||||
|
|
@ -533,7 +533,7 @@ const unifiedCache = getGlobalCache() // Singleton, 100MB default
|
|||
// MetadataIndex
|
||||
this.unifiedCache = unifiedCache
|
||||
|
||||
// HNSWIndex
|
||||
// Vector index
|
||||
this.unifiedCache = unifiedCache
|
||||
|
||||
// GraphAdjacencyIndex
|
||||
|
|
@ -549,7 +549,7 @@ this.unifiedCache = unifiedCache
|
|||
|
||||
### Rebuild Times (Typical Hardware)
|
||||
|
||||
| Dataset Size | Metadata | HNSW | Graph | Total (Parallel) |
|
||||
| Dataset Size | Metadata | Vector | Graph | Total (Parallel) |
|
||||
|--------------|----------|------|-------|------------------|
|
||||
| 1K entities | 50ms | 100ms | 30ms | **150ms** |
|
||||
| 10K entities | 200ms | 500ms | 150ms | **600ms** |
|
||||
|
|
@ -563,7 +563,7 @@ this.unifiedCache = unifiedCache
|
|||
| Index | In-Memory Overhead | Disk Storage |
|
||||
|-------|-------------------|--------------|
|
||||
| **MetadataIndex** | ~100 bytes/entity | ~500 bytes/entity (chunks) |
|
||||
| **HNSWIndex** | ~200 bytes/entity (no vectors) | ~1.5 KB/entity (vectors + connections) |
|
||||
| **Vector Index** | ~200 bytes/entity (no vectors) | ~1.5 KB/entity (vectors + connections) |
|
||||
| **GraphAdjacencyIndex** | ~128 bytes/relationship | ~200 bytes/relationship (LSM-tree) |
|
||||
| **DeletedItemsIndex** | ~40 bytes/deleted ID | ~50 bytes/deleted ID |
|
||||
|
||||
|
|
@ -573,9 +573,9 @@ this.unifiedCache = unifiedCache
|
|||
|
||||
### O(N) vs O(N log N) Comparison
|
||||
|
||||
**Before fix** (TypeAwareHNSWIndex bug):
|
||||
**Before fix** (TypeAwareVectorIndex bug):
|
||||
```typescript
|
||||
// BAD: Recomputes HNSW connections during rebuild
|
||||
// BAD: Recomputes vector index connections during rebuild
|
||||
for (const noun of nouns) {
|
||||
await index.addItem(noun) // O(log N) per item → O(N log N) total
|
||||
}
|
||||
|
|
@ -652,7 +652,7 @@ console.timeEnd('rebuild')
|
|||
|
||||
// For 10K entities:
|
||||
// - Expected: 500-800ms (loading from storage)
|
||||
// - Bug: 5-10 minutes (recomputing HNSW connections)
|
||||
// - Bug: 5-10 minutes (recomputing vector index connections)
|
||||
```
|
||||
|
||||
**Solution**: Ensure index is loading from storage, not calling `addItem()` during rebuild.
|
||||
|
|
@ -706,8 +706,8 @@ console.log('Nouns in storage:', nouns.items.length)
|
|||
## Version History
|
||||
|
||||
- **v5.7.7** (November 2025): Added production-scale lazy loading with `ensureIndexesLoaded()` helper. Fixed critical bug where `disableAutoRebuild: true` left indexes empty forever. Added concurrency control (mutex) to prevent duplicate rebuilds from concurrent queries. Added `getIndexStatus()` diagnostic method. Zero-config operation - works automatically.
|
||||
- **v3.45.0** (October 2025): Fixed TypeAwareHNSWIndex.rebuild() to load from storage instead of recomputing. Removed all snapshot code (unnecessary with correct rebuild pattern). 200-600x speedup.
|
||||
- **v3.45.0** (October 2025): Fixed type-aware vector index `rebuild()` to load from storage instead of recomputing. Removed all snapshot code (unnecessary with correct rebuild pattern). 200-600x speedup.
|
||||
- **v3.44.0** (October 2025): GraphAdjacencyIndex migrated to LSM-tree storage for billion-scale relationships
|
||||
- **v3.42.0** (October 2025): MetadataIndex migrated to chunked sparse indexing
|
||||
- **v3.35.0** (August 2025): HNSW connections first persisted to storage
|
||||
- **v3.35.0** (August 2025): Vector index connections first persisted to storage
|
||||
- **v3.0.0** (September 2025): Initial 3-tier index architecture
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue