brainy/docs/architecture/initialization-and-rebuild.md

Initialization and Rebuild Processes

This document explains how Brainy's four indexes (MetadataIndex, HNSWIndex, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.

Core Principle: All Indexes Are Disk-Based

KEY INSIGHT: All indexes in Brainy are already disk-based. There is no need for snapshots or separate backup mechanisms. Initialization simply loads the right amount of data from storage into memory based on available resources.

What Gets Persisted

Index	Persisted Data	Storage Method	Since Version
MetadataIndex	Field registry + chunked sparse indices with bloom filters + zone maps	storage.saveMetadata()	v3.42.0 (chunks), v4.2.1 (registry)
HNSWIndex	Vector embeddings + HNSW graph connections	storage.saveHNSWData() + storage.saveHNSWSystem()	v3.35.0
GraphAdjacencyIndex	Relationships via LSM-tree SSTables	LSM-tree auto-persistence	v3.44.0
DeletedItemsIndex	Set of deleted IDs	storage.saveDeletedItems()	v3.0.0

MetadataIndex Persistence Details

The MetadataIndex now persists two components:

1. Field Registry (__metadata_field_registry__): Directory of indexed fields for O(1) discovery

   • Size: ~4-8KB (50-200 fields typical)
   • Enables instant cold starts by discovering persisted indices
   • Auto-saved during every flush operation

2. Sparse Indices (__sparse_index__<field>): Per-field index directories

   • Contains chunk metadata, zone maps, and bloom filters
   • Lazy-loaded via UnifiedCache on first query

3. Chunks (__metadata_chunk__<field>_<chunkId>): Actual inverted index data

   • Roaring bitmaps for compressed entity ID storage
   • Loaded on-demand based on query patterns

All storage operations use the StorageAdapter interface, which works with FileSystem, OPFS, S3, GCS, R2, and Memory backends.

Initialization Process

1. Lazy Initialization Pattern

All indexes use lazy initialization - they don't load data until first use:

// Example: GraphAdjacencyIndex
class GraphAdjacencyIndex {
  private initialized = false

  private async ensureInitialized(): Promise<void> {
    if (this.initialized) return

    // Initialize LSM-trees from storage
    await this.lsmTreeSource.init()
    await this.lsmTreeTarget.init()

    this.initialized = true
  }

  // Every public method calls ensureInitialized() first
  async getNeighbors(id: string): Promise<string[]> {
    await this.ensureInitialized()  // Lazy init!
    // ... actual logic
  }
}

Benefits:

• Zero-cost abstraction: No initialization overhead if index not used
• Faster startup: Indexes initialize in parallel on first use
• Lower memory: Only used indexes consume memory

2. Brain Initialization Flow

When you create a Brain instance and call init(), behavior depends on the disableAutoRebuild configuration:

Mode 1: Auto-Rebuild on init() (Default)

// src/brainy.ts (lines 192-237)
async init(): Promise<void> {
  const initStartTime = Date.now()

  // STEP 1: Initialize storage and unified cache
  await this.storage.init()

  // STEP 2: Check index sizes (lazy initialization triggers here)
  const metadataStats = await this.metadataIndex.getStats()
  const hnswIndexSize = this.index.size()
  const graphIndexSize = await this.graphIndex.size()

  // STEP 3: Rebuild empty indexes from storage in parallel
  if (metadataStats.totalEntries === 0 ||
      hnswIndexSize === 0 ||
      graphIndexSize === 0) {

    const rebuildStartTime = Date.now()
    await Promise.all([
      metadataStats.totalEntries === 0
        ? this.metadataIndex.rebuild()
        : Promise.resolve(),
      hnswIndexSize === 0
        ? this.index.rebuild()
        : Promise.resolve(),
      graphIndexSize === 0
        ? this.graphIndex.rebuild()
        : Promise.resolve()
    ])

    const rebuildDuration = Date.now() - rebuildStartTime
    console.log(`✅ All indexes rebuilt in ${rebuildDuration}ms`)
  }

  // STEP 4: Log statistics
  const stats = await this.stats()
  console.log(`📊 Brain initialized with ${stats.entities} entities`)
}

Timeline (typical cold start with 10K entities):

• 0-50ms: Storage adapter initialization
• 50-100ms: Field registry loading (O(1) discovery of persisted indices)
• 100-200ms: Index lazy initialization (LSM-tree loading)
• 200-500ms: Cache warming (preload common fields)
• No rebuild needed! Registry discovers existing indices
• Total: ~0.5-1 second (instant cold starts)

Timeline (cold start WITHOUT field registry - first run only):

• 0-50ms: Storage adapter initialization
• 50-100ms: Index lazy initialization
• 100-2000ms: One-time rebuild to create indices
• Total: ~1-3 seconds (one time only)

Mode 2: Lazy Loading on First Query

When disableAutoRebuild: true, indexes remain empty after init() and rebuild on first query:

// User code
const brain = new Brainy({
  storage: { type: 'filesystem' },
  disableAutoRebuild: true  // Enable lazy loading
})

await brain.init()  // Returns instantly (0-10ms)

// First query triggers lazy rebuild
const results = await brain.find({ limit: 10 })
// → Calls ensureIndexesLoaded() internally (brainy.ts:4617)
// → Rebuilds all 3 main indexes with concurrency control
// → Returns results (~50-200ms total for 1K-10K entities)

// Subsequent queries are instant
const more = await brain.find({ limit: 100 })  // 0ms check, instant

ensureIndexesLoaded() Implementation (brainy.ts:4617-4664):

private async ensureIndexesLoaded(): Promise<void> {
  // Fast path: Already loaded
  if (this.lazyRebuildCompleted) {
    return  // 0ms
  }

  // Concurrency control: Wait for in-progress rebuild
  if (this.lazyRebuildInProgress && this.lazyRebuildPromise) {
    await this.lazyRebuildPromise  // Wait for same rebuild
    return
  }

  // Check if storage has data
  const entities = await this.storage.getNouns({ pagination: { limit: 1 } })
  const hasData = (entities.totalCount && entities.totalCount > 0) || entities.items.length > 0

  if (!hasData) {
    this.lazyRebuildCompleted = true
    return
  }

  // Start lazy rebuild with mutex
  this.lazyRebuildInProgress = true
  this.lazyRebuildPromise = this.rebuildIndexesIfNeeded(true)
    .then(() => {
      this.lazyRebuildCompleted = true
    })
    .finally(() => {
      this.lazyRebuildInProgress = false
      this.lazyRebuildPromise = null
    })

  await this.lazyRebuildPromise
}

Lazy Loading Performance:

• First query: ~50-200ms (1K-10K entities) - triggers rebuild
• Concurrent queries: Wait for same rebuild (mutex prevents duplicates)
• Subsequent queries: 0ms check (instant)
• Zero-config: Works automatically, no code changes needed

Use Cases for Lazy Loading:

• Serverless/Edge: Minimize cold start time, indexes load on demand
• Development: Faster restarts during development
• Large datasets: Defer index loading until actually needed
• Read-heavy workloads: Write operations don't wait for index rebuild

Rebuild Process

What "Rebuild" Actually Means

IMPORTANT: "Rebuild" does NOT mean recomputing data. It means:

1. Load persisted data from storage (HNSW connections, metadata chunks, LSM-tree SSTables)
2. Populate in-memory structures (Maps, Sets, graphs)
3. Apply adaptive caching (preload vectors if small dataset, lazy load if large)

Complexity: O(N) - linear scan through storage, NOT O(N log N) recomputation!

1. HNSWIndex Rebuild (Correct Pattern)

// src/hnsw/hnswIndex.ts (lines 809-947)
public async rebuild(options: {
  lazy?: boolean
  batchSize?: number
  onProgress?: (loaded: number, total: number) => void
} = {}): Promise<void> {
  // STEP 1: Clear in-memory structures
  this.clear()

  // STEP 2: Load system data (entry point, max level)
  const systemData = await this.storage.getHNSWSystem()
  this.entryPointId = systemData.entryPointId
  this.maxLevel = systemData.maxLevel

  // STEP 3: Determine preloading strategy (adaptive caching)
  const totalNouns = await this.storage.getNounCount()
  const vectorMemory = totalNouns * 384 * 4  // 384 dims × 4 bytes
  const availableCache = this.unifiedCache.getRemainingCapacity()
  const shouldPreload = vectorMemory < availableCache * 0.3

  // STEP 4: Load entities with persisted HNSW connections
  let hasMore = true
  let cursor: string | undefined = undefined

  while (hasMore) {
    const result = await this.storage.getNouns({
      pagination: { limit: 1000, cursor }
    })

    for (const nounData of result.items) {
      // Load HNSW graph data from storage (NOT recomputed!)
      const hnswData = await this.storage.getHNSWData(nounData.id)

      // Create noun with restored connections
      const noun: HNSWNoun = {
        id: nounData.id,
        vector: shouldPreload ? nounData.vector : [],  // Adaptive!
        connections: new Map(),
        level: hnswData.level
      }

      // Restore connections from persisted data
      for (const [levelStr, nounIds] of Object.entries(hnswData.connections)) {
        const level = parseInt(levelStr, 10)
        noun.connections.set(level, new Set<string>(nounIds))
      }

      // Just add to memory (no recomputation!)
      this.nouns.set(nounData.id, noun)
    }

    hasMore = result.hasMore
    cursor = result.nextCursor
  }
}

Key Points:

• ✅ Loads HNSW connections from storage via getHNSWData()
• ✅ Uses adaptive caching (preload vectors if < 30% of available cache)
• ✅ O(N) complexity - just loads existing data
• ❌ Does NOT call addItem() which would recompute connections (O(N log N))

2. TypeAwareHNSWIndex Rebuild (Fixed in v3.45.0)

Critical Architectural Fix: TypeAwareHNSWIndex previously had TWO major bugs:

1. Bug #1: Called addItem() during rebuild → O(N log N) recomputation instead of O(N) loading
2. Bug #2: Loaded ALL nouns 31 times in parallel (once per type) → O(31*N) complexity causing timeouts

Both were fixed in v3.45.0 by loading ALL nouns ONCE and routing to correct type indexes:

// src/hnsw/typeAwareHNSWIndex.ts (lines 379-571)
public async rebuild(options?: {
  lazy?: boolean
  batchSize?: number
  onProgress?: (loaded: number, total: number) => void
}): Promise<void> {
  // STEP 1: Clear all type-specific indexes
  for (const index of this.typeIndexes.values()) {
    index.clear()
  }

  // STEP 2: Determine preloading strategy (same as HNSWIndex)
  const totalNouns = await this.storage.getNounCount()
  const vectorMemory = totalNouns * 384 * 4
  const availableCache = this.unifiedCache.getRemainingCapacity()
  const shouldPreload = vectorMemory < availableCache * 0.3

  // STEP 3: Load entities grouped by type
  for (const nounType of ALL_NOUN_TYPES) {
    const index = this.getOrCreateIndex(nounType)
    let hasMore = true
    let cursor: string | undefined = undefined

    while (hasMore) {
      const result = await this.storage.getNouns({
        type: nounType,
        pagination: { limit: 1000, cursor }
      })

      for (const nounData of result.items) {
        // CORRECT: Load persisted HNSW data (not recomputed!)
        const hnswData = await this.storage.getHNSWData(nounData.id)

        const noun = {
          id: nounData.id,
          vector: shouldPreload ? nounData.vector : [],
          connections: new Map(),
          level: hnswData.level
        }

        // Restore connections from storage
        for (const [levelStr, nounIds] of Object.entries(hnswData.connections)) {
          const level = parseInt(levelStr, 10)
          noun.connections.set(level, new Set<string>(nounIds))
        }

        // Add to in-memory index (no recomputation!)
        index.nouns.set(nounData.id, noun)
      }

      hasMore = result.hasMore
      cursor = result.nextCursor
    }
  }
}

Bug Fix: Changed from index.addItem() (recomputation) to direct nouns.set() (restoration).

Performance Impact: 200-600x speedup (5 minutes → 500ms for 10K entities)

Correct Pattern:

// Load ALL nouns ONCE (not 31 times!)
while (hasMore) {
  const result = await storage.getNounsWithPagination({ limit: 1000, cursor })

  for (const noun of result.items) {
    const type = noun.nounType || noun.metadata?.noun
    const index = this.getIndexForType(type)

    // Load persisted HNSW data
    const hnswData = await storage.getHNSWData(noun.id)

    // Restore connections (not recompute!)
    const restoredNoun = {
      id: noun.id,
      vector: shouldPreload ? noun.vector : [],
      connections: restoreConnections(hnswData),
      level: hnswData.level
    }

    // Add to correct type index
    index.nouns.set(noun.id, restoredNoun)
  }

  cursor = result.nextCursor
  hasMore = result.hasMore
}

Performance Improvements:

• 31x speedup: Load nouns ONCE instead of 31 times (O(N) vs O(31*N))
• 200-600x speedup: Load from storage instead of recomputing (O(N) vs O(N log N))
• Combined: ~6000x speedup! (150 minutes → 1.5 seconds for 10K entities)

3. MetadataIndex Rebuild (v4.2.1+ with Field Registry)

v4.2.1 Critical Fix: Field registry persistence eliminates unnecessary rebuilds!

// src/utils/metadataIndex.ts (lines 202-216)
async init(): Promise<void> {
  // STEP 1: Load field registry to discover persisted indices
  // This is THE KEY FIX - O(1) discovery of existing indices
  await this.loadFieldRegistry()

  // If registry found, fieldIndexes Map is now populated
  // getStats() will return totalEntries > 0 → skips rebuild!

  // STEP 2: Initialize EntityIdMapper
  await this.idMapper.init()

  // STEP 3: Warm cache with discovered fields
  await this.warmCache()
}

async loadFieldRegistry(): Promise<void> {
  const registry = await this.storage.getMetadata('__metadata_field_registry__')

  if (registry?.fields) {
    // Populate fieldIndexes Map from discovered fields
    // Sparse indices are lazy-loaded when first accessed
    for (const field of registry.fields) {
      this.fieldIndexes.set(field, {
        values: {},
        lastUpdated: registry.lastUpdated
      })
    }
    // Result: getStats() now returns totalEntries > 0
    // → Brain skips rebuild, cold start in 2-3 seconds!
  }
}

Rebuild Only Happens If:

1. First run (no field registry exists yet)
2. Registry corruption (rare)
3. Explicit rebuild request (manual operation)

// Only runs if field registry not found
async rebuild(): Promise<void> {
  // STEP 1: Clear in-memory structures
  this.fieldIndexes.clear()

  // STEP 2: Load all entity metadata and rebuild indices
  // Sequential batching (25/batch) to prevent socket exhaustion
  // After rebuild: Field registry saved during next flush()

  // One-time cost: ~2-3 seconds for 1K entities
}

Performance Comparison:

Version	Cold Start	Discovery Method	Rebuild Needed?
v4.2.0	8-9 min	None (always rebuild)	Always
v4.2.1	2-3 sec	Field registry O(1)	First run only

Key Points:

• ✅ Field registry enables O(1) discovery (4-8KB file)
• ✅ Sparse indices lazy-loaded on first query
• ✅ Bloom filters + zone maps loaded for fast filtering
• ✅ One-time rebuild on first run, then instant restarts forever
• ✅ Automatic: No configuration needed

4. GraphAdjacencyIndex Rebuild

// src/graph/graphAdjacencyIndex.ts (lines 279-336)
async rebuild(): Promise<void> {
  // STEP 1: Clear in-memory caches
  this.verbIndex.clear()
  this.relationshipCountsByType.clear()

  // STEP 2: Load all verbs from storage
  let hasMore = true
  let cursor: string | undefined = undefined

  while (hasMore) {
    const result = await this.storage.getVerbs({
      pagination: { limit: 1000, cursor }
    })

    for (const verb of result.items) {
      // Add to index (which updates LSM-trees)
      await this.addVerb(verb)
    }

    hasMore = result.hasMore
    cursor = result.nextCursor
  }

  // Note: LSM-trees (lsmTreeSource, lsmTreeTarget) are already
  // initialized from persisted SSTables during ensureInitialized()
}

Key Points:

• ✅ LSM-tree SSTables already loaded during init()
• ✅ Rebuild just repopulates verb cache
• ✅ O(E) complexity where E = number of edges

Adaptive Memory Management

Strategy: Preload vs Lazy Load

All indexes use the UnifiedCache to determine memory allocation:

// Decision logic (in all indexes)
const totalDataSize = estimateDataSize()
const availableCache = unifiedCache.getRemainingCapacity()

if (totalDataSize < availableCache * 0.3) {
  // PRELOAD: Dataset is small relative to available memory
  // Load everything into memory for maximum performance
  shouldPreload = true
} else {
  // LAZY LOAD: Dataset is large
  // Load on-demand with LRU eviction
  shouldPreload = false
}

Thresholds:

• < 30% of available cache: Preload all vectors
• > 30% of available cache: Lazy load on demand

Example (default 100MB cache):

• 10K entities × 1.5KB = 15MB → Preload (15MB < 30MB)
• 100K entities × 1.5KB = 150MB → Lazy load (150MB > 30MB)

UnifiedCache Integration

// All indexes share the same cache
const unifiedCache = getGlobalCache()  // Singleton, 100MB default

// MetadataIndex
this.unifiedCache = unifiedCache

// HNSWIndex
this.unifiedCache = unifiedCache

// GraphAdjacencyIndex
this.unifiedCache = unifiedCache

Benefits:

• Fair resource allocation across indexes
• Prevents any single index from monopolizing memory
• Coordinated LRU eviction system-wide

Performance Characteristics

Rebuild Times (Typical Hardware)

Dataset Size	Metadata	HNSW	Graph	Total (Parallel)
1K entities	50ms	100ms	30ms	150ms
10K entities	200ms	500ms	150ms	600ms
100K entities	1s	3s	1s	3.5s
1M entities	8s	25s	10s	28s

Note: Parallel rebuild means total time ≈ max(individual times), not sum.

Memory Overhead

Index	In-Memory Overhead	Disk Storage
MetadataIndex	~100 bytes/entity	~500 bytes/entity (chunks)
HNSWIndex	~200 bytes/entity (no vectors)	~1.5 KB/entity (vectors + connections)
GraphAdjacencyIndex	~128 bytes/relationship	~200 bytes/relationship (LSM-tree)
DeletedItemsIndex	~40 bytes/deleted ID	~50 bytes/deleted ID

Total overhead (lazy loading):

• In-memory: ~300 bytes per entity + ~128 bytes per relationship
• On-disk: ~2 KB per entity + ~200 bytes per relationship

O(N) vs O(N log N) Comparison

Before fix (TypeAwareHNSWIndex bug):

// BAD: Recomputes HNSW connections during rebuild
for (const noun of nouns) {
  await index.addItem(noun)  // O(log N) per item → O(N log N) total
}
// 10K entities: ~5 minutes

After fix (correct pattern):

// GOOD: Loads connections from storage
for (const noun of nouns) {
  const hnswData = await storage.getHNSWData(noun.id)  // O(1) per item
  noun.connections = restoreConnections(hnswData)      // O(1) per item
  index.nouns.set(noun.id, noun)                       // O(1) per item
}
// 10K entities: ~500ms (600x faster!)

Common Patterns

Cold Start (Empty Storage)

const brain = new Brain({ storage })

// First init: All indexes are empty
await brain.init()
// → No rebuild needed, indexes start empty

// Add data
await brain.add({ content: 'Hello', noun: 'message' })

// Second init: Indexes populated
const brain2 = new Brain({ storage })
await brain2.init()
// → Rebuilds all indexes from storage (~1-3s for 10K entities)

Warm Start (Storage Already Populated)

const brain = new Brain({ storage })

// Init with existing data
await brain.init()
// → Detects non-empty storage
// → Rebuilds indexes in parallel
// → Uses adaptive caching (preload if small, lazy if large)

Manual Rebuild

const brain = new Brain({ storage })
await brain.init()

// Force rebuild (e.g., after data corruption)
await brain.metadataIndex.rebuild()
await brain.index.rebuild()
await brain.graphIndex.rebuild()

Troubleshooting

Slow Rebuild Times

Symptom: Rebuild takes minutes instead of seconds

Diagnosis:

// Check if rebuild is recomputing instead of loading
console.time('rebuild')
await brain.index.rebuild()
console.timeEnd('rebuild')

// For 10K entities:
// - Expected: 500-800ms (loading from storage)
// - Bug: 5-10 minutes (recomputing HNSW connections)

Solution: Ensure index is loading from storage, not calling addItem() during rebuild.

High Memory Usage

Symptom: Memory usage exceeds expectations

Diagnosis:

// Check if vectors are being preloaded
const stats = brain.index.getStats()
console.log('Preloaded vectors:', stats.preloadedVectors)

// Expected:
// - Small dataset (< 30% cache): Most vectors preloaded
// - Large dataset (> 30% cache): Few vectors preloaded

Solution: Adjust UnifiedCache size or force lazy loading:

const brain = new Brain({
  storage,
  cache: { maxSize: 50 * 1024 * 1024 }  // 50MB cache
})

Missing Data After Rebuild

Symptom: Entities disappear after restart

Diagnosis:

// Check storage persistence
const nouns = await storage.getNouns({ pagination: { limit: 10 } })
console.log('Nouns in storage:', nouns.items.length)

// If empty: Storage not persisting
// If populated: Rebuild not loading correctly

Solution: Verify storage adapter is configured correctly (e.g., FileSystem path exists).

Related Documentation

• Index Architecture - Data structures and operations
• Storage Architecture - Storage layer details
• Performance Guide - Performance tuning
• Scaling Guide - Large dataset optimization

Version History

• v5.7.7 (November 2025): Added production-scale lazy loading with ensureIndexesLoaded() helper. Fixed critical bug where disableAutoRebuild: true left indexes empty forever. Added concurrency control (mutex) to prevent duplicate rebuilds from concurrent queries. Added getIndexStatus() diagnostic method. Zero-config operation - works automatically.
• v3.45.0 (October 2025): Fixed TypeAwareHNSWIndex.rebuild() to load from storage instead of recomputing. Removed all snapshot code (unnecessary with correct rebuild pattern). 200-600x speedup.
• v3.44.0 (October 2025): GraphAdjacencyIndex migrated to LSM-tree storage for billion-scale relationships
• v3.42.0 (October 2025): MetadataIndex migrated to chunked sparse indexing
• v3.35.0 (August 2025): HNSW connections first persisted to storage
• v3.0.0 (September 2025): Initial 3-tier index architecture