docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
This commit is contained in:
parent
2626ab8d62
commit
adda1570f3
22 changed files with 570 additions and 2658 deletions
|
|
@ -10,14 +10,14 @@ Brainy has **3 main indexes** at the top level, each with multiple sub-indexes m
|
|||
|
||||
| Index | Purpose | Data Structure | Complexity | File Location | rebuild() Method |
|
||||
|-------|---------|----------------|------------|---------------|------------------|
|
||||
| **TypeAwareHNSWIndex** | Type-aware vector similarity search | 42 type-specific HNSW hierarchical graphs | O(log n) search | `src/hnsw/typeAwareHNSWIndex.ts` | ✅ Line 403 |
|
||||
| **TypeAwareVectorIndex** | Type-aware vector similarity search | 42 type-specific hierarchical graphs | O(log n) search | `src/hnsw/typeAwareHNSWIndex.ts` | ✅ Line 403 |
|
||||
| **MetadataIndexManager** | Fast metadata filtering | Chunked sparse indices with bloom filters + zone maps + roaring bitmaps | O(1) exact, O(log n) ranges | `src/utils/metadataIndex.ts` | ✅ Line 2318 |
|
||||
| **GraphAdjacencyIndex** | Relationship traversal | 4 LSM-trees + bidirectional adjacency maps | O(1) per hop | `src/graph/graphAdjacencyIndex.ts` | ✅ Line 389 |
|
||||
|
||||
### Sub-Indexes (Level 2)
|
||||
|
||||
**TypeAwareHNSWIndex contains:**
|
||||
- **42 type-specific HNSW indexes** - One per NounType (automatically rebuilt via parent)
|
||||
**TypeAwareVectorIndex contains:**
|
||||
- **42 type-specific vector indexes** - One per NounType (automatically rebuilt via parent)
|
||||
|
||||
**MetadataIndexManager contains:**
|
||||
- **ChunkManager** - Adaptive chunked sparse indexing
|
||||
|
|
@ -398,14 +398,16 @@ const DEFAULT_EXCLUDE_FIELDS = [
|
|||
|
||||
**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of they are indexed with automatic bucketing.
|
||||
|
||||
## 2. HNSWIndex - Vector Similarity Search
|
||||
## 2. Vector Index - Vector Similarity Search
|
||||
|
||||
**Purpose**: O(log n) semantic similarity search using vector embeddings.
|
||||
|
||||
The default JS implementation is `JsHnswVectorIndex`; an optional native acceleration package (`@soulcraft/cortex`) can register a higher-performing `VectorIndexProvider` through the plugin system. The public API stays the same either way.
|
||||
|
||||
### Internal Architecture
|
||||
|
||||
```typescript
|
||||
class HNSWIndex {
|
||||
class JsHnswVectorIndex {
|
||||
// Per-noun indexes for efficiency
|
||||
private nouns: Map<string, HNSWNoun> = new Map()
|
||||
|
||||
|
|
@ -437,7 +439,7 @@ class HNSWNode {
|
|||
|
||||
### Hierarchical Graph Structure
|
||||
|
||||
HNSW builds a multi-layered graph:
|
||||
The default vector index builds a multi-layered graph:
|
||||
|
||||
```
|
||||
Layer 2: [entry] ←→ [node1] (sparse, long-range connections)
|
||||
|
|
@ -603,7 +605,7 @@ class UnifiedCache {
|
|||
// Each index gets the same cache instance
|
||||
const unifiedCache = new UnifiedCache({ maxSize: 1000 })
|
||||
this.metadataIndex = new MetadataIndexManager(storage, { unifiedCache })
|
||||
this.hnswIndex = new HNSWIndex(storage, { unifiedCache })
|
||||
this.vectorIndex = new JsHnswVectorIndex(storage, { unifiedCache })
|
||||
this.graphIndex = new GraphAdjacencyIndex(storage, { unifiedCache })
|
||||
```
|
||||
|
||||
|
|
@ -623,7 +625,7 @@ Each index uses different key prefixes:
|
|||
// Metadata index
|
||||
cache.set(`meta:${field}:${value}`, indexEntry)
|
||||
|
||||
// HNSW index
|
||||
// Vector index
|
||||
cache.set(`vector:${id}`, vectorData)
|
||||
|
||||
// Graph index
|
||||
|
|
@ -645,7 +647,7 @@ async add(params: AddParams): Promise<string> {
|
|||
// Add to metadata index (field filtering)
|
||||
await this.metadataIndex.addToIndex(id, params.metadata)
|
||||
|
||||
// Add to HNSW index (vector search)
|
||||
// Add to vector index (vector search)
|
||||
await this.index.addEntity(id, vector, params.noun)
|
||||
|
||||
// Relationships added via separate relate() calls
|
||||
|
|
@ -700,7 +702,7 @@ async update(params: UpdateParams): Promise<void> {
|
|||
await this.metadataIndex.removeFromIndex(params.id, existing.metadata)
|
||||
await this.metadataIndex.addToIndex(params.id, params.metadata)
|
||||
|
||||
// Update HNSW index (re-embed if content changed)
|
||||
// Update vector index (re-embed if content changed)
|
||||
if (params.content) {
|
||||
const newVector = await this.embedder(params.content)
|
||||
await this.index.updateEntity(params.id, newVector)
|
||||
|
|
@ -729,7 +731,7 @@ async stats(): Promise<Statistics> {
|
|||
// From deleted items index
|
||||
deletedItems: this.deletedItemsIndex.getDeletedCount(),
|
||||
|
||||
// From HNSW index
|
||||
// From vector index
|
||||
vectorIndexSize: this.index.getSize()
|
||||
}
|
||||
}
|
||||
|
|
@ -746,16 +748,16 @@ async stats(): Promise<Statistics> {
|
|||
async init(): Promise<void> {
|
||||
// When disableAutoRebuild: false (default)
|
||||
const metadataStats = await this.metadataIndex.getStats()
|
||||
const hnswIndexSize = this.index.size()
|
||||
const vectorIndexSize = this.index.size()
|
||||
const graphIndexSize = await this.graphIndex.size()
|
||||
|
||||
if (metadataStats.totalEntries === 0 ||
|
||||
hnswIndexSize === 0 ||
|
||||
vectorIndexSize === 0 ||
|
||||
graphIndexSize === 0) {
|
||||
// Rebuild all indexes in parallel
|
||||
await Promise.all([
|
||||
metadataStats.totalEntries === 0 ? this.metadataIndex.rebuild() : Promise.resolve(),
|
||||
hnswIndexSize === 0 ? this.index.rebuild() : Promise.resolve(),
|
||||
vectorIndexSize === 0 ? this.index.rebuild() : Promise.resolve(),
|
||||
graphIndexSize === 0 ? this.graphIndex.rebuild() : Promise.resolve()
|
||||
])
|
||||
}
|
||||
|
|
@ -795,7 +797,7 @@ The **TripleIntelligenceSystem** (`src/triple/TripleIntelligenceSystem.ts`) comb
|
|||
class TripleIntelligenceSystem {
|
||||
constructor(
|
||||
private metadataIndex: MetadataIndexManager,
|
||||
private hnswIndex: HNSWIndex,
|
||||
private vectorIndex: VectorIndexProvider,
|
||||
private graphIndex: GraphAdjacencyIndex,
|
||||
private embedder: EmbedderFunction,
|
||||
private storage: BaseStorage
|
||||
|
|
@ -808,7 +810,7 @@ class TripleIntelligenceSystem {
|
|||
// Execute across all three indexes
|
||||
const [metadataResults, vectorResults, graphResults] = await Promise.all([
|
||||
this.metadataIndex.getIdsForFilter(parsed.filters),
|
||||
this.hnswIndex.search(parsed.vector, parsed.limit),
|
||||
this.vectorIndex.search(parsed.vector, parsed.limit),
|
||||
this.graphIndex.traverse(parsed.graphConstraints)
|
||||
])
|
||||
|
||||
|
|
@ -822,7 +824,7 @@ class TripleIntelligenceSystem {
|
|||
|
||||
### Operation Complexity by Index
|
||||
|
||||
| Operation | MetadataIndexManager | TypeAwareHNSWIndex | GraphAdjacencyIndex |
|
||||
| Operation | MetadataIndexManager | TypeAwareVectorIndex | GraphAdjacencyIndex |
|
||||
|-----------|---------------------|-------------------|---------------------|
|
||||
| **Add** | O(1) per field | O(log n) | O(1) |
|
||||
| **Remove** | O(1) per field | O(log n) | O(1) |
|
||||
|
|
@ -837,14 +839,14 @@ Where:
|
|||
- n = total number of entities
|
||||
- k = number of matching results
|
||||
|
||||
**Note**: All 3 main indexes have rebuild() methods that load persisted data (O(n)) rather than recomputing (which would be O(n log n) for HNSW).
|
||||
**Note**: All 3 main indexes have rebuild() methods that load persisted data (O(n)) rather than recomputing (which would be O(n log n) for the vector index).
|
||||
|
||||
### Memory Footprint
|
||||
|
||||
| Index | Per-Entity Memory | Notes |
|
||||
|-------|-------------------|-------|
|
||||
| **MetadataIndexManager** | ~100 bytes | Depends on field count and cardinality (RoaringBitmap32 compression) |
|
||||
| **TypeAwareHNSWIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections across 42 type-specific indexes |
|
||||
| **TypeAwareVectorIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections across 42 type-specific indexes |
|
||||
| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional references + metadata in 4 LSM-trees |
|
||||
|
||||
**Total overhead**: ~1.6 KB per entity + ~50 bytes per relationship
|
||||
|
|
@ -868,7 +870,7 @@ All indexes scale gracefully:
|
|||
**Key observations**:
|
||||
- Graph queries stay O(1) regardless of scale
|
||||
- Metadata filtering scales sub-linearly
|
||||
- Vector search degrades gracefully due to HNSW
|
||||
- Vector search degrades gracefully due to the hierarchical index
|
||||
- Combined queries remain fast even at scale
|
||||
|
||||
## Best Practices
|
||||
|
|
@ -881,7 +883,7 @@ All indexes scale gracefully:
|
|||
- Field discovery (what filters are available)
|
||||
- Type-based querying (find all characters, all items)
|
||||
|
||||
**HNSWIndex**:
|
||||
**Vector Index**:
|
||||
- Semantic similarity search ("find similar documents")
|
||||
- Content-based retrieval ("find posts about AI")
|
||||
- Fuzzy matching (when exact matches aren't required)
|
||||
|
|
@ -906,9 +908,9 @@ All indexes scale gracefully:
|
|||
### Memory Management
|
||||
|
||||
1. **Configure UnifiedCache appropriately** - Balance between speed and memory
|
||||
2. **Use lazy loading** - HNSW loads vectors on-demand
|
||||
2. **Use lazy loading** - Vector index loads vectors on-demand
|
||||
3. **Monitor cache hit rates** - Adjust cache size if hit rate is low
|
||||
4. **Consider storage adapter** - Memory storage = fastest, S3 = most scalable
|
||||
4. **Consider storage adapter** - Memory = fastest, filesystem = persistent
|
||||
|
||||
## Related Documentation
|
||||
|
||||
|
|
@ -922,13 +924,13 @@ All indexes scale gracefully:
|
|||
|
||||
### Level 1: Main Indexes (3)
|
||||
All have rebuild() methods and are covered by lazy loading:
|
||||
1. **TypeAwareHNSWIndex** - `src/hnsw/typeAwareHNSWIndex.ts:403`
|
||||
1. **TypeAwareVectorIndex** - `src/hnsw/typeAwareHNSWIndex.ts:403`
|
||||
2. **MetadataIndexManager** - `src/utils/metadataIndex.ts:2318`
|
||||
3. **GraphAdjacencyIndex** - `src/graph/graphAdjacencyIndex.ts:389`
|
||||
|
||||
### Level 2: Sub-Indexes (~50+)
|
||||
Automatically managed by parent rebuild():
|
||||
- **42 type-specific HNSW indexes** (one per NounType)
|
||||
- **42 type-specific vector indexes** (one per NounType)
|
||||
- **6 metadata components** (ChunkManager, EntityIdMapper, FieldTypeInference, Field Sparse Indexes, Sorted Indexes)
|
||||
- **4 LSM-trees** (lsmTreeSource, lsmTreeTarget, lsmTreeVerbsBySource, lsmTreeVerbsByTarget)
|
||||
- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue