soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix: exclude __words__ keyword index from corruption detection and getStats()

Browse source 
The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:

1. validateConsistency() to falsely detect corruption on every startup,
   triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
   and report inflated totalEntries/totalIds stats

Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
This commit is contained in:
David Snelling 2026-01-27 15:38:21 -08:00
parent 32dbdcec61
commit 364360d447
128 changed files with 5637 additions and 5682 deletions
Download patch file Download diff file Expand all files Collapse all files

40
docs/API_REFERENCE.md
View file

@ -90,8 +90,8 @@ Adds a new entity to the brain.
- `id` - Custom ID (auto-generated if not provided)
- `vector` - Pre-computed embedding vector
- `service` - Service name for multi-tenancy
- `confidence` - Type classification confidence (0-1) ✨ *New in v4.3.0*
- `weight` - Entity importance/salience (0-1) ✨ *New in v4.3.0*
- `confidence` - Type classification confidence (0-1) ✨
- `weight` - Entity importance/salience (0-1) ✨
- `writeOnly` - Skip validation for high-speed ingestion
**Returns:** Entity ID
@ -116,7 +116,7 @@ const id = await brain.add({
### `async get(id: string, options?: GetOptions): Promise<Entity | null>`
Retrieves an entity by ID.
✨ **v5.11.1 Performance Optimization**:
✨ **Performance Optimization**:
- **Default (metadata-only)**: 76-81% faster, 95% less bandwidth - perfect for VFS, existence checks, metadata access
- **Opt-in vectors**: Use `{ includeVectors: true }` when computing similarity
@ -127,7 +127,7 @@ Retrieves an entity by ID.
**Returns:** Entity object or null if not found
**Entity Properties:** ✨ *Updated in v4.3.0, v5.11.1*
**Entity Properties:**
- `id` - Unique identifier
- `type` - NounType classification
- `data` - Original content
@ -172,8 +172,8 @@ Updates an existing entity.
- `metadata` - New or partial metadata
- `merge` - Merge metadata (true) or replace (false), default: true
- `vector` - New embedding vector
- `confidence` - Update type classification confidence ✨ *New in v4.3.0*
- `weight` - Update entity importance/salience ✨ *New in v4.3.0*
- `confidence` - Update type classification confidence ✨
- `weight` - Update entity importance/salience ✨
**Example:**
```typescript
@ -406,7 +406,7 @@ Universal search with Triple Intelligence fusion.
**Returns:** Array of Result objects with scores
**Result Properties:** ✨ *Enhanced in v4.3.0*
**Result Properties:** ✨
- `id` - Entity ID
- `score` - Relevance score (0-1)
- `type` - Entity type (flattened for convenience) *Enhanced*
@ -422,7 +422,7 @@ Universal search with Triple Intelligence fusion.
// Natural language search
const results = await brain.find('recent product launches')
// NEW in v4.3.0: Direct access to flattened fields
// Direct access to flattened fields
console.log(results[0].metadata) // Direct access (convenient!)
console.log(results[0].confidence) // Type confidence
console.log(results[0].weight) // Entity importance
@ -469,7 +469,7 @@ Finds similar entities using vector similarity.
- `type` - Filter by type(s)
- `where` - Metadata filters
**Returns:** Array of Result objects (same structure as `find()`) ✨ *Enhanced in v4.3.0*
**Returns:** Array of Result objects (same structure as `find()`) ✨
**Example:**
```typescript
@ -480,7 +480,7 @@ const similar = await brain.similar({
type: NounType.Document
})
// NEW in v4.3.0: Access flattened fields directly
// Access flattened fields directly
for (const result of similar) {
console.log(`Similarity: ${result.score}`)
console.log(`Type: ${result.type}`) // Flattened
@ -491,7 +491,7 @@ for (const result of similar) {
---
### `async embed(data: any): Promise<Vector>` ✨ *v7.1.0*
### `async embed(data: any): Promise<Vector>`
Generates an embedding vector from data.
**Parameters:**
@ -507,7 +507,7 @@ console.log(vector.length) // 384
---
### `async embedBatch(texts: string[]): Promise<number[][]>` ✨ *v7.1.0, Optimized v7.9.0*
### `async embedBatch(texts: string[]): Promise<number[][]>`
Batch embed multiple texts using native WASM batch API (single forward pass).
**Parameters:**
@ -515,7 +515,7 @@ Batch embed multiple texts using native WASM batch API (single forward pass).
**Returns:** Array of 384-dimensional vectors
> **v7.9.0**: Uses the engine's native `embed_batch()` for a single model forward pass instead of N individual `embed()` calls.
> Uses the engine's native `embed_batch()` for a single model forward pass instead of N individual `embed()` calls.
**Example:**
```typescript
@ -530,7 +530,7 @@ console.log(embeddings[0].length) // 384
---
### `async similarity(textA: string, textB: string): Promise<number>` ✨ *v7.1.0*
### `async similarity(textA: string, textB: string): Promise<number>`
Calculate semantic similarity between two texts.
**Parameters:**
@ -550,7 +550,7 @@ console.log(score) // ~0.85 (high semantic similarity)
---
### `async neighbors(entityId: string, options?): Promise<string[]>` ✨ *v7.1.0*
### `async neighbors(entityId: string, options?): Promise<string[]>`
Get graph neighbors of an entity.
**Parameters:**
@ -582,7 +582,7 @@ const extended = await brain.neighbors(entityId, {
---
### `async findDuplicates(options?): Promise<DuplicateResult[]>` ✨ *v7.1.0*
### `async findDuplicates(options?): Promise<DuplicateResult[]>`
Find semantic duplicates in the database.
**Parameters:**
@ -614,7 +614,7 @@ const personDupes = await brain.findDuplicates({
---
### `async indexStats(): Promise<IndexStats>` ✨ *v7.1.0*
### `async indexStats(): Promise<IndexStats>`
Get comprehensive index statistics.
**Returns:**
@ -639,7 +639,7 @@ console.log(`Fields: ${stats.metadataFields.join(', ')}`)
---
### `async cluster(options?): Promise<ClusterResult[]>` ✨ *v7.1.0*
### `async cluster(options?): Promise<ClusterResult[]>`
Cluster entities by semantic similarity.
Groups entities into clusters based on their embedding similarity using
@ -1665,7 +1665,7 @@ Executes the pipeline.
### Core Interfaces
✨ *Updated in v4.3.0 - Added confidence/weight to Entity, flattened Result fields*
✨ *Updated in Added confidence/weight to Entity, flattened Result fields*
```typescript
interface Entity<T = any> {
@ -1714,7 +1714,7 @@ interface Result<T = any> {
}
```
**Key Changes in v4.3.0:**
**Key Changes:**
- ✅ `Entity` now exposes `confidence` and `weight`
- ✅ `Result` flattens commonly-used entity fields to top level
- ✅ Direct access: `result.metadata` instead of `result.entity.metadata`

23
docs/BATCHING.md
View file

@ -1,23 +1,22 @@
# Batch Operations API v5.12.0
# Batch Operations API
> **Enterprise Production-Ready** | Zero N+1 Query Patterns | 90%+ Performance Improvement
## Overview
Brainy v5.12.0 introduces comprehensive batch operations at the storage layer, eliminating N+1 query patterns and dramatically improving performance for VFS operations, relationship queries, and entity retrieval on cloud storage.
Brainy introduces comprehensive batch operations at the storage layer, eliminating N+1 query patterns and dramatically improving performance for VFS operations, relationship queries, and entity retrieval on cloud storage.
### Problem Solved
**Before v5.12.0:**
**Before optimization:**
- VFS `getTreeStructure()` on cloud storage: **12.7 seconds** for directory with 12 files
- N+1 query pattern: 1 directory query + N individual file queries (22 sequential calls × 580ms latency)
**After v5.12.0:**
**After optimization:**
- VFS `getTreeStructure()`: **<1 second** for 12 files (90%+ improvement)
- 2-3 batched calls instead of 22 sequential calls
- Native cloud storage batch APIs for maximum throughput
**IMPORTANT:** The v5.12.0 batch optimizations apply **ONLY to `getTreeStructure()`**, not to `readFile()` or individual `get()` operations. See v6.0.0 changes for comprehensive storage path optimizations.
**IMPORTANT:** The batch optimizations apply **ONLY to `getTreeStructure()`**, not to `readFile()` or individual `get()` operations. See changes for comprehensive storage path optimizations.
---
@ -56,7 +55,7 @@ results.size // → 3 (number of found entities)
### 2. `storage.getNounMetadataBatch(ids)`
Batch metadata retrieval with direct O(1) path construction (v6.0.0+).
Batch metadata retrieval with direct O(1) path construction.
```typescript
const storage = brain.storage as BaseStorage
@ -254,7 +253,7 @@ VFS operations automatically use batch APIs for maximum performance.
// OLD: Sequential N+1 pattern (12.7 seconds for 12 files)
const tree = await brain.vfs.getTreeStructure('/my-dir')
// NEW v5.12.0: Parallel breadth-first with batching (<1 second)
// NEW Parallel breadth-first with batching (<1 second)
// ✅ PathResolver.getChildren() uses brain.batchGet() internally
// ✅ Parallel traversal of directories at same tree level
// ✅ 2-3 batched calls instead of 22 sequential calls
@ -285,11 +284,11 @@ VFS.getTreeStructure()
## Advanced Features Compatibility
### ✅ ID-First Storage Architecture (v6.0.0+)
### ✅ ID-First Storage Architecture
All batch operations use direct ID-first paths - no type lookup needed!
**NEW v6.0.0 Path Structure:**
**ID-First Path Structure:**
```
entities/nouns/{SHARD}/{ID}/metadata.json
entities/verbs/{SHARD}/{ID}/metadata.json
@ -391,7 +390,7 @@ Historical queries use `HistoricalStorageAdapter` which wraps batch operations t
### VFS Operations (12 Files)
| Storage | Before v5.12.0 | After v5.12.0 | Improvement |
| Storage | Before optimization | After optimization | Improvement |
|---------|---------------|---------------|-------------|
| **GCS** | 12.7s | <1s | **92% faster** |
| **S3** | 13.2s | <1s | **92% faster** |
@ -658,7 +657,7 @@ if (typeof selfWithBatch.readBatch === 'function') {
- Zero N+1 query patterns
**Compatibility:**
- ✅ ID-first storage (v6.0.0+)
- ✅ ID-first storage
- ✅ Sharding (256 shards)
- ✅ COW (branch isolation, inheritance)
- ✅ fork() and checkout()

20
docs/CREATING-AUGMENTATIONS.md
View file

@ -1,6 +1,6 @@
# Creating Augmentations for Brainy
> **Updated for v4.0.0** - Includes metadata structure changes and type system improvements
> **Updated** - Includes metadata structure changes and type system improvements
## The BrainyAugmentation Interface
@ -23,13 +23,13 @@ interface BrainyAugmentation {
}
```
## v4.0.0 Breaking Changes for Augmentation Developers
## Breaking Changes for Augmentation Developers
### 1. Metadata Structure Separation
v4.0.0 introduces strict metadata/vector separation for billion-scale performance:
Brainy introduces strict metadata/vector separation for billion-scale performance:
```typescript
// ✅ v4.0.0: Metadata has required type field
// ✅ Metadata has required type field
interface NounMetadata {
noun: NounType // Required! Must be a valid noun type
[key: string]: any // Your custom metadata
@ -61,7 +61,7 @@ The verb relationship field changed from `type` to `verb`:
// ❌ v3.x
verb.type === 'relatedTo'
// ✅ v4.0.0
// ✅ Current
verb.verb === 'relatedTo'
```
@ -69,7 +69,7 @@ verb.verb === 'relatedTo'
Storage augmentations are special - they provide the storage backend for Brainy.
### Important: v4.0.0 Storage Requirements
### Important: Storage Requirements
Your storage adapter MUST:
1. **Wrap metadata** with required `noun`/`verb` fields
@ -96,7 +96,7 @@ export class MyCustomStorage extends BaseStorageAdapter {
const noun = await this._getNoun(id)
if (!noun) return null
// Fetch metadata separately (v4.0.0 pattern)
// Fetch metadata separately
const metadata = await this.getNounMetadata(id)
return {
@ -109,14 +109,14 @@ export class MyCustomStorage extends BaseStorageAdapter {
async saveNoun(noun: HNSWNoun, metadata?: NounMetadata): Promise<void> {
// Validate metadata has required 'noun' field
if (!metadata?.noun) {
throw new Error('v4.0.0: NounMetadata requires "noun" field')
throw new Error('NounMetadata requires "noun" field')
}
await this.database.save({
id: noun.id,
vector: noun.vector,
nounType: noun.nounType,
metadata: metadata // Stored separately in v4.0.0
metadata: metadata // Stored separately
})
}
}
@ -356,7 +356,7 @@ Future capability for premium augmentations:
6. **Log appropriately** - Use context.log() for consistent output
7. **Document your augmentation** - Include examples
### v4.0.0 Specific Best Practices
### Specific Best Practices
8. **Always include `noun` field** when creating/modifying NounMetadata:
```typescript

4
docs/DEVELOPER_LEARNING_PATH.md
View file

@ -613,7 +613,7 @@ vfsFiles.forEach(f => {
})
// 3. VFS FILTERING IN KNOWLEDGE QUERIES
console.log('\n🔍 Understanding VFS filtering (v4.4.0)...\n')
console.log('\n🔍 Understanding VFS filtering...\n')
// Create some knowledge entities
const conceptId = await brain.add({
@ -758,7 +758,7 @@ await brain.close()
### Key Concepts
#### 1. **VFS Filtering Architecture (v4.4.0)**
#### 1. **VFS Filtering Architecture**
```typescript
// 🎯 DEFAULT BEHAVIOR: Clean Separation

25
docs/FIND_SYSTEM.md
View file

@ -13,8 +13,7 @@ Brainy's `find()` method is the most advanced query system in any vector databas
- **Data Structure**: Multi-layer graph with 16 connections per node
- **Use Cases**: "Find similar documents", "Content like this"
### 2. Text Intelligence (Word Index) - v7.7.0
- **Purpose**: Keyword/exact text matching
### 2. Text Intelligence (Word Index) - **Purpose**: Keyword/exact text matching
- **Algorithm**: Inverted word index with FNV-1a hashing
- **Performance**: O(log C) where C = chunks (~50 values each)
- **Data Structure**: `__words__ → hash → Roaring Bitmap of entity IDs`
@ -92,7 +91,7 @@ await brain.find({
})
```
### 4. Hybrid Search (v7.7.0+)
### 4. Hybrid Search
```typescript
// Zero-config hybrid: automatically combines text + semantic search
await brain.find({ query: "David Smith" })
@ -113,7 +112,7 @@ await brain.find({ query: "search term", hybridAlpha: 0.3 })
- Medium queries (3-4 words): alpha = 0.5 (balanced)
- Long queries (5+ words): alpha = 0.7 (favor semantic matching)
### 5. Match Visibility (v7.8.0)
### 5. Match Visibility
Search results include match details showing what matched:
@ -132,7 +131,7 @@ Use this for:
- **Explaining** why a result was found (matchSource)
- **Debugging** search behavior (separate scores)
### 6. Semantic Highlighting (v7.8.0)
### 6. Semantic Highlighting
Highlight which concepts/words in text matched your query:
@ -446,7 +445,7 @@ await brain.find({
// Total performance: ~1.2ms for 100K entities
```
## Filter Syntax Reference (v5.8.0+)
## Filter Syntax Reference
### Where Clause: Complete Operator Guide
@ -771,7 +770,7 @@ await brain.find({
// Returns: Verified people with high reputation who work at Stanford AI Lab
```
#### Pagination with Graph Queries (v5.8.0+)
#### Pagination with Graph Queries
```typescript
// Page through high-degree nodes efficiently
@ -792,7 +791,7 @@ const verbIds = await brain.graphIndex.getVerbIdsBySource('source-id', {
**Note**: See `src/graph/graphAdjacencyIndex.ts` for low-level graph operations.
### Sorting Results (v4.5.4+)
### Sorting Results
Sort query results by any field, including timestamps:
@ -895,7 +894,7 @@ const page1 = await getPaginatedResults(0) // First 20
const page2 = await getPaginatedResults(1) // Next 20
```
**Graph pagination** (v5.8.0+):
**Graph pagination**:
```typescript
// Paginate through high-degree node relationships
async function getNeighborPage(entityId: string, page: number, pageSize: number = 50) {
@ -1208,7 +1207,7 @@ await brain.find({ type: NounType.Document })
where: { age: { greaterThan: 18 } } // Old API
// ✅ Correct: Use canonical operators
where: { age: { gt: 18 } } // v5.0.0+
where: { age: { gt: 18 } }
```
### Graph Traversal Issues
@ -1291,7 +1290,7 @@ console.log(`Results: ${results.length}, Unique: ${uniqueIds.size}`)
// Brainy should never return duplicates - report if found
```
## VFS (Virtual File System) Visibility (v4.7.0+)
## VFS (Virtual File System) Visibility
### Default Behavior
@ -1342,7 +1341,7 @@ await brain.find({
### Migration from v4.6.x
**BREAKING CHANGE** (v4.7.0): The `includeVFS` parameter has been removed:
**BREAKING CHANGE**: The `includeVFS` parameter has been removed:
```typescript
// ❌ Old (v4.6.x and earlier)
@ -1351,7 +1350,7 @@ await brain.find({
includeVFS: true // No longer needed!
})
// ✅ New (v4.7.0+)
// ✅ New
await brain.find({
query: 'docs' // VFS included by default
})

8
docs/PERFORMANCE.md
View file

@ -24,11 +24,11 @@ Where:
- `f` = number of fields for entity type
- `t` = number of types (42 nouns, 127 verbs)
### v5.11.1: brain.get() Metadata-Only Optimization
### brain.get() Metadata-Only Optimization
✨ **Massive Performance Improvement**: `brain.get()` is now **76-81% faster** by default!
| Operation | Before (v5.11.0) | After (v5.11.1) | Speedup | Use Case |
| Operation | Before | After | Speedup | Use Case |
|-----------|------------------|-----------------|---------|----------|
| **brain.get() (metadata-only)** | 43ms, 6KB | **10ms, 300 bytes** | **76-81%** | VFS, existence checks, metadata |
| **brain.get({ includeVectors: true })** | 43ms, 6KB | 43ms, 6KB | 0% | Similarity calculations |
@ -308,7 +308,7 @@ const results = await Promise.all(searchPromises)
- ✅ **Horizontally Scalable**: Stateless operations support clustering
- ✅ **Zero Stubs**: Every line of code is production-ready
## Lazy Loading Performance (v5.7.7+)
## Lazy Loading Performance
Brainy supports two initialization modes for optimal performance across different use cases:
@ -324,7 +324,7 @@ await brain.init() // Rebuilds indexes during init (~500ms-3s for 10K entities)
- First query: Instant (indexes already loaded)
- Use case: Traditional applications, long-running servers
### Mode 2: Lazy Loading (v5.7.7+)
### Mode 2: Lazy Loading
```javascript
const brain = new Brainy({ disableAutoRebuild: true })

29
docs/README.md
View file

@ -1,8 +1,8 @@
# Brainy Documentation (v6.5.0)
# Brainy Documentation
Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI database with Triple Intelligence Engine.
## 🆕 What's New in v4.0.0
## 🆕 What's New in
**Production-Ready Cost Optimization:**
- **Lifecycle Management**: Automatic tier transitions for S3, GCS, Azure (96% cost savings!)
@ -14,7 +14,7 @@ Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI
**Cost Impact Example (500TB dataset):**
- Before: $138,000/year
- After v4.0.0: $5,940/year
- After $5,940/year
- **Savings: $132,060/year (96%)**
## 📊 Implementation Status
@ -117,7 +117,7 @@ const results = await brain.find("highly rated technology articles by researcher
| [API Reference](./api/README.md) | **START HERE** - Complete API documentation with examples |
| [VFS Quick Start](./vfs/QUICK_START.md) | Virtual filesystem in 30 seconds |
### 🆕 v4.0.0 Migration & Optimization
### 🆕 Migration & Optimization
| Document | Description |
|----------|-------------|
@ -135,15 +135,15 @@ const results = await brain.find("highly rated technology articles by researcher
| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | Revolutionary data model - entities and relationships |
| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Field unified query system |
| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment |
| [Storage Architecture](./architecture/storage-architecture.md) | **v4.0.0** - Storage adapters and optimization |
| [Data Storage Architecture](./architecture/data-storage-architecture.md) | **v4.0.0** - Metadata/vector separation, sharding |
| [Storage Architecture](./architecture/storage-architecture.md) | Storage adapters and optimization |
| [Data Storage Architecture](./architecture/data-storage-architecture.md) | Metadata/vector separation, sharding |
| [Index Architecture](./architecture/index-architecture.md) | HNSW, Graph, and Metadata indexing |
### 💾 Storage & Deployment
| Document | Description |
|----------|-------------|
| [Cloud Deployment Guide](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | **v4.0.0** - Deploy on AWS, GCP, Azure, Cloudflare |
| [Cloud Deployment Guide](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | Deploy on AWS, GCP, Azure, Cloudflare |
| [AWS Deployment](./deployment/aws-deployment.md) | AWS-specific deployment patterns |
| [GCP Deployment](./deployment/gcp-deployment.md) | Google Cloud deployment |
| [Kubernetes Deployment](./deployment/kubernetes-deployment.md) | K8s deployment configurations |
@ -258,7 +258,7 @@ const results = await brain.find("highly rated technology articles by researcher
```
docs/
├── README.md (this file) # Complete documentation index
├── MIGRATION-V3-TO-V4.md # v4.0.0 migration guide
├── MIGRATION-V3-TO-V4.md # Migration guide
│
├── guides/ # User guides
│ ├── natural-language.md
@ -276,9 +276,7 @@ docs/
│ ├── noun-verb-taxonomy.md
│ ├── triple-intelligence.md
│ ├── zero-config.md
│ ├── storage-architecture.md # v4.0.0
│ ├── data-storage-architecture.md # v4.0.0
│ ├── index-architecture.md
│ ├── storage-architecture.md│ ├── data-storage-architecture.md│ ├── index-architecture.md
│ ├── distributed-storage.md
│ ├── augmentations.md
│ ├── augmentation-system-audit.md
@ -287,15 +285,10 @@ docs/
│ └── ...
│
├── operations/ # Operations guides
│ ├── cost-optimization-aws-s3.md # v4.0.0
│ ├── cost-optimization-gcs.md # v4.0.0
│ ├── cost-optimization-azure.md # v4.0.0
│ ├── cost-optimization-cloudflare-r2.md # v4.0.0
│ └── capacity-planning.md
│ ├── cost-optimization-aws-s3.md│ ├── cost-optimization-gcs.md│ ├── cost-optimization-azure.md│ ├── cost-optimization-cloudflare-r2.md│ └── capacity-planning.md
│
├── deployment/ # Deployment guides
│ ├── CLOUD_DEPLOYMENT_GUIDE.md # v4.0.0
│ ├── aws-deployment.md
│ ├── CLOUD_DEPLOYMENT_GUIDE.md│ ├── aws-deployment.md
│ ├── gcp-deployment.md
│ └── kubernetes-deployment.md
│

2
docs/RELEASE-GUIDE.md
View file

@ -48,7 +48,7 @@ git commit -m "chore: remove unused dependency"
# DON'T use BREAKING CHANGE for internal changes
git commit -m "feat: improve model delivery
BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers v3.0.0
BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers
```
## Release Workflow Checklist

2
docs/STAGE3-CANONICAL-TAXONOMY.md
View file

@ -1,4 +1,4 @@
# Brainy Stage 3: Canonical Taxonomy (v6.0.0)
# Brainy Stage 3: Canonical Taxonomy
**Status:** FINAL - This is the definitive, timeless taxonomy
**Total Types:** 169 (42 nouns + 127 verbs)

6
docs/api/COMPREHENSIVE_API_OVERVIEW.md
View file

@ -1,6 +1,6 @@
# Brainy Complete Public API Reference
> **Accurate API documentation for Brainy v6.5.0+**
> **Accurate API documentation for Brainy**
## Initialization
@ -20,7 +20,7 @@ const brain = new Brainy({
await brain.init()
```
## Readiness API (v7.3.0+)
## Readiness API
For reliable initialization detection, especially in cloud environments with progressive initialization:
@ -355,7 +355,7 @@ await vfs.mkdir('/project/src', { recursive: true })
const files = await vfs.readdir('/project')
await vfs.rmdir('/project', { recursive: true })
// Bulk operations (v6.5.0+)
// Bulk operations
const result = await vfs.bulkWrite([
{ type: 'mkdir', path: '/data' },
{ type: 'write', path: '/data/config.json', data: '{}' },

32
docs/architecture/data-storage-architecture.md
View file

@ -1,4 +1,4 @@
# Brainy Data Storage Architecture (v5.11.0)
# Brainy Data Storage Architecture
**Complete file structure reference for all storage backends**
@ -13,7 +13,7 @@ This document explains how Brainy stores, indexes, and scales data across all st
3. [The 4 Indexes](#3-the-4-indexes)
4. [Sharding Strategy](#4-sharding-strategy)
5. [COW (Copy-on-Write) Architecture](#5-cow-copy-on-write-architecture)
6. [ID-First Storage Architecture (v6.0.0+)](#6-id-first-storage-architecture-v600)
6. [ID-First Storage Architecture](#6-id-first-storage-architecture-v600)
7. [VFS (Virtual File System)](#7-vfs-virtual-file-system)
8. [Storage Backend Mapping](#8-storage-backend-mapping)
9. [Performance Characteristics](#9-performance-characteristics)
@ -438,7 +438,7 @@ Unlike entities and relationships, system metadata consists of **index files** t
Understanding how Brainy constructs storage paths is critical for debugging and optimization.
### Path Construction Steps (v6.0.0+)
### Path Construction Steps
**For an entity (noun)**:
```typescript
@ -498,7 +498,7 @@ const fieldIndexPath = `_system/metadata_indexes/__metadata_field_index__${field
const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
```
### Path Patterns Summary (v6.0.0+)
### Path Patterns Summary
| Data Type | Path Pattern | Sharded? | Branched? |
|-----------|--------------|----------|-----------|
@ -516,7 +516,7 @@ const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
| **HNSW node** | `_system/hnsw/nodes/{shard}/{uuid}.json` | ✅ Yes (UUID) | ❌ No |
| **Field index** | `_system/metadata_indexes/__metadata_field_index__{field}.json` | ❌ No | ❌ No |
### Key Principles (v6.0.0+)
### Key Principles
1. **Shard Extraction**: Always use first 2 hex characters of UUID/SHA-256
2. **ID-First**: Shard + ID come BEFORE type (type is in metadata)
@ -549,7 +549,7 @@ Brainy uses four complementary index systems for different query patterns.
- Memory (standard): ~200MB per 100K entities
- Memory (lazy): ~15-33MB per 100K entities (5-10x less!)
**Automatic Lazy Mode** (v3.36.0+): Enables automatically when vectors don't fit in UnifiedCache
**Automatic Lazy Mode**: Enables automatically when vectors don't fit in UnifiedCache
---
@ -559,7 +559,7 @@ Brainy uses four complementary index systems for different query patterns.
**Location**: MetadataIndexManager index on `noun` field
**Data Structure**: RoaringBitmap32 per type value
**How It Works** (v6.0.0+):
**How It Works**:
```typescript
// Find all Person entities
const people = await brain.getNouns({ type: 'person' })
@ -721,7 +721,7 @@ COW is Brainy's **git-like versioning system** that enables:
- ✅ **Deduplication** (identical data stored only once)
- ✅ **Version history** (full audit trail of all changes)
**Status**: ALWAYS ENABLED (v5.11.0+) - cannot be disabled
**Status**: ALWAYS ENABLED - cannot be disabled
---
@ -782,19 +782,19 @@ _cow/blobs/ab/abc123...sha256.bin (used by both entities)
---
## 6. ID-First Storage Architecture (v6.0.0+)
## 6. ID-First Storage Architecture
### 6.1 What is ID-First?
**ID-first storage** organizes entities by **ID shard only** - no type directories! This eliminates 42-type sequential searches that caused 20-21 second delays on cloud storage.
**Old type-first structure** (v5.4.0-v5.12.0):
**Old type-first structure** (v5.12.0):
```
branches/main/entities/nouns/{TYPE}/metadata/00/001234...uuid.json
# Problem: Requires knowing type OR searching 42 type directories!
```
**NEW ID-first structure** (v6.0.0+):
**NEW ID-first structure**:
```
branches/main/entities/nouns/00/001234...uuid/metadata.json
# Direct O(1) lookup - no type needed!
@ -809,7 +809,7 @@ branches/main/entities/nouns/00/001234...uuid/metadata.json
// v5.x: Had to search 42 types if type unknown
// Result: 21 seconds on GCS (42 types × 500ms)
// v6.0.0: Direct path from ID
// Direct path from ID
const id = '001234...'
const shard = id.substring(0, 2) // '00'
const path = `branches/main/entities/nouns/${shard}/${id}/metadata.json`
@ -1165,7 +1165,7 @@ opfs://root/brainy/
### 10.2 clear() Operation
**What clear() deletes** (v5.11.0+):
**What clear() deletes**:
✅ Deletes:
- `branches/` → ALL entity data (all types, all shards, all branches, all forks)
@ -1184,7 +1184,7 @@ opfs://root/brainy/
**Example**:
```typescript
await brain.storage.clear() // ✅ Deletes ALL data correctly (v5.11.0+)
await brain.storage.clear() // ✅ Deletes ALL data correctly
await brain.add({ data: 'Alice', type: 'person' }) // ✅ COW reinitializes automatically
```
@ -1353,7 +1353,7 @@ const historicalData = await yesterday.getNouns({ type: 'Character' })
await brain.storage.clear()
```
**What happens in storage** (v5.11.0+):
**What happens in storage**:
```
1. Delete all entity data:
→ Remove: branches/ (entire directory)
@ -1452,7 +1452,7 @@ await brain.addBatch([
## 11. Summary
**Complete Storage Structure (v6.0.0)**:
**Complete Storage Structure**:
- **3 storage layers**: branches/ (data), _cow/ (versions), _system/ (indexes)
- **2 files per entity**: metadata.json + vector.json (optimized I/O)
- **4 indexes**: HNSW (semantic), Type Index (metadata-based), Graph (relationships), Metadata (fields)

32
docs/architecture/index-architecture.md
View file

@ -25,7 +25,7 @@ Brainy has **3 main indexes** at the top level, each with multiple sub-indexes m
- **FieldTypeInference** - DuckDB-inspired value-based field type detection
- **Field Sparse Indexes** - Per-field sparse indexes with roaring bitmaps (dynamic count)
- **Sorted Indexes** - Support orderBy queries (automatically maintained)
- **Word Index (`__words__`)** - Text search via FNV-1a word hashes (v7.7.0)
- **Word Index (`__words__`)** - Text search via FNV-1a word hashes
**GraphAdjacencyIndex contains:**
- **lsmTreeSource** - Source → Targets (outgoing edges)
@ -39,7 +39,7 @@ All indexes share a **UnifiedCache** for coordinated memory management, ensuring
**Purpose**: Enable O(1) field-value lookups and O(log n) range queries on metadata fields using adaptive chunked sparse indexing.
### Internal Architecture (v3.42.0)
### Internal Architecture
```typescript
class MetadataIndexManager {
@ -64,7 +64,7 @@ class MetadataIndexManager {
### Key Data Structures
#### Chunked Sparse Index (NEW in v3.42.0)
#### Chunked Sparse Index
```typescript
// SparseIndex: Directory of chunks for a field
// Example: field="status"
@ -87,7 +87,7 @@ interface ChunkDescriptor {
class ChunkData {
chunkId: number
field: string
entries: Map<value, RoaringBitmap32> // ~50 values per chunk (v3.43.0: roaring bitmaps!)
entries: Map<value, RoaringBitmap32> // ~50 values per chunk (roaring bitmaps!)
}
```
@ -96,7 +96,7 @@ class ChunkData {
- O(log n) range queries with zone maps
- 630x file reduction (560k flat files → 89 chunk files)
#### Roaring Bitmap Optimization (NEW in v3.43.0)
#### Roaring Bitmap Optimization
**Problem Solved**: JavaScript `Set<string>` for storing entity IDs was inefficient:
- Memory overhead: ~40 bytes per UUID string (36 chars + overhead)
@ -150,7 +150,7 @@ class ChunkData {
**Multi-Field Intersection (THE BIG WIN!)**:
```typescript
// Before (v3.42.0): JavaScript array filtering
// Before: JavaScript array filtering
async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string[]> {
// 1. Fetch UUID arrays for each field
const statusIds = await this.getIds('status', 'active') // ["uuid1", "uuid2", ...]
@ -160,7 +160,7 @@ async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string
return statusIds.filter(id => roleIds.includes(id)) // O(n*m) array filtering
}
// After (v3.43.0): Roaring bitmap intersection
// After: Roaring bitmap intersection
async getIdsForMultipleFields(pairs: [{field, value}, ...]): Promise<string[]> {
// 1. Fetch roaring bitmaps (integers, not UUIDs)
const bitmaps: RoaringBitmap32[] = []
@ -229,7 +229,7 @@ interface ZoneMap {
**Use case**: Enables NLP to understand "find characters named John" → knows 'name' is a character field
#### Word Index (`__words__`) - v7.7.0
#### Word Index (`__words__`) -
```typescript
// Special field for text/keyword search
// Entity text content is tokenized and indexed as word hashes
@ -253,14 +253,14 @@ interface ZoneMap {
- **Lowercase normalization**: Case-insensitive matching
- **Automatic integration**: Words extracted via `extractIndexableFields()`
**Hybrid Search** (v7.7.0): Text results combined with vector results using Reciprocal Rank Fusion (RRF):
**Hybrid Search**: Text results combined with vector results using Reciprocal Rank Fusion (RRF):
```typescript
// RRF formula: score(d) = sum(1 / (k + rank(d)))
// where k = 60 (standard constant)
// alpha = weight for semantic (0 = text only, 1 = semantic only)
```
### Query Algorithm (v3.42.0)
### Query Algorithm
**Exact Match Query**:
```typescript
@ -317,7 +317,7 @@ async getIdsForRange(field: string, min: any, max: any): Promise<string[]> {
- Adaptive chunking: ~50 values per chunk optimizes I/O
- Immediate flushing: No need for dirty tracking or batch writes
### Temporal Bucketing (v3.41.0)
### Temporal Bucketing
**Problem Solved**: High-cardinality timestamp fields created massive file pollution.
- Example: 575 entities with unique timestamps → 358,407 index files (98.7% pollution!)
@ -396,7 +396,7 @@ const DEFAULT_EXCLUDE_FIELDS = [
]
```
**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of v3.41.0 - they are indexed with automatic bucketing.
**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of they are indexed with automatic bucketing.
## 2. HNSWIndex - Vector Similarity Search
@ -735,7 +735,7 @@ async stats(): Promise<Statistics> {
}
```
### 5. Index Rebuilding (v5.7.7: Lazy Loading Support)
### 5. Index Rebuilding (Lazy Loading Support)
**Two modes of index loading:**
@ -762,7 +762,7 @@ async init(): Promise<void> {
}
```
#### Mode 2: Lazy Loading on First Query (v5.7.7+)
#### Mode 2: Lazy Loading on First Query
```typescript
// When disableAutoRebuild: true
@ -918,7 +918,7 @@ All indexes scale gracefully:
- [Performance Guide](../PERFORMANCE.md) - Performance tuning
- [Overview](./overview.md) - High-level architecture
## Summary: Index Hierarchy (v5.7.7)
## Summary: Index Hierarchy
### Level 1: Main Indexes (3)
All have rebuild() methods and are covered by lazy loading:
@ -933,7 +933,7 @@ Automatically managed by parent rebuild():
- **4 LSM-trees** (lsmTreeSource, lsmTreeTarget, lsmTreeVerbsBySource, lsmTreeVerbsByTarget)
- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex)
### Lazy Loading (v5.7.7)
### Lazy Loading
- **Mode 1**: Auto-rebuild on init() (default)
- **Mode 2**: Lazy rebuild on first query (when `disableAutoRebuild: true`)
- **Concurrency-safe**: Mutex prevents duplicate rebuilds

8
docs/architecture/initialization-and-rebuild.md
View file

@ -15,7 +15,7 @@ This document explains how Brainy's four indexes (MetadataIndex, HNSWIndex, Grap
| **GraphAdjacencyIndex** | Relationships via LSM-tree SSTables | LSM-tree auto-persistence | v3.44.0 |
| **DeletedItemsIndex** | Set of deleted IDs | `storage.saveDeletedItems()` | v3.0.0 |
#### MetadataIndex Persistence Details (v4.2.1+)
#### MetadataIndex Persistence Details
The MetadataIndex now persists two components:
@ -129,7 +129,7 @@ async init(): Promise<void> {
- 100-2000ms: One-time rebuild to create indices
- Total: ~1-3 seconds (one time only)
#### Mode 2: Lazy Loading on First Query (v5.7.7+)
#### Mode 2: Lazy Loading on First Query
When `disableAutoRebuild: true`, indexes remain empty after init() and rebuild on first query:
@ -350,7 +350,7 @@ public async rebuild(options?: {
**Performance Impact**: 200-600x speedup (5 minutes → 500ms for 10K entities)
**Correct Pattern** (v3.45.0):
**Correct Pattern**:
```typescript
// Load ALL nouns ONCE (not 31 times!)
while (hasMore) {
@ -392,7 +392,7 @@ while (hasMore) {
```typescript
// src/utils/metadataIndex.ts (lines 202-216)
async init(): Promise<void> {
// STEP 1: Load field registry to discover persisted indices (v4.2.1)
// STEP 1: Load field registry to discover persisted indices
// This is THE KEY FIX - O(1) discovery of existing indices
await this.loadFieldRegistry()

32
docs/architecture/storage-architecture.md
View file

@ -1,12 +1,12 @@
# Storage Architecture (v4.0.0)
# Storage Architecture
> **Updated for v4.0.0**: Metadata/vector separation, UUID-based sharding, lifecycle management
> **Updated**: Metadata/vector separation, UUID-based sharding, lifecycle management
## Storage Structure
### v4.0.0 Architecture: Metadata/Vector Separation
### Architecture: Metadata/Vector Separation
In v4.0.0, entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
```
brainy-data/
@ -64,7 +64,7 @@ const shard = uuid.substring(0, 2) // "3f"
## Storage Adapters
Brainy provides multiple storage adapters with identical APIs and v4.0.0 production features:
Brainy provides multiple storage adapters with identical APIs and production features:
### FileSystem Storage (Node.js)
```typescript
@ -72,14 +72,14 @@ const brain = new Brainy({
storage: {
type: 'filesystem',
path: './data',
compression: true // v4.0.0: Gzip compression (60-80% space savings)
compression: true // Gzip compression (60-80% space savings)
}
})
```
- **Use case**: Server applications, CLI tools
- **Performance**: Direct file I/O with optional compression
- **Persistence**: Permanent on disk
- **v4.0.0 Features**:
- **Features**:
- **Gzip Compression**: 60-80% storage savings with minimal CPU overhead
- **Batch Delete**: Efficient bulk deletion with retries
- **UUID Sharding**: Automatic 256-shard distribution
@ -101,7 +101,7 @@ const brain = new Brainy({
- **Use case**: Distributed applications, cloud deployments
- **Performance**: Network dependent, with intelligent caching
- **Persistence**: Cloud storage durability (99.999999999%)
- **v4.0.0 Features**:
- **Features**:
- **Lifecycle Policies**: Automatic tier transitions (Standard → IA → Glacier → Deep Archive)
- **Intelligent-Tiering**: Automatic optimization based on access patterns (up to 95% savings)
- **Batch Delete**: Efficient bulk deletion (1000 objects per request)
@ -120,7 +120,7 @@ const brain = new Brainy({
- **Use case**: Google Cloud deployments
- **Performance**: Global CDN with edge caching
- **Persistence**: 99.999999999% durability
- **v4.0.0 Features**:
- **Features**:
- **Lifecycle Policies**: Automatic tier transitions (Standard → Nearline → Coldline → Archive)
- **Autoclass**: Intelligent automatic tier optimization
- **Batch Delete**: Efficient bulk operations
@ -139,7 +139,7 @@ const brain = new Brainy({
- **Use case**: Azure cloud deployments
- **Performance**: Global replication with CDN
- **Persistence**: LRS, ZRS, GRS, RA-GRS options
- **v4.0.0 Features**:
- **Features**:
- **Blob Tier Management**: Hot/Cool/Archive tiers (99% cost savings)
- **Lifecycle Policies**: Automatic tier transitions and deletions
- **Batch Delete**: BlobBatchClient for efficient bulk operations
@ -157,7 +157,7 @@ const brain = new Brainy({
- **Use case**: Browser applications, PWAs
- **Performance**: Near-native file system speed
- **Persistence**: Permanent in browser (with quota limits)
- **v4.0.0 Features**:
- **Features**:
- **Quota Monitoring**: Real-time quota tracking and warnings
- **Batch Delete**: Efficient bulk deletion
- **Storage Status**: Detailed usage/available reporting
@ -244,7 +244,7 @@ Ensures durability and enables recovery:
2. Replay operations from last checkpoint
3. Verify checksums for integrity
## Storage Optimization (v4.0.0)
## Storage Optimization
### 1. Lifecycle Policies (Cloud Storage)
@ -359,7 +359,7 @@ const brain = new Brainy({
### 5. Batch Operations
```typescript
// v4.0.0: Efficient batch delete
// Efficient batch delete
await storage.batchDelete([
'entities/nouns/vectors/00/00123456-....json',
'entities/nouns/metadata/00/00123456-....json',
@ -522,7 +522,7 @@ console.log(stats)
// }
```
## Best Practices (v4.0.0)
## Best Practices
### Choose the Right Adapter
1. **Development**: FileSystem with compression (local persistence, small storage footprint)
@ -537,7 +537,7 @@ console.log(stats)
4. **Archival**: Cloud storage with lifecycle policies (96% cost savings!)
5. **Large-scale**: Metadata/vector separation + UUID sharding + lifecycle policies
### v4.0.0 Cost Optimization
### Cost Optimization
1. **Enable lifecycle policies** for cloud storage (automated cost reduction)
2. **Use Intelligent-Tiering (S3)** or Autoclass (GCS) for automatic optimization
3. **Enable compression** for FileSystem storage (60-80% space savings)
@ -547,7 +547,7 @@ console.log(stats)
**Example Cost Savings (500TB dataset):**
- Without lifecycle policies: **$138,000/year**
- With v4.0.0 lifecycle policies: **$5,940/year**
- With lifecycle policies: **$5,940/year**
- **Savings: $132,060/year (96%)**
### Monitor and Maintain

8
docs/augmentations/COMPLETE-REFERENCE.md
View file

@ -1,8 +1,8 @@
# 🔌 Brainy v4.0.0 Augmentations Complete Reference
# 🔌 Brainy Augmentations Complete Reference
> **All augmentations that power Brainy's extensibility - with locations, usage, and examples**
>
> **⚠️ v4.0.0 Update**: Updated for metadata structure changes and billion-scale optimizations
> **⚠️ Update**: Updated for metadata structure changes and billion-scale optimizations
## Quick Start
@ -19,7 +19,7 @@ const brain = new Brainy({
await brain.init() // Augmentations initialize automatically
```
## v4.0.0 Augmentation Architecture
## Augmentation Architecture
### Key Improvements for Billion-Scale Performance
@ -40,7 +40,7 @@ await brain.init() // Augmentations initialize automatically
### What This Means for Augmentation Users
**✅ If you use built-in augmentations**: No changes needed! They're all updated for v4.0.0.
**✅ If you use built-in augmentations**: No changes needed! They're all updated.
**⚠️ If you created custom storage augmentations**: Update your storage adapter to:
- Wrap metadata with required `noun`/`verb` fields

10
docs/augmentations/DEVELOPER-GUIDE.md
View file

@ -1,10 +1,10 @@
# 🛠️ Brainy Augmentation Developer Guide
> **How to create, test, and use augmentations in Brainy v4.0.0**
> **How to create, test, and use augmentations in Brainy**
>
> **⚠️ v4.0.0 Update**: This guide has been updated with breaking changes for metadata structure and type system improvements.
> **⚠️ Update**: This guide has been updated with breaking changes for metadata structure and type system improvements.
## v4.0.0 Migration Guide
## Migration Guide
### What Changed?
@ -31,7 +31,7 @@ const verb = {
}
if (verb.type === 'relatedTo') { ... }
// ✅ v4.0.0
// ✅ Current
const verb = {
verb: 'relatedTo',
sourceId: 'a',
@ -70,7 +70,7 @@ export class MyFirstAugmentation extends BaseAugmentation {
if (operation === 'add') {
console.log('Noun added:', params.noun)
// v4.0.0: Access metadata correctly
// Access metadata correctly
if (params.noun?.metadata) {
console.log('Noun type:', params.noun.metadata.noun) // Required field
}

16
docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md
View file

@ -1,8 +1,8 @@
# Brainy v4.0.0 Cloud Deployment Guide
# Brainy Cloud Deployment Guide
This guide provides production-ready deployment configurations for Brainy using S3CompatibleStorage (preferred) or FileSystemStorage across major cloud platforms. All examples are verified against the actual Brainy v4.0.0 codebase.
This guide provides production-ready deployment configurations for Brainy using S3CompatibleStorage (preferred) or FileSystemStorage across major cloud platforms. All examples are verified against the actual Brainy codebase.
## 🆕 v4.0.0 Production Features
## 🆕 Production Features
**Cost Optimization at Scale:**
- **Lifecycle Policies**: Automatic tier transitions for 96% cost savings
@ -12,7 +12,7 @@ This guide provides production-ready deployment configurations for Brainy using
**Example Impact (500TB dataset):**
- Before: $138,000/year
- With v4.0.0 lifecycle policies: $5,940/year
- With lifecycle policies: $5,940/year
- **Savings: $132,060/year (96%)**
See the [Cost Optimization](#cost-optimization-v40) section below for implementation details.
@ -727,7 +727,7 @@ S3CompatibleStorage constructor parameters (verified from source):
4. **Implement rate limiting** to prevent abuse
5. **Configure CORS** appropriately for your use case
## Cost Optimization (v4.0.0)
## Cost Optimization
### Enable Lifecycle Policies
@ -765,7 +765,7 @@ await storage.enableIntelligentTiering('entities/', 'auto-optimize')
**Efficient bulk deletions:**
```javascript
// v4.0.0: Batch delete (1000 objects per request)
// Batch delete (1000 objects per request)
const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => [
`entities/nouns/vectors/${id.substring(0, 2)}/${id}.json`,
@ -793,8 +793,8 @@ const brain = new Brainy({ storage })
2. **Use S3CompatibleStorage** for cloud deployments (better scalability)
3. **Enable the cache augmentation** for frequently accessed data
4. **Configure appropriate memory limits** for your runtime
5. **v4.0.0**: Enable lifecycle policies to reduce storage costs by 96%
6. **v4.0.0**: Use batch operations for cleanup tasks
5. Enable lifecycle policies to reduce storage costs by 96%
6. Use batch operations for cleanup tasks
## Troubleshooting

4
docs/deployment/aws-deployment.md
View file

@ -387,7 +387,7 @@ jobs:
3. **Cold Starts (Lambda)**
**v7.3.0+ Progressive Initialization (Zero-Config)**
**Progressive Initialization (Zero-Config)**
Brainy automatically detects Lambda environments (AWS_LAMBDA_FUNCTION_NAME)
and uses progressive initialization for <200ms cold starts:
@ -441,7 +441,7 @@ jobs:
}
```
**Readiness Detection (v7.3.0+)**
**Readiness Detection**
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:

4
docs/deployment/gcp-deployment.md
View file

@ -507,7 +507,7 @@ const brain = new Brainy({
2. **Cold Starts**
**v7.3.0+ Progressive Initialization (Zero-Config)**
**Progressive Initialization (Zero-Config)**
Brainy automatically detects Cloud Run and Cloud Functions environments
and uses progressive initialization for <200ms cold starts:
@ -557,7 +557,7 @@ const brain = new Brainy({
})
```
**Readiness Detection (v7.3.0+)**
**Readiness Detection**
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:

31
docs/features/instant-fork.md
View file

@ -61,7 +61,7 @@ if (success) {
Brainy uses **Snowflake-style Copy-on-Write (COW)** for instant forking:
### Architecture (v5.0.0)
### Architecture
1. **HNSW Index COW** (The Performance Bottleneck):
- **Shallow Copy**: O(1) Map reference copying (~10ms for 1M+ nodes)
@ -79,7 +79,7 @@ Brainy uses **Snowflake-style Copy-on-Write (COW)** for instant forking:
- **BlobStorage**: Content-addressable with deduplication
- **All Adapters**: Memory, FileSystem, S3, R2, GCS, Azure, OPFS
**Performance (v5.0.0)**:
**Performance**:
- **Fork time**: < 100ms @ 10K entities (MEASURED in tests)
- **Storage overhead**: 10-20% (shared blobs, only changed data duplicated)
- **Memory overhead**: 10-20% (shared HNSW nodes, only modified nodes duplicated)
@ -316,7 +316,7 @@ console.log(originalData[0].data.version) // 1 (original state!)
// Clean up
await snapshot.destroy()
// Note: Time-travel queries (asOf) planned for v5.1.0
// Note: Time-travel queries (asOf) Planned
```
---
@ -339,9 +339,9 @@ const fork3 = await brain.fork('test', {
})
```
### Branch Management (v5.0.0)
### Branch Management
**NEW in v5.0.0:** Full branch management now available!
Full branch management now available!
```javascript
// List all branches
@ -359,9 +359,9 @@ await brain.checkout('experiment')
await brain.deleteBranch('old-experiment')
```
### Commit Tracking (v5.0.0)
### Commit Tracking
**NEW in v5.0.0:** Git-style commit tracking!
Git-style commit tracking!
```javascript
// Create a commit (snapshot of current state)
@ -375,9 +375,9 @@ const commitHash = await brain.commit({
console.log(commitHash) // 'a3f2c1b9...'
```
### Commit History (v5.0.0)
### Commit History
**NEW in v5.0.0:** View commit history!
View commit history!
```javascript
// Get commit history for current branch
@ -425,7 +425,7 @@ Savings: 40% less memory
## Zero Configuration
**Fork is enabled by default in v5.0.0+. No setup required.**
**Fork is enabled by default. No setup required.**
```javascript
// This is all you need:
@ -612,8 +612,7 @@ const fork = await brain.fork() // Done!
## What's Implemented vs. What's Next
### ✅ Available in v5.0.0:
- ✅ `fork()` - Instant clone in <100ms
### ✅ Available in - ✅ `fork()` - Instant clone in <100ms
- ✅ `listBranches()` - List all forks
- ✅ `getCurrentBranch()` - Get active branch
- ✅ `checkout()` - Switch between branches
@ -621,14 +620,14 @@ const fork = await brain.fork() // Done!
- ✅ `commit()` - Create state snapshots
- ✅ `getHistory()` - View commit history
### 🔮 Planned for v5.1.0+:
### 🔮 Planned for:
**Temporal Features:**
- `asOf(timestamp)` - Query data at specific time (✅ v5.0.0+)
- `asOf(timestamp)` - Query data at specific time (✅ available)
- `rollback(commitHash)` - Restore to previous state
- Full audit trail for all changes
These features require additional temporal infrastructure and are being carefully designed for v5.1.0+
These features require additional temporal infrastructure and are being carefully designed
---
@ -691,4 +690,4 @@ console.log('Fork created in < 2 seconds! 🚀')
---
**Brainy v5.0.0+** | [GitHub](https://github.com/soulcraftlabs/brainy) | [npm](https://npmjs.com/package/@soulcraft/brainy)
**Brainy** | [GitHub](https://github.com/soulcraftlabs/brainy) | [npm](https://npmjs.com/package/@soulcraft/brainy)

16
docs/guides/import-anything.md
View file

@ -92,7 +92,7 @@ await brain.import(yaml, { format: 'yaml' })
// ✨ Hierarchical data becomes a connected graph!
```
### 📄 Import Word Documents (DOCX) - v4.2.0
### 📄 Import Word Documents (DOCX) -
```javascript
// From file path
await brain.import('research-paper.docx')
@ -121,7 +121,7 @@ await brain.import('https://api.example.com/data.json')
await brain.import('https://data.gov/census.csv')
// ✨ Fetches CSV from web, parses, imports!
// With authentication (v4.2.0)
// With authentication
await brain.import({
type: 'url',
data: 'https://api.example.com/private/data.xlsx',
@ -132,7 +132,7 @@ await brain.import({
})
// ✨ Supports basic authentication for protected resources!
// With custom headers (v4.2.0)
// With custom headers
await brain.import({
type: 'url',
data: 'https://api.example.com/data.json',
@ -163,9 +163,9 @@ When you import data, Brainy:
2. **Intelligent parsing** - CSV (encoding/delimiter), Excel (multi-sheet), PDF (text/tables), DOCX (headings/paragraphs)
3. **Identifies entity types** - Uses AI to classify as Person, Document, Product, etc. (31 types!)
4. **Finds relationships** - Detects connections like "belongsTo", "createdBy", "references" (40 types!)
5. **Scores confidence & weight** - Every entity and relationship gets quality metrics (v4.2.0)
5. **Scores confidence & weight** - Every entity and relationship gets quality metrics
6. **Creates embeddings** - Makes everything semantically searchable
7. **Indexes metadata** - Enables lightning-fast filtering with range queries (v4.2.0)
7. **Indexes metadata** - Enables lightning-fast filtering with range queries
## Intelligent Type Detection
@ -204,8 +204,7 @@ await brain.import(data)
// - Bob "memberOf" Engineering
```
## Confidence & Weight Scoring - v4.2.0
## Confidence & Weight Scoring -
Every entity and relationship gets confidence and weight scores:
```javascript
@ -236,8 +235,7 @@ const mediumConfidence = await brain.find({
**Weights** indicate importance/relevance within the document context.
## Per-Sheet Excel Extraction - v4.2.0
## Per-Sheet Excel Extraction -
Excel files with multiple sheets can be organized by sheet:
```javascript

4
docs/guides/import-flow.md
View file

@ -39,7 +39,7 @@ Each phase adds intelligence and structure to your raw data, transforming it int
---
## 🌊 Always-On Streaming Architecture (v4.2.0+)
## 🌊 Always-On Streaming Architecture
All imports use streaming with **progressive flush intervals**:
@ -1062,7 +1062,7 @@ if (!fromEntity || !toEntity) {
}
```
#### 5.3b: Check for Duplicates (v3.43.2 Critical Fix)
#### 5.3b: Check for Duplicates (Critical Fix)
**The Bug**: Without duplicate checking, re-importing would create:
```

2
docs/guides/import-progress-examples.md
View file

@ -2,7 +2,7 @@
**How to Use Progress Tracking in Your Applications**
Brainy v4.5.0+ provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX).
Brainy provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX).
> **⚠️ KEY FEATURE:** The progress API is **100% standardized**. Write your progress handler ONCE and it works for ALL formats with zero format-specific code! See [Standard Import Progress API](./standard-import-progress.md) for the complete interface documentation.

6
docs/guides/import-progress-implementation.md
View file

@ -74,7 +74,7 @@ interface ImporterProgressCallback {
## 🎯 Overview
As of v4.5.0, Brainy supports comprehensive, multi-dimensional progress tracking for imports:
Brainy supports comprehensive, multi-dimensional progress tracking for imports:
- **Bytes processed** (always available, most deterministic)
- **Entities extracted** (AI extraction phase)
- **Stage-specific metrics** (parsing: MB/s, extraction: entities/s)
@ -120,13 +120,13 @@ export interface FormatHandlerOptions {
// ... existing options ...
/**
* Progress hooks (v4.5.0)
* Progress hooks
* Handlers call these to report progress during processing
*/
progressHooks?: FormatHandlerProgressHooks
/**
* Total file size in bytes (v4.5.0)
* Total file size in bytes
* Used for progress percentage calculation
*/
totalBytes?: number

4
docs/guides/import-quick-reference.md
View file

@ -91,7 +91,7 @@ await brain.import(file, {
})
```
### Import Tracking (v4.10.0+)
### Import Tracking
Track and organize imports by project:
@ -149,7 +149,7 @@ await brain.import(file, {
})
```
### Always-On Streaming (v4.2.0+)
### Always-On Streaming
All imports use streaming with adaptive flush intervals. Query data as it's imported:

2
docs/guides/standard-import-progress.md
View file

@ -65,7 +65,7 @@ interface ImportProgress {
/** Estimated time remaining (milliseconds) */
eta?: number
/** Whether data is queryable at this point (v4.2.0+) */
/** Whether data is queryable at this point */
queryable?: boolean
}
```

4
docs/guides/streaming-imports.md
View file

@ -300,7 +300,7 @@ interface ImportProgress {
relationships?: number // Relationships inferred so far
/**
* Whether data is queryable (v4.2.0+)
* Whether data is queryable
*
* true = Indexes flushed, queries will be fast and complete
* false/undefined = Data in storage but indexes not flushed yet
@ -376,7 +376,7 @@ No changes required! Streaming is now always enabled with optimal defaults:
// Before (v3.x, v4.0, v4.1): Works the same
await brain.import(file)
// After (v4.2.0+): Streaming always on, zero config
// After: Streaming always on, zero config
await brain.import(file)
```

5
docs/operations/capacity-planning.md
View file

@ -1,6 +1,6 @@
# Capacity Planning & Operations Guide
**Brainy v3.36.0+ Enterprise Operations**
**Brainy Enterprise Operations**
This guide provides production-ready capacity planning formulas, deployment strategies, and operational guidelines for scaling Brainy from development (2GB) to enterprise (128GB+) deployments.
@ -704,8 +704,7 @@ if (stats.fairness.fairnessViolation) {
## 📚 Additional Resources
- **[Migration Guide](../guides/migration-3.36.0.md)** - Upgrading to v3.36.0
- **[Architecture Overview](../architecture/data-storage-architecture.md)** - Deep dive into storage and caching
- **[Migration Guide](../guides/migration-3.36.0.md)** - Upgrading to **[Architecture Overview](../architecture/data-storage-architecture.md)** - Deep dive into storage and caching
- **[GitHub Issues](https://github.com/soulcraftlabs/brainy/issues)** - Report problems or ask questions
---

8
docs/operations/cost-optimization-aws-s3.md
View file

@ -1,10 +1,9 @@
# AWS S3 Cost Optimization Guide for Brainy v4.0.0
# AWS S3 Cost Optimization Guide for Brainy
> **Cost Impact**: Reduce storage costs from $138k/year to $5.9k/year at 500TB scale (**96% savings**)
## Overview
Brainy v4.0.0 provides enterprise-grade cost optimization features for AWS S3 storage, enabling automatic tier transitions that dramatically reduce storage costs while maintaining performance.
Brainy provides enterprise-grade cost optimization features for AWS S3 storage, enabling automatic tier transitions that dramatically reduce storage costs while maintaining performance.
## Cost Breakdown (Before Optimization)
@ -250,7 +249,7 @@ Note: Retrieval costs may be significant if archived data is accessed frequently
### Efficient Cleanup
```typescript
// v4.0.0: Batch delete (1000 objects per request)
// Batch delete (1000 objects per request)
const idsToDelete = [/* array of entity IDs */]
// Generate paths for both vector and metadata files
@ -396,6 +395,5 @@ await storage.enableIntelligentTiering('entities/', 'new-config')
---
**Version**: v4.0.0
**Last Updated**: 2025-10-17
**Cloud Provider**: AWS S3

8
docs/operations/cost-optimization-azure.md
View file

@ -1,10 +1,9 @@
# Azure Blob Storage Cost Optimization Guide for Brainy v4.0.0
# Azure Blob Storage Cost Optimization Guide for Brainy
> **Cost Impact**: Reduce storage costs from $107k/year to $5k/year at 500TB scale (**95% savings**)
## Overview
Brainy v4.0.0 provides enterprise-grade cost optimization features for Azure Blob Storage, including manual tier management, lifecycle policies, and batch operations.
Brainy provides enterprise-grade cost optimization features for Azure Blob Storage, including manual tier management, lifecycle policies, and batch operations.
## Cost Breakdown (Before Optimization)
@ -333,7 +332,7 @@ Examples:
### Efficient Bulk Deletions
```typescript
// v4.0.0: Batch delete (256 blobs per request)
// Batch delete (256 blobs per request)
const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => {
@ -549,6 +548,5 @@ const storage = new AzureBlobStorage({
---
**Version**: v4.0.0
**Last Updated**: 2025-10-17
**Cloud Provider**: Azure Blob Storage

12
docs/operations/cost-optimization-cloudflare-r2.md
View file

@ -1,10 +1,9 @@
# Cloudflare R2 Cost Optimization Guide for Brainy v4.0.0
# Cloudflare R2 Cost Optimization Guide for Brainy
> **Cost Impact**: $0 egress fees + 96% storage savings = Lowest cloud storage costs
## Overview
Cloudflare R2 is an S3-compatible object storage with **zero egress fees**, making it ideal for high-traffic applications. Brainy v4.0.0 fully supports R2 with lifecycle policies and batch operations.
Cloudflare R2 is an S3-compatible object storage with **zero egress fees**, making it ideal for high-traffic applications. Brainy fully supports R2 with lifecycle policies and batch operations.
## Cost Breakdown
@ -49,7 +48,7 @@ Savings vs AWS: $156,000/year (62%)
- Cloudflare plans to add infrequent access tiers
**When lifecycle features arrive:**
- Brainy v4.0.0 is already prepared with `setLifecyclePolicy()` support
- Brainy is already prepared with `setLifecyclePolicy()` support
- Will work seamlessly once Cloudflare enables lifecycle management
## Strategy 1: Use R2 Standard (Current Best Practice)
@ -187,7 +186,7 @@ Savings: $193,624/year (77%)
### Efficient Bulk Deletions
```typescript
// v4.0.0: R2 supports S3 batch delete API
// R2 supports S3 batch delete API
const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => {
@ -361,7 +360,7 @@ ROI: 3.4 months
### Prepared for Future Features
```typescript
// Brainy v4.0.0 is ready for R2 lifecycle features
// Brainy is ready for R2 lifecycle features
await storage.setLifecyclePolicy({
rules: [{
id: 'archive-old-data',
@ -446,7 +445,6 @@ const storage = new S3CompatibleStorage({
---
**Version**: v4.0.0
**Last Updated**: 2025-10-17
**Cloud Provider**: Cloudflare R2
**Key Advantage**: **$0 egress fees forever**

8
docs/operations/cost-optimization-gcs.md
View file

@ -1,10 +1,9 @@
# Google Cloud Storage Cost Optimization Guide for Brainy v4.0.0
# Google Cloud Storage Cost Optimization Guide for Brainy
> **Cost Impact**: Reduce storage costs from $138k/year to $8.3k/year at 500TB scale (**94% savings**)
## Overview
Brainy v4.0.0 provides enterprise-grade cost optimization features for Google Cloud Storage, including lifecycle policies and Autoclass for automatic tier management.
Brainy provides enterprise-grade cost optimization features for Google Cloud Storage, including lifecycle policies and Autoclass for automatic tier management.
## Cost Breakdown (Before Optimization)
@ -241,7 +240,7 @@ Warning: High retrieval costs if archived data is accessed frequently
### Efficient Cleanup
```typescript
// v4.0.0: Batch delete (100 objects per request for GCS)
// Batch delete (100 objects per request for GCS)
const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => {
@ -417,6 +416,5 @@ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
---
**Version**: v4.0.0
**Last Updated**: 2025-10-17
**Cloud Provider**: Google Cloud Storage

8
docs/transactions.md
View file

@ -1,6 +1,6 @@
# Transaction System
**Status:** ✅ Production Ready (v5.8.0+)
**Status:** ✅ Production Ready
## Overview
@ -123,7 +123,7 @@ await brain.relate({ from: id1, to: id2, type: VerbType.RelatesTo })
- Transaction operations don't need to know about shards
- Rollback works across all shards involved
### ID-First Storage (v6.0.0+)
### ID-First Storage
✅ **Fully Compatible**
@ -152,7 +152,7 @@ await brain.update({
**How It Works:**
- Type information stored in metadata.noun field
- Storage layer uses O(1) ID-first path construction
- No type cache needed (removed in v6.0.0)
- No type cache needed (removed in a previous version)
- Type counters adjusted on commit/rollback
- 40x faster on cloud storage (eliminates 42-type search)
@ -563,7 +563,7 @@ interface StorageAdapter {
## Version History
- **v5.8.0**: Initial transaction system release
- Initial transaction system release
- Atomic operations with rollback
- Compatible with COW, sharding, type-aware, distributed
- 36/36 unit tests passing

2
docs/vfs/QUICK_START.md
View file

@ -294,7 +294,7 @@ npm install @soulcraft/brainy@latest # Update if needed
### "VFS not initialized" errors
```typescript
// v5.1.0+: Just await brain.init() - VFS is auto-initialized!
// Just await brain.init() - VFS is auto-initialized!
await brain.init()
// VFS ready - use brain.vfs directly
await brain.vfs.writeFile('/test.txt', 'data')

6
docs/vfs/README.md
View file

@ -48,11 +48,11 @@ const results = await vfs.search('files about authentication')
await vfs.addRelationship('/docs/spec.md', '/projects/my-app/', 'implements')
```
## ⚡ Performance (v5.11.1)
## ⚡ Performance
**75% Faster File Operations!** VFS now automatically benefits from brain.get() metadata-only optimization:
| Operation | Before (v5.11.0) | After (v5.11.1) | Speedup |
| Operation | Before | After | Speedup |
|-----------|------------------|-----------------|---------|
| `readFile()` | 53ms | **~13ms** | **75%** |
| `stat()` | 53ms | **~13ms** | **75%** |
@ -60,7 +60,7 @@ await vfs.addRelationship('/docs/spec.md', '/projects/my-app/', 'implements')
**Zero configuration** - automatic optimization for all VFS operations!
VFS operations only need metadata (path, size, timestamps), not 384-dimensional vector embeddings. The v5.11.1 optimization automatically uses metadata-only reads, saving 95% bandwidth and 76-81% time.
VFS operations only need metadata (path, size, timestamps), not 384-dimensional vector embeddings. The optimization automatically uses metadata-only reads, saving 95% bandwidth and 76-81% time.
## Core Features

6
docs/vfs/TROUBLESHOOTING.md
View file

@ -12,8 +12,8 @@
**Root Cause:**
The root directory entity exists but doesn't have the proper metadata structure.
**Solution (v3.15.0+):**
This issue has been fixed in v3.15.0. The VFS now:
**Solution:**
This issue has been fixed. The VFS now:
1. Ensures root directory has `vfsType: 'directory'` metadata
2. Adds compatibility layer for entities with malformed metadata
3. Automatically repairs metadata on entity retrieval
@ -47,7 +47,7 @@ await brain.update({
**Root Cause:**
Contains relationships are missing between parent directories and files.
**Solution (v3.15.0+):**
**Solution:**
This issue has been fixed. The VFS now:
1. Creates Contains relationships when writing new files
2. Ensures Contains relationships exist when updating files

4
docs/vfs/VFS_CORE.md
View file

@ -418,7 +418,7 @@ await vfs.importDirectory()// Import directory from filesystem
## Bulk Write Operations
**v6.5.0**: `bulkWrite` efficiently processes multiple VFS operations with automatic ordering to prevent race conditions.
`bulkWrite` efficiently processes multiple VFS operations with automatic ordering to prevent race conditions.
```javascript
const result = await vfs.bulkWrite([
@ -439,7 +439,7 @@ console.log(`Successful: ${result.successful}, Failed: ${result.failed.length}`)
- `delete` - Delete file
- `update` - Update file metadata only
**Operation ordering (v6.5.0):**
**Operation ordering:**
1. `mkdir` operations run **first**, sequentially, sorted by path depth (shallowest first)
2. Other operations (`write`, `delete`, `update`) run **after** in parallel batches of 10

8
docs/vfs/VFS_INITIALIZATION.md
View file

@ -1,4 +1,4 @@
# VFS Initialization Guide (v5.1.0+)
# VFS Initialization Guide
## Quick Start
@ -30,7 +30,7 @@ await vfs.init() // ❌ Separate initialization
await vfs.writeFile(...)
```
### After (v5.1.0+):
### After:
```javascript
const brain = new Brainy(...)
await brain.init() // VFS auto-initialized!
@ -53,7 +53,7 @@ await vfs.init()
await vfs.writeFile('/test.txt', 'data')
```
### New Pattern (v5.1.0+):
### New Pattern:
```javascript
// Just remove the () and init() call
await brain.vfs.writeFile('/test.txt', 'data')
@ -142,7 +142,7 @@ await brain.init() // Required!
await brain.vfs.writeFile(...) // Now this works
```
## Fork Support (v5.0.0+)
## Fork Support
VFS works seamlessly with Brainy's Copy-on-Write (COW) fork feature:

12
src/api/ConfigAPI.ts
View file

@ -60,7 +60,7 @@ export class ConfigAPI {
// Store in cache
this.configCache.set(key, entry)
// v4.0.0: Persist to storage as NounMetadata
// Persist to storage as NounMetadata
const configId = this.CONFIG_NOUN_PREFIX + key
const nounMetadata = {
noun: 'config' as any,
@ -94,7 +94,7 @@ export class ConfigAPI {
return defaultValue
}
// v4.0.0: Extract ConfigEntry from NounMetadata
// Extract ConfigEntry from NounMetadata
const createdAtVal = typeof metadata.createdAt === 'object' && metadata.createdAt !== null && 'seconds' in metadata.createdAt
? metadata.createdAt.seconds * 1000 + Math.floor((metadata.createdAt.nanoseconds || 0) / 1000000)
: ((metadata.createdAt as unknown as number) || Date.now())
@ -140,7 +140,7 @@ export class ConfigAPI {
// Remove from cache
this.configCache.delete(key)
// v4.0.0: Remove from storage
// Remove from storage
const configId = this.CONFIG_NOUN_PREFIX + key
await this.storage.deleteNounMetadata(configId)
}
@ -149,7 +149,7 @@ export class ConfigAPI {
* List all configuration keys
*/
async list(): Promise<string[]> {
// v4.0.0: Get all nouns and filter for config entries
// Get all nouns and filter for config entries
const result = await this.storage.getNouns({ pagination: { limit: 10000 } })
const configKeys: string[] = []
@ -213,7 +213,7 @@ export class ConfigAPI {
for (const [key, entry] of Object.entries(config)) {
this.configCache.set(key, entry)
const configId = this.CONFIG_NOUN_PREFIX + key
// v4.0.0: Convert ConfigEntry to NounMetadata
// Convert ConfigEntry to NounMetadata
const nounMetadata = {
noun: 'config' as any,
...entry,
@ -240,7 +240,7 @@ export class ConfigAPI {
return null
}
// v4.0.0: Extract ConfigEntry from NounMetadata
// Extract ConfigEntry from NounMetadata
const createdAtVal = typeof metadata.createdAt === 'object' && metadata.createdAt !== null && 'seconds' in metadata.createdAt
? metadata.createdAt.seconds * 1000 + Math.floor((metadata.createdAt.nanoseconds || 0) / 1000000)
: ((metadata.createdAt as unknown as number) || Date.now())

2
src/api/DataAPI.ts
View file

@ -210,7 +210,7 @@ export class DataAPI {
this.validateImportItem(mapped)
}
// v4.0.0: Save entity - separate vector and metadata
// Save entity - separate vector and metadata
const id = mapped.id || this.generateId()
const noun: HNSWNoun = {
id,

10
src/augmentations/brainyAugmentation.ts
View file

@ -375,7 +375,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
}
/**
* Get CommitLog for temporal features (v5.0.0+)
* Get CommitLog for temporal features
*
* Provides access to commit history for time-travel queries, audit trails,
* and branch management. Available after initialize() is called.
@ -414,7 +414,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
}
/**
* Get BlobStorage for content-addressable storage (v5.0.0+)
* Get BlobStorage for content-addressable storage
*
* Provides access to the underlying blob storage system for storing
* and retrieving content-addressed data. Available after initialize() is called.
@ -454,7 +454,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
}
/**
* Get RefManager for branch/ref management (v5.0.0+)
* Get RefManager for branch/ref management
*
* Provides access to the reference manager for creating, updating,
* and managing Git-style branches and refs. Available after initialize() is called.
@ -496,7 +496,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
}
/**
* Get current branch name (v5.0.0+)
* Get current branch name
*
* Convenience helper for getting the current branch from the Brainy instance.
* Available after initialize() is called.
@ -525,7 +525,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
if (typeof brain.getCurrentBranch !== 'function') {
throw new Error(
`${this.name}: getCurrentBranch() not available on Brainy instance. ` +
`This method requires Brainy v5.0.0+.`
`This method requires Brainy with VFS support.`
)
}

2
src/augmentations/intelligentImport/FormatHandlerRegistry.ts
View file

@ -1,5 +1,5 @@
/**
* Format Handler Registry (v5.2.0)
* Format Handler Registry
*
* Central registry for format handlers with:
* - MIME type-based routing

2
src/augmentations/intelligentImport/IntelligentImportAugmentation.ts
View file

@ -35,7 +35,7 @@ export class IntelligentImportAugmentation extends BaseAugmentation {
enableCSV: true,
enableExcel: true,
enablePDF: true,
enableImage: true, // v5.2.0: Image handler enabled by default
enableImage: true, // Image handler enabled by default
maxFileSize: 100 * 1024 * 1024, // 100MB default
enableCache: true,
cacheTTL: 24 * 60 * 60 * 1000, // 24 hours

12
src/augmentations/intelligentImport/handlers/csvHandler.ts
View file

@ -40,7 +40,7 @@ export class CSVHandler extends BaseFormatHandler {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8')
const totalBytes = buffer.length
// v4.5.0: Report total bytes for progress tracking
// Report total bytes for progress tracking
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0)
}
@ -55,7 +55,7 @@ export class CSVHandler extends BaseFormatHandler {
// Detect delimiter if not specified
const delimiter = options.csvDelimiter || this.detectDelimiter(text)
// v4.5.0: Report progress - parsing started
// Report progress - parsing started
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`)
}
@ -75,7 +75,7 @@ export class CSVHandler extends BaseFormatHandler {
cast: false // We'll do type inference ourselves
})
// v4.5.0: Report bytes processed (entire file parsed)
// Report bytes processed (entire file parsed)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
@ -83,7 +83,7 @@ export class CSVHandler extends BaseFormatHandler {
// Convert to array of objects
const data = Array.isArray(records) ? records : [records]
// v4.5.0: Report data extraction progress
// Report data extraction progress
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(data.length, data.length)
}
@ -101,7 +101,7 @@ export class CSVHandler extends BaseFormatHandler {
converted[key] = this.convertValue(value, types[key] || 'string')
}
// v4.5.0: Report progress every 1000 rows
// Report progress every 1000 rows
if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`)
}
@ -111,7 +111,7 @@ export class CSVHandler extends BaseFormatHandler {
const processingTime = Date.now() - startTime
// v4.5.0: Final progress update
// Final progress update
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`)
}

18
src/augmentations/intelligentImport/handlers/excelHandler.ts
View file

@ -27,7 +27,7 @@ export class ExcelHandler extends BaseFormatHandler {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'binary')
const totalBytes = buffer.length
// v4.5.0: Report start
// Report start
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0)
}
@ -47,7 +47,7 @@ export class ExcelHandler extends BaseFormatHandler {
// Determine which sheets to process
const sheetsToProcess = this.getSheetsToProcess(workbook, options)
// v4.5.0: Report workbook loaded
// Report workbook loaded
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Processing ${sheetsToProcess.length} sheets...`)
}
@ -59,7 +59,7 @@ export class ExcelHandler extends BaseFormatHandler {
for (let sheetIndex = 0; sheetIndex < sheetsToProcess.length; sheetIndex++) {
const sheetName = sheetsToProcess[sheetIndex]
// v4.5.0: Report current sheet
// Report current sheet
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(
`Reading sheet: ${sheetName} (${sheetIndex + 1}/${sheetsToProcess.length})`
@ -117,19 +117,19 @@ export class ExcelHandler extends BaseFormatHandler {
headers
}
// v4.5.0: Estimate bytes processed (sheets are sequential)
// Estimate bytes processed (sheets are sequential)
const bytesProcessed = Math.floor(((sheetIndex + 1) / sheetsToProcess.length) * totalBytes)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed)
}
// v4.5.0: Report extraction progress
// Report extraction progress
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until complete
}
}
// v4.5.0: Report data extraction complete
// Report data extraction complete
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Extracted ${allData.length} rows, inferring types...`)
}
@ -152,7 +152,7 @@ export class ExcelHandler extends BaseFormatHandler {
}
}
// v4.5.0: Report progress every 1000 rows (avoid spam)
// Report progress every 1000 rows (avoid spam)
if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
progressHooks.onCurrentItem(`Converting types: ${index}/${allData.length} rows...`)
}
@ -160,14 +160,14 @@ export class ExcelHandler extends BaseFormatHandler {
return converted
})
// v4.5.0: Final progress - all bytes processed
// Final progress - all bytes processed
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
const processingTime = Date.now() - startTime
// v4.5.0: Report completion
// Report completion
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(
`Excel complete: ${sheetsToProcess.length} sheets, ${convertedData.length} rows`

6
src/augmentations/intelligentImport/handlers/imageHandler.ts
View file

@ -1,5 +1,5 @@
/**
* Image Import Handler (v5.8.0 - Pure JavaScript)
* Image Import Handler
*
* Handles image files with:
* - EXIF metadata extraction (camera, GPS, timestamps) via exifr
@ -7,7 +7,7 @@
* - Support for JPEG, PNG, WebP, GIF, TIFF, BMP, SVG
*
* NO NATIVE DEPENDENCIES - Pure JavaScript implementation
* Replaces Sharp (v5.7.x) with lightweight pure-JS alternatives
* Replaces Sharp with lightweight pure-JS alternatives
*/
import { BaseFormatHandler } from './base.js'
@ -96,7 +96,7 @@ export interface ImageHandlerOptions extends FormatHandlerOptions {
* Enables developers to import images into the knowledge graph with
* full metadata extraction.
*
* v5.8.0: Pure JavaScript implementation (no native dependencies)
* Pure JavaScript implementation (no native dependencies)
*/
export class ImageHandler extends BaseFormatHandler {
readonly format = 'image'

14
src/augmentations/intelligentImport/handlers/pdfHandler.ts
View file

@ -52,7 +52,7 @@ export class PDFHandler extends BaseFormatHandler {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'binary')
const totalBytes = buffer.length
// v4.5.0: Report start
// Report start
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0)
}
@ -74,7 +74,7 @@ export class PDFHandler extends BaseFormatHandler {
const metadata = await pdfDoc.getMetadata()
const numPages = pdfDoc.numPages
// v4.5.0: Report document loaded
// Report document loaded
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Processing ${numPages} pages...`)
}
@ -85,7 +85,7 @@ export class PDFHandler extends BaseFormatHandler {
let detectedTables = 0
for (let pageNum = 1; pageNum <= numPages; pageNum++) {
// v4.5.0: Report current page
// Report current page
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Processing page ${pageNum} of ${numPages}`)
}
@ -131,19 +131,19 @@ export class PDFHandler extends BaseFormatHandler {
}
}
// v4.5.0: Estimate bytes processed (pages are sequential)
// Estimate bytes processed (pages are sequential)
const bytesProcessed = Math.floor((pageNum / numPages) * totalBytes)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed)
}
// v4.5.0: Report extraction progress
// Report extraction progress
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until complete
}
}
// v4.5.0: Final progress - all bytes processed
// Final progress - all bytes processed
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
@ -153,7 +153,7 @@ export class PDFHandler extends BaseFormatHandler {
const processingTime = Date.now() - startTime
// v4.5.0: Report completion
// Report completion
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(
`PDF complete: ${numPages} pages, ${allData.length} items extracted`

4
src/augmentations/intelligentImport/index.ts
View file

@ -18,14 +18,14 @@ export { ExcelHandler } from './handlers/excelHandler.js'
export { PDFHandler } from './handlers/pdfHandler.js'
export { ImageHandler } from './handlers/imageHandler.js'
// Format Handler Registry (v5.2.0)
// Format Handler Registry
export {
FormatHandlerRegistry,
globalHandlerRegistry
} from './FormatHandlerRegistry.js'
export type { HandlerRegistration } from './FormatHandlerRegistry.js'
// Image Handler Types (v5.2.0)
// Image Handler Types
export type {
ImageMetadata,
EXIFData,

8
src/augmentations/intelligentImport/types.ts
View file

@ -88,13 +88,13 @@ export interface FormatHandlerOptions {
streaming?: boolean
/**
* Progress hooks (v4.5.0)
* Progress hooks
* Handlers call these to report progress during processing
*/
progressHooks?: FormatHandlerProgressHooks
/**
* Total file size in bytes (v4.5.0)
* Total file size in bytes
* Used for progress percentage calculation
*/
totalBytes?: number
@ -168,7 +168,7 @@ export interface IntelligentImportConfig {
/** Enable PDF handler */
enablePDF: boolean
/** Enable Image handler (v5.2.0) */
/** Enable Image handler */
enableImage: boolean
/** Default options for CSV */
@ -180,7 +180,7 @@ export interface IntelligentImportConfig {
/** Default options for PDF */
pdfDefaults?: Partial<FormatHandlerOptions>
/** Default options for Image (v5.2.0) */
/** Default options for Image */
imageDefaults?: Partial<FormatHandlerOptions>
/** Maximum file size to process (bytes) */

2
src/augmentations/typeMatching/brainyTypes.ts
View file

@ -168,7 +168,7 @@ export interface TypeMatchResult {
/**
* BrainyTypes - Intelligent type detection for nouns and verbs
* PRODUCTION OPTIMIZATION (v3.33.0): Uses pre-computed type embeddings
* PRODUCTION OPTIMIZATION: Uses pre-computed type embeddings
* Type embeddings are loaded instantly; only input objects are embedded at runtime
*/
export class BrainyTypes {

4
src/cli/commands/core.ts
View file

@ -172,7 +172,7 @@ export const coreCommands = {
metadata
}
// v4.3.x: Add confidence and weight if provided
// Add confidence and weight if provided
if (options.confidence) {
addParams.confidence = parseFloat(options.confidence)
}
@ -347,7 +347,7 @@ export const coreCommands = {
searchParams.includeRelations = true
}
// v4.7.0: VFS is now part of the knowledge graph (included by default)
// VFS is now part of the knowledge graph (included by default)
// Users can exclude VFS with --where vfsType exists:false if needed
// Triple Intelligence Fusion - custom weighting

4
src/cli/commands/cow.ts
View file

@ -312,7 +312,7 @@ ${chalk.cyan('Fork Statistics:')}
},
/**
* Migrate from v4.x to v5.0.0 (one-time)
* Migrate storage format (one-time)
*/
async migrate(options: MigrateOptions) {
let spinner: any = null
@ -388,7 +388,7 @@ ${chalk.cyan('Fork Statistics:')}
await oldBrain.init()
// Create new brain (v5.0.0)
// Create new brain
const newBrain = new Brainy({
storage: {
type: 'filesystem',

4
src/cli/commands/neural.ts
View file

@ -117,7 +117,7 @@ export const neuralCommand = {
await handleNeighborsCommand(neural, argv)
break
case 'path':
console.log(chalk.yellow('\n⚠️ Semantic path finding coming in v3.21.0'))
console.log(chalk.yellow('\n⚠️ Semantic path finding coming soon'))
console.log(chalk.dim('This feature requires implementing graph traversal algorithms'))
console.log(chalk.dim('Use "neighbors" and "hierarchy" commands to explore connections'))
break
@ -148,7 +148,7 @@ async function promptForAction(): Promise<string> {
{ name: '🎯 Find semantic clusters', value: 'clusters' },
{ name: '🌳 Show item hierarchy', value: 'hierarchy' },
{ name: '🕸️ Find semantic neighbors', value: 'neighbors' },
{ name: '🛣️ Find semantic path between items (v3.21.0)', value: 'path', disabled: true },
{ name: '🛣️ Find semantic path between items (coming soon)', value: 'path', disabled: true },
{ name: '🚨 Detect outliers', value: 'outliers' },
{ name: '📊 Generate visualization data', value: 'visualize' }
]

2
src/cli/commands/storage.ts
View file

@ -1,5 +1,5 @@
/**
* 💾 Storage Management Commands - v4.0.0
* 💾 Storage Management Commands
*
* Modern interactive CLI for storage lifecycle, cost optimization, and management
*/

2
src/cli/commands/utility.ts
View file

@ -133,7 +133,7 @@ export const utilityCommands = {
// removeOrphans and rebuildIndex would require new Brainy APIs
if (options.removeOrphans || options.rebuildIndex) {
spinner.warn('Advanced cleanup options not yet implemented')
console.log(chalk.yellow('\n⚠️ Advanced cleanup features coming in v3.21.0:'))
console.log(chalk.yellow('\n⚠️ Advanced cleanup features coming soon:'))
console.log(chalk.dim(' • --remove-orphans: Remove disconnected items'))
console.log(chalk.dim(' • --rebuild-index: Rebuild vector index'))
console.log(chalk.dim('\nUse "brainy clean" without options to clear the database'))

6
src/cli/commands/vfs.ts
View file

@ -21,7 +21,7 @@ let brainyInstance: Brainy | null = null
const getBrainy = async (): Promise<Brainy> => {
if (!brainyInstance) {
brainyInstance = new Brainy()
await brainyInstance.init() // v5.0.1: Initialize brain (VFS auto-initialized here!)
await brainyInstance.init() // Initialize brain (VFS auto-initialized here!)
}
return brainyInstance
}
@ -52,8 +52,8 @@ export const vfsCommands = {
const spinner = ora('Reading file...').start()
try {
const brain = await getBrainy() // v5.0.1: Await async getBrainy
// v5.0.1: VFS auto-initialized, no need for vfs.init()
const brain = await getBrainy() // Await async getBrainy
// VFS auto-initialized, no need for vfs.init()
const buffer = await brain.vfs.readFile(path, {
encoding: options.encoding as any
})

10
src/cli/index.ts
View file

@ -68,7 +68,7 @@ ${chalk.cyan('Examples:')}
$ brainy vfs search "React components"
$ brainy vfs similar /code/Button.tsx
${chalk.dim('# Storage management (v4.0.0)')}
${chalk.dim('# Storage management')}
$ brainy storage status --quota
$ brainy storage lifecycle set ${chalk.dim('# Interactive mode')}
$ brainy storage cost-estimate
@ -217,11 +217,11 @@ program
program
.command('path <from> <to>')
.description('Find semantic path between items (v3.21.0)')
.description('Find semantic path between items')
.option('--steps', 'Show step-by-step path')
.option('--max-hops <number>', 'Maximum path length', '5')
.action(() => {
console.log(chalk.yellow('\n⚠️ Semantic path finding coming in v3.21.0'))
console.log(chalk.yellow('\n⚠️ Semantic path finding coming soon'))
console.log(chalk.dim('This feature requires implementing graph traversal algorithms'))
console.log(chalk.dim('Use "brainy neighbors" and "brainy hierarchy" to explore connections'))
})
@ -446,7 +446,7 @@ program
vfsCommands.tree(path, options)
})
// ===== Storage Management Commands (v4.0.0) =====
// ===== Storage Management Commands =====
program
.command('storage')
@ -610,7 +610,7 @@ program
.option('--iterations <n>', 'Number of iterations', '100')
.action(utilityCommands.benchmark)
// ===== COW Commands (v5.0.0) - Instant Fork & Branching =====
// ===== COW Commands - Instant Fork & Branching =====
program
.command('fork [name]')

2
src/cli/interactive.ts
View file

@ -625,7 +625,7 @@ export async function promptCommand(): Promise<string> {
*/
export async function startInteractiveMode() {
console.log(chalk.cyan('\n🧠 Brainy Interactive Mode\n'))
console.log(chalk.yellow('Interactive REPL mode coming in v3.20.0\n'))
console.log(chalk.yellow('Interactive REPL mode coming soon\n'))
console.log(chalk.dim('Use specific commands for now: brainy add, brainy search, etc.'))
process.exit(0)
}

68
src/coreTypes.ts
View file

@ -57,7 +57,7 @@ export type DistanceFunction = (a: Vector, b: Vector) => number
/**
* Embedding function for converting data to vectors
* v4.0.0: Now properly typed - accepts string, string array (batch), or object, no `any`
* Now properly typed - accepts string, string array (batch), or object, no `any`
*/
export type EmbeddingFunction = (data: string | string[] | Record<string, unknown>) => Promise<Vector>
@ -72,7 +72,7 @@ export interface EmbeddingModel {
/**
* Embed data into a vector
* v4.0.0: Now properly typed - accepts string, string array (batch), or object, no `any`
* Now properly typed - accepts string, string array (batch), or object, no `any`
*/
embed(data: string | string[] | Record<string, unknown>): Promise<Vector>
@ -83,9 +83,9 @@ export interface EmbeddingModel {
}
/**
* HNSW graph noun - Pure vector structure (v4.0.0)
* HNSW graph noun - Pure vector structure
*
* v4.0.0 BREAKING CHANGE: metadata field removed
* metadata field removed
* - Stores ONLY vector data for optimal memory usage
* - Metadata stored separately and combined on retrieval
* - 25% memory reduction @ 1B scale (no in-memory metadata)
@ -100,15 +100,15 @@ export interface HNSWNoun {
}
/**
* Lightweight verb for HNSW index storage - Core relational structure (v4.0.0)
* Lightweight verb for HNSW index storage - Core relational structure
*
* Core fields (v3.50.1+): verb/sourceId/targetId are first-class fields
* Core fields: verb/sourceId/targetId are first-class fields
* These are NOT metadata - they're the essence of what a verb IS:
* - verb: The relationship type (creates, contains, etc.) - needed for routing & display
* - sourceId: What entity this verb connects FROM - needed for graph traversal
* - targetId: What entity this verb connects TO - needed for graph traversal
*
* v4.0.0 BREAKING CHANGE: metadata field removed
* metadata field removed
* - Stores ONLY vector + core relational data
* - User metadata (weight, custom fields) stored separately
* - 10x faster metadata-only updates (skip HNSW rebuild)
@ -134,9 +134,9 @@ export interface HNSWVerb {
}
/**
* Noun metadata structure (v4.8.0)
* Noun metadata structure
*
* v4.8.0 BREAKING CHANGE: Now contains ONLY custom user-defined fields
* Now contains ONLY custom user-defined fields
* - Standard fields (confidence, weight, timestamps, etc.) moved to top-level in HNSWNounWithMetadata
* - This interface represents custom metadata stored separately from vector data
* - Storage format unchanged (backward compatible at storage layer)
@ -162,9 +162,9 @@ export interface NounMetadata {
}
/**
* Verb metadata structure (v4.8.0)
* Verb metadata structure
*
* v4.8.0 BREAKING CHANGE: Now contains ONLY custom user-defined fields
* Now contains ONLY custom user-defined fields
* - Standard fields (weight, confidence, timestamps, etc.) moved to top-level in HNSWVerbWithMetadata
* - This interface represents custom metadata stored separately from vector + core relational data
* - Storage format unchanged (backward compatible at storage layer)
@ -190,9 +190,9 @@ export interface VerbMetadata {
}
/**
* Combined noun structure for transport/API boundaries (v4.8.0)
* Combined noun structure for transport/API boundaries
*
* v4.8.0 BREAKING CHANGE: Standard fields moved to top-level
* Standard fields moved to top-level
* - ALL standard fields (confidence, weight, timestamps, etc.) are now at top-level
* - metadata contains ONLY custom user-defined fields
* - Provides clean, predictable API: entity.confidence always works
@ -230,9 +230,9 @@ export interface HNSWNounWithMetadata {
}
/**
* Combined verb structure for transport/API boundaries (v4.8.0)
* Combined verb structure for transport/API boundaries
*
* v4.8.0 BREAKING CHANGE: Standard fields moved to top-level
* Standard fields moved to top-level
* - ALL standard fields (weight, confidence, timestamps, etc.) are now at top-level
* - metadata contains ONLY custom user-defined fields
* - Provides clean, predictable API: verb.weight always works
@ -308,7 +308,7 @@ export interface HNSWConfig {
efSearch: number // Size of the dynamic candidate list during search
ml: number // Maximum level
useDiskBasedIndex?: boolean // Whether to use disk-based index
maxConcurrentNeighborWrites?: number // Maximum concurrent neighbor updates during insert (v4.10.0+). Default: unlimited (full concurrency)
maxConcurrentNeighborWrites?: number // Maximum concurrent neighbor updates during insert. Default: unlimited (full concurrency)
}
/**
@ -548,7 +548,7 @@ export interface StatisticsData {
}
/**
* Change record for getChangesSince (v4.0.0)
* Change record for getChangesSince
* Replaces `any[]` with properly typed structure
*/
export interface Change {
@ -563,27 +563,27 @@ export interface StorageAdapter {
init(): Promise<void>
/**
* Save noun - Pure HNSW vector data only (v4.0.0)
* Save noun - Pure HNSW vector data only
* @param noun Pure HNSW vector data (no metadata)
* Note: Use saveNounMetadata() to save metadata separately
*/
saveNoun(noun: HNSWNoun): Promise<void>
/**
* Save noun metadata separately (v4.0.0)
* Save noun metadata separately
* @param id Noun ID
* @param metadata Noun metadata
*/
saveNounMetadata(id: string, metadata: NounMetadata): Promise<void>
/**
* Delete noun metadata (v4.0.0)
* Delete noun metadata
* @param id Noun ID
*/
deleteNounMetadata(id: string): Promise<void>
/**
* Get noun with metadata combined (v4.0.0)
* Get noun with metadata combined
* @returns Combined HNSWNounWithMetadata or null
*/
getNoun(id: string): Promise<HNSWNounWithMetadata | null>
@ -622,14 +622,14 @@ export interface StorageAdapter {
deleteNoun(id: string): Promise<void>
/**
* Save verb - Pure HNSW verb with core fields only (v4.0.0)
* Save verb - Pure HNSW verb with core fields only
* @param verb Pure HNSW verb data (vector + core fields, no user metadata)
* Note: Use saveVerbMetadata() to save metadata separately
*/
saveVerb(verb: HNSWVerb): Promise<void>
/**
* Get verb with metadata combined (v4.0.0)
* Get verb with metadata combined
* @returns Combined HNSWVerbWithMetadata or null
*/
getVerb(id: string): Promise<HNSWVerbWithMetadata | null>
@ -686,14 +686,14 @@ export interface StorageAdapter {
deleteVerb(id: string): Promise<void>
/**
* Save metadata (v4.0.0: now typed)
* Save metadata
* @param id Entity ID
* @param metadata Typed noun metadata
*/
saveMetadata(id: string, metadata: NounMetadata): Promise<void>
/**
* Get metadata (v4.0.0: now typed)
* Get metadata
* @param id Entity ID
* @returns Typed noun metadata or null
*/
@ -702,26 +702,26 @@ export interface StorageAdapter {
/**
* Get multiple metadata objects in batches (prevents socket exhaustion)
* @param ids Array of IDs to get metadata for
* @returns Promise that resolves to a Map of id -> metadata (v4.0.0: typed)
* @returns Promise that resolves to a Map of id -> metadata
*/
getMetadataBatch?(ids: string[]): Promise<Map<string, NounMetadata>>
/**
* Get noun metadata from storage (v4.0.0: now typed)
* Get noun metadata from storage
* @param id The ID of the noun
* @returns Promise that resolves to the metadata or null if not found
*/
getNounMetadata(id: string): Promise<NounMetadata | null>
/**
* Batch get multiple nouns with vectors (v6.2.0 - N+1 fix)
* Batch get multiple nouns with vectors
* @param ids Array of noun IDs to fetch
* @returns Map of id → HNSWNounWithMetadata (only successful reads included)
*/
getNounBatch?(ids: string[]): Promise<Map<string, HNSWNounWithMetadata>>
/**
* Save verb metadata to storage (v4.0.0: now typed)
* Save verb metadata to storage
* @param id The ID of the verb
* @param metadata The metadata to save
* @returns Promise that resolves when the metadata is saved
@ -729,14 +729,14 @@ export interface StorageAdapter {
saveVerbMetadata(id: string, metadata: VerbMetadata): Promise<void>
/**
* Get verb metadata from storage (v4.0.0: now typed)
* Get verb metadata from storage
* @param id The ID of the verb
* @returns Promise that resolves to the metadata or null if not found
*/
getVerbMetadata(id: string): Promise<VerbMetadata | null>
/**
* Batch get multiple verbs (v6.2.0 - N+1 fix)
* Batch get multiple verbs
* @param ids Array of verb IDs to fetch
* @returns Map of id → HNSWVerbWithMetadata (only successful reads included)
*/
@ -745,7 +745,7 @@ export interface StorageAdapter {
clear(): Promise<void>
/**
* Batch delete multiple objects from storage (v4.0.0)
* Batch delete multiple objects from storage
* Efficient deletion of large numbers of entities using cloud provider batch APIs.
* Significantly faster and cheaper than individual deletes (up to 1000x speedup).
*
@ -853,7 +853,7 @@ export interface StorageAdapter {
flushStatisticsToStorage(): Promise<void>
/**
* Track field names from a JSON document (v4.0.0: now typed)
* Track field names from a JSON document
* @param jsonDocument The JSON document to extract field names from
* @param service The service that inserted the data
*/
@ -872,7 +872,7 @@ export interface StorageAdapter {
getStandardFieldMappings(): Promise<Record<string, Record<string, string[]>>>
/**
* Get changes since a specific timestamp (v4.0.0: now typed)
* Get changes since a specific timestamp
* @param timestamp The timestamp to get changes since
* @param limit Optional limit on the number of changes to return
* @returns Promise that resolves to an array of properly typed changes

4
src/embeddings/candle-wasm/src/lib.rs
View file

@ -26,7 +26,7 @@ use js_sys::{Array, Float32Array};
use tokenizers::Tokenizer;
use wasm_bindgen::prelude::*;
// v7.2.0: Model weights are NO LONGER embedded in WASM
// Model weights are NO LONGER embedded in WASM
//
// Previous design: 90MB WASM with model weights embedded via include_bytes!()
// Problem: WASM parsing/compilation took 139+ seconds on throttled CPU (Cloud Run)
@ -78,7 +78,7 @@ impl EmbeddingEngine {
/// Load the model and tokenizer from bytes
///
/// v7.2.0: This is now the ONLY way to initialize the engine.
/// This is now the ONLY way to initialize the engine.
/// Model weights are no longer embedded in WASM for faster initialization.
///
/// # Arguments

10
src/embeddings/wasm/CandleEmbeddingEngine.ts
View file

@ -5,7 +5,7 @@
* Pure Rust/WASM implementation for sentence embeddings.
* Works with Bun, Node.js, Bun --compile, and browsers.
*
* v7.2.0 Architecture (20x faster initialization):
* Architecture (20x faster initialization):
* - WASM file: ~2.4MB (inference code only)
* - Model files: ~88MB (loaded separately as raw bytes)
* - Init time: ~5-7 seconds (vs 139 seconds with embedded model)
@ -34,7 +34,7 @@ interface CandleWasmModule {
}
interface CandleEngineInstance {
// v7.2.0: load() is the only initialization method (no more embedded model)
// load() is the only initialization method (no more embedded model)
load(modelBytes: Uint8Array, tokenizerBytes: Uint8Array, configBytes: Uint8Array): void
is_ready(): boolean
embed(text: string): Float32Array
@ -54,7 +54,7 @@ let globalInitPromise: Promise<void> | null = null
* Uses the Candle ML framework (Rust/WASM) for inference.
* Supports all-MiniLM-L6-v2 with 384-dimensional embeddings.
*
* v7.2.0: Model weights are loaded separately from WASM for 20x faster init.
* Model weights are loaded separately from WASM for 20x faster init.
* For bun --compile deployments, both WASM and model files are automatically
* embedded in the binary - single file deployment still works.
*/
@ -104,7 +104,7 @@ export class CandleEmbeddingEngine {
/**
* Perform actual initialization
*
* v7.2.0: WASM and model files are loaded separately for 20x faster init.
* WASM and model files are loaded separately for 20x faster init.
* - WASM (~2.4MB): Compiles in ~3-5 seconds
* - Model (~88MB): Loads as raw bytes in ~1-2 seconds
* - Total: ~5-7 seconds (vs 139 seconds with embedded model)
@ -153,7 +153,7 @@ export class CandleEmbeddingEngine {
/**
* Load the WASM module
*
* v7.2.0: WASM is now only ~2.4MB (inference code only, no model weights).
* WASM is now only ~2.4MB (inference code only, no model weights).
* Uses wasmLoader.ts for cross-environment compatibility.
*/
private async loadWasmModule(): Promise<CandleWasmModule> {

2
src/embeddings/wasm/modelLoader.ts
View file

@ -1,7 +1,7 @@
/**
* Universal Model Loader for Candle Embeddings
*
* v7.2.0: Model weights are now loaded separately from WASM for faster initialization.
* Model weights are now loaded separately from WASM for faster initialization.
* This reduces WASM compilation time from 139 seconds to ~3-5 seconds.
*
* Loads model files from appropriate source based on environment:

37
src/graph/graphAdjacencyIndex.ts
View file

@ -45,7 +45,7 @@ export class GraphAdjacencyIndex {
private lsmTreeVerbsBySource: LSMTree // sourceId -> verbIds
private lsmTreeVerbsByTarget: LSMTree // targetId -> verbIds
// v5.7.0: ID-only tracking for billion-scale memory optimization
// ID-only tracking for billion-scale memory optimization
// Previous: Map<string, GraphVerb> stored full objects (128GB @ 1B verbs)
// Now: Set<string> stores only IDs (~100KB @ 1B verbs) = 1,280,000x reduction
private verbIdSet = new Set<string>()
@ -117,7 +117,7 @@ export class GraphAdjacencyIndex {
/**
* Initialize the graph index (lazy initialization)
* v6.3.0: Added defensive auto-rebuild check for verbIdSet consistency
* Added defensive auto-rebuild check for verbIdSet consistency
*/
private async ensureInitialized(): Promise<void> {
if (this.initialized) {
@ -129,7 +129,7 @@ export class GraphAdjacencyIndex {
await this.lsmTreeVerbsBySource.init()
await this.lsmTreeVerbsByTarget.init()
// v6.3.0: Defensive check - if LSM-trees have data but verbIdSet is empty,
// Defensive check - if LSM-trees have data but verbIdSet is empty,
// the index was created without proper rebuild (shouldn't happen with singleton
// pattern but protects against edge cases and future refactoring)
const lsmTreeSize = this.lsmTreeVerbsBySource.size()
@ -151,7 +151,7 @@ export class GraphAdjacencyIndex {
}
/**
* Populate verbIdSet from storage without full rebuild (v6.3.0)
* Populate verbIdSet from storage without full rebuild
* Lighter weight than full rebuild - only loads verb IDs, not all verb data
* @private
*/
@ -192,7 +192,7 @@ export class GraphAdjacencyIndex {
* Core API - Neighbor lookup with LSM-tree storage
*
* O(log n) with bloom filter optimization (90% of queries skip disk I/O)
* v5.8.0: Added pagination support for high-degree nodes
* Added pagination support for high-degree nodes
*
* @param id Entity ID to get neighbors for
* @param optionsOrDirection Optional: direction string OR options object
@ -252,7 +252,7 @@ export class GraphAdjacencyIndex {
// Convert to array for pagination
let result = Array.from(neighbors)
// Apply pagination if requested (v5.8.0)
// Apply pagination if requested
if (options?.limit !== undefined || options?.offset !== undefined) {
const offset = options.offset || 0
const limit = options.limit !== undefined ? options.limit : result.length
@ -273,8 +273,8 @@ export class GraphAdjacencyIndex {
* Get verb IDs by source - Billion-scale optimization for getVerbsBySource
*
* O(log n) LSM-tree lookup with bloom filter optimization
* v5.7.1: Filters out deleted verb IDs (tombstone deletion workaround)
* v5.8.0: Added pagination support for entities with many relationships
* Filters out deleted verb IDs (tombstone deletion workaround)
* Added pagination support for entities with many relationships
*
* @param sourceId Source entity ID
* @param options Optional configuration
@ -318,7 +318,7 @@ export class GraphAdjacencyIndex {
const allIds = verbIds || []
let result = allIds.filter(id => this.verbIdSet.has(id))
// Apply pagination if requested (v5.8.0)
// Apply pagination if requested
if (options?.limit !== undefined || options?.offset !== undefined) {
const offset = options.offset || 0
const limit = options.limit !== undefined ? options.limit : result.length
@ -332,8 +332,8 @@ export class GraphAdjacencyIndex {
* Get verb IDs by target - Billion-scale optimization for getVerbsByTarget
*
* O(log n) LSM-tree lookup with bloom filter optimization
* v5.7.1: Filters out deleted verb IDs (tombstone deletion workaround)
* v5.8.0: Added pagination support for popular target entities
* Filters out deleted verb IDs (tombstone deletion workaround)
* Added pagination support for popular target entities
*
* @param targetId Target entity ID
* @param options Optional configuration
@ -377,7 +377,7 @@ export class GraphAdjacencyIndex {
const allIds = verbIds || []
let result = allIds.filter(id => this.verbIdSet.has(id))
// Apply pagination if requested (v5.8.0)
// Apply pagination if requested
if (options?.limit !== undefined || options?.offset !== undefined) {
const offset = options.offset || 0
const limit = options.limit !== undefined ? options.limit : result.length
@ -414,7 +414,7 @@ export class GraphAdjacencyIndex {
}
/**
* Batch get multiple verbs with caching (v6.2.0 - N+1 fix)
* Batch get multiple verbs with caching
*
* **Performance**: Eliminates N+1 pattern for verb loading
* - Current: N × getVerbCached() = N × 50ms on GCS = 250ms for 5 verbs
@ -433,7 +433,6 @@ export class GraphAdjacencyIndex {
* @param verbIds Array of verb IDs to fetch
* @returns Map of verbId → GraphVerb (only successful reads included)
*
* @since v6.2.0
*/
async getVerbsBatchCached(verbIds: string[]): Promise<Map<string, GraphVerb>> {
const results = new Map<string, GraphVerb>()
@ -514,7 +513,7 @@ export class GraphAdjacencyIndex {
const targetStats = this.lsmTreeTarget.getStats()
// Note: Exact unique node counts would require full LSM-tree scan
// v5.7.0: Using verbIdSet (ID-only tracking) for memory efficiency
// Using verbIdSet (ID-only tracking) for memory efficiency
const uniqueSourceNodes = this.verbIdSet.size
const uniqueTargetNodes = this.verbIdSet.size
const totalNodes = this.verbIdSet.size
@ -622,13 +621,13 @@ export class GraphAdjacencyIndex {
// Clear current index
this.verbIdSet.clear()
this.totalRelationshipsIndexed = 0
// v6.2.4: CRITICAL FIX - Clear relationship counts to prevent accumulation
// CRITICAL FIX - Clear relationship counts to prevent accumulation
this.relationshipCountsByType.clear()
// Note: LSM-trees will be recreated from storage via their own initialization
// Verb data will be loaded on-demand via UnifiedCache
// Adaptive loading strategy based on storage type (v4.2.4)
// Adaptive loading strategy based on storage type
const storageType = this.storage?.constructor.name || ''
const isLocalStorage =
storageType === 'FileSystemStorage' ||
@ -754,7 +753,7 @@ export class GraphAdjacencyIndex {
bytes += targetStats.memTableMemory
// Verb ID set (memory-efficient: IDs only, ~8 bytes per ID pointer)
// v5.7.0: Previous verbIndex Map stored full objects (128 bytes each = 128GB @ 1B verbs)
// Previous verbIndex Map stored full objects (128 bytes each = 128GB @ 1B verbs)
// Now: verbIdSet stores only IDs (~8 bytes each = ~100KB @ 1B verbs) = 1,280,000x reduction
bytes += this.verbIdSet.size * 8
@ -792,7 +791,7 @@ export class GraphAdjacencyIndex {
/**
* Flush LSM-tree MemTables to disk
* CRITICAL FIX (v3.43.2): Now public so it can be called from brain.flush()
* CRITICAL FIX: Now public so it can be called from brain.flush()
*/
async flush(): Promise<void> {
if (!this.initialized) {

61
src/hnsw/hnswIndex.ts
View file

@ -35,18 +35,18 @@ export class HNSWIndex {
private distanceFunction: DistanceFunction
private dimension: number | null = null
private useParallelization: boolean = true // Whether to use parallelization for performance-critical operations
private storage: BaseStorage | null = null // Storage adapter for HNSW persistence (v3.35.0+)
private storage: BaseStorage | null = null // Storage adapter for HNSW persistence
// Universal memory management (v3.36.0+)
// Universal memory management
private unifiedCache: UnifiedCache // Shared cache with Graph and Metadata indexes
// Always-adaptive caching (v3.36.0+) - no "mode" concept, system adapts automatically
// Always-adaptive caching - no "mode" concept, system adapts automatically
// COW (Copy-on-Write) support - v5.0.0
// COW (Copy-on-Write) support
private cowEnabled: boolean = false
private cowModifiedNodes: Set<string> = new Set()
private cowParent: HNSWIndex | null = null
// v6.2.8: Deferred HNSW persistence for cloud storage performance
// Deferred HNSW persistence for cloud storage performance
// In deferred mode, HNSW connections are only persisted on flush/close
// This reduces GCS operations from 70 to 2-3 per add() (30-50× faster)
private persistMode: 'immediate' | 'deferred' = 'immediate'
@ -86,7 +86,7 @@ export class HNSWIndex {
}
/**
* v6.2.8: Flush dirty HNSW data to storage
* Flush dirty HNSW data to storage
*
* In deferred persistence mode, HNSW connections are tracked as dirty but not
* immediately persisted. Call flush() to persist all pending changes.
@ -361,6 +361,19 @@ export class HNSWIndex {
this.entryPointId = id
this.maxLevel = nounLevel
this.nouns.set(id, noun)
// Persist system data for first noun (previously skipped)
if (this.storage && this.persistMode === 'immediate') {
await this.storage.saveHNSWSystem({
entryPointId: this.entryPointId,
maxLevel: this.maxLevel
}).catch(error => {
console.error('Failed to persist initial HNSW system data:', error)
})
} else if (this.persistMode === 'deferred') {
this.dirtySystem = true
}
return id
}
@ -438,7 +451,7 @@ export class HNSWIndex {
)
// Add bidirectional connections
// PERFORMANCE OPTIMIZATION (v4.10.0): Collect all neighbor updates for concurrent execution
// PERFORMANCE OPTIMIZATION: Collect all neighbor updates for concurrent execution
const neighborUpdates: Array<{
neighborId: string
promise: Promise<void>
@ -467,9 +480,9 @@ export class HNSWIndex {
await this.pruneConnections(neighbor, level)
}
// Persist updated neighbor HNSW data (v3.35.0+)
// Persist updated neighbor HNSW data
//
// v6.2.8: Deferred persistence mode for cloud storage performance
// Deferred persistence mode for cloud storage performance
// In deferred mode, we track dirty nodes instead of persisting immediately
// This reduces GCS operations from 70 to 2-3 per add() (30-50× faster)
if (this.storage && this.persistMode === 'immediate') {
@ -561,8 +574,8 @@ export class HNSWIndex {
this.highLevelNodes.get(nounLevel)!.add(id)
}
// Persist HNSW graph data to storage (v3.35.0+)
// v6.2.8: Respect persistMode setting
// Persist HNSW graph data to storage
// Respect persistMode setting
if (this.storage && this.persistMode === 'immediate') {
// IMMEDIATE MODE: Original behavior - persist new entity and system data
const connectionsObj: Record<string, string[]> = {}
@ -594,7 +607,7 @@ export class HNSWIndex {
}
/**
* O(1) entry point recovery using highLevelNodes index (v6.2.3).
* O(1) entry point recovery using highLevelNodes index.
* At any reasonable scale (1000+ nodes), level 2+ nodes are guaranteed to exist.
* For tiny indexes with only level 0-1 nodes, any node works as entry point.
*/
@ -640,7 +653,7 @@ export class HNSWIndex {
}
// Start from the entry point
// If entry point is null but nouns exist, attempt O(1) recovery (v6.2.3)
// If entry point is null but nouns exist, attempt O(1) recovery
if (!this.entryPointId && this.nouns.size > 0) {
const { id: recoveredId, level: recoveredLevel } = this.recoverEntryPointO1()
if (recoveredId) {
@ -656,7 +669,7 @@ export class HNSWIndex {
let entryPoint = this.nouns.get(this.entryPointId)
if (!entryPoint) {
// Entry point ID exists but noun was deleted - O(1) recovery (v6.2.3)
// Entry point ID exists but noun was deleted - O(1) recovery
if (this.nouns.size > 0) {
const { id: recoveredId, level: recoveredLevel } = this.recoverEntryPointO1()
if (recoveredId) {
@ -944,7 +957,7 @@ export class HNSWIndex {
/**
* Get vector safely (always uses adaptive caching via UnifiedCache)
*
* Production-grade adaptive caching (v3.36.0+):
* Production-grade adaptive caching:
* - Vector already loaded: Returns immediately (O(1))
* - Vector in cache: Loads from UnifiedCache (O(1) hash lookup)
* - Vector on disk: Loads from storage → UnifiedCache (O(disk))
@ -990,7 +1003,7 @@ export class HNSWIndex {
}
/**
* Get vector synchronously if available in memory (v3.36.0+)
* Get vector synchronously if available in memory
*
* Sync fast path optimization:
* - Vector in memory: Returns immediately (zero overhead)
@ -1048,7 +1061,7 @@ export class HNSWIndex {
}
/**
* Calculate distance with sync fast path (v3.36.0+)
* Calculate distance with sync fast path
*
* Eliminates async overhead when vectors are in memory:
* - Sync path: Vector in memory → returns number (zero overhead)
@ -1093,7 +1106,7 @@ export class HNSWIndex {
}
/**
* Rebuild HNSW index from persisted graph data (v3.35.0+)
* Rebuild HNSW index from persisted graph data
*
* This is a production-grade O(N) rebuild that restores the pre-computed graph structure
* from storage. Much faster than re-building which is O(N log N).
@ -1157,7 +1170,7 @@ export class HNSWIndex {
)
}
// Step 4: Adaptive loading strategy based on storage type (v4.2.4)
// Step 4: Adaptive loading strategy based on storage type
// FileSystem/Memory/OPFS: Load all at once (avoids repeated getAllShardedFiles() calls)
// Cloud (GCS/S3/R2): Use pagination (efficient native cloud APIs)
const storageType = this.storage?.constructor.name || ''
@ -1238,7 +1251,7 @@ export class HNSWIndex {
prodLog.info(`HNSW: Using cloud pagination strategy (${storageType})`)
let hasMore = true
let offset = 0 // v5.7.11: Use offset-based pagination instead of cursor (bug fix for infinite loop)
let offset = 0 // Use offset-based pagination instead of cursor (bug fix for infinite loop)
while (hasMore) {
// Fetch batch of nouns from storage (cast needed as method is not in base interface)
@ -1249,7 +1262,7 @@ export class HNSWIndex {
nextCursor?: string
} = await (this.storage as any).getNounsWithPagination({
limit: batchSize,
offset // v5.7.11: Pass offset for proper pagination (previously passed cursor which was ignored)
offset // Pass offset for proper pagination (previously passed cursor which was ignored)
})
// Set total count on first batch
@ -1307,11 +1320,11 @@ export class HNSWIndex {
// Check for more data
hasMore = result.hasMore
offset += batchSize // v5.7.11: Increment offset for next page
offset += batchSize // Increment offset for next page
}
}
// Step 5: CRITICAL - Recover entry point if missing (v6.2.3 - O(1))
// Step 5: CRITICAL - Recover entry point if missing)
// This ensures consistency even if getHNSWSystem() returned null
if (this.nouns.size > 0 && this.entryPointId === null) {
prodLog.warn('HNSW rebuild: Entry point was null after loading nouns - recovering with O(1) lookup')
@ -1426,7 +1439,7 @@ export class HNSWIndex {
}
/**
* Get cache performance statistics for monitoring and diagnostics (v3.36.0+)
* Get cache performance statistics for monitoring and diagnostics
*
* Production-grade monitoring:
* - Adaptive caching strategy (preloading vs on-demand)

26
src/hnsw/typeAwareHNSWIndex.ts
View file

@ -118,7 +118,7 @@ export class TypeAwareHNSWIndex {
}
/**
* v6.2.8: Flush dirty HNSW data to storage for all type-specific indexes
* Flush dirty HNSW data to storage for all type-specific indexes
*
* In deferred persistence mode, HNSW connections are tracked as dirty but not
* immediately persisted. Call flush() to persist all pending changes across
@ -502,7 +502,7 @@ export class TypeAwareHNSWIndex {
// Load ALL nouns ONCE and route to correct type indexes
// This is O(N) instead of O(42*N) from the previous parallel approach
let offset = 0 // v5.7.11: Use offset-based pagination instead of cursor (bug fix for infinite loop)
let offset = 0 // Use offset-based pagination instead of cursor (bug fix for infinite loop)
let hasMore = true
let totalLoaded = 0
const loadedByType = new Map<NounType, number>()
@ -515,13 +515,13 @@ export class TypeAwareHNSWIndex {
totalCount?: number
} = await (this.storage as any).getNounsWithPagination({
limit: batchSize,
offset // v5.7.11: Pass offset for proper pagination (previously passed cursor which was ignored)
offset // Pass offset for proper pagination (previously passed cursor which was ignored)
})
// Route each noun to its type index
for (const nounData of result.items) {
try {
// v7.3.0: Use 'type' property from HNSWNounWithMetadata (not 'nounType')
// Use 'type' property from HNSWNounWithMetadata (not 'nounType')
// Previously accessed wrong property, causing N+1 metadata fetches
// getNounsWithPagination already includes type in response
const nounType = nounData.type
@ -578,7 +578,7 @@ export class TypeAwareHNSWIndex {
}
hasMore = result.hasMore
offset += batchSize // v5.7.11: Increment offset for next page
offset += batchSize // Increment offset for next page
// Progress logging
if (totalLoaded % 1000 === 0) {
@ -593,12 +593,26 @@ export class TypeAwareHNSWIndex {
let entryPointId: string | null = null
for (const [id, noun] of (index as any).nouns.entries()) {
if (noun.level > maxLevel) {
if (entryPointId === null || noun.level > maxLevel) {
maxLevel = noun.level
entryPointId = id
}
}
// Recovery: if still null after loop but nouns exist
if ((index as any).nouns.size > 0 && !entryPointId) {
const { id: recoveredId, level: recoveredLevel } = (index as any).recoverEntryPointO1()
entryPointId = recoveredId
maxLevel = recoveredLevel
}
// Validation: if entry point doesn't exist in loaded nouns
if (entryPointId && !(index as any).nouns.has(entryPointId)) {
const { id: recoveredId, level: recoveredLevel } = (index as any).recoverEntryPointO1()
entryPointId = recoveredId
maxLevel = recoveredLevel
}
;(index as any).entryPointId = entryPointId
;(index as any).maxLevel = maxLevel

8
src/import/FormatDetector.ts
View file

@ -111,7 +111,7 @@ export class FormatDetector {
return 'docx'
}
// Images (v5.2.0: ImageHandler support)
// Images (ImageHandler support)
if (mimeType.startsWith('image/')) {
return 'image'
}
@ -134,7 +134,7 @@ export class FormatDetector {
}
}
// YAML detection (v4.2.0)
// YAML detection
if (this.looksLikeYAML(trimmed)) {
return {
format: 'yaml',
@ -206,7 +206,7 @@ export class FormatDetector {
}
}
// Image formats (v5.2.0)
// Image formats
// JPEG: FF D8 FF
if (buffer[0] === 0xFF && buffer[1] === 0xD8 && buffer[2] === 0xFF) {
return {
@ -384,7 +384,7 @@ export class FormatDetector {
/**
* Check if content looks like YAML
* v4.2.0: Added YAML detection
* Added YAML detection
*/
private looksLikeYAML(content: string): boolean {
const lines = content.split('\n').filter(l => l.trim()).slice(0, 20)

110
src/import/ImportCoordinator.ts
View file

@ -37,10 +37,10 @@ export interface ImportSource {
/** Optional filename hint */
filename?: string
/** HTTP headers for URL imports (v4.2.0) */
/** HTTP headers for URL imports */
headers?: Record<string, string>
/** Basic authentication for URL imports (v4.2.0) */
/** Basic authentication for URL imports */
auth?: {
username: string
password: string
@ -93,7 +93,7 @@ export interface ValidImportOptions {
/** Create relationships in knowledge graph */
createRelationships?: boolean
/** Create provenance relationships (document → entity) [v4.9.0] */
/** Create provenance relationships (document → entity) */
createProvenanceLinks?: boolean
/** Preserve source file in VFS */
@ -144,7 +144,7 @@ export interface ValidImportOptions {
customMetadata?: Record<string, any>
/**
* Progress callback for tracking import progress (v4.2.0+)
* Progress callback for tracking import progress
*
* **Streaming Architecture** (always enabled):
* - Indexes are flushed periodically during import (adaptive intervals)
@ -227,21 +227,21 @@ export type ImportOptions = ValidImportOptions & DeprecatedImportOptions
export interface ImportProgress {
stage: 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'relationships' | 'complete'
/** Phase of import - extraction or relationship building (v3.49.0) */
/** Phase of import - extraction or relationship building */
phase?: 'extraction' | 'relationships'
message: string
processed?: number
/** Alias for processed, used in relationship phase (v3.49.0) */
/** Alias for processed, used in relationship phase */
current?: number
total?: number
entities?: number
relationships?: number
/** Rows per second (v3.38.0) */
/** Rows per second */
throughput?: number
/** Estimated time remaining in ms (v3.38.0) */
/** Estimated time remaining in ms */
eta?: number
/**
* Whether data is queryable at this point (v4.2.0+)
* Whether data is queryable at this point
*
* When true, indexes have been flushed and queries will return up-to-date results.
* When false, data exists in storage but indexes may not be current (queries may be slower/incomplete).
@ -357,7 +357,7 @@ export class ImportCoordinator {
/**
* Import from any source with auto-detection
* v4.2.0: Now supports URL imports with authentication
* Now supports URL imports with authentication
*/
async import(
source: Buffer | string | object | ImportSource,
@ -365,10 +365,10 @@ export class ImportCoordinator {
): Promise<ImportResult> {
const startTime = Date.now()
// Validate options (v4.0.0+: Reject deprecated v3.x options)
// Validate options (Reject deprecated options)
this.validateOptions(options)
// Normalize source (v4.2.0: handles URL fetching)
// Normalize source (handles URL fetching)
const normalizedSource = await this.normalizeSource(source, options.format)
// Report detection stage
@ -387,10 +387,10 @@ export class ImportCoordinator {
}
// Set defaults early (needed for tracking context)
// CRITICAL FIX (v4.3.2): Spread options FIRST, then apply defaults
// CRITICAL FIX: Spread options FIRST, then apply defaults
// Previously: ...options at the end overwrote normalized defaults with undefined
// Now: Defaults properly override undefined values
// v4.4.0: Enable AI features by default for smarter imports
// Enable AI features by default for smarter imports
const opts = {
...options, // Spread first to get all options
vfsPath: options.vfsPath || `/imports/${Date.now()}`,
@ -399,13 +399,13 @@ export class ImportCoordinator {
createRelationships: options.createRelationships !== false,
preserveSource: options.preserveSource !== false,
enableDeduplication: options.enableDeduplication !== false,
enableNeuralExtraction: options.enableNeuralExtraction !== false, // v4.4.0: Default true
enableRelationshipInference: options.enableRelationshipInference !== false, // v4.4.0: Default true
enableNeuralExtraction: options.enableNeuralExtraction !== false, // Default true
enableRelationshipInference: options.enableRelationshipInference !== false, // Default true
enableConceptExtraction: options.enableConceptExtraction !== false, // Already defaults to true
deduplicationThreshold: options.deduplicationThreshold || 0.85
}
// Generate tracking context (v4.10.0+: Unified import/project tracking)
// Generate tracking context (Unified import/project tracking)
const importId = options.importId || uuidv4()
const projectId = options.projectId || this.deriveProjectId(opts.vfsPath)
const trackingContext: TrackingContext = {
@ -441,13 +441,13 @@ export class ImportCoordinator {
groupBy: opts.groupBy,
customGrouping: opts.customGrouping,
preserveSource: opts.preserveSource,
// v5.1.2: Fix sourceBuffer for file paths - type is 'path' not 'buffer' from normalizeSource()
// Fix sourceBuffer for file paths - type is 'path' not 'buffer' from normalizeSource()
sourceBuffer: Buffer.isBuffer(normalizedSource.data) ? normalizedSource.data as Buffer : undefined,
sourceFilename: normalizedSource.filename || `import.${detection.format}`,
createRelationshipFile: true,
createMetadataFile: true,
trackingContext, // v4.10.0: Pass tracking metadata to VFS
// v4.11.1: Pass progress callback for VFS creation updates
trackingContext, // Pass tracking metadata to VFS
// Pass progress callback for VFS creation updates
onProgress: (vfsProgress) => {
options.onProgress?.({
stage: 'storing-vfs',
@ -473,7 +473,7 @@ export class ImportCoordinator {
sourceFilename: normalizedSource.filename || `import.${detection.format}`,
format: detection.format
},
trackingContext // v4.10.0: Pass tracking metadata to graph creation
trackingContext // Pass tracking metadata to graph creation
)
// Report complete
@ -520,7 +520,7 @@ export class ImportCoordinator {
)
}
// CRITICAL FIX (v3.43.2): Auto-flush all indexes before returning
// CRITICAL FIX: Auto-flush all indexes before returning
// Ensures imported data survives server restarts
// Bug #5: Import data was only in memory, lost on restart
options.onProgress?.({
@ -535,7 +535,7 @@ export class ImportCoordinator {
/**
* Normalize source to ImportSource
* v4.2.0: Now async to support URL fetching
* Now async to support URL fetching
*/
private async normalizeSource(
source: Buffer | string | object | ImportSource,
@ -616,7 +616,7 @@ export class ImportCoordinator {
/**
* Fetch content from URL
* v4.2.0: Supports authentication and custom headers
* Supports authentication and custom headers
*/
private async fetchUrl(source: ImportSource): Promise<ImportSource> {
const url = typeof source.data === 'string' ? source.data : String(source.data)
@ -722,7 +722,7 @@ export class ImportCoordinator {
format: SupportedFormat,
options: ImportOptions
): Promise<any> {
// v5.2.0: Check if IntelligentImportAugmentation already extracted data
// Check if IntelligentImportAugmentation already extracted data
if ((options as any)._intelligentImport && (options as any)._extractedData) {
const extractedData = (options as any)._extractedData
// Convert extracted data to ExtractedRow format
@ -760,7 +760,7 @@ export class ImportCoordinator {
enableConceptExtraction: options.enableConceptExtraction !== false,
confidenceThreshold: options.confidenceThreshold || 0.6,
onProgress: (stats: any) => {
// Enhanced progress reporting (v3.38.0) with throughput and ETA
// Enhanced progress reporting with throughput and ETA
const message = stats.throughput
? `Extracting entities from ${format} (${stats.throughput} rows/sec, ETA: ${Math.round(stats.eta / 1000)}s)...`
: `Extracting entities from ${format}...`
@ -825,7 +825,7 @@ export class ImportCoordinator {
return await this.docxImporter.extract(docxBuffer, extractOptions)
case 'image':
// v5.2.0: Images are handled by IntelligentImportAugmentation
// Images are handled by IntelligentImportAugmentation
// If we reach here, augmentation didn't process it - return minimal result
const imageName = source.filename || 'image'
const imageId = `image-${Date.now()}`
@ -867,7 +867,7 @@ export class ImportCoordinator {
/**
* Create entities and relationships in knowledge graph
* v4.9.0: Added sourceInfo parameter for document entity creation
* Added sourceInfo parameter for document entity creation
*/
private async createGraphEntities(
extractionResult: any,
@ -877,7 +877,7 @@ export class ImportCoordinator {
sourceFilename: string
format: string
},
trackingContext?: TrackingContext // v4.10.0: Import/project tracking
trackingContext?: TrackingContext // Import/project tracking
): Promise<{
entities: Array<{ id: string; name: string; type: NounType; vfsPath?: string; metadata?: Record<string, any> }>
relationships: Array<{ id: string; from: string; to: string; type: VerbType }>
@ -891,7 +891,7 @@ export class ImportCoordinator {
let mergedCount = 0
let newCount = 0
// CRITICAL FIX (v4.3.2): Default to true when undefined
// CRITICAL FIX: Default to true when undefined
// Previously: if (!options.createEntities) treated undefined as false
// Now: Only skip when explicitly set to false
if (options.createEntities === false) {
@ -908,7 +908,7 @@ export class ImportCoordinator {
// Extract rows/sections/entities from result (unified across formats)
const rows = extractionResult.rows || extractionResult.sections || extractionResult.entities || []
// Progressive flush interval - adjusts based on current count (v4.2.0+)
// Progressive flush interval - adjusts based on current count
// Starts at 100, increases to 1000 at 1K entities, then 5000 at 10K
// This works for both known totals (files) and unknown totals (streaming APIs)
let currentFlushInterval = 100 // Start with frequent updates for better UX
@ -938,7 +938,7 @@ export class ImportCoordinator {
}
// ============================================
// v4.9.0: Create document entity for import source
// Create document entity for import source
// ============================================
let documentEntityId: string | null = null
let provenanceCount = 0
@ -957,7 +957,7 @@ export class ImportCoordinator {
vfsPath: vfsResult.rootPath,
totalRows: rows.length,
byType: this.countByType(rows),
// v4.10.0: Import tracking metadata
// Import tracking metadata
...(trackingContext && {
importIds: [trackingContext.importId],
projectId: trackingContext.projectId,
@ -973,7 +973,7 @@ export class ImportCoordinator {
}
// ============================================
// v4.11.0: Batch entity creation using addMany()
// Batch entity creation using addMany()
// Replaces entity-by-entity loop for 10-100x performance improvement on cloud storage
// ============================================
@ -1038,7 +1038,7 @@ export class ImportCoordinator {
name: entity.name,
type: entity.type,
vfsPath: vfsFile?.path,
metadata: entity.metadata // v5.2.0: Include metadata in return (for ImageHandler, etc)
metadata: entity.metadata // Include metadata in return (for ImageHandler, etc)
})
newCount++
}
@ -1090,7 +1090,7 @@ export class ImportCoordinator {
const importSource = vfsResult.rootPath
let entityId: string
// v5.7.0: No deduplication during import (12-24x speedup)
// No deduplication during import (12-24x speedup)
// Background deduplication runs 5 minutes after import completes
entityId = await this.brain.add({
data: entity.description || entity.name,
@ -1101,7 +1101,7 @@ export class ImportCoordinator {
confidence: entity.confidence,
vfsPath: vfsFile?.path,
importedFrom: 'import-coordinator',
// v4.10.0: Import tracking metadata
// Import tracking metadata
...(trackingContext && {
importId: trackingContext.importId, // Used for background dedup
importIds: [trackingContext.importId],
@ -1126,11 +1126,11 @@ export class ImportCoordinator {
name: entity.name,
type: entity.type,
vfsPath: vfsFile?.path,
metadata: entity.metadata // v5.2.0: Include metadata in return (for ImageHandler, etc)
metadata: entity.metadata // Include metadata in return (for ImageHandler, etc)
})
// ============================================
// v4.9.0: Create provenance relationship (document → entity)
// Create provenance relationship (document → entity)
// ============================================
if (documentEntityId && options.createProvenanceLinks !== false) {
await this.brain.relate({
@ -1144,7 +1144,7 @@ export class ImportCoordinator {
rowNumber: row.rowNumber,
extractedAt: Date.now(),
format: sourceInfo?.format,
// v4.10.0: Import tracking metadata
// Import tracking metadata
...(trackingContext && {
importIds: [trackingContext.importId],
projectId: trackingContext.projectId,
@ -1161,7 +1161,7 @@ export class ImportCoordinator {
if (options.createRelationships && row.relationships) {
for (const rel of row.relationships) {
try {
// CRITICAL FIX (v3.43.2): Prevent infinite placeholder creation loop
// CRITICAL FIX: Prevent infinite placeholder creation loop
// Find or create target entity using EXACT matching only
let targetEntityId: string | undefined
@ -1195,7 +1195,7 @@ export class ImportCoordinator {
name: rel.to,
placeholder: true,
inferredFrom: entity.name,
// v4.10.0: Import tracking metadata
// Import tracking metadata
...(trackingContext && {
importIds: [trackingContext.importId],
projectId: trackingContext.projectId,
@ -1221,11 +1221,11 @@ export class ImportCoordinator {
from: entityId,
to: targetEntityId,
type: rel.type,
confidence: rel.confidence, // v4.2.0: Top-level field
weight: rel.weight || 1.0, // v4.2.0: Top-level field
confidence: rel.confidence, // Top-level field
weight: rel.weight || 1.0, // Top-level field
metadata: {
evidence: rel.evidence,
// v4.10.0: Import tracking metadata (will be merged in batch creation)
// Import tracking metadata (will be merged in batch creation)
...(trackingContext && {
importIds: [trackingContext.importId],
projectId: trackingContext.projectId,
@ -1242,7 +1242,7 @@ export class ImportCoordinator {
}
}
// Streaming import: Progressive flush with dynamic interval adjustment (v4.2.0+)
// Streaming import: Progressive flush with dynamic interval adjustment
entitiesSinceFlush++
if (entitiesSinceFlush >= currentFlushInterval) {
@ -1307,7 +1307,7 @@ export class ImportCoordinator {
}
// Batch create all relationships using brain.relateMany() for performance
// v4.9.0: Enhanced with type-based inference and semantic metadata
// Enhanced with type-based inference and semantic metadata
if (options.createRelationships && relationships.length > 0) {
try {
const relationshipParams = relationships.map(rel => {
@ -1331,7 +1331,7 @@ export class ImportCoordinator {
type: verbType, // Enhanced type
metadata: {
...((rel as any).metadata || {}),
relationshipType: 'semantic', // v4.9.0: Distinguish from VFS/provenance
relationshipType: 'semantic', // Distinguish from VFS/provenance
inferredType: verbType !== rel.type, // Track if type was enhanced
originalType: rel.type
}
@ -1369,7 +1369,7 @@ export class ImportCoordinator {
}
}
// v5.7.0: Schedule background deduplication (debounced 5 minutes)
// Schedule background deduplication (debounced 5 minutes)
if (trackingContext && trackingContext.importId) {
this.backgroundDedup.scheduleDedup(trackingContext.importId)
}
@ -1457,7 +1457,7 @@ export class ImportCoordinator {
}
}
// YAML: entities -> rows (v4.2.0)
// YAML: entities -> rows
if (format === 'yaml') {
const rows = result.entities.map((entity: any) => ({
entity,
@ -1477,7 +1477,7 @@ export class ImportCoordinator {
}
}
// DOCX: entities -> rows (v4.2.0)
// DOCX: entities -> rows
if (format === 'docx') {
const rows = result.entities.map((entity: any) => ({
entity,
@ -1502,7 +1502,7 @@ export class ImportCoordinator {
}
/**
* Validate options and reject deprecated v3.x options (v4.0.0+)
* Validate options and reject deprecated v3.x options
* Throws clear errors with migration guidance
*/
private validateOptions(options: any): void {
@ -1657,7 +1657,7 @@ ${optionDetails}
}
/**
* Get progressive flush interval based on CURRENT entity count (v4.2.0+)
* Get progressive flush interval based on CURRENT entity count
*
* Unlike adaptive intervals (which require knowing total count upfront),
* progressive intervals adjust dynamically as import proceeds.
@ -1699,7 +1699,7 @@ ${optionDetails}
/**
* Infer relationship type based on entity types and context
* v4.9.0: Semantic relationship enhancement
* Semantic relationship enhancement
*
* @param sourceType - Type of source entity
* @param targetType - Type of target entity
@ -1769,7 +1769,7 @@ ${optionDetails}
/**
* Count entities by type for document metadata
* v4.9.0: Used for document entity statistics
* Used for document entity statistics
*
* @param rows - Extracted rows from import
* @returns Record of entity type counts

14
src/importers/SmartCSVImporter.ts
View file

@ -42,17 +42,17 @@ export interface SmartCSVOptions extends FormatHandlerOptions {
csvDelimiter?: string
csvHeaders?: boolean
/** Progress callback (v3.39.0: Enhanced with performance metrics) */
/** Progress callback (Enhanced with performance metrics) */
onProgress?: (stats: {
processed: number
total: number
entities: number
relationships: number
/** Rows per second (v3.39.0) */
/** Rows per second */
throughput?: number
/** Estimated time remaining in ms (v3.39.0) */
/** Estimated time remaining in ms */
eta?: number
/** Current phase (v3.39.0) */
/** Current phase */
phase?: string
}) => void
}
@ -169,7 +169,7 @@ export class SmartCSVImporter {
}
// Parse CSV using existing handler
// v4.5.0: Pass progress hooks to handler for file parsing progress
// Pass progress hooks to handler for file parsing progress
const processedData = await this.csvHandler.process(buffer, {
...options,
csvDelimiter: opts.csvDelimiter,
@ -217,7 +217,7 @@ export class SmartCSVImporter {
// Detect column names
const columns = this.detectColumns(rows[0], opts)
// Process each row with BATCHED PARALLEL PROCESSING (v3.39.0)
// Process each row with BATCHED PARALLEL PROCESSING
const extractedRows: ExtractedRow[] = []
const entityMap = new Map<string, string>()
const stats = {
@ -375,7 +375,7 @@ export class SmartCSVImporter {
total: rows.length,
entities: extractedRows.reduce((sum, row) => sum + 1 + row.relatedEntities.length, 0),
relationships: extractedRows.reduce((sum, row) => sum + row.relationships.length, 0),
// Additional performance metrics (v3.39.0)
// Additional performance metrics
throughput: Math.round(rowsPerSecond * 10) / 10,
eta: Math.round(estimatedTimeRemaining),
phase: 'extracting'

6
src/importers/SmartDOCXImporter.ts
View file

@ -9,7 +9,7 @@
* - NaturalLanguageProcessor for relationship inference
* - Hierarchical relationship creation based on heading hierarchy
*
* v4.2.0: New format handler
* New format handler
* NO MOCKS - Production-ready implementation
*/
@ -187,7 +187,7 @@ export class SmartDOCXImporter {
await this.init()
}
// v4.5.0: Report parsing start
// Report parsing start
options.onProgress?.({
processed: 0,
entities: 0,
@ -200,7 +200,7 @@ export class SmartDOCXImporter {
// Extract HTML for structure analysis (headings, tables)
const htmlResult = await mammoth.convertToHtml({ buffer })
// v4.5.0: Report parsing complete
// Report parsing complete
options.onProgress?.({
processed: 0,
entities: 0,

28
src/importers/SmartExcelImporter.ts
View file

@ -36,17 +36,17 @@ export interface SmartExcelOptions extends FormatHandlerOptions {
typeColumn?: string // e.g., "Type", "Category"
relatedColumn?: string // e.g., "Related Terms", "See Also"
/** Progress callback (v3.38.0: Enhanced with performance metrics) */
/** Progress callback (Enhanced with performance metrics) */
onProgress?: (stats: {
processed: number
total: number
entities: number
relationships: number
/** Rows per second (v3.38.0) */
/** Rows per second */
throughput?: number
/** Estimated time remaining in ms (v3.38.0) */
/** Estimated time remaining in ms */
eta?: number
/** Current phase (v3.38.0) */
/** Current phase */
phase?: string
}) => void
}
@ -111,7 +111,7 @@ export interface SmartExcelResult {
}
}
/** Sheet-specific data for VFS extraction (v4.2.0) */
/** Sheet-specific data for VFS extraction */
sheets?: Array<{
name: string
rows: ExtractedRow[]
@ -161,7 +161,7 @@ export class SmartExcelImporter {
const opts = {
enableNeuralExtraction: true,
enableRelationshipInference: true,
// CONCEPT EXTRACTION PRODUCTION-READY (v3.33.0+):
// CONCEPT EXTRACTION PRODUCTION-READY:
// Type embeddings are now pre-computed at build time - zero runtime cost!
// All 42 noun types + 127 verb types instantly available
//
@ -184,7 +184,7 @@ export class SmartExcelImporter {
}
// Parse Excel using existing handler
// v4.5.0: Pass progress hooks to handler for file parsing progress
// Pass progress hooks to handler for file parsing progress
const processedData = await this.excelHandler.process(buffer, {
...options,
totalBytes: buffer.length,
@ -227,7 +227,7 @@ export class SmartExcelImporter {
return this.emptyResult(startTime)
}
// CRITICAL FIX (v4.8.6): Detect columns per-sheet, not globally
// CRITICAL FIX: Detect columns per-sheet, not globally
// Different sheets may have different column structures (Term vs Name, etc.)
// Group rows by sheet and detect columns for each sheet separately
const rowsBySheet = new Map<string, typeof rows>()
@ -247,7 +247,7 @@ export class SmartExcelImporter {
}
}
// Process each row with BATCHED PARALLEL PROCESSING (v3.38.0)
// Process each row with BATCHED PARALLEL PROCESSING
const extractedRows: ExtractedRow[] = []
const entityMap = new Map<string, string>()
const stats = {
@ -269,7 +269,7 @@ export class SmartExcelImporter {
chunk.map(async (row, chunkIndex) => {
const i = chunkStart + chunkIndex
// CRITICAL FIX (v4.8.6): Use sheet-specific column mapping
// CRITICAL FIX: Use sheet-specific column mapping
const sheet = row._sheet || 'default'
const columns = columnsBySheet.get(sheet) || this.detectColumns(row, opts)
@ -353,7 +353,7 @@ export class SmartExcelImporter {
}
// ============================================
// v4.9.0: Enhanced column-based relationship detection
// Enhanced column-based relationship detection
// ============================================
// Parse explicit "Related Terms" column
if (relatedTerms) {
@ -379,7 +379,7 @@ export class SmartExcelImporter {
}
}
// v4.9.0: Check for additional relationship-indicating columns
// Check for additional relationship-indicating columns
// Expanded patterns for various relationship types
const relationshipColumnPatterns = [
{ pattern: /^(location|home|lives in|resides|dwelling|place)$/i, defaultType: VerbType.LocatedAt },
@ -485,14 +485,14 @@ export class SmartExcelImporter {
total: rows.length,
entities: extractedRows.reduce((sum, row) => sum + 1 + row.relatedEntities.length, 0),
relationships: extractedRows.reduce((sum, row) => sum + row.relationships.length, 0),
// Additional performance metrics (v3.38.0)
// Additional performance metrics
throughput: Math.round(rowsPerSecond * 10) / 10,
eta: Math.round(estimatedTimeRemaining),
phase: 'extracting'
} as any)
}
// Group rows by sheet for VFS extraction (v4.2.0)
// Group rows by sheet for VFS extraction
const sheetGroups = new Map<string, ExtractedRow[]>()
extractedRows.forEach((extractedRow, index) => {
const originalRow = rows[index]

6
src/importers/SmartJSONImporter.ts
View file

@ -167,7 +167,7 @@ export class SmartJSONImporter {
...options
}
// v4.5.0: Report parsing start
// Report parsing start
opts.onProgress({
processed: 0,
entities: 0,
@ -186,7 +186,7 @@ export class SmartJSONImporter {
jsonData = data
}
// v4.5.0: Report parsing complete, starting traversal
// Report parsing complete, starting traversal
opts.onProgress({
processed: 0,
entities: 0,
@ -228,7 +228,7 @@ export class SmartJSONImporter {
}
)
// v4.5.0: Report completion
// Report completion
opts.onProgress({
processed: nodesProcessed,
entities: entities.length,

6
src/importers/SmartMarkdownImporter.ts
View file

@ -174,7 +174,7 @@ export class SmartMarkdownImporter {
...options
}
// v4.5.0: Report parsing start
// Report parsing start
opts.onProgress({
processed: 0,
total: 0,
@ -185,7 +185,7 @@ export class SmartMarkdownImporter {
// Parse markdown into sections
const parsedSections = this.parseMarkdown(markdown, opts)
// v4.5.0: Report parsing complete
// Report parsing complete
opts.onProgress({
processed: 0,
total: parsedSections.length,
@ -218,7 +218,7 @@ export class SmartMarkdownImporter {
})
}
// v4.5.0: Report completion
// Report completion
const totalEntities = sections.reduce((sum, s) => sum + s.entities.length, 0)
const totalRelationships = sections.reduce((sum, s) => sum + s.relationships.length, 0)
opts.onProgress({

16
src/importers/SmartPDFImporter.ts
View file

@ -40,17 +40,17 @@ export interface SmartPDFOptions extends FormatHandlerOptions {
/** Group by page or full document */
groupBy?: 'page' | 'document'
/** Progress callback (v3.39.0: Enhanced with performance metrics) */
/** Progress callback (Enhanced with performance metrics) */
onProgress?: (stats: {
processed: number
total: number
entities: number
relationships: number
/** Sections per second (v3.39.0) */
/** Sections per second */
throughput?: number
/** Estimated time remaining in ms (v3.39.0) */
/** Estimated time remaining in ms */
eta?: number
/** Current phase (v3.39.0) */
/** Current phase */
phase?: string
}) => void
}
@ -181,7 +181,7 @@ export class SmartPDFImporter {
}
// Parse PDF using existing handler
// v4.5.0: Pass progress hooks to handler for file parsing progress
// Pass progress hooks to handler for file parsing progress
const processedData = await this.pdfHandler.process(buffer, {
...options,
totalBytes: buffer.length,
@ -228,7 +228,7 @@ export class SmartPDFImporter {
// Group data by page or combine into single document
const grouped = this.groupData(data, opts)
// Process each group with BATCHED PARALLEL PROCESSING (v3.39.0)
// Process each group with BATCHED PARALLEL PROCESSING
const sections: ExtractedSection[] = []
const entityMap = new Map<string, string>()
const stats = {
@ -270,7 +270,7 @@ export class SmartPDFImporter {
total: totalGroups,
entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0),
// Additional performance metrics (v3.39.0)
// Additional performance metrics
throughput: Math.round(sectionsPerSecond * 10) / 10,
eta: Math.round(estimatedTimeRemaining),
phase: 'extracting'
@ -367,7 +367,7 @@ export class SmartPDFImporter {
const combinedText = texts.join('\n\n')
// Parallel extraction: entities AND concepts at the same time (v3.39.0)
// Parallel extraction: entities AND concepts at the same time
const [extractedEntities, concepts] = await Promise.all([
// Extract entities if enabled
options.enableNeuralExtraction && combinedText.length > 0

6
src/importers/SmartYAMLImporter.ts
View file

@ -8,7 +8,7 @@
* - NaturalLanguageProcessor for relationship inference
* - Hierarchical relationship creation (parent-child, contains, etc.)
*
* v4.2.0: New format handler
* New format handler
* NO MOCKS - Production-ready implementation
*/
@ -159,7 +159,7 @@ export class SmartYAMLImporter {
): Promise<SmartYAMLResult> {
const startTime = Date.now()
// v4.5.0: Report parsing start
// Report parsing start
options.onProgress?.({
processed: 0,
entities: 0,
@ -178,7 +178,7 @@ export class SmartYAMLImporter {
throw new Error(`Failed to parse YAML: ${error.message}`)
}
// v4.5.0: Report parsing complete
// Report parsing complete
options.onProgress?.({
processed: 0,
entities: 0,

26
src/importers/VFSStructureGenerator.ts
View file

@ -40,10 +40,10 @@ export interface VFSStructureOptions {
/** Create metadata file */
createMetadataFile?: boolean
/** Import tracking context (v4.10.0) */
/** Import tracking context */
trackingContext?: TrackingContext
/** Progress callback (v4.11.1) - Reports VFS creation progress */
/** Progress callback - Reports VFS creation progress */
onProgress?: (progress: {
stage: 'directories' | 'entities' | 'metadata'
message: string
@ -98,7 +98,7 @@ export class VFSStructureGenerator {
// Get brain's cached VFS instance (creates if doesn't exist)
this.vfs = this.brain.vfs
// CRITICAL FIX (v4.10.2): Always call vfs.init() explicitly
// CRITICAL FIX: Always call vfs.init() explicitly
// The previous code tried to check if initialized via stat('/') but this was unreliable
// vfs.init() is idempotent, so calling it multiple times is safe
await this.vfs.init()
@ -123,7 +123,7 @@ export class VFSStructureGenerator {
// Ensure VFS is initialized
await this.init()
// v4.11.1: Calculate total operations for progress tracking
// Calculate total operations for progress tracking
const groups = this.groupEntities(importResult, options)
const totalEntities = Array.from(groups.values()).reduce((sum, entities) => sum + entities.length, 0)
const totalOperations =
@ -161,7 +161,7 @@ export class VFSStructureGenerator {
try {
await this.vfs.mkdir(options.rootPath, {
recursive: true,
metadata: trackingMetadata // v4.10.0: Add tracking metadata
metadata: trackingMetadata // Add tracking metadata
})
result.directories.push(options.rootPath)
result.operations++
@ -179,7 +179,7 @@ export class VFSStructureGenerator {
if (options.preserveSource && options.sourceBuffer && options.sourceFilename) {
const sourcePath = `${options.rootPath}/_source${this.getExtension(options.sourceFilename)}`
await this.vfs.writeFile(sourcePath, options.sourceBuffer, {
metadata: trackingMetadata // v4.10.0: Add tracking metadata
metadata: trackingMetadata // Add tracking metadata
})
result.files.push({
path: sourcePath,
@ -200,7 +200,7 @@ export class VFSStructureGenerator {
try {
await this.vfs.mkdir(groupPath, {
recursive: true,
metadata: trackingMetadata // v4.10.0: Add tracking metadata
metadata: trackingMetadata // Add tracking metadata
})
result.directories.push(groupPath)
result.operations++
@ -242,7 +242,7 @@ export class VFSStructureGenerator {
await this.vfs.writeFile(entityPath, JSON.stringify(entityJson, null, 2), {
metadata: {
...trackingMetadata, // v4.10.0: Add tracking metadata
...trackingMetadata, // Add tracking metadata
entityId: extracted.entity.id
}
})
@ -253,7 +253,7 @@ export class VFSStructureGenerator {
})
result.operations++
// v4.11.1: Report progress every 10 entities (or on last entity)
// Report progress every 10 entities (or on last entity)
if (entityCount % 10 === 0 || entityCount === entities.length) {
reportProgress('entities', `Created ${entityCount}/${entities.length} ${groupName} files`)
}
@ -280,7 +280,7 @@ export class VFSStructureGenerator {
}
await this.vfs.writeFile(relationshipsPath, JSON.stringify(relationshipsJson, null, 2), {
metadata: trackingMetadata // v4.10.0: Add tracking metadata
metadata: trackingMetadata // Add tracking metadata
})
result.files.push({
path: relationshipsPath,
@ -322,7 +322,7 @@ export class VFSStructureGenerator {
}
await this.vfs.writeFile(metadataPath, JSON.stringify(metadataJson, null, 2), {
metadata: trackingMetadata // v4.10.0: Add tracking metadata
metadata: trackingMetadata // Add tracking metadata
})
result.files.push({
path: metadataPath,
@ -332,7 +332,7 @@ export class VFSStructureGenerator {
reportProgress('metadata', 'Created metadata file')
}
// v4.11.1: Final progress update
// Final progress update
if (options.onProgress) {
options.onProgress({
stage: 'metadata',
@ -355,7 +355,7 @@ export class VFSStructureGenerator {
): Map<string, typeof importResult.rows> {
const groups = new Map<string, typeof importResult.rows>()
// Handle sheet-based grouping (v4.2.0)
// Handle sheet-based grouping
if (options.groupBy === 'sheet' && importResult.sheets && importResult.sheets.length > 0) {
for (const sheet of importResult.sheets) {
groups.set(sheet.name, sheet.rows)

6
src/index.ts
View file

@ -70,7 +70,7 @@ export type {
NeuralImportOptions
} from './cortex/neuralImport.js'
// Export Neural Entity Extraction (v5.7.6 - Workshop request)
// Export Neural Entity Extraction
export { NeuralEntityExtractor } from './neural/entityExtractor.js'
export { SmartExtractor } from './neural/SmartExtractor.js'
export { SmartRelationshipExtractor } from './neural/SmartRelationshipExtractor.js'
@ -220,7 +220,7 @@ export {
// FileSystemStorage is exported separately to avoid browser build issues
export { FileSystemStorage } from './storage/adapters/fileSystemStorage.js'
// Export COW (Copy-on-Write) infrastructure for v5.0.0
// Export COW (Copy-on-Write) infrastructure
// Enables premium augmentations to implement temporal features
import { CommitLog } from './storage/cow/CommitLog.js'
import { CommitObject, CommitBuilder } from './storage/cow/CommitObject.js'
@ -543,7 +543,7 @@ export type {
MCPTool
}
// ============= Integration Hub (v7.4.0) =============
// ============= Integration Hub =============
// Connect Brainy to Excel, Power BI, Google Sheets, and more
// Enable with: new Brainy({ integrations: true })

2
src/integrations/index.ts
View file

@ -7,7 +7,7 @@
* - Real-time dashboards (SSE streaming)
* - External webhooks (push notifications)
*
* @example Enable integrations (v7.4.0 - recommended)
* @example Enable integrations (recommended)
* ```typescript
* import { Brainy } from '@soulcraft/brainy'
*

6
src/interfaces/IIndex.ts
View file

@ -1,5 +1,5 @@
/**
* Unified Index Interface (v3.35.0+)
* Unified Index Interface
*
* Standardizes index lifecycle across all index types in Brainy.
* All indexes (HNSW Vector, Graph Adjacency, Metadata Field) implement this interface
@ -57,7 +57,7 @@ export interface RebuildOptions {
* - On-demand: Large datasets loaded adaptively via UnifiedCache
*
* This option is kept for backwards compatibility but is ignored.
* The system always uses adaptive caching (v3.36.0+).
* The system always uses adaptive caching.
*/
lazy?: boolean
@ -104,7 +104,7 @@ export interface IIndex {
* - Provide progress reporting for large datasets
* - Recover gracefully from partial failures
*
* Adaptive Caching (v3.36.0+):
* Adaptive Caching:
* System automatically chooses optimal strategy:
* - Small datasets: Preload all data at init for zero-latency access
* - Large datasets: Load on-demand via UnifiedCache for memory efficiency

20
src/neural/entityExtractor.ts
View file

@ -2,7 +2,7 @@
* Neural Entity Extractor using Brainy's NounTypes
* Uses embeddings and similarity matching for accurate type detection
*
* v4.2.0: Now powered by SmartExtractor for ultra-neural classification
* Now powered by SmartExtractor for ultra-neural classification
* PRODUCTION-READY with caching support
*/
@ -24,7 +24,7 @@ export interface ExtractedEntity {
type: NounType
position: { start: number; end: number }
confidence: number
weight?: number // v4.2.0: Entity importance/salience
weight?: number // Entity importance/salience
vector?: Vector
metadata?: any
}
@ -39,7 +39,7 @@ export class NeuralEntityExtractor {
// Entity extraction cache
private cache: EntityExtractionCache
// Runtime embedding cache for performance (v3.38.0)
// Runtime embedding cache for performance
// Caches candidate embeddings during an extraction session to avoid redundant model calls
private embeddingCache: Map<string, Vector> = new Map()
private embeddingCacheStats = {
@ -48,7 +48,7 @@ export class NeuralEntityExtractor {
size: 0
}
// v4.2.0: SmartExtractor for ultra-neural classification
// SmartExtractor for ultra-neural classification
private smartExtractor: SmartExtractor
constructor(brain: Brainy | Brainy<any>, cacheOptions?: EntityCacheOptions) {
@ -63,7 +63,7 @@ export class NeuralEntityExtractor {
/**
* Initialize type embeddings for neural matching
* PRODUCTION OPTIMIZATION (v3.33.0): Uses pre-computed embeddings from build time
* PRODUCTION OPTIMIZATION: Uses pre-computed embeddings from build time
* Zero runtime cost - embeddings are loaded instantly from embedded data
*/
private async initializeTypeEmbeddings(requestedTypes?: NounType[]): Promise<void> {
@ -109,7 +109,7 @@ export class NeuralEntityExtractor {
}
}
): Promise<ExtractedEntity[]> {
// PRODUCTION OPTIMIZATION (v3.33.0): Load pre-computed type embeddings
// PRODUCTION OPTIMIZATION: Load pre-computed type embeddings
// Zero runtime cost - embeddings were computed at build time
await this.initializeTypeEmbeddings(options?.types)
@ -138,7 +138,7 @@ export class NeuralEntityExtractor {
// Step 1: Extract potential entities using patterns
const candidates = await this.extractCandidates(text)
// Step 2: Classify each candidate using SmartExtractor (v4.2.0)
// Step 2: Classify each candidate using SmartExtractor
for (const candidate of candidates) {
// Use SmartExtractor for unified neural + rule-based classification
const classification = await this.smartExtractor.extract(candidate.text, {
@ -357,7 +357,7 @@ export class NeuralEntityExtractor {
}
/**
* Get embedding for text with caching (v3.38.0)
* Get embedding for text with caching
*
* PERFORMANCE OPTIMIZATION: Caches embeddings during extraction session
* to avoid redundant model calls for repeated text (common in large imports)
@ -509,7 +509,7 @@ export class NeuralEntityExtractor {
}
/**
* Clear embedding cache (v3.38.0)
* Clear embedding cache
*
* Clears the runtime embedding cache. Useful for:
* - Freeing memory after large imports
@ -525,7 +525,7 @@ export class NeuralEntityExtractor {
}
/**
* Get embedding cache statistics (v3.38.0)
* Get embedding cache statistics
*
* Returns performance metrics for the embedding cache:
* - hits: Number of cache hits (avoided model calls)

2
src/neural/naturalLanguageProcessor.ts
View file

@ -99,7 +99,7 @@ export class NaturalLanguageProcessor {
/**
* Initialize embeddings for all NounTypes and VerbTypes
* PRODUCTION OPTIMIZATION (v3.33.0): Uses pre-computed type embeddings
* PRODUCTION OPTIMIZATION: Uses pre-computed type embeddings
* Zero runtime cost - embeddings are loaded instantly from embedded data
*/
private async initializeTypeEmbeddings(): Promise<void> {

4
src/neural/signals/PatternSignal.ts
View file

@ -122,7 +122,7 @@ export class PatternSignal {
])
// Location patterns - MEDIUM PRIORITY (city/country format - requires more context)
// v4.11.2: Lower priority to avoid matching person names with commas
// Lower priority to avoid matching person names with commas
this.addPatterns(NounType.Location, 0.75, [
/\b[A-Z][a-z]+,\s*(?:Japan|China|France|Germany|Italy|Spain|Canada|Mexico|Brazil|India|Australia|Russia|UK|USA)\b/
])
@ -170,7 +170,7 @@ export class PatternSignal {
// Technology patterns (Thing type)
this.addPatterns(NounType.Thing, 0.82, [
/\b(?:JavaScript|TypeScript|Python|Java|Go|Rust|Swift|Kotlin)\b/,
/\bC\+\+(?!\w)/, // v4.11.2: Special handling for C++ (word boundary doesn't work with +)
/\bC\+\+(?!\w)/, // Special handling for C++ (word boundary doesn't work with +)
/\b(?:React|Vue|Angular|Node|Express|Django|Flask|Rails)\b/,
/\b(?:AWS|Azure|GCP|Docker|Kubernetes|Git|GitHub|GitLab)\b/,
/\b(?:API|SDK|CLI|IDE|framework|library|package|module)\b/i,

2
src/setup.ts
View file

@ -1,7 +1,7 @@
/**
* Brainy Setup - Minimal Polyfills
*
* ARCHITECTURE (v7.0.0):
* ARCHITECTURE:
* Brainy uses Candle WASM (Rust-based) for embeddings.
* No transformers.js or ONNX Runtime dependency, no hacks required.
*

108
src/storage/adapters/azureBlobStorage.ts
View file

@ -9,7 +9,7 @@
* 4. SAS Token
* 5. Azure AD (OAuth2) via DefaultAzureCredential
*
* v4.0.0: Fully compatible with metadata/vector separation architecture
* Fully compatible with metadata/vector separation architecture
*/
import {
@ -65,7 +65,7 @@ const MAX_AZURE_PAGE_SIZE = 5000
* 3. Storage Account Key - if accountName + accountKey provided
* 4. SAS Token - if accountName + sasToken provided
*
* v5.4.0: Type-aware storage now built into BaseStorage
* Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed pagination overrides
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -105,7 +105,7 @@ export class AzureBlobStorage extends BaseStorage {
// Request coalescer for deduplication
private requestCoalescer: RequestCoalescer | null = null
// v6.2.7: Write buffering always enabled for consistent performance
// Write buffering always enabled for consistent performance
// Removes dynamic mode switching complexity - cloud storage always benefits from batching
// Multi-level cache manager for efficient data access
@ -115,7 +115,7 @@ export class AzureBlobStorage extends BaseStorage {
// Module logger
private logger = createModuleLogger('AzureBlobStorage')
// v5.4.0: HNSW mutex locks to prevent read-modify-write races
// HNSW mutex locks to prevent read-modify-write races
private hnswLocks = new Map<string, Promise<void>>()
/**
@ -162,7 +162,7 @@ export class AzureBlobStorage extends BaseStorage {
}
/**
* Initialization mode for fast cold starts (v7.3.0+)
* Initialization mode for fast cold starts
*
* - `'auto'` (default): Progressive in cloud environments (Azure Functions),
* strict locally. Zero-config optimization.
@ -171,7 +171,6 @@ export class AzureBlobStorage extends BaseStorage {
* - `'strict'`: Traditional blocking init. Validates container and loads counts
* before init() returns.
*
* @since v7.3.0
*/
initMode?: InitMode
@ -185,7 +184,7 @@ export class AzureBlobStorage extends BaseStorage {
this.sasToken = options.sasToken
this.readOnly = options.readOnly || false
// v7.3.0: Handle initMode
// Handle initMode
if (options.initMode) {
this.initMode = options.initMode
}
@ -201,7 +200,7 @@ export class AzureBlobStorage extends BaseStorage {
this.nounCacheManager = new CacheManager<HNSWNode>(options.cacheConfig)
this.verbCacheManager = new CacheManager<Edge>(options.cacheConfig)
// v6.2.7: Write buffering always enabled - no env var check needed
// Write buffering always enabled - no env var check needed
}
/**
@ -216,7 +215,7 @@ export class AzureBlobStorage extends BaseStorage {
* Recent Azure improvements make parallel downloads very efficient
*
* @returns Azure Blob-optimized batch configuration
* @since v5.12.0 - Updated for native batch API
* Updated for native batch API
*/
public getBatchConfig(): StorageBatchConfig {
return {
@ -241,7 +240,6 @@ export class AzureBlobStorage extends BaseStorage {
*
* @param paths - Array of Azure blob paths to read
* @returns Map of path -> parsed JSON data (only successful reads)
* @since v5.12.0
*/
public async readBatch(paths: string[]): Promise<Map<string, any>> {
await this.ensureInitialized()
@ -297,7 +295,7 @@ export class AzureBlobStorage extends BaseStorage {
/**
* Initialize the storage adapter
*
* v7.3.0: Supports progressive initialization for fast cold starts
* Supports progressive initialization for fast cold starts
*
* | Mode | Init Time | When |
* |------|-----------|------|
@ -404,7 +402,7 @@ export class AzureBlobStorage extends BaseStorage {
this.verbCacheManager.clear()
prodLog.info('✅ Cache cleared - starting fresh')
// v7.3.0: Progressive vs Strict initialization
// Progressive vs Strict initialization
if (effectiveMode === 'progressive') {
// PROGRESSIVE MODE: Fast init, background validation
// Mark as initialized immediately - ready to accept operations
@ -413,7 +411,7 @@ export class AzureBlobStorage extends BaseStorage {
prodLog.info(`✅ Azure progressive init complete: ${this.containerName} (validation deferred)`)
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
// Schedule background tasks (non-blocking)
@ -434,7 +432,7 @@ export class AzureBlobStorage extends BaseStorage {
await this.initializeCounts()
this.countsLoaded = true
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
// Mark background tasks as complete (nothing to do in background)
@ -447,7 +445,7 @@ export class AzureBlobStorage extends BaseStorage {
}
// =============================================
// Progressive Initialization (v7.3.0+)
// Progressive Initialization
// =============================================
/**
@ -461,7 +459,6 @@ export class AzureBlobStorage extends BaseStorage {
*
* @protected
* @override
* @since v7.3.0
*/
protected async runBackgroundInit(): Promise<void> {
const startTime = Date.now()
@ -485,7 +482,6 @@ export class AzureBlobStorage extends BaseStorage {
* bucketValidated/bucketValidationError for lazy use.
*
* @private
* @since v7.3.0
*/
private async validateContainerInBackground(): Promise<void> {
try {
@ -517,7 +513,6 @@ export class AzureBlobStorage extends BaseStorage {
* Load counts from storage in background.
*
* @private
* @since v7.3.0
*/
private async loadCountsInBackground(): Promise<void> {
try {
@ -541,7 +536,6 @@ export class AzureBlobStorage extends BaseStorage {
* @throws Error if container validation fails
* @protected
* @override
* @since v7.3.0
*/
protected async ensureValidatedForWrite(): Promise<void> {
// If already validated, nothing to do
@ -665,7 +659,7 @@ export class AzureBlobStorage extends BaseStorage {
}
}
// v6.2.7: Removed checkVolumeMode() - write buffering always enabled for cloud storage
// Removed checkVolumeMode() - write buffering always enabled for cloud storage
/**
* Flush noun buffer to Azure
@ -697,20 +691,20 @@ export class AzureBlobStorage extends BaseStorage {
await Promise.all(writes)
}
// v5.4.0: Removed saveNoun_internal - now inherit from BaseStorage's type-first implementation
// Removed saveNoun_internal - now inherit from BaseStorage's type-first implementation
/**
* Save a node to storage
* v6.2.7: Always uses write buffer for consistent performance
* Always uses write buffer for consistent performance
*/
protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching
// Always use write buffer - cloud storage benefits from batching
if (this.nounWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency
// Populate cache BEFORE buffering for read-after-write consistency
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node)
}
@ -791,7 +785,7 @@ export class AzureBlobStorage extends BaseStorage {
}
}
// v5.4.0: Removed getNoun_internal - now inherit from BaseStorage's type-first implementation
// Removed getNoun_internal - now inherit from BaseStorage's type-first implementation
/**
* Get a node from storage
@ -889,18 +883,18 @@ export class AzureBlobStorage extends BaseStorage {
}
}
// v5.4.0: Removed deleteNoun_internal - now inherit from BaseStorage's type-first implementation
// Removed deleteNoun_internal - now inherit from BaseStorage's type-first implementation
/**
* Write an object to a specific path in Azure
* Primitive operation required by base class
*
* v7.3.0: Performs lazy container validation on first write in progressive mode.
* Performs lazy container validation on first write in progressive mode.
* @protected
*/
protected async writeObjectToPath(path: string, data: any): Promise<void> {
await this.ensureInitialized()
// v7.3.0: Lazy container validation for progressive init
// Lazy container validation for progressive init
await this.ensureValidatedForWrite()
try {
@ -954,12 +948,12 @@ export class AzureBlobStorage extends BaseStorage {
* Delete an object from a specific path in Azure
* Primitive operation required by base class
*
* v7.3.0: Performs lazy container validation on first delete in progressive mode.
* Performs lazy container validation on first delete in progressive mode.
* @protected
*/
protected async deleteObjectFromPath(path: string): Promise<void> {
await this.ensureInitialized()
// v7.3.0: Lazy container validation for progressive init
// Lazy container validation for progressive init
await this.ensureValidatedForWrite()
try {
@ -1209,20 +1203,20 @@ export class AzureBlobStorage extends BaseStorage {
})
}
// v5.4.0: Removed saveVerb_internal - now inherit from BaseStorage's type-first implementation
// Removed saveVerb_internal - now inherit from BaseStorage's type-first implementation
/**
* Save an edge to storage
* v6.2.7: Always uses write buffer for consistent performance
* Always uses write buffer for consistent performance
*/
protected async saveEdge(edge: Edge): Promise<void> {
await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching
// Always use write buffer - cloud storage benefits from batching
if (this.verbWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding verb ${edge.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency
// Populate cache BEFORE buffering for read-after-write consistency
this.verbCacheManager.set(edge.id, edge)
await this.verbWriteBuffer.add(edge.id, edge)
@ -1255,7 +1249,7 @@ export class AzureBlobStorage extends BaseStorage {
])
),
// CORE RELATIONAL DATA (v4.0.0)
// CORE RELATIONAL DATA
verb: edge.verb,
sourceId: edge.sourceId,
targetId: edge.targetId,
@ -1280,7 +1274,7 @@ export class AzureBlobStorage extends BaseStorage {
// Update cache
this.verbCacheManager.set(edge.id, edge)
// Count tracking happens in baseStorage.saveVerbMetadata_internal (v4.1.2)
// Count tracking happens in baseStorage.saveVerbMetadata_internal
// This fixes the race condition where metadata didn't exist yet
this.logger.trace(`Edge ${edge.id} saved successfully`)
@ -1298,7 +1292,7 @@ export class AzureBlobStorage extends BaseStorage {
}
}
// v5.4.0: Removed getVerb_internal - now inherit from BaseStorage's type-first implementation
// Removed getVerb_internal - now inherit from BaseStorage's type-first implementation
/**
* Get an edge from storage
@ -1335,7 +1329,7 @@ export class AzureBlobStorage extends BaseStorage {
connections.set(Number(level), new Set(verbIds as string[]))
}
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field)
// Return HNSWVerb with core relational fields (NO metadata field)
const edge: Edge = {
id: data.id,
vector: data.vector,
@ -1346,7 +1340,7 @@ export class AzureBlobStorage extends BaseStorage {
sourceId: data.sourceId,
targetId: data.targetId
// ✅ NO metadata field in v4.0.0
// ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata()
}
@ -1375,13 +1369,13 @@ export class AzureBlobStorage extends BaseStorage {
}
}
// v5.4.0: Removed deleteVerb_internal - now inherit from BaseStorage's type-first implementation
// Removed deleteVerb_internal - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed getNounsWithPagination - now inherit from BaseStorage's type-first implementation
// Removed getNounsWithPagination - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed getNounsByNounType_internal - now inherit from BaseStorage's type-first implementation
// Removed getNounsByNounType_internal - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed 3 verb query *_internal methods (getVerbsBySource, getVerbsByTarget, getVerbsByType) - now inherit from BaseStorage's type-first implementation
// Removed 3 verb query *_internal methods (getVerbsBySource, getVerbsByTarget, getVerbsByType) - now inherit from BaseStorage's type-first implementation
/**
* Clear all data from storage
@ -1393,7 +1387,7 @@ export class AzureBlobStorage extends BaseStorage {
this.logger.info('🧹 Clearing all data from Azure container...')
// Delete all blobs in container
// v5.6.1: listBlobsFlat() returns ALL blobs including _cow/ prefix
// listBlobsFlat() returns ALL blobs including _cow/ prefix
// This correctly deletes COW version control data (commits, trees, blobs, refs)
for await (const blob of this.containerClient!.listBlobsFlat()) {
if (blob.name) {
@ -1402,7 +1396,7 @@ export class AzureBlobStorage extends BaseStorage {
}
}
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled)
// Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use
this.refManager = undefined
this.blobStorage = undefined
@ -1461,12 +1455,12 @@ export class AzureBlobStorage extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts
* Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker blob exists, false otherwise
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
@ -1643,7 +1637,7 @@ export class AzureBlobStorage extends BaseStorage {
/**
* Get a noun's vector for HNSW rebuild
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id)
@ -1653,7 +1647,7 @@ export class AzureBlobStorage extends BaseStorage {
/**
* Save HNSW graph data for a noun
*
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths)
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Uses mutex locking to prevent read-modify-write races
*/
public async saveHNSWData(nounId: string, hnswData: {
@ -1662,7 +1656,7 @@ export class AzureBlobStorage extends BaseStorage {
}): Promise<void> {
const lockKey = `hnsw/${nounId}`
// CRITICAL FIX (v4.10.1): Mutex lock to prevent read-modify-write races
// CRITICAL FIX: Mutex lock to prevent read-modify-write races
// Problem: Without mutex, concurrent operations can:
// 1. Thread A reads noun (connections: [1,2,3])
// 2. Thread B reads noun (connections: [1,2,3])
@ -1682,7 +1676,7 @@ export class AzureBlobStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise)
try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths)
// Use BaseStorage's getNoun (type-first paths)
// Read existing noun data (if exists)
const existingNoun = await this.getNoun(nounId)
@ -1704,7 +1698,7 @@ export class AzureBlobStorage extends BaseStorage {
connections: connectionsMap
}
// v5.4.0: Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
// Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
await this.saveNoun(updatedNoun)
} finally {
// Release lock (ALWAYS runs, even if error thrown)
@ -1715,7 +1709,7 @@ export class AzureBlobStorage extends BaseStorage {
/**
* Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getHNSWData(nounId: string): Promise<{
level: number
@ -1744,7 +1738,7 @@ export class AzureBlobStorage extends BaseStorage {
/**
* Save HNSW system data (entry point, max level)
*
* CRITICAL FIX (v4.10.1): Optimistic locking with ETags to prevent race conditions
* CRITICAL FIX: Optimistic locking with ETags to prevent race conditions
*/
public async saveHNSWSystem(systemData: {
entryPointId: string | null
@ -1840,7 +1834,7 @@ export class AzureBlobStorage extends BaseStorage {
}
/**
* Set the access tier for a specific blob (v4.0.0 cost optimization)
* Set the access tier for a specific blob (cost optimization)
* Azure Blob Storage tiers:
* - Hot: $0.0184/GB/month - Frequently accessed data
* - Cool: $0.01/GB/month - Infrequently accessed data (45% cheaper)
@ -1906,7 +1900,7 @@ export class AzureBlobStorage extends BaseStorage {
}
/**
* Set access tier for multiple blobs in batch (v4.0.0 cost optimization)
* Set access tier for multiple blobs in batch (cost optimization)
* Efficiently move large numbers of blobs between tiers for cost optimization
*
* @param blobs - Array of blob names and their target tiers
@ -2128,7 +2122,7 @@ export class AzureBlobStorage extends BaseStorage {
}
/**
* Set lifecycle management policy for automatic tier transitions and deletions (v4.0.0)
* Set lifecycle management policy for automatic tier transitions and deletions
* Automates cost optimization by moving old data to cheaper tiers or deleting it
*
* Azure Lifecycle Management rules run once per day and apply to the entire container.

43
src/storage/adapters/baseStorageAdapter.ts
View file

@ -19,7 +19,7 @@ import { extractFieldNamesFromJson, mapToStandardField } from '../../utils/field
import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
// =============================================================================
// Progressive Initialization Types (v7.3.0+)
// Progressive Initialization Types
// =============================================================================
/**
@ -29,7 +29,6 @@ import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
* fast cold starts in serverless environments while maintaining strict
* validation for local development.
*
* @since v7.3.0
*
* | Mode | Description | Use Case |
* |------|-------------|----------|
@ -96,7 +95,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
abstract getVerbMetadata(id: string): Promise<any | null>
// HNSW Index Persistence (v3.35.0+)
// HNSW Index Persistence
// These methods enable HNSW index rebuilding after container restarts
abstract getNounVector(id: string): Promise<number[] | null>
@ -139,7 +138,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* relateMany(), and import operations to automatically adapt to storage capabilities.
*
* @returns Batch configuration optimized for this storage type
* @since v4.11.0
*/
public getBatchConfig(): StorageBatchConfig {
// Conservative defaults that work safely across all storage types
@ -295,7 +293,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
}> = new Map()
// =============================================
// Progressive Initialization State (v7.3.0+)
// Progressive Initialization State
// =============================================
// These properties enable fast cold starts in cloud environments
// by deferring validation and count loading to background tasks.
@ -308,7 +306,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* - `'strict'`: Always validate during init (traditional behavior)
*
* @protected
* @since v7.3.0
*/
protected initMode: InitMode = 'auto'
@ -319,7 +316,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* after the first successful write operation or background validation.
*
* @protected
* @since v7.3.0
*/
protected bucketValidated = false
@ -329,7 +325,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* Stored here so subsequent operations can fail fast with the same error.
*
* @protected
* @since v7.3.0
*/
protected bucketValidationError: Error | null = null
@ -340,7 +335,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* Operations work immediately; counts become accurate after background load.
*
* @protected
* @since v7.3.0
*/
protected countsLoaded = false
@ -350,7 +344,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* Useful for tests and diagnostics to ensure full initialization.
*
* @protected
* @since v7.3.0
*/
protected backgroundTasksComplete = false
@ -361,7 +354,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* before proceeding (e.g., in tests).
*
* @protected
* @since v7.3.0
*/
protected backgroundInitPromise: Promise<void> | null = null
@ -1057,7 +1049,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
protected readonly COUNT_CACHE_TTL = 60000 // 1 minute cache TTL
// =============================================
// Smart Count Batching (v3.32.3+)
// Smart Count Batching
// =============================================
// Count batching state - mirrors statistics batching pattern
@ -1112,7 +1104,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
const mutex = getGlobalMutex()
await mutex.runExclusive(`count-entity-${type}`, async () => {
this.incrementEntityCount(type)
// Smart batching (v3.32.3+): Adapts to storage type
// Smart batching: Adapts to storage type
// - Cloud storage (GCS, S3): Batches 10 ops OR 5 seconds
// - Local storage (File, Memory): Persists immediately
await this.scheduleCountPersist()
@ -1147,13 +1139,13 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
const mutex = getGlobalMutex()
await mutex.runExclusive(`count-entity-${type}`, async () => {
this.decrementEntityCount(type)
// Smart batching (v3.32.3+): Adapts to storage type
// Smart batching: Adapts to storage type
await this.scheduleCountPersist()
})
}
/**
* Increment verb count - O(1) operation (v4.1.2: now synchronous)
* Increment verb count - O(1) operation (now synchronous)
* Protected by storage-specific mechanisms (mutex, distributed consensus, etc.)
* @param type The verb type
*/
@ -1168,7 +1160,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
}
/**
* Thread-safe increment for verb counts (v4.1.2)
* Thread-safe increment for verb counts
* Uses mutex for single-node, distributed consensus for multi-node
* @param type The verb type
*/
@ -1176,13 +1168,13 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
const mutex = getGlobalMutex()
await mutex.runExclusive(`count-verb-${type}`, async () => {
this.incrementVerbCount(type)
// Smart batching (v3.32.3+): Adapts to storage type
// Smart batching: Adapts to storage type
await this.scheduleCountPersist()
})
}
/**
* Decrement verb count - O(1) operation (v4.1.2: now synchronous)
* Decrement verb count - O(1) operation (now synchronous)
* @param type The verb type
*/
protected decrementVerbCount(type: string): void {
@ -1203,20 +1195,20 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
}
/**
* Thread-safe decrement for verb counts (v4.1.2)
* Thread-safe decrement for verb counts
* @param type The verb type
*/
protected async decrementVerbCountSafe(type: string): Promise<void> {
const mutex = getGlobalMutex()
await mutex.runExclusive(`count-verb-${type}`, async () => {
this.decrementVerbCount(type)
// Smart batching (v3.32.3+): Adapts to storage type
// Smart batching: Adapts to storage type
await this.scheduleCountPersist()
})
}
// =============================================
// Smart Batching Methods (v3.32.3+)
// Smart Batching Methods
// =============================================
/**
@ -1235,7 +1227,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
}
// =============================================
// Progressive Initialization Methods (v7.3.0+)
// Progressive Initialization Methods
// =============================================
/**
@ -1254,7 +1246,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
*
* @returns `true` if running in a detected cloud environment
* @protected
* @since v7.3.0
*/
protected detectCloudEnvironment(): boolean {
return !!(
@ -1274,7 +1265,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
*
* @returns The resolved init mode (`'progressive'` or `'strict'`)
* @protected
* @since v7.3.0
*/
protected resolveInitMode(): 'progressive' | 'strict' {
if (this.initMode === 'auto') {
@ -1295,7 +1285,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* Always call `super.scheduleBackgroundInit()` first.
*
* @protected
* @since v7.3.0
*/
protected scheduleBackgroundInit(): void {
// Create a promise that tracks all background work
@ -1320,7 +1309,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* The default implementation does nothing (for local storage adapters).
*
* @protected
* @since v7.3.0
*/
protected async runBackgroundInit(): Promise<void> {
// Default implementation: nothing to do for local storage
@ -1348,7 +1336,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* ```
*
* @public
* @since v7.3.0
*/
public async awaitBackgroundInit(): Promise<void> {
if (this.backgroundInitPromise) {
@ -1361,7 +1348,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
*
* @returns `true` if all background tasks are complete
* @public
* @since v7.3.0
*/
public isBackgroundInitComplete(): boolean {
return this.backgroundTasksComplete
@ -1379,7 +1365,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
*
* @throws Error if bucket validation fails
* @protected
* @since v7.3.0
*/
protected async ensureValidatedForWrite(): Promise<void> {
// If already validated, nothing to do

109
src/storage/adapters/fileSystemStorage.ts
View file

@ -59,7 +59,7 @@ try {
* File system storage adapter for Node.js environments
* Uses the file system to store data in the specified directory structure
*
* v5.4.0: Type-aware storage now built into BaseStorage
* Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination)
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -91,12 +91,12 @@ export class FileSystemStorage extends BaseStorage {
private lockTimers: Map<string, NodeJS.Timeout> = new Map() // Track timers for cleanup
private allTimers: Set<NodeJS.Timeout> = new Set() // Track all timers for cleanup
// CRITICAL FIX (v4.10.1): Mutex locks for HNSW concurrency control
// CRITICAL FIX: Mutex locks for HNSW concurrency control
// Prevents read-modify-write races during concurrent neighbor updates at scale (1000+ ops)
// Matches MemoryStorage and OPFSStorage behavior (tested in production)
private hnswLocks = new Map<string, Promise<void>>()
// Compression configuration (v4.0.0)
// Compression configuration
private compressionEnabled: boolean = true // Enable gzip compression by default for 60-80% disk savings
private compressionLevel: number = 6 // zlib compression level (1-9, default: 6 = balanced)
@ -136,7 +136,6 @@ export class FileSystemStorage extends BaseStorage {
* - Parallel processing supported
*
* @returns FileSystem-optimized batch configuration
* @since v4.11.0
*/
public getBatchConfig(): StorageBatchConfig {
return {
@ -179,7 +178,7 @@ export class FileSystemStorage extends BaseStorage {
try {
// Initialize directory paths now that path module is loaded
// Clean directory structure (v4.7.2+)
// Clean directory structure
this.nounsDir = path.join(this.rootDir, 'entities/nouns/hnsw')
this.verbsDir = path.join(this.rootDir, 'entities/verbs/hnsw')
this.metadataDir = path.join(this.rootDir, 'entities/nouns/metadata') // Legacy reference
@ -245,7 +244,7 @@ export class FileSystemStorage extends BaseStorage {
// Always use fixed depth after migration/detection
this.cachedShardingDepth = this.SHARDING_DEPTH
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
} catch (error) {
console.error('Error initializing FileSystemStorage:', error)
@ -281,7 +280,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Save a node to storage
* CRITICAL FIX (v4.10.3): Added atomic write pattern to prevent file corruption during concurrent imports
* CRITICAL FIX: Added atomic write pattern to prevent file corruption during concurrent imports
*/
protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized()
@ -303,7 +302,7 @@ export class FileSystemStorage extends BaseStorage {
const tempPath = `${filePath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}`
try {
// ATOMIC WRITE SEQUENCE (v4.10.3):
// ATOMIC WRITE SEQUENCE:
// 1. Write to temp file
await this.ensureDirectoryExists(path.dirname(tempPath))
await fs.promises.writeFile(tempPath, JSON.stringify(serializableNode, null, 2))
@ -320,7 +319,7 @@ export class FileSystemStorage extends BaseStorage {
throw error
}
// Count tracking happens in baseStorage.saveNounMetadata_internal (v4.1.2)
// Count tracking happens in baseStorage.saveNounMetadata_internal
// This fixes the race condition where metadata didn't exist yet
}
@ -361,7 +360,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Get all nodes from storage
* CRITICAL FIX (v3.43.2): Now scans sharded subdirectories (depth=1)
* CRITICAL FIX: Now scans sharded subdirectories (depth=1)
* Previously only scanned flat directory, causing rebuild to find 0 entities
*/
protected async getAllNodes(): Promise<HNSWNode[]> {
@ -406,7 +405,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Get nodes by noun type
* CRITICAL FIX (v3.43.2): Now scans sharded subdirectories (depth=1)
* CRITICAL FIX: Now scans sharded subdirectories (depth=1)
* @param nounType The noun type to filter by
* @returns Promise that resolves to an array of nodes of the specified noun type
*/
@ -462,7 +461,7 @@ export class FileSystemStorage extends BaseStorage {
const filePath = this.getNodePath(id)
// Load metadata to get type for count update (v4.0.0: separate storage)
// Load metadata to get type for count update (separate storage)
try {
const metadata = await this.getNounMetadata(id)
if (metadata) {
@ -490,13 +489,13 @@ export class FileSystemStorage extends BaseStorage {
/**
* Save an edge to storage
* CRITICAL FIX (v4.10.3): Added atomic write pattern to prevent file corruption during concurrent imports
* CRITICAL FIX: Added atomic write pattern to prevent file corruption during concurrent imports
*/
protected async saveEdge(edge: Edge): Promise<void> {
await this.ensureInitialized()
// Convert connections Map to a serializable format
// ARCHITECTURAL FIX (v3.50.1): Include core relational fields in verb vector file
// ARCHITECTURAL FIX: Include core relational fields in verb vector file
// These fields are essential for 90% of operations - no metadata lookup needed
const serializableEdge = {
id: edge.id,
@ -505,7 +504,7 @@ export class FileSystemStorage extends BaseStorage {
Array.from(set as Set<string>)
),
// CORE RELATIONAL DATA (v3.50.1+)
// CORE RELATIONAL DATA
verb: edge.verb,
sourceId: edge.sourceId,
targetId: edge.targetId,
@ -518,7 +517,7 @@ export class FileSystemStorage extends BaseStorage {
const tempPath = `${filePath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}`
try {
// ATOMIC WRITE SEQUENCE (v4.10.3):
// ATOMIC WRITE SEQUENCE:
// 1. Write to temp file
await this.ensureDirectoryExists(path.dirname(tempPath))
await fs.promises.writeFile(tempPath, JSON.stringify(serializableEdge, null, 2))
@ -535,7 +534,7 @@ export class FileSystemStorage extends BaseStorage {
throw error
}
// Count tracking happens in baseStorage.saveVerbMetadata_internal (v4.1.2)
// Count tracking happens in baseStorage.saveVerbMetadata_internal
// This fixes the race condition where metadata didn't exist yet
}
@ -556,7 +555,7 @@ export class FileSystemStorage extends BaseStorage {
connections.set(Number(level), new Set(nodeIds as string[]))
}
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field)
// Return HNSWVerb with core relational fields (NO metadata field)
return {
id: parsedEdge.id,
vector: parsedEdge.vector,
@ -567,7 +566,7 @@ export class FileSystemStorage extends BaseStorage {
sourceId: parsedEdge.sourceId,
targetId: parsedEdge.targetId
// ✅ NO metadata field in v4.0.0
// ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata()
}
} catch (error: any) {
@ -580,7 +579,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Get all edges from storage
* CRITICAL FIX (v3.43.2): Now scans sharded subdirectories (depth=1)
* CRITICAL FIX: Now scans sharded subdirectories (depth=1)
* Previously only scanned flat directory, causing rebuild to find 0 relationships
*/
protected async getAllEdges(): Promise<Edge[]> {
@ -608,7 +607,7 @@ export class FileSystemStorage extends BaseStorage {
connections.set(Number(level), new Set(nodeIds as string[]))
}
// v4.0.0: Include core relational fields (NO metadata field)
// Include core relational fields (NO metadata field)
allEdges.push({
id: parsedEdge.id,
vector: parsedEdge.vector,
@ -619,7 +618,7 @@ export class FileSystemStorage extends BaseStorage {
sourceId: parsedEdge.sourceId,
targetId: parsedEdge.targetId
// ✅ NO metadata field in v4.0.0
// ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata()
})
}
@ -696,8 +695,8 @@ export class FileSystemStorage extends BaseStorage {
/**
* Primitive operation: Write object to path
* All metadata operations use this internally via base class routing
* v4.0.0: Supports gzip compression for 60-80% disk savings
* CRITICAL FIX (v4.10.3): Added atomic write pattern to prevent file corruption during concurrent imports
* Supports gzip compression for 60-80% disk savings
* CRITICAL FIX: Added atomic write pattern to prevent file corruption during concurrent imports
*/
protected async writeObjectToPath(pathStr: string, data: any): Promise<void> {
await this.ensureInitialized()
@ -711,7 +710,7 @@ export class FileSystemStorage extends BaseStorage {
const tempPath = `${compressedPath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}`
try {
// ATOMIC WRITE SEQUENCE (v4.10.3):
// ATOMIC WRITE SEQUENCE:
// 1. Compress and write to temp file
const jsonString = JSON.stringify(data, null, 2)
const compressed = await new Promise<Buffer>((resolve, reject) => {
@ -748,7 +747,7 @@ export class FileSystemStorage extends BaseStorage {
const tempPath = `${fullPath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}`
try {
// ATOMIC WRITE SEQUENCE (v4.10.3):
// ATOMIC WRITE SEQUENCE:
// 1. Write to temp file
await fs.promises.writeFile(tempPath, JSON.stringify(data, null, 2))
@ -770,7 +769,7 @@ export class FileSystemStorage extends BaseStorage {
* Primitive operation: Read object from path
* All metadata operations use this internally via base class routing
* Enhanced error handling for corrupted metadata files (Bug #3 mitigation)
* v4.0.0: Supports reading both compressed (.gz) and uncompressed files for backward compatibility
* Supports reading both compressed (.gz) and uncompressed files for backward compatibility
*/
protected async readObjectFromPath(pathStr: string): Promise<any | null> {
await this.ensureInitialized()
@ -822,7 +821,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Primitive operation: Delete object from path
* All metadata operations use this internally via base class routing
* v4.0.0: Deletes both compressed and uncompressed versions (for cleanup)
* Deletes both compressed and uncompressed versions (for cleanup)
*/
protected async deleteObjectFromPath(pathStr: string): Promise<void> {
await this.ensureInitialized()
@ -863,7 +862,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Primitive operation: List objects under path prefix
* All metadata operations use this internally via base class routing
* v4.0.0: Handles both .json and .json.gz files, normalizes paths
* Handles both .json and .json.gz files, normalizes paths
*/
protected async listObjectsUnderPath(prefix: string): Promise<string[]> {
await this.ensureInitialized()
@ -877,7 +876,7 @@ export class FileSystemStorage extends BaseStorage {
for (const entry of entries) {
if (entry.isFile()) {
// v5.3.5: Handle multiple compression formats for broad compatibility
// Handle multiple compression formats for broad compatibility
// - .json.gz: Standard entity/metadata files (JSON compressed)
// - .gz: COW files (refs, blobs, commits - raw compressed)
// - .json: Uncompressed JSON files
@ -890,7 +889,7 @@ export class FileSystemStorage extends BaseStorage {
seen.add(normalizedPath)
}
} else if (entry.name.endsWith('.gz')) {
// v5.3.5 fix: COW files stored as .gz (not .json.gz)
// COW files stored as .gz (not .json.gz)
// Strip .gz extension and return path
const normalizedName = entry.name.slice(0, -3) // Remove .gz
const normalizedPath = path.join(prefix, normalizedName)
@ -966,7 +965,7 @@ export class FileSystemStorage extends BaseStorage {
* Get nouns with pagination support
* @param options Pagination options
*/
// v5.4.0: Removed getNounsWithPagination override - now uses BaseStorage's type-first implementation
// Removed getNounsWithPagination override - now uses BaseStorage's type-first implementation
/**
* Clear all data from storage
@ -1002,7 +1001,7 @@ export class FileSystemStorage extends BaseStorage {
}
}
// v5.10.4: Clear the entire branches/ directory (branch-based storage)
// Clear the entire branches/ directory (branch-based storage)
// Bug fix: Data is stored in branches/main/entities/, not just entities/
// The branch-based structure was introduced for COW support
const branchesDir = path.join(this.rootDir, 'branches')
@ -1016,7 +1015,7 @@ export class FileSystemStorage extends BaseStorage {
await removeDirectoryContents(this.indexDir)
}
// v5.6.1: Remove COW (copy-on-write) version control data
// Remove COW (copy-on-write) version control data
// This directory stores all git-like versioning data (commits, trees, blobs, refs)
// Must be deleted to fully clear all data including version history
const cowDir = path.join(this.rootDir, '_cow')
@ -1024,7 +1023,7 @@ export class FileSystemStorage extends BaseStorage {
// Delete the entire _cow/ directory (not just contents)
await fs.promises.rm(cowDir, { recursive: true, force: true })
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled)
// Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use
this.refManager = undefined
this.blobStorage = undefined
@ -1035,12 +1034,12 @@ export class FileSystemStorage extends BaseStorage {
this.statisticsCache = null
this.statisticsModified = false
// v5.6.1: Reset entity counters (inherited from BaseStorageAdapter)
// Reset entity counters (inherited from BaseStorageAdapter)
// These in-memory counters must be reset to 0 after clearing all data
;(this as any).totalNounCount = 0
;(this as any).totalVerbCount = 0
// v7.3.1: Clear write-through cache (inherited from BaseStorage)
// Clear write-through cache (inherited from BaseStorage)
// Without this, readWithInheritance() would return stale cached data
// after clear(), causing "ghost" entities to appear
this.clearWriteCache()
@ -1074,12 +1073,12 @@ export class FileSystemStorage extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts
* Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker file exists, false otherwise
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
@ -1152,7 +1151,7 @@ export class FileSystemStorage extends BaseStorage {
totalSize = nounsDirSize + verbsDirSize + metadataDirSize + indexDirSize
// CRITICAL FIX (v3.43.2): Use persisted counts instead of directory reads
// CRITICAL FIX: Use persisted counts instead of directory reads
// This is O(1) instead of O(n), and handles sharded structure correctly
const nounsCount = this.totalNounCount
const verbsCount = this.totalVerbCount
@ -1210,8 +1209,8 @@ export class FileSystemStorage extends BaseStorage {
}
}
// v5.4.0: Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed 2 pagination methods (getNounsWithPagination, getVerbsWithPagination) - use BaseStorage's implementation
// Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
// Removed 2 pagination methods (getNounsWithPagination, getVerbsWithPagination) - use BaseStorage's implementation
/**
* Acquire a file-based lock for coordinating operations across multiple processes
@ -1503,14 +1502,14 @@ export class FileSystemStorage extends BaseStorage {
this.totalVerbCount = validVerbFiles.length
// Sample some files to get type distribution (don't read all)
// v4.0.0: Load metadata separately for type information
// Load metadata separately for type information
const sampleSize = Math.min(100, validNounFiles.length)
for (let i = 0; i < sampleSize; i++) {
try {
const file = validNounFiles[i]
const id = file.replace('.json', '')
// v4.0.0: Load metadata from separate storage for type info
// Load metadata from separate storage for type info
const metadata = await this.getNounMetadata(id)
if (metadata) {
const type = metadata.noun || 'default'
@ -2148,7 +2147,7 @@ export class FileSystemStorage extends BaseStorage {
const edge = JSON.parse(data)
const metadata = await this.getVerbMetadata(id)
// v4.8.1: Don't skip verbs without metadata - metadata is optional
// Don't skip verbs without metadata - metadata is optional
// FIX: This was the root cause of the VFS bug (11 versions)
// Verbs can exist without metadata files (e.g., from imports/migrations)
@ -2162,7 +2161,7 @@ export class FileSystemStorage extends BaseStorage {
connections = connectionsMap
}
// v4.8.0: Extract standard fields from metadata to top-level
// Extract standard fields from metadata to top-level
const metadataObj = (metadata || {}) as VerbMetadata
const { createdAt, updatedAt, confidence, weight, service, data: dataField, createdBy, ...customMetadata } = metadataObj
@ -2309,12 +2308,12 @@ export class FileSystemStorage extends BaseStorage {
}
// =============================================
// HNSW Index Persistence (v3.35.0+)
// HNSW Index Persistence
// =============================================
/**
* Get vector for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id)
@ -2324,7 +2323,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Save HNSW graph data for a noun
*
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths)
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Preserves mutex locking to prevent read-modify-write races
*/
public async saveHNSWData(nounId: string, hnswData: {
@ -2333,7 +2332,7 @@ export class FileSystemStorage extends BaseStorage {
}): Promise<void> {
const lockKey = `hnsw/${nounId}`
// CRITICAL FIX (v4.10.1): Mutex lock to prevent read-modify-write races
// CRITICAL FIX: Mutex lock to prevent read-modify-write races
// Problem: Without mutex, concurrent operations can:
// 1. Thread A reads noun (connections: [1,2,3])
// 2. Thread B reads noun (connections: [1,2,3])
@ -2353,7 +2352,7 @@ export class FileSystemStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise)
try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths)
// Use BaseStorage's getNoun (type-first paths)
// Read existing noun data (if exists)
const existingNoun = await this.getNoun(nounId)
@ -2375,7 +2374,7 @@ export class FileSystemStorage extends BaseStorage {
connections: connectionsMap
}
// v5.4.0: Use BaseStorage's saveNoun (type-first paths, atomic write)
// Use BaseStorage's saveNoun (type-first paths, atomic write)
await this.saveNoun(updatedNoun)
} finally {
// Release lock (ALWAYS runs, even if error thrown)
@ -2386,7 +2385,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getHNSWData(nounId: string): Promise<{
level: number
@ -2415,7 +2414,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Save HNSW system data (entry point, max level)
*
* CRITICAL FIX (v4.10.1): Mutex lock + atomic write to prevent race conditions
* CRITICAL FIX: Mutex lock + atomic write to prevent race conditions
*/
public async saveHNSWSystem(systemData: {
entryPointId: string | null
@ -2425,7 +2424,7 @@ export class FileSystemStorage extends BaseStorage {
const lockKey = 'hnsw/system'
// CRITICAL FIX (v4.10.1): Mutex lock to serialize system updates
// CRITICAL FIX: Mutex lock to serialize system updates
// System data (entry point, max level) updated frequently during HNSW construction
// Without mutex, concurrent updates can lose data (same as entity-level problem)

135
src/storage/adapters/gcsStorage.ts
View file

@ -65,7 +65,7 @@ const MAX_GCS_PAGE_SIZE = 5000
* 3. Service Account Credentials Object (if credentials provided)
* 4. HMAC Keys (if accessKeyId/secretAccessKey provided)
*
* v5.4.0: Type-aware storage now built into BaseStorage
* Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination)
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -110,7 +110,7 @@ export class GcsStorage extends BaseStorage {
// Request coalescer for deduplication
private requestCoalescer: RequestCoalescer | null = null
// v6.2.7: Write buffering always enabled for consistent performance
// Write buffering always enabled for consistent performance
// Removes dynamic mode switching complexity - cloud storage always benefits from batching
// Multi-level cache manager for efficient data access
@ -120,7 +120,7 @@ export class GcsStorage extends BaseStorage {
// Module logger
private logger = createModuleLogger('GcsStorage')
// v5.4.0: HNSW mutex locks to prevent read-modify-write races
// HNSW mutex locks to prevent read-modify-write races
private hnswLocks = new Map<string, Promise<void>>()
// Configuration options
@ -166,7 +166,7 @@ export class GcsStorage extends BaseStorage {
secretAccessKey?: string
/**
* Initialization mode for fast cold starts (v7.3.0+)
* Initialization mode for fast cold starts
*
* - `'auto'` (default): Progressive in cloud environments (Cloud Run, Lambda),
* strict locally. Zero-config optimization.
@ -175,19 +175,18 @@ export class GcsStorage extends BaseStorage {
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts
* before init() returns.
*
* @since v7.3.0
*/
initMode?: InitMode
/**
* @deprecated Use `initMode: 'progressive'` instead.
* Will be removed in v8.0.0.
* Will be removed in a future version.
*/
skipInitialScan?: boolean
/**
* @deprecated Use `initMode: 'progressive'` instead.
* Will be removed in v8.0.0.
* Will be removed in a future version.
*/
skipCountsFile?: boolean
@ -206,7 +205,7 @@ export class GcsStorage extends BaseStorage {
this.accessKeyId = options.accessKeyId
this.secretAccessKey = options.secretAccessKey
// v7.3.0: Handle initMode and deprecated skip* flags
// Handle initMode and deprecated skip* flags
if (options.initMode) {
this.initMode = options.initMode
}
@ -215,14 +214,14 @@ export class GcsStorage extends BaseStorage {
if (options.skipInitialScan) {
console.warn(
'[GcsStorage] DEPRECATION WARNING: skipInitialScan is deprecated. ' +
'Use initMode: "progressive" instead. Will be removed in v8.0.0.'
'Use initMode: "progressive" instead. Will be removed in a future version.'
)
this.skipInitialScan = true
}
if (options.skipCountsFile) {
console.warn(
'[GcsStorage] DEPRECATION WARNING: skipCountsFile is deprecated. ' +
'Use initMode: "progressive" instead. Will be removed in v8.0.0.'
'Use initMode: "progressive" instead. Will be removed in a future version.'
)
this.skipCountsFile = true
}
@ -240,13 +239,13 @@ export class GcsStorage extends BaseStorage {
this.nounCacheManager = new CacheManager<HNSWNode>(options.cacheConfig)
this.verbCacheManager = new CacheManager<Edge>(options.cacheConfig)
// v6.2.7: Write buffering always enabled - no env var check needed
// Write buffering always enabled - no env var check needed
}
/**
* Initialize the storage adapter
*
* v7.3.0: Supports progressive initialization for fast cold starts
* Supports progressive initialization for fast cold starts
*
* | Mode | Init Time | When |
* |------|-----------|------|
@ -337,14 +336,14 @@ export class GcsStorage extends BaseStorage {
}
)
// CRITICAL FIX (v3.37.7): Clear any stale cache entries from previous runs
// CRITICAL FIX: Clear any stale cache entries from previous runs
// This prevents cache poisoning from causing silent failures on container restart
prodLog.info('🧹 Clearing cache from previous run to prevent cache poisoning')
this.nounCacheManager.clear()
this.verbCacheManager.clear()
prodLog.info('✅ Cache cleared - starting fresh')
// v7.3.0: Progressive vs Strict initialization
// Progressive vs Strict initialization
if (effectiveMode === 'progressive') {
// PROGRESSIVE MODE: Fast init, background validation
// Mark as initialized immediately - ready to accept operations
@ -353,7 +352,7 @@ export class GcsStorage extends BaseStorage {
prodLog.info(`✅ GCS progressive init complete: ${this.bucketName} (validation deferred)`)
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
// Schedule background tasks (non-blocking)
@ -373,7 +372,7 @@ export class GcsStorage extends BaseStorage {
await this.initializeCounts()
this.countsLoaded = true
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
// Mark background tasks as complete (nothing to do in background)
@ -386,7 +385,7 @@ export class GcsStorage extends BaseStorage {
}
// =============================================
// Progressive Initialization (v7.3.0+)
// Progressive Initialization
// =============================================
/**
@ -400,7 +399,6 @@ export class GcsStorage extends BaseStorage {
*
* @protected
* @override
* @since v7.3.0
*/
protected async runBackgroundInit(): Promise<void> {
const startTime = Date.now()
@ -423,7 +421,6 @@ export class GcsStorage extends BaseStorage {
* Stores result in bucketValidated/bucketValidationError for lazy use.
*
* @private
* @since v7.3.0
*/
private async validateBucketInBackground(): Promise<void> {
try {
@ -451,7 +448,6 @@ export class GcsStorage extends BaseStorage {
* Uses the existing initializeCounts() logic but in background.
*
* @private
* @since v7.3.0
*/
private async loadCountsInBackground(): Promise<void> {
try {
@ -475,7 +471,6 @@ export class GcsStorage extends BaseStorage {
* @throws Error if bucket does not exist or is not accessible
* @protected
* @override
* @since v7.3.0
*/
protected async ensureValidatedForWrite(): Promise<void> {
// If already validated, nothing to do
@ -562,7 +557,7 @@ export class GcsStorage extends BaseStorage {
}
/**
* Override base class to enable smart batching for cloud storage (v3.32.3+)
* Override base class to enable smart batching for cloud storage
*
* GCS is cloud storage with network latency (~50ms per write).
* Smart batching reduces writes from 1000 ops → 100 batches.
@ -596,7 +591,7 @@ export class GcsStorage extends BaseStorage {
}
}
// v6.2.7: Removed checkVolumeMode() - write buffering always enabled for cloud storage
// Removed checkVolumeMode() - write buffering always enabled for cloud storage
/**
* Flush noun buffer to GCS
@ -628,20 +623,20 @@ export class GcsStorage extends BaseStorage {
await Promise.all(writes)
}
// v5.4.0: Removed saveNoun_internal - now inherit from BaseStorage's type-first implementation
// Removed saveNoun_internal - now inherit from BaseStorage's type-first implementation
/**
* Save a node to storage
* v6.2.7: Always uses write buffer for consistent performance
* Always uses write buffer for consistent performance
*/
protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching
// Always use write buffer - cloud storage benefits from batching
if (this.nounWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency
// Populate cache BEFORE buffering for read-after-write consistency
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node)
}
@ -657,10 +652,10 @@ export class GcsStorage extends BaseStorage {
/**
* Save a node directly to GCS (bypass buffer)
*
* v7.3.0: Performs lazy bucket validation on first write in progressive mode.
* Performs lazy bucket validation on first write in progressive mode.
*/
private async saveNodeDirect(node: HNSWNode): Promise<void> {
// v7.3.0: Lazy bucket validation for progressive init
// Lazy bucket validation for progressive init
await this.ensureValidatedForWrite()
// Apply backpressure before starting operation
@ -695,14 +690,14 @@ export class GcsStorage extends BaseStorage {
resumable: false // For small objects, non-resumable is faster
})
// CRITICAL FIX (v3.37.8): Only cache nodes with non-empty vectors
// CRITICAL FIX: Only cache nodes with non-empty vectors
// This prevents cache pollution from HNSW's lazy-loading nodes (vector: [])
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node)
}
// Note: Empty vectors are intentional during HNSW lazy mode - not logged
// Count tracking happens in baseStorage.saveNounMetadata_internal (v4.1.2)
// Count tracking happens in baseStorage.saveNounMetadata_internal
// This fixes the race condition where metadata didn't exist yet
this.logger.trace(`Node ${node.id} saved successfully`)
@ -721,7 +716,7 @@ export class GcsStorage extends BaseStorage {
}
}
// v5.4.0: Removed getNoun_internal - now inherit from BaseStorage's type-first implementation
// Removed getNoun_internal - now inherit from BaseStorage's type-first implementation
/**
* Get a node from storage
@ -732,7 +727,7 @@ export class GcsStorage extends BaseStorage {
// Check cache first
const cached: HNSWNode | null = await this.nounCacheManager.get(id)
// Validate cached object before returning (v3.37.8+)
// Validate cached object before returning
if (cached !== undefined && cached !== null) {
// Validate cached object has required fields (including non-empty vector!)
if (!cached.id || !cached.vector || !Array.isArray(cached.vector) || cached.vector.length === 0) {
@ -831,18 +826,18 @@ export class GcsStorage extends BaseStorage {
}
}
// v5.4.0: Removed deleteNoun_internal - now inherit from BaseStorage's type-first implementation
// Removed deleteNoun_internal - now inherit from BaseStorage's type-first implementation
/**
* Write an object to a specific path in GCS
* Primitive operation required by base class
*
* v7.3.0: Performs lazy bucket validation on first write in progressive mode.
* Performs lazy bucket validation on first write in progressive mode.
* @protected
*/
protected async writeObjectToPath(path: string, data: any): Promise<void> {
await this.ensureInitialized()
// v7.3.0: Lazy bucket validation for progressive init
// Lazy bucket validation for progressive init
await this.ensureValidatedForWrite()
try {
@ -892,7 +887,7 @@ export class GcsStorage extends BaseStorage {
}
/**
* Batch read multiple objects from GCS (v5.12.0 - Cloud Storage Optimization)
* Batch read multiple objects from GCS
*
* **Performance**: GCS-optimized parallel downloads
* - Uses Promise.all() for concurrent requests
@ -908,7 +903,6 @@ export class GcsStorage extends BaseStorage {
* @returns Map of path → data (only successful reads included)
*
* @public - Called by baseStorage.readBatchFromAdapter()
* @since v5.12.0
*/
public async readBatch(paths: string[]): Promise<Map<string, any>> {
await this.ensureInitialized()
@ -960,12 +954,11 @@ export class GcsStorage extends BaseStorage {
}
/**
* Get GCS-specific batch configuration (v5.12.0)
* Get GCS-specific batch configuration
*
* GCS performs well with high concurrency due to HTTP/2 multiplexing
*
* @public - Overrides BaseStorage.getBatchConfig()
* @since v5.12.0
*/
public getBatchConfig(): StorageBatchConfig {
return {
@ -984,12 +977,12 @@ export class GcsStorage extends BaseStorage {
* Delete an object from a specific path in GCS
* Primitive operation required by base class
*
* v7.3.0: Performs lazy bucket validation on first delete in progressive mode.
* Performs lazy bucket validation on first delete in progressive mode.
* @protected
*/
protected async deleteObjectFromPath(path: string): Promise<void> {
await this.ensureInitialized()
// v7.3.0: Lazy bucket validation for progressive init
// Lazy bucket validation for progressive init
await this.ensureValidatedForWrite()
try {
@ -1034,20 +1027,20 @@ export class GcsStorage extends BaseStorage {
}
}
// v5.4.0: Removed saveVerb_internal - now inherit from BaseStorage's type-first implementation
// Removed saveVerb_internal - now inherit from BaseStorage's type-first implementation
/**
* Save an edge to storage
* v6.2.7: Always uses write buffer for consistent performance
* Always uses write buffer for consistent performance
*/
protected async saveEdge(edge: Edge): Promise<void> {
await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching
// Always use write buffer - cloud storage benefits from batching
if (this.verbWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding verb ${edge.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency
// Populate cache BEFORE buffering for read-after-write consistency
this.verbCacheManager.set(edge.id, edge)
await this.verbWriteBuffer.add(edge.id, edge)
@ -1061,10 +1054,10 @@ export class GcsStorage extends BaseStorage {
/**
* Save an edge directly to GCS (bypass buffer)
*
* v7.3.0: Performs lazy bucket validation on first write in progressive mode.
* Performs lazy bucket validation on first write in progressive mode.
*/
private async saveEdgeDirect(edge: Edge): Promise<void> {
// v7.3.0: Lazy bucket validation for progressive init
// Lazy bucket validation for progressive init
await this.ensureValidatedForWrite()
const requestId = await this.applyBackpressure()
@ -1073,7 +1066,7 @@ export class GcsStorage extends BaseStorage {
this.logger.trace(`Saving edge ${edge.id}`)
// Convert connections Map to serializable format
// ARCHITECTURAL FIX (v3.50.1): Include core relational fields in verb vector file
// ARCHITECTURAL FIX: Include core relational fields in verb vector file
// These fields are essential for 90% of operations - no metadata lookup needed
const serializableEdge = {
id: edge.id,
@ -1085,7 +1078,7 @@ export class GcsStorage extends BaseStorage {
])
),
// CORE RELATIONAL DATA (v3.50.1+)
// CORE RELATIONAL DATA
verb: edge.verb,
sourceId: edge.sourceId,
targetId: edge.targetId,
@ -1107,7 +1100,7 @@ export class GcsStorage extends BaseStorage {
// Update cache
this.verbCacheManager.set(edge.id, edge)
// Count tracking happens in baseStorage.saveVerbMetadata_internal (v4.1.2)
// Count tracking happens in baseStorage.saveVerbMetadata_internal
// This fixes the race condition where metadata didn't exist yet
this.logger.trace(`Edge ${edge.id} saved successfully`)
@ -1125,7 +1118,7 @@ export class GcsStorage extends BaseStorage {
}
}
// v5.4.0: Removed getVerb_internal - now inherit from BaseStorage's type-first implementation
// Removed getVerb_internal - now inherit from BaseStorage's type-first implementation
/**
* Get an edge from storage
@ -1161,7 +1154,7 @@ export class GcsStorage extends BaseStorage {
connections.set(Number(level), new Set(verbIds as string[]))
}
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field)
// Return HNSWVerb with core relational fields (NO metadata field)
const edge: Edge = {
id: data.id,
vector: data.vector,
@ -1172,7 +1165,7 @@ export class GcsStorage extends BaseStorage {
sourceId: data.sourceId,
targetId: data.targetId
// ✅ NO metadata field in v4.0.0
// ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata()
}
@ -1201,13 +1194,13 @@ export class GcsStorage extends BaseStorage {
}
}
// v5.4.0: Removed deleteVerb_internal - now inherit from BaseStorage's type-first implementation
// Removed deleteVerb_internal - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed pagination overrides - use BaseStorage's type-first implementation
// Removed pagination overrides - use BaseStorage's type-first implementation
// - getNounsWithPagination, getNodesWithPagination, getVerbsWithPagination
// - getNouns, getVerbs (public wrappers)
// v5.4.0: Removed 4 query *_internal methods - now inherit from BaseStorage's type-first implementation
// Removed 4 query *_internal methods - now inherit from BaseStorage's type-first implementation
// (getNounsByNounType_internal, getVerbsBySource_internal, getVerbsByTarget_internal, getVerbsByType_internal)
/**
@ -1280,7 +1273,7 @@ export class GcsStorage extends BaseStorage {
}
}
// v5.11.0: Clear ALL data using correct paths
// Clear ALL data using correct paths
// Delete entire branches/ directory (includes ALL entities, ALL types, ALL VFS data, ALL forks)
await deleteObjectsWithPrefix('branches/')
@ -1290,7 +1283,7 @@ export class GcsStorage extends BaseStorage {
// Delete system metadata
await deleteObjectsWithPrefix('_system/')
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled)
// Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use
this.refManager = undefined
this.blobStorage = undefined
@ -1352,12 +1345,12 @@ export class GcsStorage extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts
* Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker object exists, false otherwise
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
@ -1413,7 +1406,7 @@ export class GcsStorage extends BaseStorage {
}
} catch (error: any) {
if (error.code === 404) {
// CRITICAL FIX (v3.37.4): Statistics file doesn't exist yet (first restart)
// CRITICAL FIX: Statistics file doesn't exist yet (first restart)
// Return minimal stats with counts instead of null
// This prevents HNSW from seeing entityCount=0 during index rebuild
this.logger.trace('Statistics file not found - returning minimal stats with counts')
@ -1607,11 +1600,11 @@ export class GcsStorage extends BaseStorage {
}
}
// HNSW Index Persistence (v3.35.0+)
// HNSW Index Persistence
/**
* Get a noun's vector for HNSW rebuild
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id)
@ -1621,7 +1614,7 @@ export class GcsStorage extends BaseStorage {
/**
* Save HNSW graph data for a noun
*
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths)
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Uses mutex locking to prevent read-modify-write races
*/
public async saveHNSWData(nounId: string, hnswData: {
@ -1630,7 +1623,7 @@ export class GcsStorage extends BaseStorage {
}): Promise<void> {
const lockKey = `hnsw/${nounId}`
// CRITICAL FIX (v4.10.1): Mutex lock to prevent read-modify-write races
// CRITICAL FIX: Mutex lock to prevent read-modify-write races
// Problem: Without mutex, concurrent operations can:
// 1. Thread A reads noun (connections: [1,2,3])
// 2. Thread B reads noun (connections: [1,2,3])
@ -1650,7 +1643,7 @@ export class GcsStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise)
try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths)
// Use BaseStorage's getNoun (type-first paths)
// Read existing noun data (if exists)
const existingNoun = await this.getNoun(nounId)
@ -1672,7 +1665,7 @@ export class GcsStorage extends BaseStorage {
connections: connectionsMap
}
// v5.4.0: Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
// Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
await this.saveNoun(updatedNoun)
} finally {
// Release lock (ALWAYS runs, even if error thrown)
@ -1683,7 +1676,7 @@ export class GcsStorage extends BaseStorage {
/**
* Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getHNSWData(nounId: string): Promise<{
level: number
@ -1713,7 +1706,7 @@ export class GcsStorage extends BaseStorage {
* Save HNSW system data (entry point, max level)
* Storage path: system/hnsw-system.json
*
* CRITICAL FIX (v4.10.1): Optimistic locking with generation numbers to prevent race conditions
* CRITICAL FIX: Optimistic locking with generation numbers to prevent race conditions
*/
public async saveHNSWSystem(systemData: {
entryPointId: string | null
@ -1807,7 +1800,7 @@ export class GcsStorage extends BaseStorage {
}
// ============================================================================
// GCS Lifecycle Management & Autoclass (v4.0.0)
// GCS Lifecycle Management & Autoclass
// Cost optimization through automatic tier transitions and Autoclass
// ============================================================================

10
src/storage/adapters/historicalStorageAdapter.ts
View file

@ -25,7 +25,7 @@
* - Lazy loading: only loads entities when accessed
* - No eager-loading of entire commit state
*
* v5.4.0: Production-ready, billion-scale historical queries
* Production-ready, billion-scale historical queries
*/
import { BaseStorage } from '../baseStorage.js'
@ -144,7 +144,7 @@ export class HistoricalStorageAdapter extends BaseStorage {
*/
public async init(): Promise<void> {
// Get COW components from underlying storage
// v6.2.4: Fixed property names - use public properties without underscore prefix
// Fixed property names - use public properties without underscore prefix
this.commitLog = this.underlyingStorage.commitLog
this.blobStorage = this.underlyingStorage.blobStorage
@ -161,7 +161,7 @@ export class HistoricalStorageAdapter extends BaseStorage {
throw new Error(`Commit not found: ${this.commitId}`)
}
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
}
@ -310,12 +310,12 @@ export class HistoricalStorageAdapter extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: No-op for HistoricalStorageAdapter (read-only, doesn't manage COW)
* No-op for HistoricalStorageAdapter (read-only, doesn't manage COW)
* @returns Always false (read-only adapter doesn't manage COW state)
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/

49
src/storage/adapters/memoryStorage.ts
View file

@ -24,7 +24,7 @@ import { PaginatedResult } from '../../types/paginationTypes.js'
* Uses Maps to store data in memory
*/
export class MemoryStorage extends BaseStorage {
// v5.4.0: Removed redundant Maps (nouns, verbs) - objectStore handles all storage via type-first paths
// Removed redundant Maps (nouns, verbs) - objectStore handles all storage via type-first paths
private statistics: StatisticsData | null = null
// Unified object store for primitive operations (replaces metadata, nounMetadata, verbMetadata)
@ -55,7 +55,6 @@ export class MemoryStorage extends BaseStorage {
* - Parallel processing maximizes throughput
*
* @returns Memory-optimized batch configuration
* @since v4.11.0
*/
public getBatchConfig(): StorageBatchConfig {
return {
@ -72,27 +71,27 @@ export class MemoryStorage extends BaseStorage {
/**
* Initialize the storage adapter
* v6.0.0: Calls super.init() to initialize GraphAdjacencyIndex and type statistics
* Calls super.init() to initialize GraphAdjacencyIndex and type statistics
*/
public async init(): Promise<void> {
await super.init()
}
// v5.4.0: Removed saveNoun_internal and getNoun_internal - using BaseStorage's type-first implementation
// Removed saveNoun_internal and getNoun_internal - using BaseStorage's type-first implementation
/**
* Get nouns with pagination and filtering
* v4.0.0: Returns HNSWNounWithMetadata[] (includes metadata field)
* Returns HNSWNounWithMetadata[] (includes metadata field)
* @param options Pagination and filtering options
* @returns Promise that resolves to a paginated result of nouns with metadata
*/
// v5.4.0: Removed public method overrides (getNouns, getNounsWithPagination, getVerbs) - using BaseStorage's type-first implementation
// Removed public method overrides (getNouns, getNounsWithPagination, getVerbs) - using BaseStorage's type-first implementation
// v5.4.0: Removed getNounsByNounType_internal and deleteNoun_internal - using BaseStorage's type-first implementation
// Removed getNounsByNounType_internal and deleteNoun_internal - using BaseStorage's type-first implementation
// v5.4.0: Removed saveVerb_internal and getVerb_internal - using BaseStorage's type-first implementation
// Removed saveVerb_internal and getVerb_internal - using BaseStorage's type-first implementation
// v5.4.0: Removed verb *_internal method overrides - using BaseStorage's type-first implementation
// Removed verb *_internal method overrides - using BaseStorage's type-first implementation
/**
* Primitive operation: Write object to path
@ -159,8 +158,8 @@ export class MemoryStorage extends BaseStorage {
/**
* Clear all data from storage
* v5.4.0: Clears objectStore (type-first paths)
* v7.3.1: Also clears writeCache to prevent stale data after clear
* Clears objectStore (type-first paths)
* Also clears writeCache to prevent stale data after clear
*/
public async clear(): Promise<void> {
this.objectStore.clear()
@ -174,7 +173,7 @@ export class MemoryStorage extends BaseStorage {
this.statisticsCache = null
this.statisticsModified = false
// v7.3.1: Clear write-through cache (inherited from BaseStorage)
// Clear write-through cache (inherited from BaseStorage)
// Without this, readWithInheritance() would return stale cached data
// after clear(), causing "ghost" entities to appear
this.clearWriteCache()
@ -182,7 +181,7 @@ export class MemoryStorage extends BaseStorage {
/**
* Get information about storage usage and capacity
* v5.4.0: Uses BaseStorage counts
* Uses BaseStorage counts
*/
public async getStorageStatus(): Promise<{
type: string
@ -204,12 +203,12 @@ export class MemoryStorage extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: No-op for MemoryStorage (doesn't persist)
* No-op for MemoryStorage (doesn't persist)
* @returns Always false (marker doesn't persist in memory)
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
@ -252,7 +251,7 @@ export class MemoryStorage extends BaseStorage {
*/
protected async getStatisticsData(): Promise<StatisticsData | null> {
if (!this.statistics) {
// CRITICAL FIX (v3.37.4): Statistics don't exist yet (first init)
// CRITICAL FIX: Statistics don't exist yet (first init)
// Return minimal stats with counts instead of null
// This prevents HNSW from seeing entityCount=0 during index rebuild
return {
@ -299,10 +298,10 @@ export class MemoryStorage extends BaseStorage {
}
/**
* Initialize counts from in-memory storage - O(1) operation (v4.0.0)
* Initialize counts from in-memory storage - O(1) operation
*/
protected async initializeCounts(): Promise<void> {
// v6.0.0: Scan objectStore paths (ID-first structure) to count entities
// Scan objectStore paths (ID-first structure) to count entities
this.entityCounts.clear()
this.verbCounts.clear()
@ -314,14 +313,14 @@ export class MemoryStorage extends BaseStorage {
// Count nouns (entities/nouns/{shard}/{id}/vectors.json)
const nounMatch = path.match(/^entities\/nouns\/[0-9a-f]{2}\/[^/]+\/vectors\.json$/)
if (nounMatch) {
// v6.0.0: Type is in metadata, not path - just count total
// Type is in metadata, not path - just count total
totalNouns++
}
// Count verbs (entities/verbs/{shard}/{id}/vectors.json)
const verbMatch = path.match(/^entities\/verbs\/[0-9a-f]{2}\/[^/]+\/vectors\.json$/)
if (verbMatch) {
// v6.0.0: Type is in metadata, not path - just count total
// Type is in metadata, not path - just count total
totalVerbs++
}
}
@ -339,26 +338,26 @@ export class MemoryStorage extends BaseStorage {
}
// =============================================
// HNSW Index Persistence (v3.35.0+)
// HNSW Index Persistence
// =============================================
/**
* Get vector for a noun
* v5.4.0: Uses BaseStorage's type-first implementation
* Uses BaseStorage's type-first implementation
*/
public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id)
return noun ? [...noun.vector] : null
}
// CRITICAL FIX (v4.10.1): Mutex locks for HNSW concurrency control
// CRITICAL FIX: Mutex locks for HNSW concurrency control
// Even in-memory operations need serialization to prevent async race conditions
private hnswLocks = new Map<string, Promise<void>>()
/**
* Save HNSW graph data for a noun
*
* CRITICAL FIX (v4.10.1): Mutex locking to prevent race conditions during concurrent HNSW updates
* CRITICAL FIX: Mutex locking to prevent race conditions during concurrent HNSW updates
* Even in-memory operations can race due to async/await interleaving
* Prevents data corruption when multiple entities connect to same neighbor simultaneously
*/
@ -417,7 +416,7 @@ export class MemoryStorage extends BaseStorage {
/**
* Save HNSW system data (entry point, max level)
*
* CRITICAL FIX (v4.10.1): Mutex locking to prevent race conditions
* CRITICAL FIX: Mutex locking to prevent race conditions
*/
public async saveHNSWSystem(systemData: {
entryPointId: string | null

43
src/storage/adapters/opfsStorage.ts
View file

@ -57,7 +57,7 @@ const ROOT_DIR = 'opfs-vector-db'
* OPFS storage adapter for browser environments
* Uses the Origin Private File System API to store data persistently
*
* v5.4.0: Type-aware storage now built into BaseStorage
* Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination)
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -97,7 +97,6 @@ export class OPFSStorage extends BaseStorage {
* - Sequential processing preferred for stability
*
* @returns OPFS-optimized batch configuration
* @since v4.11.0
*/
public getBatchConfig(): StorageBatchConfig {
return {
@ -172,7 +171,7 @@ export class OPFSStorage extends BaseStorage {
// Initialize counts from storage
await this.initializeCounts()
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
} catch (error) {
console.error('Failed to initialize OPFS storage:', error)
@ -232,7 +231,7 @@ export class OPFSStorage extends BaseStorage {
}
}
// v5.4.0: Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
// Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
/**
* Delete an edge from storage
@ -482,14 +481,14 @@ export class OPFSStorage extends BaseStorage {
// Remove all files in the index directory
await removeDirectoryContents(this.indexDir!)
// v5.6.1: Remove COW (copy-on-write) version control data
// Remove COW (copy-on-write) version control data
// This directory stores all git-like versioning data (commits, trees, blobs, refs)
// Must be deleted to fully clear all data including version history
try {
// Delete the entire _cow/ directory (not just contents)
await this.rootDir!.removeEntry('_cow', { recursive: true })
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled)
// Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use
this.refManager = undefined
this.blobStorage = undefined
@ -505,7 +504,7 @@ export class OPFSStorage extends BaseStorage {
this.statisticsCache = null
this.statisticsModified = false
// v5.6.1: Reset entity counters (inherited from BaseStorageAdapter)
// Reset entity counters (inherited from BaseStorageAdapter)
// These in-memory counters must be reset to 0 after clearing all data
;(this as any).totalNounCount = 0
;(this as any).totalVerbCount = 0
@ -517,16 +516,16 @@ export class OPFSStorage extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts
* Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker file exists, false otherwise
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
// Quota monitoring configuration (v4.0.0)
// Quota monitoring configuration
private quotaWarningThreshold = 0.8 // Warn at 80% usage
private quotaCriticalThreshold = 0.95 // Critical at 95% usage
private lastQuotaCheck: number = 0
@ -675,7 +674,7 @@ export class OPFSStorage extends BaseStorage {
}
/**
* Get detailed quota status with warnings (v4.0.0)
* Get detailed quota status with warnings
* Monitors storage usage and warns when approaching quota limits
*
* @returns Promise that resolves to quota status with warning levels
@ -769,7 +768,7 @@ export class OPFSStorage extends BaseStorage {
}
/**
* Monitor quota during operations (v4.0.0)
* Monitor quota during operations
* Automatically checks quota at regular intervals and warns if approaching limits
* Call this before write operations to ensure quota is available
*
@ -1185,7 +1184,7 @@ export class OPFSStorage extends BaseStorage {
}
}
} catch (error) {
// CRITICAL FIX (v3.37.4): No statistics files exist (first init)
// CRITICAL FIX: No statistics files exist (first init)
// Return minimal stats with counts instead of null
// This prevents HNSW from seeing entityCount=0 during index rebuild
return {
@ -1228,7 +1227,7 @@ export class OPFSStorage extends BaseStorage {
* @param options Pagination and filter options
* @returns Promise that resolves to a paginated result of nouns
*/
// v5.4.0: Removed pagination overrides (getNounsWithPagination, getVerbsWithPagination) - use BaseStorage's type-first implementation
// Removed pagination overrides (getNounsWithPagination, getVerbsWithPagination) - use BaseStorage's type-first implementation
/**
* Initialize counts from OPFS storage
@ -1313,28 +1312,28 @@ export class OPFSStorage extends BaseStorage {
}
}
// HNSW Index Persistence (v3.35.0+)
// HNSW Index Persistence
/**
* Get a noun's vector for HNSW rebuild
*/
/**
* Get vector for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id)
return noun ? noun.vector : null
}
// CRITICAL FIX (v4.10.1): Mutex locks for HNSW concurrency control
// CRITICAL FIX: Mutex locks for HNSW concurrency control
// Browser environments are single-threaded but async operations can still interleave
private hnswLocks = new Map<string, Promise<void>>()
/**
* Save HNSW graph data for a noun
*
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths)
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Preserves mutex locking to prevent read-modify-write races
*/
public async saveHNSWData(nounId: string, hnswData: {
@ -1354,7 +1353,7 @@ export class OPFSStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise)
try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths)
// Use BaseStorage's getNoun (type-first paths)
const existingNoun = await this.getNoun(nounId)
if (!existingNoun) {
@ -1374,7 +1373,7 @@ export class OPFSStorage extends BaseStorage {
connections: connectionsMap
}
// v5.4.0: Use BaseStorage's saveNoun (type-first paths)
// Use BaseStorage's saveNoun (type-first paths)
await this.saveNoun(updatedNoun)
} finally {
// Release lock
@ -1385,7 +1384,7 @@ export class OPFSStorage extends BaseStorage {
/**
* Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getHNSWData(nounId: string): Promise<{
level: number
@ -1415,7 +1414,7 @@ export class OPFSStorage extends BaseStorage {
* Save HNSW system data (entry point, max level)
* Storage path: index/hnsw-system.json
*
* CRITICAL FIX (v4.10.1): Mutex locking to prevent race conditions
* CRITICAL FIX: Mutex locking to prevent race conditions
*/
public async saveHNSWSystem(systemData: {
entryPointId: string | null

4
src/storage/adapters/optimizedS3Search.ts
View file

@ -107,7 +107,7 @@ export class OptimizedS3Search {
}
// Determine if there are more items
const hasMore = listResult.hasMore || nouns.length > limit // v5.7.11: Fixed >= to > (was causing infinite loop)
const hasMore = listResult.hasMore || nouns.length > limit // Fixed >= to > (was causing infinite loop)
// Set next cursor
let nextCursor: string | undefined
@ -190,7 +190,7 @@ export class OptimizedS3Search {
}
// Determine if there are more items
const hasMore = listResult.hasMore || verbs.length > limit // v5.7.11: Fixed >= to > (was causing infinite loop)
const hasMore = listResult.hasMore || verbs.length > limit // Fixed >= to > (was causing infinite loop)
// Set next cursor
let nextCursor: string | undefined

49
src/storage/adapters/r2Storage.ts
View file

@ -58,7 +58,7 @@ const MAX_R2_PAGE_SIZE = 1000
* Dedicated Cloudflare R2 storage adapter
* Optimized for R2's unique characteristics and global edge network
*
* v5.4.0: Type-aware storage now built into BaseStorage
* Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed getNounsWithPagination override
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -112,7 +112,7 @@ export class R2Storage extends BaseStorage {
// Request coalescer for deduplication
private requestCoalescer: RequestCoalescer | null = null
// v6.2.7: Write buffering always enabled for consistent performance
// Write buffering always enabled for consistent performance
// Removes dynamic mode switching complexity - cloud storage always benefits from batching
// Multi-level cache manager for efficient data access
@ -122,7 +122,7 @@ export class R2Storage extends BaseStorage {
// Module logger
private logger = createModuleLogger('R2Storage')
// v5.4.0: HNSW mutex locks to prevent read-modify-write races
// HNSW mutex locks to prevent read-modify-write races
private hnswLocks = new Map<string, Promise<void>>()
/**
@ -168,7 +168,7 @@ export class R2Storage extends BaseStorage {
})
this.verbCacheManager = new CacheManager<Edge>(options.cacheConfig)
// v6.2.7: Write buffering always enabled - no env var check needed
// Write buffering always enabled - no env var check needed
}
/**
@ -183,7 +183,7 @@ export class R2Storage extends BaseStorage {
* Zero egress fees enable aggressive caching and parallel downloads
*
* @returns R2-optimized batch configuration
* @since v5.12.0 - Updated for native batch API
* Updated for native batch API
*/
public getBatchConfig(): StorageBatchConfig {
return {
@ -208,7 +208,6 @@ export class R2Storage extends BaseStorage {
*
* @param paths - Array of R2 object keys to read
* @returns Map of path -> parsed JSON data (only successful reads)
* @since v5.12.0
*/
public async readBatch(paths: string[]): Promise<Map<string, any>> {
await this.ensureInitialized()
@ -336,7 +335,7 @@ export class R2Storage extends BaseStorage {
this.nounCacheManager.clear()
this.verbCacheManager.clear()
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
} catch (error) {
this.logger.error('Failed to initialize R2 storage:', error)
@ -409,7 +408,7 @@ export class R2Storage extends BaseStorage {
}
}
// v6.2.7: Removed checkVolumeMode() - write buffering always enabled for cloud storage
// Removed checkVolumeMode() - write buffering always enabled for cloud storage
/**
* Flush noun buffer to R2
@ -444,16 +443,16 @@ export class R2Storage extends BaseStorage {
/**
* Save a node to storage
* v6.2.7: Always uses write buffer for consistent performance
* Always uses write buffer for consistent performance
*/
protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching
// Always use write buffer - cloud storage benefits from batching
if (this.nounWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency
// Populate cache BEFORE buffering for read-after-write consistency
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node)
}
@ -728,14 +727,14 @@ export class R2Storage extends BaseStorage {
/**
* Save an edge to storage
* v6.2.7: Always uses write buffer for consistent performance
* Always uses write buffer for consistent performance
*/
protected async saveEdge(edge: Edge): Promise<void> {
await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching
// Always use write buffer - cloud storage benefits from batching
if (this.verbWriteBuffer) {
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency
// Populate cache BEFORE buffering for read-after-write consistency
this.verbCacheManager.set(edge.id, edge)
await this.verbWriteBuffer.add(edge.id, edge)
@ -750,7 +749,7 @@ export class R2Storage extends BaseStorage {
const requestId = await this.applyBackpressure()
try {
// ARCHITECTURAL FIX (v3.50.1): Include core relational fields in verb vector file
// ARCHITECTURAL FIX: Include core relational fields in verb vector file
// These fields are essential for 90% of operations - no metadata lookup needed
const serializableEdge = {
id: edge.id,
@ -762,7 +761,7 @@ export class R2Storage extends BaseStorage {
])
),
// CORE RELATIONAL DATA (v3.50.1+)
// CORE RELATIONAL DATA
verb: edge.verb,
sourceId: edge.sourceId,
targetId: edge.targetId,
@ -785,7 +784,7 @@ export class R2Storage extends BaseStorage {
this.verbCacheManager.set(edge.id, edge)
// Count tracking happens in baseStorage.saveVerbMetadata_internal (v4.1.2)
// Count tracking happens in baseStorage.saveVerbMetadata_internal
// This fixes the race condition where metadata didn't exist yet
this.releaseBackpressure(true, requestId)
@ -831,7 +830,7 @@ export class R2Storage extends BaseStorage {
connections.set(Number(level), new Set(verbIds as string[]))
}
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field)
// Return HNSWVerb with core relational fields (NO metadata field)
const edge: Edge = {
id: data.id,
vector: data.vector,
@ -842,7 +841,7 @@ export class R2Storage extends BaseStorage {
sourceId: data.sourceId,
targetId: data.targetId
// ✅ NO metadata field in v4.0.0
// ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata()
}
@ -1081,7 +1080,7 @@ export class R2Storage extends BaseStorage {
prodLog.info('🧹 R2: Clearing all data from bucket...')
// v5.11.0: Clear ALL data using correct paths
// Clear ALL data using correct paths
// Delete entire branches/ directory (includes ALL entities, ALL types, ALL VFS data, ALL forks)
const branchObjects = await this.listObjectsUnderPath('branches/')
for (const key of branchObjects) {
@ -1100,7 +1099,7 @@ export class R2Storage extends BaseStorage {
await this.deleteObjectFromPath(key)
}
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled)
// Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use
this.refManager = undefined
this.blobStorage = undefined
@ -1143,17 +1142,17 @@ export class R2Storage extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts
* Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker object exists, false otherwise
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
// v5.4.0: Removed getNounsWithPagination override - use BaseStorage's type-first implementation
// Removed getNounsWithPagination override - use BaseStorage's type-first implementation
// v5.4.0: Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
// Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
}

98
src/storage/adapters/s3CompatibleStorage.ts
View file

@ -85,7 +85,7 @@ type S3Command = any
* - endpoint: GCS endpoint (e.g., 'https://storage.googleapis.com')
* - bucketName: GCS bucket name
*
* v5.4.0: Type-aware storage now built into BaseStorage
* Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination)
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -152,7 +152,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Request coalescer for deduplication
private requestCoalescer: RequestCoalescer | null = null
// v6.2.7: Write buffering always enabled for consistent performance
// Write buffering always enabled for consistent performance
// Removes dynamic mode switching complexity - cloud storage always benefits from batching
// Operation executors for timeout and retry handling
@ -165,7 +165,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Module logger
private logger = createModuleLogger('S3Storage')
// v5.4.0: HNSW mutex locks to prevent read-modify-write races
// HNSW mutex locks to prevent read-modify-write races
private hnswLocks = new Map<string, Promise<void>>()
/**
@ -210,7 +210,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
/**
* Initialization mode for fast cold starts (v7.3.0+)
* Initialization mode for fast cold starts
*
* - `'auto'` (default): Progressive in cloud environments (Lambda, Cloud Run),
* strict locally. Zero-config optimization.
@ -219,7 +219,6 @@ export class S3CompatibleStorage extends BaseStorage {
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts
* before init() returns.
*
* @since v7.3.0
*/
initMode?: InitMode
@ -236,7 +235,7 @@ export class S3CompatibleStorage extends BaseStorage {
this.serviceType = options.serviceType || 's3'
this.readOnly = options.readOnly || false
// v7.3.0: Handle initMode
// Handle initMode
if (options.initMode) {
this.initMode = options.initMode
}
@ -270,7 +269,7 @@ export class S3CompatibleStorage extends BaseStorage {
* S3 supports ~5000 operations/second with burst capacity up to 10,000
*
* @returns S3-optimized batch configuration
* @since v5.12.0 - Updated for native batch API
* Updated for native batch API
*/
public getBatchConfig(): StorageBatchConfig {
return {
@ -295,7 +294,6 @@ export class S3CompatibleStorage extends BaseStorage {
*
* @param paths - Array of S3 object keys to read
* @returns Map of path -> parsed JSON data (only successful reads)
* @since v5.12.0
*/
public async readBatch(paths: string[]): Promise<Map<string, any>> {
await this.ensureInitialized()
@ -358,7 +356,7 @@ export class S3CompatibleStorage extends BaseStorage {
/**
* Initialize the storage adapter
*
* v7.3.0: Supports progressive initialization for fast cold starts
* Supports progressive initialization for fast cold starts
*
* | Mode | Init Time | When |
* |------|-----------|------|
@ -514,7 +512,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Initialize request coalescer
this.initializeCoalescer()
// CRITICAL FIX (v3.37.7): Clear any stale cache entries from previous runs
// CRITICAL FIX: Clear any stale cache entries from previous runs
// This prevents cache poisoning from causing silent failures on container restart
const nodeCacheSize = this.nodeCache?.size || 0
if (nodeCacheSize > 0) {
@ -524,7 +522,7 @@ export class S3CompatibleStorage extends BaseStorage {
prodLog.info('🧹 Node cache is empty - starting fresh')
}
// v7.3.0: Progressive vs Strict initialization
// Progressive vs Strict initialization
if (effectiveMode === 'progressive') {
// PROGRESSIVE MODE: Fast init, background validation
// Mark as initialized immediately - ready to accept operations
@ -533,7 +531,7 @@ export class S3CompatibleStorage extends BaseStorage {
prodLog.info(`✅ S3 progressive init complete: ${this.bucketName} (validation deferred)`)
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
// Schedule background tasks (non-blocking)
@ -556,7 +554,7 @@ export class S3CompatibleStorage extends BaseStorage {
await this.initializeCounts()
this.countsLoaded = true
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
// Mark background tasks as complete (nothing to do in background)
@ -573,7 +571,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
// =============================================
// Progressive Initialization (v7.3.0+)
// Progressive Initialization
// =============================================
/**
@ -588,7 +586,6 @@ export class S3CompatibleStorage extends BaseStorage {
*
* @protected
* @override
* @since v7.3.0
*/
protected async runBackgroundInit(): Promise<void> {
const startTime = Date.now()
@ -614,7 +611,6 @@ export class S3CompatibleStorage extends BaseStorage {
* Stores result in bucketValidated/bucketValidationError for lazy use.
*
* @private
* @since v7.3.0
*/
private async validateBucketInBackground(): Promise<void> {
try {
@ -638,7 +634,6 @@ export class S3CompatibleStorage extends BaseStorage {
* Load counts from storage in background.
*
* @private
* @since v7.3.0
*/
private async loadCountsInBackground(): Promise<void> {
try {
@ -662,7 +657,6 @@ export class S3CompatibleStorage extends BaseStorage {
* @throws Error if bucket does not exist or is not accessible
* @protected
* @override
* @since v7.3.0
*/
protected async ensureValidatedForWrite(): Promise<void> {
// If already validated, nothing to do
@ -918,7 +912,7 @@ export class S3CompatibleStorage extends BaseStorage {
)
}
// v6.2.7: Removed checkVolumeMode() - write buffering always enabled for cloud storage
// Removed checkVolumeMode() - write buffering always enabled for cloud storage
/**
* Bulk write nouns to S3
@ -1179,20 +1173,20 @@ export class S3CompatibleStorage extends BaseStorage {
return this.socketManager.getBatchSize()
}
// v5.4.0: Removed 10 *_internal method overrides (lines 984-2069) - now inherit from BaseStorage's type-first implementation
// Removed 10 *_internal method overrides (lines 984-2069) - now inherit from BaseStorage's type-first implementation
/**
* Save a node to storage
* v6.2.7: Always uses write buffer for consistent performance
* Always uses write buffer for consistent performance
*/
protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching
// Always use write buffer - cloud storage benefits from batching
if (this.nounWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency
// Populate cache BEFORE buffering for read-after-write consistency
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node)
}
@ -1242,7 +1236,7 @@ export class S3CompatibleStorage extends BaseStorage {
this.logger.debug(`Node ${node.id} saved successfully`)
// Log the change for efficient synchronization (v4.0.0: no metadata on node)
// Log the change for efficient synchronization (no metadata on node)
await this.appendToChangeLog({
timestamp: Date.now(),
operation: 'add', // Could be 'update' if we track existing nodes
@ -1250,7 +1244,7 @@ export class S3CompatibleStorage extends BaseStorage {
entityId: node.id,
data: {
vector: node.vector
// ✅ NO metadata field in v4.0.0 - stored separately
// ✅ NO metadata field - stored separately
}
})
@ -1296,7 +1290,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
}
// v5.4.0: Removed getNoun_internal override - uses BaseStorage type-first implementation
// Removed getNoun_internal override - uses BaseStorage type-first implementation
/**
* Get a node from storage
@ -1307,7 +1301,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Check cache first
const cached = this.nodeCache.get(id)
// Validate cached object before returning (v3.37.8+)
// Validate cached object before returning
if (cached !== undefined && cached !== null) {
// Validate cached object has required fields (including non-empty vector!)
if (!cached.id || !cached.vector || !Array.isArray(cached.vector) || cached.vector.length === 0) {
@ -1607,7 +1601,7 @@ export class S3CompatibleStorage extends BaseStorage {
return nodes
}
// v5.4.0: Removed 4 *_internal method overrides (getNounsByNounType_internal, deleteNoun_internal, saveVerb_internal, getVerb_internal)
// Removed 4 *_internal method overrides (getNounsByNounType_internal, deleteNoun_internal, saveVerb_internal, getVerb_internal)
// Now inherit from BaseStorage's type-first implementation
@ -1788,23 +1782,23 @@ export class S3CompatibleStorage extends BaseStorage {
return true // Return all edges since filtering requires metadata
}
// v5.4.0: Removed getVerbsWithPagination override - use BaseStorage's type-first implementation
// Removed getVerbsWithPagination override - use BaseStorage's type-first implementation
// v5.4.0: Removed 4 more *_internal method overrides (getVerbsBySource, getVerbsByTarget, getVerbsByType, deleteVerb)
// Removed 4 more *_internal method overrides (getVerbsBySource, getVerbsByTarget, getVerbsByType, deleteVerb)
// Total: 8 *_internal methods removed - all now inherit from BaseStorage's type-first implementation
/**
* Primitive operation: Write object to path
* All metadata operations use this internally via base class routing
*
* v7.3.0: Performs lazy bucket validation on first write in progressive mode.
* Performs lazy bucket validation on first write in progressive mode.
*/
protected async writeObjectToPath(path: string, data: any): Promise<void> {
await this.ensureInitialized()
// v7.3.0: Lazy bucket validation for progressive init
// Lazy bucket validation for progressive init
await this.ensureValidatedForWrite()
// Apply backpressure before starting operation
@ -1897,11 +1891,11 @@ export class S3CompatibleStorage extends BaseStorage {
* Primitive operation: Delete object from path
* All metadata operations use this internally via base class routing
*
* v7.3.0: Performs lazy bucket validation on first delete in progressive mode.
* Performs lazy bucket validation on first delete in progressive mode.
*/
protected async deleteObjectFromPath(path: string): Promise<void> {
await this.ensureInitialized()
// v7.3.0: Lazy bucket validation for progressive init
// Lazy bucket validation for progressive init
await this.ensureValidatedForWrite()
try {
@ -2315,7 +2309,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
}
// v5.11.0: Clear ALL data using correct paths
// Clear ALL data using correct paths
// Delete entire branches/ directory (includes ALL entities, ALL types, ALL VFS data, ALL forks)
await deleteObjectsWithPrefix('branches/')
@ -2325,7 +2319,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Delete system metadata
await deleteObjectsWithPrefix('_system/')
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled)
// Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use
this.refManager = undefined
this.blobStorage = undefined
@ -2335,7 +2329,7 @@ export class S3CompatibleStorage extends BaseStorage {
this.statisticsCache = null
this.statisticsModified = false
// v5.6.1: Reset entity counters (inherited from BaseStorageAdapter)
// Reset entity counters (inherited from BaseStorageAdapter)
// These in-memory counters must be reset to 0 after clearing all data
;(this as any).totalNounCount = 0
;(this as any).totalVerbCount = 0
@ -2457,12 +2451,12 @@ export class S3CompatibleStorage extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts
* Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker object exists, false otherwise
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
@ -2851,7 +2845,7 @@ export class S3CompatibleStorage extends BaseStorage {
totalEdges: this.totalVerbCount
}
}
// CRITICAL FIX (v3.37.4): Statistics file doesn't exist yet (first restart)
// CRITICAL FIX: Statistics file doesn't exist yet (first restart)
// Return minimal stats with counts instead of null
// This prevents HNSW from seeing entityCount=0 during index rebuild
return {
@ -3415,7 +3409,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
}
// v5.4.0: Removed getNounsWithPagination override - use BaseStorage's type-first implementation
// Removed getNounsWithPagination override - use BaseStorage's type-first implementation
/**
* Estimate total noun count by listing objects across all shards
@ -3549,7 +3543,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
/**
* Override base class to enable smart batching for cloud storage (v3.32.3+)
* Override base class to enable smart batching for cloud storage
*
* S3 is cloud storage with network latency (~50ms per write).
* Smart batching reduces writes from 1000 ops → 100 batches.
@ -3560,11 +3554,11 @@ export class S3CompatibleStorage extends BaseStorage {
return true // S3 benefits from batching
}
// HNSW Index Persistence (v3.35.0+)
// HNSW Index Persistence
/**
* Get a noun's vector for HNSW rebuild
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id)
@ -3574,7 +3568,7 @@ export class S3CompatibleStorage extends BaseStorage {
/**
* Save HNSW graph data for a noun
*
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths)
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Uses mutex locking to prevent read-modify-write races
*/
public async saveHNSWData(nounId: string, hnswData: {
@ -3583,7 +3577,7 @@ export class S3CompatibleStorage extends BaseStorage {
}): Promise<void> {
const lockKey = `hnsw/${nounId}`
// CRITICAL FIX (v4.10.1): Mutex lock to prevent read-modify-write races
// CRITICAL FIX: Mutex lock to prevent read-modify-write races
// Problem: Without mutex, concurrent operations can:
// 1. Thread A reads noun (connections: [1,2,3])
// 2. Thread B reads noun (connections: [1,2,3])
@ -3603,7 +3597,7 @@ export class S3CompatibleStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise)
try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths)
// Use BaseStorage's getNoun (type-first paths)
// Read existing noun data (if exists)
const existingNoun = await this.getNoun(nounId)
@ -3625,7 +3619,7 @@ export class S3CompatibleStorage extends BaseStorage {
connections: connectionsMap
}
// v5.4.0: Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
// Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
await this.saveNoun(updatedNoun)
} finally {
// Release lock (ALWAYS runs, even if error thrown)
@ -3636,7 +3630,7 @@ export class S3CompatibleStorage extends BaseStorage {
/**
* Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getHNSWData(nounId: string): Promise<{
level: number
@ -3666,7 +3660,7 @@ export class S3CompatibleStorage extends BaseStorage {
* Save HNSW system data (entry point, max level)
* Storage path: system/hnsw-system.json
*
* CRITICAL FIX (v4.10.1): Optimistic locking with ETags to prevent race conditions
* CRITICAL FIX: Optimistic locking with ETags to prevent race conditions
*/
public async saveHNSWSystem(systemData: {
entryPointId: string | null
@ -3781,7 +3775,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
/**
* Set S3 lifecycle policy for automatic tier transitions and deletions (v4.0.0)
* Set S3 lifecycle policy for automatic tier transitions and deletions
* Automates cost optimization by moving old data to cheaper storage classes
*
* S3 Storage Classes:
@ -3976,7 +3970,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
/**
* Enable S3 Intelligent-Tiering for automatic cost optimization (v4.0.0)
* Enable S3 Intelligent-Tiering for automatic cost optimization
* Automatically moves objects between access tiers based on usage patterns
*
* Intelligent-Tiering automatically saves up to 95% on storage costs:

4
src/storage/backwardCompatibility.ts
View file

@ -1,6 +1,6 @@
/**
* DEPRECATED (v4.7.2): Backward compatibility stubs
* TODO: Remove in v4.7.3 after migrating s3CompatibleStorage
* DEPRECATED: Backward compatibility stubs
* TODO: Remove after migrating s3CompatibleStorage
*/
export class StorageCompatibilityLayer {

339
src/storage/baseStorage.ts
View file

File diff suppressed because it is too large Load diff

14
src/storage/cow/BlobStorage.ts
View file

@ -161,7 +161,7 @@ export class BlobStorage {
}
/**
* v5.7.5: Ensure compression is ready before write operations
* Ensure compression is ready before write operations
* Fixes race condition where write happens before async compression init completes
*/
private async ensureCompressionReady(): Promise<void> {
@ -206,7 +206,7 @@ export class BlobStorage {
return hash
}
// v5.7.5: Ensure compression is initialized before writing
// Ensure compression is initialized before writing
// Fixes race condition where write happens before async init completes
await this.ensureCompressionReady()
@ -223,7 +223,7 @@ export class BlobStorage {
}
// Create metadata
// v5.7.5: Store ACTUAL compression state, not intended
// Store ACTUAL compression state, not intended
// Prevents corruption if compression failed to initialize
const actualCompression = finalData === data ? 'none' : compression
const metadata: BlobMetadata = {
@ -277,7 +277,7 @@ export class BlobStorage {
* @returns Blob data
*/
async read(hash: string, options: BlobReadOptions = {}): Promise<Buffer> {
// v5.3.4 fix: Guard against NULL hash (sentinel value)
// Guard against NULL hash (sentinel value)
// NULL_HASH ('0000...0000') is used as a sentinel for "no parent" or "empty tree"
// It should NEVER be read as actual blob data
if (isNullHash(hash)) {
@ -309,7 +309,7 @@ export class BlobStorage {
metadataBuffer = await this.adapter.get(`${tryPrefix}-meta:${hash}`)
if (metadataBuffer) {
prefix = tryPrefix
// v5.10.1: Unwrap metadata before parsing (defense-in-depth)
// Unwrap metadata before parsing (defense-in-depth)
// Metadata should be JSON, but adapter might return wrapped format
const unwrappedMetadata = unwrapBinaryData(metadataBuffer)
metadata = JSON.parse(unwrappedMetadata.toString())
@ -338,8 +338,8 @@ export class BlobStorage {
finalData = await this.zstdDecompress(data)
}
// v5.10.1: Defense-in-depth unwrap (CRITICAL FIX for blob integrity regression)
// Even though COW adapter should unwrap (v5.7.5), verify it happened and re-unwrap if needed
// Defense-in-depth unwrap (CRITICAL FIX for blob integrity regression)
// Even though COW adapter should unwrap, verify it happened and re-unwrap if needed
// This prevents "Blob integrity check failed" errors if adapter returns wrapped data
// Uses shared binaryDataCodec utility (single source of truth for unwrap logic)
const unwrappedData = unwrapBinaryData(finalData)

2
src/storage/cow/CommitLog.ts
View file

@ -43,7 +43,7 @@ export interface CommitLogStats {
/**
* CommitLog: Efficient commit history traversal and querying
*
* Pure v5.0.0 implementation - modern, clean, fast
* Pure implementation - modern, clean, fast
*/
export class CommitLog {
private blobStorage: BlobStorage

2
src/storage/cow/CommitObject.ts
View file

@ -337,7 +337,7 @@ export class CommitObject {
let currentHash: string | null = startHash
let depth = 0
// v5.3.4 fix: Guard against NULL hash (sentinel for "no parent")
// Guard against NULL hash (sentinel for "no parent")
// The initial commit has parent = null or NULL_HASH ('0000...0000')
// We must stop walking when we reach it, not try to read it
while (currentHash && !isNullHash(currentHash)) {

2
src/storage/cow/RefManager.ts
View file

@ -55,7 +55,7 @@ export interface RefUpdateOptions {
/**
* RefManager: Manages branches, tags, and HEAD pointer
*
* Pure implementation for v5.0.0 - no backward compatibility
* Pure implementation - no backward compatibility
*/
export class RefManager {
private adapter: COWStorageAdapter

4
src/storage/cow/binaryDataCodec.ts
View file

@ -86,7 +86,7 @@ export function unwrapBinaryData(data: any): Buffer {
/**
* Wrap binary data for JSON storage
*
* ⚠️ WARNING: DO NOT USE THIS ON WRITE PATH! (v6.2.0)
* ⚠️ WARNING: DO NOT USE THIS ON WRITE PATH!
* ⚠️ Use key-based dispatch in baseStorage.ts COW adapter instead.
* ⚠️ This function exists for legacy/compatibility only.
*
@ -94,7 +94,7 @@ export function unwrapBinaryData(data: any): Buffer {
* This is FRAGILE because compressed binary can accidentally parse as valid JSON,
* causing blob integrity failures.
*
* v6.2.0 SOLUTION: baseStorage.ts COW adapter now uses key naming convention:
* SOLUTION: baseStorage.ts COW adapter now uses key naming convention:
* - Keys with '-meta:' or 'ref:' prefix → Always JSON
* - Keys with 'blob:', 'commit:', 'tree:' prefix → Always binary
* No guessing needed!

23
src/storage/storageFactory.ts
View file

@ -10,7 +10,7 @@ import { S3CompatibleStorage } from './adapters/s3CompatibleStorage.js'
import { R2Storage } from './adapters/r2Storage.js'
import { GcsStorage } from './adapters/gcsStorage.js'
import { AzureBlobStorage } from './adapters/azureBlobStorage.js'
// TypeAwareStorageAdapter removed in v5.4.0 - type-aware now built into BaseStorage
// TypeAwareStorageAdapter removed - type-aware now built into BaseStorage
// FileSystemStorage is dynamically imported to avoid issues in browser environments
import { isBrowser } from '../utils/environment.js'
import { OperationConfig } from '../utils/operationUtils.js'
@ -94,7 +94,7 @@ export interface StorageOptions {
sessionToken?: string
/**
* Initialization mode for fast cold starts (v7.3.0+)
* Initialization mode for fast cold starts
*
* - `'auto'` (default): Progressive in cloud environments (Lambda),
* strict locally. Zero-config optimization.
@ -103,7 +103,6 @@ export interface StorageOptions {
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts
* before init() returns.
*
* @since v7.3.0
*/
initMode?: 'progressive' | 'strict' | 'auto'
}
@ -215,7 +214,7 @@ export interface StorageOptions {
skipCountsFile?: boolean
/**
* Initialization mode for fast cold starts (v7.3.0+)
* Initialization mode for fast cold starts
*
* - `'auto'` (default): Progressive in cloud environments (Cloud Run),
* strict locally. Zero-config optimization.
@ -224,7 +223,6 @@ export interface StorageOptions {
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts
* before init() returns.
*
* @since v7.3.0
*/
initMode?: 'progressive' | 'strict' | 'auto'
}
@ -259,7 +257,7 @@ export interface StorageOptions {
sasToken?: string
/**
* Initialization mode for fast cold starts (v7.3.0+)
* Initialization mode for fast cold starts
*
* - `'auto'` (default): Progressive in cloud environments (Azure Functions),
* strict locally. Zero-config optimization.
@ -268,7 +266,6 @@ export interface StorageOptions {
* - `'strict'`: Traditional blocking init. Validates container and loads counts
* before init() returns.
*
* @since v7.3.0
*/
initMode?: 'progressive' | 'strict' | 'auto'
}
@ -388,7 +385,7 @@ export interface StorageOptions {
/**
* COW (Copy-on-Write) configuration for instant fork() capability
* v5.0.1: COW is now always enabled (automatic, zero-config)
* COW is now always enabled (automatic, zero-config)
*/
branch?: string // Current branch name (default: 'main')
enableCompression?: boolean // Enable zstd compression for COW blobs (default: true)
@ -410,14 +407,14 @@ function getFileSystemPath(options: StorageOptions): string {
}
/**
* Configure COW (Copy-on-Write) options on a storage adapter (v5.4.0)
* Configure COW (Copy-on-Write) options on a storage adapter
* TypeAware is now built-in to all adapters, no wrapper needed!
*
* @param storage - The storage adapter
* @param options - Storage options (for COW configuration)
*/
function configureCOW(storage: any, options?: StorageOptions): void {
// v5.0.1: COW will be initialized AFTER storage.init() in Brainy
// COW will be initialized AFTER storage.init() in Brainy
// Store COW options for later initialization
if (typeof storage.initializeCOW === 'function') {
storage._cowOptions = {
@ -672,10 +669,10 @@ export async function createStorage(
}
case 'type-aware':
// v5.0.0: TypeAware is now the default for ALL adapters
// TypeAware is now the default for ALL adapters
// Redirect to the underlying type instead
console.warn(
'⚠️ type-aware is deprecated in v5.0.0 - TypeAware is now always enabled.'
'⚠️ type-aware is deprecated - TypeAware is now always enabled.'
)
console.warn(
' Just use the underlying storage type (e.g., "filesystem", "s3", etc.)'
@ -866,7 +863,7 @@ export async function createStorage(
}
/**
* Export storage adapters (v5.4.0: TypeAware is now built-in, no separate export)
* Export storage adapters (TypeAware is now built-in, no separate export)
*/
export {
MemoryStorage,

Some files were not shown because too many files have changed in this diff Show more