feat: brain.find() excludes VFS by default (Option 3C)
Added includeVFS parameter to FindParams:
- brain.find() excludes VFS entities by default (clean knowledge graph)
- Opt-in with brain.find({ includeVFS: true })
- Automatically excludes VFS in all query paths (empty, metadata, vector)
- Respects explicit where: { isVFS: ... } queries
Implementation:
- Empty query path: Apply VFS filtering even with no criteria
- Metadata query path: Filter out isVFS: true by default
- Vector search path: Apply VFS filter after search
- Skip auto-exclusion if where clause explicitly queries isVFS
Architecture (Option 3C):
- VFS entities are first-class graph entities
- Marked with isVFS: true flag
- Separated via filtering, not storage
- Enables VFS-knowledge relationships
Moved internal docs to .strategy/:
- README_STORAGE_EXPLORATION.md
- EXPLORATION_SUMMARY.md
- STORAGE_FILES_REFERENCE.md
- STORAGE_ADAPTER_QUICK_REFERENCE.md
- SECURITY.md
This commit is contained in:
parent
86f5956d59
commit
014b8104da
14 changed files with 1172 additions and 1807 deletions
|
|
@ -1,393 +0,0 @@
|
||||||
# Brainy Storage Adapter Architecture - Exploration Summary
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
This exploration analyzed the complete storage adapter architecture in Brainy to understand how it works and determine whether a TypeAwareStorageAdapter can be added alongside existing adapters.
|
|
||||||
|
|
||||||
## Key Findings
|
|
||||||
|
|
||||||
### 1. Architecture is Clean and Extensible
|
|
||||||
|
|
||||||
Brainy implements a **well-designed, modular storage adapter architecture** using:
|
|
||||||
- **Interface-based design** (StorageAdapter interface in coreTypes.ts)
|
|
||||||
- **Abstract base classes** for common functionality
|
|
||||||
- **Concrete implementations** for specific backends
|
|
||||||
- **Factory pattern** for runtime adapter selection
|
|
||||||
|
|
||||||
### 2. Six Storage Adapters Currently Exist
|
|
||||||
|
|
||||||
| Adapter | Platform | Backend | File | Lines |
|
|
||||||
|---------|----------|---------|------|-------|
|
|
||||||
| FileSystemStorage | Node.js | Local filesystem | fileSystemStorage.ts | 2,677 |
|
|
||||||
| MemoryStorage | Browser/Node.js | In-memory Maps | memoryStorage.ts | 822 |
|
|
||||||
| S3CompatibleStorage | Node.js | AWS S3, Cloudflare R2, GCS (S3 API) | s3CompatibleStorage.ts | 5,000+ |
|
|
||||||
| GcsStorage | Node.js | Google Cloud Storage (native SDK) | gcsStorage.ts | 1,835 |
|
|
||||||
| OPFSStorage | Browser | Origin Private File System | opfsStorage.ts | - |
|
|
||||||
| R2Storage | Node.js | Alias for S3CompatibleStorage | (alias) | - |
|
|
||||||
|
|
||||||
### 3. Inheritance Hierarchy is Clean
|
|
||||||
|
|
||||||
```
|
|
||||||
StorageAdapter (interface - 27 methods)
|
|
||||||
↓
|
|
||||||
BaseStorageAdapter (abstract - 1,156 lines)
|
|
||||||
├─ Statistics management
|
|
||||||
├─ Throttling detection
|
|
||||||
├─ Count management (O(1))
|
|
||||||
└─ Service tracking
|
|
||||||
↓
|
|
||||||
BaseStorage (abstract - 1,098 lines)
|
|
||||||
├─ 2-file system (vectors + metadata)
|
|
||||||
├─ UUID-based sharding (256 shards)
|
|
||||||
├─ Pagination support
|
|
||||||
└─ Metadata routing
|
|
||||||
↓
|
|
||||||
Concrete Adapters (FileSystem, Memory, S3, GCS, OPFS)
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. Core Components
|
|
||||||
|
|
||||||
**Storage System Files (~13,000+ lines total):**
|
|
||||||
- `src/coreTypes.ts` - StorageAdapter interface
|
|
||||||
- `src/storage/baseStorageAdapter.ts` - Abstract base (1,156 lines)
|
|
||||||
- `src/storage/baseStorage.ts` - Core layer (1,098 lines)
|
|
||||||
- `src/storage/storageFactory.ts` - Factory for selection
|
|
||||||
- `src/storage/adapters/*.ts` - Concrete implementations
|
|
||||||
- `src/storage/sharding.ts` - UUID sharding utilities
|
|
||||||
- `src/storage/cacheManager.ts` - LRU cache
|
|
||||||
|
|
||||||
**Supporting Utilities:**
|
|
||||||
- `src/utils/writeBuffer.ts` - Batch operations
|
|
||||||
- `src/utils/adaptiveBackpressure.ts` - Flow control
|
|
||||||
- `src/utils/requestCoalescer.ts` - Request deduplication
|
|
||||||
- `src/storage/backwardCompatibility.ts` - Migration support
|
|
||||||
|
|
||||||
### 5. Storage Path Structure
|
|
||||||
|
|
||||||
**Modern Entity-Based Structure:**
|
|
||||||
```
|
|
||||||
entities/
|
|
||||||
├── nouns/vectors/{shard}/{id}.json (vector data)
|
|
||||||
├── nouns/metadata/{shard}/{id}.json (flexible metadata)
|
|
||||||
├── nouns/hnsw/{shard}/{id}.json (HNSW graph)
|
|
||||||
├── verbs/vectors/{shard}/{id}.json
|
|
||||||
├── verbs/metadata/{shard}/{id}.json
|
|
||||||
└── verbs/hnsw/{shard}/{id}.json
|
|
||||||
|
|
||||||
_system/
|
|
||||||
├── statistics.json (aggregate counts)
|
|
||||||
├── counts.json (O(1) totals)
|
|
||||||
└── hnsw-system.json (HNSW metadata)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Sharding:** UUID first 2 hex chars = 256 shard directories (00-ff)
|
|
||||||
|
|
||||||
### 6. 2-File System Design
|
|
||||||
|
|
||||||
Brainy separates **vector data** from **metadata** for scalability:
|
|
||||||
- **File 1:** `vectors/{id}.json` - Vector, HNSW connections (lightweight)
|
|
||||||
- **File 2:** `metadata/{id}.json` - Flexible metadata (any schema)
|
|
||||||
|
|
||||||
**Benefits:**
|
|
||||||
- Decouple vector operations from metadata queries
|
|
||||||
- Enable type-aware queries without loading vectors
|
|
||||||
- Independent scaling of vector vs metadata storage
|
|
||||||
- Support for metadata-only updates
|
|
||||||
|
|
||||||
### 7. Brainy Integration
|
|
||||||
|
|
||||||
How Brainy uses storage:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// In brainy.ts
|
|
||||||
class Brainy {
|
|
||||||
private storage!: BaseStorage
|
|
||||||
|
|
||||||
async init(config: BrainyConfig): Promise<void> {
|
|
||||||
// Factory creates appropriate adapter
|
|
||||||
this.storage = await createStorage(config.storage) as BaseStorage
|
|
||||||
await this.storage.init()
|
|
||||||
|
|
||||||
// Pass to HNSW index
|
|
||||||
this.index = new HNSWIndex(this.storage, ...)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Key insight: **Brainy only knows about `BaseStorage` interface, not specific adapters**
|
|
||||||
|
|
||||||
### 8. Design Patterns Used
|
|
||||||
|
|
||||||
1. **Factory Pattern** - `createStorage()` selects adapter at runtime
|
|
||||||
2. **Strategy Pattern** - Adapters are interchangeable
|
|
||||||
3. **Template Method** - BaseStorage defines skeleton, adapters fill details
|
|
||||||
4. **Adapter Pattern** - Maps different backends to same interface
|
|
||||||
5. **Decorator Pattern** - Could wrap adapters (e.g., TypeAware wrapper)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Answer: Can TypeAwareStorageAdapter Be Added?
|
|
||||||
|
|
||||||
### YES - DEFINITIVELY
|
|
||||||
|
|
||||||
**TypeAwareStorageAdapter can be added as a new adapter alongside existing ones WITHOUT replacing them.**
|
|
||||||
|
|
||||||
### Reasons
|
|
||||||
|
|
||||||
1. **Factory Pattern:** Multiple adapters coexist via factory function
|
|
||||||
2. **No Coupling:** Brainy depends on `BaseStorage` interface, not specific adapters
|
|
||||||
3. **Clean Inheritance:** Just extend `BaseStorage` like all other adapters
|
|
||||||
4. **Isolated:** Type awareness doesn't affect other adapters
|
|
||||||
5. **Backward Compatible:** Existing code continues to work unchanged
|
|
||||||
|
|
||||||
### Implementation Path
|
|
||||||
|
|
||||||
**3 Simple Steps:**
|
|
||||||
|
|
||||||
**Step 1: Create new adapter file**
|
|
||||||
```typescript
|
|
||||||
// src/storage/adapters/typeAwareStorageAdapter.ts
|
|
||||||
export class TypeAwareStorageAdapter extends BaseStorage {
|
|
||||||
// Implement 17 abstract methods
|
|
||||||
// Add type indexing logic
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 2: Update factory**
|
|
||||||
```typescript
|
|
||||||
// src/storage/storageFactory.ts
|
|
||||||
if (options.type === 'type-aware') {
|
|
||||||
return new TypeAwareStorageAdapter(options)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 3: Update options interface**
|
|
||||||
```typescript
|
|
||||||
// src/storage/storageFactory.ts
|
|
||||||
export interface StorageOptions {
|
|
||||||
type?: 'auto' | 'memory' | 'filesystem' | 's3' | 'gcs' | 'type-aware'
|
|
||||||
typeAwareStorage?: { ... }
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**No changes needed to:**
|
|
||||||
- Brainy.ts
|
|
||||||
- coreTypes.ts (unless adding new methods)
|
|
||||||
- Existing adapters
|
|
||||||
- HNSW index
|
|
||||||
- Any other components
|
|
||||||
|
|
||||||
### Abstract Methods to Implement
|
|
||||||
|
|
||||||
When creating TypeAwareStorageAdapter, implement these 17 methods:
|
|
||||||
|
|
||||||
**Noun/Verb Operations (6):**
|
|
||||||
- `saveNoun_internal()`
|
|
||||||
- `getNoun_internal()`
|
|
||||||
- `deleteNoun_internal()`
|
|
||||||
- `saveVerb_internal()`
|
|
||||||
- `getVerb_internal()`
|
|
||||||
- `deleteVerb_internal()`
|
|
||||||
|
|
||||||
**Path Operations (4):**
|
|
||||||
- `writeObjectToPath()`
|
|
||||||
- `readObjectFromPath()`
|
|
||||||
- `deleteObjectFromPath()`
|
|
||||||
- `listObjectsUnderPath()`
|
|
||||||
|
|
||||||
**Count Management (2):**
|
|
||||||
- `initializeCounts()`
|
|
||||||
- `persistCounts()`
|
|
||||||
|
|
||||||
**Statistics (2):**
|
|
||||||
- `saveStatisticsData()`
|
|
||||||
- `getStatisticsData()`
|
|
||||||
|
|
||||||
**Lifecycle (3):**
|
|
||||||
- `init()`
|
|
||||||
- `clear()`
|
|
||||||
- `getStorageStatus()`
|
|
||||||
|
|
||||||
### Recommended Design Approach
|
|
||||||
|
|
||||||
**Option A: Direct Implementation (Recommended)**
|
|
||||||
```
|
|
||||||
TypeAwareStorageAdapter
|
|
||||||
├─ Extends BaseStorage
|
|
||||||
├─ Implements all 17 abstract methods
|
|
||||||
├─ Adds type indexing logic
|
|
||||||
└─ Can back any storage engine
|
|
||||||
```
|
|
||||||
|
|
||||||
**Option B: Wrapper/Decorator Pattern**
|
|
||||||
```
|
|
||||||
TypeAwareStorageAdapter (wrapper)
|
|
||||||
├─ Wraps any BaseStorage adapter
|
|
||||||
├─ Intercepts saveNoun/saveVerb
|
|
||||||
├─ Tracks types in separate index
|
|
||||||
└─ Delegates all operations
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Key Insights
|
|
||||||
|
|
||||||
### Storage Architecture Strengths
|
|
||||||
|
|
||||||
✅ **Well-organized:** Clear separation of concerns
|
|
||||||
✅ **Extensible:** Factory pattern makes adding adapters simple
|
|
||||||
✅ **Scalable:** Sharding, caching, batching, backpressure
|
|
||||||
✅ **Flexible:** Multiple backends coexist without conflicts
|
|
||||||
✅ **Type-safe:** Full TypeScript with proper interfaces
|
|
||||||
✅ **Production-ready:** Used in real deployments
|
|
||||||
|
|
||||||
### What Makes This Possible
|
|
||||||
|
|
||||||
1. **Interface-based design** - Adapters implement same contract
|
|
||||||
2. **Factory pattern** - Runtime selection without coupling
|
|
||||||
3. **No hardcoded dependencies** - Brainy uses `BaseStorage` type
|
|
||||||
4. **Common base class** - Shared logic prevents duplication
|
|
||||||
5. **Metadata separation** - 2-file system enables type indexing
|
|
||||||
|
|
||||||
### Storage Adapter Evolution Path
|
|
||||||
|
|
||||||
```
|
|
||||||
Current State (v3.44.0):
|
|
||||||
├─ FileSystemStorage ✅
|
|
||||||
├─ MemoryStorage ✅
|
|
||||||
├─ S3CompatibleStorage ✅
|
|
||||||
├─ GcsStorage ✅
|
|
||||||
└─ OPFSStorage ✅
|
|
||||||
|
|
||||||
Future State (proposed):
|
|
||||||
├─ FileSystemStorage ✅
|
|
||||||
├─ MemoryStorage ✅
|
|
||||||
├─ S3CompatibleStorage ✅
|
|
||||||
├─ GcsStorage ✅
|
|
||||||
├─ OPFSStorage ✅
|
|
||||||
└─ TypeAwareStorageAdapter ✅ (new)
|
|
||||||
|
|
||||||
All coexist without conflicts
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Documents Created
|
|
||||||
|
|
||||||
This exploration generated three comprehensive documents:
|
|
||||||
|
|
||||||
### 1. STORAGE_ARCHITECTURE_ANALYSIS.md (28 KB)
|
|
||||||
Complete analysis covering:
|
|
||||||
- Current storage architecture overview
|
|
||||||
- All existing storage adapters
|
|
||||||
- StorageAdapter interface specification
|
|
||||||
- How Brainy uses storage
|
|
||||||
- Storage paths and patterns
|
|
||||||
- Storage adapter pattern analysis
|
|
||||||
- Detailed implementation recommendations
|
|
||||||
- Design patterns and best practices
|
|
||||||
|
|
||||||
### 2. STORAGE_ADAPTER_QUICK_REFERENCE.md (8.6 KB)
|
|
||||||
Quick reference guide with:
|
|
||||||
- File locations
|
|
||||||
- Storage adapter hierarchy
|
|
||||||
- Abstract methods checklist (17 methods)
|
|
||||||
- Storage path structure
|
|
||||||
- 2-file system design
|
|
||||||
- Existing adapters overview
|
|
||||||
- Factory integration
|
|
||||||
- Performance characteristics
|
|
||||||
- Design patterns summary
|
|
||||||
|
|
||||||
### 3. STORAGE_FILES_REFERENCE.md (13 KB)
|
|
||||||
Complete file reference with:
|
|
||||||
- All core storage files
|
|
||||||
- Line counts and purposes
|
|
||||||
- Each adapter's features
|
|
||||||
- Integration points
|
|
||||||
- Data flow diagrams
|
|
||||||
- Statistics tracking
|
|
||||||
- Type definitions
|
|
||||||
- Summary statistics table
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Recommendations
|
|
||||||
|
|
||||||
### For TypeAwareStorageAdapter Implementation
|
|
||||||
|
|
||||||
1. **Use Direct Implementation approach** (not wrapper)
|
|
||||||
- Simpler to maintain
|
|
||||||
- Better performance
|
|
||||||
- Easier to debug
|
|
||||||
- Can back any storage engine
|
|
||||||
|
|
||||||
2. **Implement as new entry in factory**
|
|
||||||
- `type: 'type-aware'` with storage config
|
|
||||||
- Auto-detection can select it
|
|
||||||
- No changes to existing code
|
|
||||||
|
|
||||||
3. **Leverage 2-file system**
|
|
||||||
- Store type index in metadata files
|
|
||||||
- Queries don't require loading vectors
|
|
||||||
- Aligns with existing patterns
|
|
||||||
|
|
||||||
4. **Inherit common functionality**
|
|
||||||
- Throttling detection
|
|
||||||
- Statistics tracking
|
|
||||||
- Caching and batching
|
|
||||||
- Count management (O(1))
|
|
||||||
|
|
||||||
5. **Follow existing patterns**
|
|
||||||
- Sharding strategy (first 2 hex chars)
|
|
||||||
- Path structure (entities/{noun|verb}/{vectors|metadata}/{shard}/{id}.json)
|
|
||||||
- Pagination support
|
|
||||||
- Metadata separation
|
|
||||||
|
|
||||||
### For Integration
|
|
||||||
|
|
||||||
1. Add new file: `/src/storage/adapters/typeAwareStorageAdapter.ts`
|
|
||||||
2. Modify: `/src/storage/storageFactory.ts` (add case + interface)
|
|
||||||
3. Optional: `/src/coreTypes.ts` (if extending StorageAdapter interface)
|
|
||||||
4. No changes needed elsewhere
|
|
||||||
|
|
||||||
### For Testing
|
|
||||||
|
|
||||||
1. Test with MemoryStorage first (fastest)
|
|
||||||
2. Test with FileSystemStorage (persistent)
|
|
||||||
3. Ensure all existing tests still pass
|
|
||||||
4. Add type-aware specific tests
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
Brainy's storage adapter architecture is **professionally designed and inherently extensible**. Adding a TypeAwareStorageAdapter is straightforward because:
|
|
||||||
|
|
||||||
- The architecture supports multiple concurrent adapters
|
|
||||||
- Brainy uses interface-based dependency injection
|
|
||||||
- The factory pattern enables runtime selection
|
|
||||||
- No breaking changes required anywhere
|
|
||||||
|
|
||||||
**The answer is unambiguous: TypeAwareStorageAdapter can be added alongside existing adapters with minimal integration effort.**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Files Analyzed
|
|
||||||
|
|
||||||
- `/src/coreTypes.ts` - Interface definition
|
|
||||||
- `/src/storage/baseStorageAdapter.ts` - Abstract base
|
|
||||||
- `/src/storage/baseStorage.ts` - Core layer
|
|
||||||
- `/src/storage/storageFactory.ts` - Factory
|
|
||||||
- `/src/storage/adapters/fileSystemStorage.ts` - FileSystem
|
|
||||||
- `/src/storage/adapters/memoryStorage.ts` - Memory
|
|
||||||
- `/src/storage/adapters/s3CompatibleStorage.ts` - S3/R2
|
|
||||||
- `/src/storage/adapters/gcsStorage.ts` - GCS native
|
|
||||||
- `/src/storage/adapters/opfsStorage.ts` - Browser OPFS
|
|
||||||
- `/src/brainy.ts` - Main class
|
|
||||||
- Plus all supporting utilities and type definitions
|
|
||||||
|
|
||||||
**Total files analyzed:** 50+
|
|
||||||
**Total lines examined:** 13,000+
|
|
||||||
**Analysis coverage:** Complete storage system
|
|
||||||
|
|
||||||
|
|
@ -1,291 +0,0 @@
|
||||||
# Storage Adapter Architecture Exploration - Documentation Index
|
|
||||||
|
|
||||||
## Quick Answer
|
|
||||||
|
|
||||||
**Can TypeAwareStorageAdapter be added alongside existing adapters?**
|
|
||||||
|
|
||||||
**YES - ABSOLUTELY.** It can be added as a new adapter without replacing any existing ones. See **EXPLORATION_SUMMARY.md** for details.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Documentation Files
|
|
||||||
|
|
||||||
### 1. EXPLORATION_SUMMARY.md (12 KB)
|
|
||||||
**START HERE** - Overview of the entire exploration
|
|
||||||
|
|
||||||
**Contains:**
|
|
||||||
- Executive summary of findings
|
|
||||||
- Definitive answer to the main question
|
|
||||||
- Implementation roadmap (3 simple steps)
|
|
||||||
- List of 17 abstract methods to implement
|
|
||||||
- Key insights about architecture
|
|
||||||
- Recommended design approaches
|
|
||||||
|
|
||||||
**Read this when:** You want a quick understanding of what we discovered
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 2. STORAGE_ARCHITECTURE_ANALYSIS.md (28 KB)
|
|
||||||
**COMPREHENSIVE DEEP DIVE** - Complete technical analysis
|
|
||||||
|
|
||||||
**Sections:**
|
|
||||||
1. Current storage architecture overview
|
|
||||||
2. Existing storage adapters (5 detailed profiles)
|
|
||||||
3. StorageAdapter interface specification (27 methods)
|
|
||||||
4. How Brainy uses storage
|
|
||||||
5. Storage factory pattern
|
|
||||||
6. Current storage paths and patterns
|
|
||||||
7. Storage adapter pattern analysis
|
|
||||||
8. Detailed recommendations for TypeAwareStorageAdapter
|
|
||||||
9. Storage directory structure details
|
|
||||||
10. Key design patterns
|
|
||||||
11. Summary and recommendations
|
|
||||||
|
|
||||||
**Read this when:** You need comprehensive technical understanding
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 3. STORAGE_ADAPTER_QUICK_REFERENCE.md (8.6 KB)
|
|
||||||
**QUICK LOOKUP GUIDE** - Fast reference for developers
|
|
||||||
|
|
||||||
**Sections:**
|
|
||||||
- File locations (all storage files)
|
|
||||||
- Storage adapter hierarchy (visual tree)
|
|
||||||
- Abstract methods checklist (17 methods)
|
|
||||||
- Storage path structure (modern format)
|
|
||||||
- 2-file system design explanation
|
|
||||||
- Existing adapters overview (quick stats)
|
|
||||||
- Factory integration example
|
|
||||||
- Key inherited features
|
|
||||||
- Performance characteristics
|
|
||||||
- Design patterns used
|
|
||||||
- Conclusion and next steps
|
|
||||||
|
|
||||||
**Read this when:** You're implementing TypeAwareStorageAdapter
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 4. STORAGE_FILES_REFERENCE.md (13 KB)
|
|
||||||
**COMPLETE FILE REFERENCE** - Detailed information about each file
|
|
||||||
|
|
||||||
**Sections:**
|
|
||||||
1. Core storage files (6 files described)
|
|
||||||
2. Storage adapter implementations (5 adapters detailed)
|
|
||||||
3. Storage patterns and utilities (5 utilities listed)
|
|
||||||
4. Integration points (3 main integration points)
|
|
||||||
5. Data flow examples (saving and querying)
|
|
||||||
6. Storage statistics tracking (JSON examples)
|
|
||||||
7. Type definitions (HNSWNoun, GraphVerb)
|
|
||||||
8. Summary statistics table (13,000+ lines)
|
|
||||||
|
|
||||||
**Read this when:** You need details about specific storage files
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Reading Guide by Use Case
|
|
||||||
|
|
||||||
### I want a quick answer
|
|
||||||
1. Read this README (you're here!)
|
|
||||||
2. Read: EXPLORATION_SUMMARY.md - first 30% (Key Findings + Answer)
|
|
||||||
|
|
||||||
### I'm implementing TypeAwareStorageAdapter
|
|
||||||
1. Read: EXPLORATION_SUMMARY.md (full)
|
|
||||||
2. Read: STORAGE_ADAPTER_QUICK_REFERENCE.md (Implementation section)
|
|
||||||
3. Reference: STORAGE_FILES_REFERENCE.md (for specific files)
|
|
||||||
|
|
||||||
### I need comprehensive understanding
|
|
||||||
1. Read: EXPLORATION_SUMMARY.md
|
|
||||||
2. Read: STORAGE_ARCHITECTURE_ANALYSIS.md (sections 1-7)
|
|
||||||
3. Reference: STORAGE_ADAPTER_QUICK_REFERENCE.md
|
|
||||||
|
|
||||||
### I'm debugging storage issues
|
|
||||||
1. Check: STORAGE_FILES_REFERENCE.md (which file handles what)
|
|
||||||
2. Check: STORAGE_ARCHITECTURE_ANALYSIS.md (data flow sections)
|
|
||||||
3. Check: STORAGE_ADAPTER_QUICK_REFERENCE.md (path structure)
|
|
||||||
|
|
||||||
### I'm integrating with storage system
|
|
||||||
1. Read: STORAGE_ARCHITECTURE_ANALYSIS.md (sections 3-4)
|
|
||||||
2. Read: STORAGE_FILES_REFERENCE.md (integration points)
|
|
||||||
3. Reference: STORAGE_ADAPTER_QUICK_REFERENCE.md (as needed)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Key Findings Summary
|
|
||||||
|
|
||||||
### Current State
|
|
||||||
- 5 storage adapters exist (FileSystem, Memory, S3, GCS, OPFS)
|
|
||||||
- 27-method StorageAdapter interface
|
|
||||||
- 17 abstract methods to implement for new adapters
|
|
||||||
- 13,000+ lines of storage code
|
|
||||||
- Clean inheritance hierarchy
|
|
||||||
- Factory pattern for runtime selection
|
|
||||||
|
|
||||||
### Architecture Strengths
|
|
||||||
- Well-organized and modular
|
|
||||||
- Factory pattern enables multiple backends
|
|
||||||
- Interface-based design (no coupling)
|
|
||||||
- Common base class (code reuse)
|
|
||||||
- 2-file system (separation of concerns)
|
|
||||||
|
|
||||||
### For TypeAwareStorageAdapter
|
|
||||||
- Can extend BaseStorage class
|
|
||||||
- Must implement 17 abstract methods
|
|
||||||
- Simple factory integration (1 case + interface update)
|
|
||||||
- No changes to existing code
|
|
||||||
- Inherits statistics, throttling, caching
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Core Facts
|
|
||||||
|
|
||||||
### Storage Layers
|
|
||||||
```
|
|
||||||
StorageAdapter interface (27 methods)
|
|
||||||
↓
|
|
||||||
BaseStorageAdapter (1,156 lines - common functionality)
|
|
||||||
↓
|
|
||||||
BaseStorage (1,098 lines - routing & pagination)
|
|
||||||
↓
|
|
||||||
Concrete Adapters (FileSystem, Memory, S3, GCS, OPFS)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Storage Paths
|
|
||||||
```
|
|
||||||
entities/nouns/vectors/{shard}/{id}.json ← vector data
|
|
||||||
entities/nouns/metadata/{shard}/{id}.json ← flexible metadata
|
|
||||||
entities/verbs/vectors/{shard}/{id}.json
|
|
||||||
entities/verbs/metadata/{shard}/{id}.json
|
|
||||||
_system/statistics.json ← aggregate stats
|
|
||||||
_system/counts.json ← O(1) totals
|
|
||||||
```
|
|
||||||
|
|
||||||
### Sharding
|
|
||||||
- UUID first 2 hex characters (00-ff)
|
|
||||||
- 256 shard directories
|
|
||||||
- Handles 2.5M+ entities efficiently
|
|
||||||
|
|
||||||
### 2-File System
|
|
||||||
- File 1: Vectors (lightweight, always loaded)
|
|
||||||
- File 2: Metadata (flexible schema, separately loaded)
|
|
||||||
- Enables type-aware queries without loading vectors
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Implementation Checklist
|
|
||||||
|
|
||||||
For adding TypeAwareStorageAdapter:
|
|
||||||
|
|
||||||
- [ ] Create `/src/storage/adapters/typeAwareStorageAdapter.ts`
|
|
||||||
- [ ] Extend BaseStorage class
|
|
||||||
- [ ] Implement 17 abstract methods:
|
|
||||||
- [ ] saveNoun_internal()
|
|
||||||
- [ ] getNoun_internal()
|
|
||||||
- [ ] deleteNoun_internal()
|
|
||||||
- [ ] saveVerb_internal()
|
|
||||||
- [ ] getVerb_internal()
|
|
||||||
- [ ] deleteVerb_internal()
|
|
||||||
- [ ] writeObjectToPath()
|
|
||||||
- [ ] readObjectFromPath()
|
|
||||||
- [ ] deleteObjectFromPath()
|
|
||||||
- [ ] listObjectsUnderPath()
|
|
||||||
- [ ] initializeCounts()
|
|
||||||
- [ ] persistCounts()
|
|
||||||
- [ ] saveStatisticsData()
|
|
||||||
- [ ] getStatisticsData()
|
|
||||||
- [ ] init()
|
|
||||||
- [ ] clear()
|
|
||||||
- [ ] getStorageStatus()
|
|
||||||
- [ ] Update `/src/storage/storageFactory.ts`:
|
|
||||||
- [ ] Add case for 'type-aware'
|
|
||||||
- [ ] Update StorageOptions interface
|
|
||||||
- [ ] Test with MemoryStorage
|
|
||||||
- [ ] Test with FileSystemStorage
|
|
||||||
- [ ] Verify existing tests still pass
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Quick Reference
|
|
||||||
|
|
||||||
### Files to Analyze
|
|
||||||
- `/src/coreTypes.ts` - StorageAdapter interface
|
|
||||||
- `/src/storage/baseStorageAdapter.ts` - Abstract base
|
|
||||||
- `/src/storage/baseStorage.ts` - Core layer
|
|
||||||
- `/src/storage/storageFactory.ts` - Factory
|
|
||||||
- `/src/storage/adapters/memoryStorage.ts` - Simple example
|
|
||||||
|
|
||||||
### Key Classes
|
|
||||||
- `StorageAdapter` - Interface (27 methods)
|
|
||||||
- `BaseStorageAdapter` - Abstract base (1,156 lines)
|
|
||||||
- `BaseStorage` - Abstract impl (1,098 lines)
|
|
||||||
- `FileSystemStorage` - Concrete impl (2,677 lines)
|
|
||||||
- `MemoryStorage` - Simple impl (822 lines)
|
|
||||||
|
|
||||||
### Key Methods to Implement
|
|
||||||
- Node/Verb: saveNoun_internal, getNoun_internal, etc.
|
|
||||||
- Path: writeObjectToPath, readObjectFromPath, etc.
|
|
||||||
- Counts: initializeCounts, persistCounts
|
|
||||||
- Stats: saveStatisticsData, getStatisticsData
|
|
||||||
- Lifecycle: init, clear, getStorageStatus
|
|
||||||
|
|
||||||
### Design Patterns
|
|
||||||
1. Factory - `createStorage()` for adapter selection
|
|
||||||
2. Strategy - Adapters are interchangeable
|
|
||||||
3. Template Method - BaseStorage defines skeleton
|
|
||||||
4. Adapter - Maps different backends to same interface
|
|
||||||
5. Decorator - Can wrap adapters if needed
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Analysis Statistics
|
|
||||||
|
|
||||||
| Metric | Count |
|
|
||||||
|--------|-------|
|
|
||||||
| Files analyzed | 50+ |
|
|
||||||
| Lines of code examined | 13,000+ |
|
|
||||||
| Storage adapters found | 5 |
|
|
||||||
| Abstract methods to implement | 17 |
|
|
||||||
| Interface methods | 27 |
|
|
||||||
| Storage backends supported | 6 (FS, Memory, S3, GCS, OPFS, R2) |
|
|
||||||
| Documentation pages created | 4 |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Contact & Questions
|
|
||||||
|
|
||||||
For questions about:
|
|
||||||
- **Architecture:** See STORAGE_ARCHITECTURE_ANALYSIS.md
|
|
||||||
- **Specific files:** See STORAGE_FILES_REFERENCE.md
|
|
||||||
- **Quick lookup:** See STORAGE_ADAPTER_QUICK_REFERENCE.md
|
|
||||||
- **Overall findings:** See EXPLORATION_SUMMARY.md
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
Brainy's storage architecture is **professionally designed** and **inherently extensible**. TypeAwareStorageAdapter can be added as a new adapter in just a few minutes by:
|
|
||||||
|
|
||||||
1. Creating a new class extending BaseStorage
|
|
||||||
2. Implementing 17 abstract methods
|
|
||||||
3. Registering in the factory
|
|
||||||
|
|
||||||
**No breaking changes required. No existing code needs modification.**
|
|
||||||
|
|
||||||
The architecture supports multiple backends coexisting peacefully through proper use of:
|
|
||||||
- Interface-based design
|
|
||||||
- Factory pattern
|
|
||||||
- Dependency injection
|
|
||||||
- Abstract base classes
|
|
||||||
|
|
||||||
This is a textbook example of good software architecture.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Document Metadata
|
|
||||||
|
|
||||||
**Created:** October 15, 2025
|
|
||||||
**Repository:** Brainy (Neural Database)
|
|
||||||
**Version:** Analysis of v3.44.0
|
|
||||||
**Scope:** Complete storage adapter architecture
|
|
||||||
**Coverage:** 100% of storage system
|
|
||||||
|
|
||||||
Generated with thorough code analysis and deep understanding of the system.
|
|
||||||
389
SECURITY.md
389
SECURITY.md
|
|
@ -1,389 +0,0 @@
|
||||||
# Security Best Practices for Brainy
|
|
||||||
|
|
||||||
## 🔒 Data Security
|
|
||||||
|
|
||||||
### Encryption at Rest
|
|
||||||
```typescript
|
|
||||||
const brain = new Brainy({
|
|
||||||
storage: {
|
|
||||||
type: 's3',
|
|
||||||
options: {
|
|
||||||
encryption: 'AES256', // Server-side encryption
|
|
||||||
kmsKeyId: process.env.KMS_KEY_ID // Optional KMS key
|
|
||||||
}
|
|
||||||
}
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
### Encryption in Transit
|
|
||||||
- Always use HTTPS/TLS for API endpoints
|
|
||||||
- Enable SSL for database connections
|
|
||||||
- Use VPN or private networks for internal communication
|
|
||||||
|
|
||||||
## 🔑 Authentication & Authorization
|
|
||||||
|
|
||||||
### API Key Management
|
|
||||||
```typescript
|
|
||||||
// Middleware example
|
|
||||||
app.use('/api/brainy', (req, res, next) => {
|
|
||||||
const apiKey = req.headers['x-api-key']
|
|
||||||
|
|
||||||
if (!apiKey || !isValidApiKey(apiKey)) {
|
|
||||||
return res.status(401).json({ error: 'Unauthorized' })
|
|
||||||
}
|
|
||||||
|
|
||||||
// Rate limit by API key
|
|
||||||
const limit = getRateLimitForKey(apiKey)
|
|
||||||
if (exceedsRateLimit(apiKey, limit)) {
|
|
||||||
return res.status(429).json({ error: 'Rate limit exceeded' })
|
|
||||||
}
|
|
||||||
|
|
||||||
next()
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
### JWT Authentication
|
|
||||||
```typescript
|
|
||||||
import jwt from 'jsonwebtoken'
|
|
||||||
|
|
||||||
// Verify JWT token
|
|
||||||
app.use('/api/brainy', (req, res, next) => {
|
|
||||||
const token = req.headers.authorization?.split(' ')[1]
|
|
||||||
|
|
||||||
try {
|
|
||||||
const decoded = jwt.verify(token, process.env.JWT_SECRET)
|
|
||||||
req.user = decoded
|
|
||||||
next()
|
|
||||||
} catch (error) {
|
|
||||||
return res.status(401).json({ error: 'Invalid token' })
|
|
||||||
}
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🛡️ Input Validation & Sanitization
|
|
||||||
|
|
||||||
### Query Validation
|
|
||||||
```typescript
|
|
||||||
import { z } from 'zod'
|
|
||||||
|
|
||||||
const SearchSchema = z.object({
|
|
||||||
query: z.string().min(1).max(1000),
|
|
||||||
limit: z.number().min(1).max(100).default(10),
|
|
||||||
metadata: z.record(z.unknown()).optional()
|
|
||||||
})
|
|
||||||
|
|
||||||
app.post('/api/search', async (req, res) => {
|
|
||||||
try {
|
|
||||||
const params = SearchSchema.parse(req.body)
|
|
||||||
const results = await brain.find(params)
|
|
||||||
res.json(results)
|
|
||||||
} catch (error) {
|
|
||||||
if (error instanceof z.ZodError) {
|
|
||||||
return res.status(400).json({ error: 'Invalid input', details: error.errors })
|
|
||||||
}
|
|
||||||
throw error
|
|
||||||
}
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
### Metadata Sanitization
|
|
||||||
```typescript
|
|
||||||
function sanitizeMetadata(metadata: any): any {
|
|
||||||
// Remove potential XSS vectors
|
|
||||||
const sanitized = {}
|
|
||||||
|
|
||||||
for (const [key, value] of Object.entries(metadata)) {
|
|
||||||
// Sanitize keys
|
|
||||||
const cleanKey = key.replace(/[<>'"]/g, '')
|
|
||||||
|
|
||||||
// Sanitize values
|
|
||||||
if (typeof value === 'string') {
|
|
||||||
sanitized[cleanKey] = value.replace(/[<>'"]/g, '')
|
|
||||||
} else if (typeof value === 'object' && value !== null) {
|
|
||||||
sanitized[cleanKey] = sanitizeMetadata(value)
|
|
||||||
} else {
|
|
||||||
sanitized[cleanKey] = value
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return sanitized
|
|
||||||
}
|
|
||||||
|
|
||||||
// Use before adding to brain
|
|
||||||
const sanitizedData = {
|
|
||||||
text: sanitizeText(input.text),
|
|
||||||
metadata: sanitizeMetadata(input.metadata)
|
|
||||||
}
|
|
||||||
await brain.add(sanitizedData)
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🚦 Rate Limiting
|
|
||||||
|
|
||||||
### Per-IP Rate Limiting
|
|
||||||
```typescript
|
|
||||||
import rateLimit from 'express-rate-limit'
|
|
||||||
|
|
||||||
const limiter = rateLimit({
|
|
||||||
windowMs: 15 * 60 * 1000, // 15 minutes
|
|
||||||
max: 100, // Limit each IP to 100 requests per windowMs
|
|
||||||
message: 'Too many requests from this IP'
|
|
||||||
})
|
|
||||||
|
|
||||||
app.use('/api/brainy', limiter)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Per-User Rate Limiting
|
|
||||||
```typescript
|
|
||||||
const userLimits = new Map()
|
|
||||||
|
|
||||||
function checkUserRateLimit(userId: string, limit = 1000): boolean {
|
|
||||||
const now = Date.now()
|
|
||||||
const userRequests = userLimits.get(userId) || []
|
|
||||||
|
|
||||||
// Remove old requests (older than 1 hour)
|
|
||||||
const recentRequests = userRequests.filter((time: number) =>
|
|
||||||
now - time < 3600000
|
|
||||||
)
|
|
||||||
|
|
||||||
if (recentRequests.length >= limit) {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
recentRequests.push(now)
|
|
||||||
userLimits.set(userId, recentRequests)
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🔍 Audit Logging
|
|
||||||
|
|
||||||
### Comprehensive Audit Trail
|
|
||||||
```typescript
|
|
||||||
interface AuditLog {
|
|
||||||
timestamp: Date
|
|
||||||
userId: string
|
|
||||||
action: string
|
|
||||||
resource: string
|
|
||||||
details: any
|
|
||||||
ip: string
|
|
||||||
userAgent: string
|
|
||||||
}
|
|
||||||
|
|
||||||
class AuditLogger {
|
|
||||||
async log(entry: AuditLog): Promise<void> {
|
|
||||||
// Log to secure storage
|
|
||||||
await this.storage.append('audit.log', JSON.stringify(entry) + '\n')
|
|
||||||
|
|
||||||
// Alert on suspicious activity
|
|
||||||
if (this.isSuspicious(entry)) {
|
|
||||||
await this.alertSecurityTeam(entry)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private isSuspicious(entry: AuditLog): boolean {
|
|
||||||
// Check for patterns like:
|
|
||||||
// - Multiple failed auth attempts
|
|
||||||
// - Unusual data access patterns
|
|
||||||
// - Bulk data exports
|
|
||||||
// - Access from new locations
|
|
||||||
return false // Implement your logic
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Use in your API
|
|
||||||
app.use(async (req, res, next) => {
|
|
||||||
const entry: AuditLog = {
|
|
||||||
timestamp: new Date(),
|
|
||||||
userId: req.user?.id || 'anonymous',
|
|
||||||
action: req.method,
|
|
||||||
resource: req.path,
|
|
||||||
details: req.body,
|
|
||||||
ip: req.ip,
|
|
||||||
userAgent: req.headers['user-agent']
|
|
||||||
}
|
|
||||||
|
|
||||||
await auditLogger.log(entry)
|
|
||||||
next()
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🗑️ Data Privacy & GDPR Compliance
|
|
||||||
|
|
||||||
### Right to Deletion
|
|
||||||
```typescript
|
|
||||||
async function deleteUserData(userId: string): Promise<void> {
|
|
||||||
// Find all items belonging to user
|
|
||||||
const userItems = await brain.find({
|
|
||||||
metadata: { userId }
|
|
||||||
})
|
|
||||||
|
|
||||||
// Delete each item
|
|
||||||
for (const item of userItems) {
|
|
||||||
await brain.delete(item.id)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Log the deletion
|
|
||||||
await auditLogger.log({
|
|
||||||
timestamp: new Date(),
|
|
||||||
userId,
|
|
||||||
action: 'DELETE_USER_DATA',
|
|
||||||
resource: 'user_data',
|
|
||||||
details: { itemCount: userItems.length },
|
|
||||||
ip: 'system',
|
|
||||||
userAgent: 'gdpr-compliance'
|
|
||||||
})
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Data Export
|
|
||||||
```typescript
|
|
||||||
async function exportUserData(userId: string): Promise<any> {
|
|
||||||
// Get all user data
|
|
||||||
const items = await brain.find({
|
|
||||||
metadata: { userId }
|
|
||||||
})
|
|
||||||
|
|
||||||
// Get all relationships
|
|
||||||
const relationships = []
|
|
||||||
for (const item of items) {
|
|
||||||
const relations = await brain.getRelations(item.id)
|
|
||||||
relationships.push(...relations)
|
|
||||||
}
|
|
||||||
|
|
||||||
return {
|
|
||||||
exportDate: new Date().toISOString(),
|
|
||||||
userId,
|
|
||||||
items,
|
|
||||||
relationships,
|
|
||||||
metadata: {
|
|
||||||
itemCount: items.length,
|
|
||||||
relationshipCount: relationships.length
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🚨 Security Headers
|
|
||||||
|
|
||||||
### Express.js Security Headers
|
|
||||||
```typescript
|
|
||||||
import helmet from 'helmet'
|
|
||||||
|
|
||||||
app.use(helmet({
|
|
||||||
contentSecurityPolicy: {
|
|
||||||
directives: {
|
|
||||||
defaultSrc: ["'self'"],
|
|
||||||
styleSrc: ["'self'", "'unsafe-inline'"],
|
|
||||||
scriptSrc: ["'self'"],
|
|
||||||
imgSrc: ["'self'", "data:", "https:"],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
hsts: {
|
|
||||||
maxAge: 31536000,
|
|
||||||
includeSubDomains: true,
|
|
||||||
preload: true
|
|
||||||
}
|
|
||||||
}))
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🔐 Environment Variables
|
|
||||||
|
|
||||||
### Secure Configuration
|
|
||||||
```bash
|
|
||||||
# .env.production
|
|
||||||
NODE_ENV=production
|
|
||||||
JWT_SECRET=<use-strong-random-secret>
|
|
||||||
DATABASE_URL=<encrypted-connection-string>
|
|
||||||
AWS_ACCESS_KEY_ID=<use-iam-roles-in-production>
|
|
||||||
AWS_SECRET_ACCESS_KEY=<use-iam-roles-in-production>
|
|
||||||
REDIS_PASSWORD=<strong-password>
|
|
||||||
ENCRYPTION_KEY=<32-byte-random-key>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Runtime Validation
|
|
||||||
```typescript
|
|
||||||
import { z } from 'zod'
|
|
||||||
|
|
||||||
const EnvSchema = z.object({
|
|
||||||
NODE_ENV: z.enum(['development', 'production', 'test']),
|
|
||||||
JWT_SECRET: z.string().min(32),
|
|
||||||
DATABASE_URL: z.string().url(),
|
|
||||||
AWS_REGION: z.string(),
|
|
||||||
REDIS_HOST: z.string(),
|
|
||||||
REDIS_PORT: z.string().transform(Number),
|
|
||||||
ENCRYPTION_KEY: z.string().length(64) // Hex encoded 32 bytes
|
|
||||||
})
|
|
||||||
|
|
||||||
// Validate on startup
|
|
||||||
try {
|
|
||||||
const env = EnvSchema.parse(process.env)
|
|
||||||
console.log('✅ Environment configuration valid')
|
|
||||||
} catch (error) {
|
|
||||||
console.error('❌ Invalid environment configuration:', error)
|
|
||||||
process.exit(1)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🛠️ Security Checklist
|
|
||||||
|
|
||||||
### Development
|
|
||||||
- [ ] Use `.env` files for secrets (never commit)
|
|
||||||
- [ ] Enable TypeScript strict mode
|
|
||||||
- [ ] Run security linting (eslint-plugin-security)
|
|
||||||
- [ ] Use dependency scanning (npm audit)
|
|
||||||
- [ ] Implement unit tests for auth logic
|
|
||||||
|
|
||||||
### Staging
|
|
||||||
- [ ] Penetration testing
|
|
||||||
- [ ] Load testing with security scenarios
|
|
||||||
- [ ] Review audit logs
|
|
||||||
- [ ] Test rate limiting
|
|
||||||
- [ ] Verify encryption working
|
|
||||||
|
|
||||||
### Production
|
|
||||||
- [ ] Enable all security headers
|
|
||||||
- [ ] Configure WAF (Web Application Firewall)
|
|
||||||
- [ ] Set up intrusion detection
|
|
||||||
- [ ] Enable DDoS protection
|
|
||||||
- [ ] Configure automated backups
|
|
||||||
- [ ] Set up security alerts
|
|
||||||
- [ ] Regular security audits
|
|
||||||
- [ ] Incident response plan
|
|
||||||
|
|
||||||
## 📊 Monitoring & Alerts
|
|
||||||
|
|
||||||
### Security Metrics
|
|
||||||
```typescript
|
|
||||||
// Track and alert on:
|
|
||||||
const securityMetrics = {
|
|
||||||
failedAuthAttempts: 0,
|
|
||||||
rateLimitHits: 0,
|
|
||||||
suspiciousQueries: 0,
|
|
||||||
largeDataExports: 0,
|
|
||||||
unusualAccessPatterns: 0
|
|
||||||
}
|
|
||||||
|
|
||||||
// Alert thresholds
|
|
||||||
const alertThresholds = {
|
|
||||||
failedAuthAttempts: 10, // per minute
|
|
||||||
rateLimitHits: 100, // per minute
|
|
||||||
suspiciousQueries: 5, // per minute
|
|
||||||
largeDataExports: 10, // per hour
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🚪 Incident Response
|
|
||||||
|
|
||||||
### Response Plan
|
|
||||||
1. **Detect** - Monitoring alerts trigger
|
|
||||||
2. **Contain** - Isolate affected systems
|
|
||||||
3. **Investigate** - Review audit logs
|
|
||||||
4. **Remediate** - Fix vulnerability
|
|
||||||
5. **Recover** - Restore normal operations
|
|
||||||
6. **Review** - Post-incident analysis
|
|
||||||
|
|
||||||
### Emergency Contacts
|
|
||||||
- Security Team: security@yourcompany.com
|
|
||||||
- On-call Engineer: Use PagerDuty
|
|
||||||
- Legal Team: legal@yourcompany.com
|
|
||||||
- PR Team: pr@yourcompany.com
|
|
||||||
|
|
@ -1,291 +0,0 @@
|
||||||
# Brainy Storage Adapter - Quick Reference Guide
|
|
||||||
|
|
||||||
## File Locations
|
|
||||||
|
|
||||||
```
|
|
||||||
src/storage/
|
|
||||||
├── baseStorageAdapter.ts # Abstract base class (1,156 lines)
|
|
||||||
├── baseStorage.ts # Implementation layer (1,098 lines)
|
|
||||||
├── storageFactory.ts # Factory for adapter selection
|
|
||||||
├── sharding.ts # UUID-based sharding utilities
|
|
||||||
├── cacheManager.ts # Cache implementation
|
|
||||||
└── adapters/
|
|
||||||
├── fileSystemStorage.ts # Node.js file system (2,677 lines)
|
|
||||||
├── memoryStorage.ts # In-memory storage (822 lines)
|
|
||||||
├── s3CompatibleStorage.ts # AWS S3 / R2 / GCS compat (5000+ lines)
|
|
||||||
├── gcsStorage.ts # Google Cloud Storage native (1,835 lines)
|
|
||||||
├── opfsStorage.ts # Browser OPFS storage
|
|
||||||
└── baseStorageAdapter.ts # Base class
|
|
||||||
|
|
||||||
src/coreTypes.ts
|
|
||||||
└── StorageAdapter interface (27 methods)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Storage Adapter Hierarchy
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─ StorageAdapter (interface)
|
|
||||||
│ ├─ StorageAdapter.init()
|
|
||||||
│ ├─ StorageAdapter.saveNoun()
|
|
||||||
│ ├─ StorageAdapter.getNouns()
|
|
||||||
│ └─ ... (23 more methods)
|
|
||||||
│
|
|
||||||
└─ BaseStorageAdapter (abstract class)
|
|
||||||
├─ Statistics management
|
|
||||||
├─ Throttling detection
|
|
||||||
├─ Count tracking (O(1))
|
|
||||||
├─ Service-level statistics
|
|
||||||
└─ Abstract methods for subclasses
|
|
||||||
|
|
||||||
└─ BaseStorage (abstract class)
|
|
||||||
├─ 2-file system routing
|
|
||||||
├─ Pagination support
|
|
||||||
├─ Metadata handling
|
|
||||||
├─ Public API (saveNoun, getNoun, etc.)
|
|
||||||
└─ Abstract internal methods
|
|
||||||
|
|
||||||
└─ Concrete Adapters
|
|
||||||
├─ FileSystemStorage
|
|
||||||
├─ MemoryStorage
|
|
||||||
├─ S3CompatibleStorage
|
|
||||||
├─ GcsStorage
|
|
||||||
└─ OPFSStorage
|
|
||||||
```
|
|
||||||
|
|
||||||
## Abstract Methods to Implement
|
|
||||||
|
|
||||||
When creating a new adapter, extend `BaseStorage` and implement:
|
|
||||||
|
|
||||||
### Noun/Verb Operations (6 methods)
|
|
||||||
```typescript
|
|
||||||
protected abstract saveNoun_internal(noun: HNSWNoun): Promise<void>
|
|
||||||
protected abstract getNoun_internal(id: string): Promise<HNSWNoun | null>
|
|
||||||
protected abstract deleteNoun_internal(id: string): Promise<void>
|
|
||||||
protected abstract saveVerb_internal(verb: HNSWVerb): Promise<void>
|
|
||||||
protected abstract getVerb_internal(id: string): Promise<HNSWVerb | null>
|
|
||||||
protected abstract deleteVerb_internal(id: string): Promise<void>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Path Operations (4 methods)
|
|
||||||
```typescript
|
|
||||||
protected abstract writeObjectToPath(path: string, data: any): Promise<void>
|
|
||||||
protected abstract readObjectFromPath(path: string): Promise<any | null>
|
|
||||||
protected abstract deleteObjectFromPath(path: string): Promise<void>
|
|
||||||
protected abstract listObjectsUnderPath(prefix: string): Promise<string[]>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Count Management (2 methods)
|
|
||||||
```typescript
|
|
||||||
protected abstract initializeCounts(): Promise<void>
|
|
||||||
protected abstract persistCounts(): Promise<void>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Statistics (2 methods)
|
|
||||||
```typescript
|
|
||||||
protected abstract saveStatisticsData(statistics: StatisticsData): Promise<void>
|
|
||||||
protected abstract getStatisticsData(): Promise<StatisticsData | null>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Lifecycle (3 methods)
|
|
||||||
```typescript
|
|
||||||
abstract init(): Promise<void>
|
|
||||||
abstract clear(): Promise<void>
|
|
||||||
abstract getStorageStatus(): Promise<StorageStatus>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Total: 17 abstract methods to implement**
|
|
||||||
|
|
||||||
## Storage Path Structure
|
|
||||||
|
|
||||||
### Modern Entity-Based Structure
|
|
||||||
```
|
|
||||||
storage-root/
|
|
||||||
├── entities/
|
|
||||||
│ ├── nouns/vectors/{shard}/{id}.json ← Vector data
|
|
||||||
│ ├── nouns/metadata/{shard}/{id}.json ← Metadata
|
|
||||||
│ ├── nouns/hnsw/{shard}/{id}.json ← HNSW graph
|
|
||||||
│ ├── verbs/vectors/{shard}/{id}.json
|
|
||||||
│ ├── verbs/metadata/{shard}/{id}.json
|
|
||||||
│ └── verbs/hnsw/{shard}/{id}.json
|
|
||||||
├── indexes/
|
|
||||||
│ ├── metadata/... ← Search indexes
|
|
||||||
│ └── graph/...
|
|
||||||
└── _system/
|
|
||||||
├── statistics.json ← Aggregate stats
|
|
||||||
├── counts.json ← O(1) totals
|
|
||||||
└── hnsw-system.json ← HNSW metadata
|
|
||||||
```
|
|
||||||
|
|
||||||
### Shard Format
|
|
||||||
- First 2 hex chars of UUID (00-ff) = 256 shards
|
|
||||||
- Example: `ab123456-...` → stored in `ab/` directory
|
|
||||||
- Enables 2.5M+ entities with consistent performance
|
|
||||||
|
|
||||||
## 2-File System Design
|
|
||||||
|
|
||||||
### Vector File (always loaded with HNSW)
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"id": "ab123456-...",
|
|
||||||
"vector": [0.1, 0.2, ...],
|
|
||||||
"connections": { "0": [...], "1": [...] },
|
|
||||||
"level": 2
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Metadata File (loaded separately)
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"noun": "Person",
|
|
||||||
"name": "Alice",
|
|
||||||
"email": "alice@example.com",
|
|
||||||
"createdAt": "...",
|
|
||||||
"service": "user-service"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Benefit:** Decouple vector operations from flexible metadata queries
|
|
||||||
|
|
||||||
## Existing Adapters Overview
|
|
||||||
|
|
||||||
### FileSystemStorage (Node.js)
|
|
||||||
- 2,677 lines
|
|
||||||
- Sharding with migration support
|
|
||||||
- File-based locking for multi-process
|
|
||||||
- Production-ready
|
|
||||||
|
|
||||||
### MemoryStorage (Testing)
|
|
||||||
- 822 lines
|
|
||||||
- In-memory Maps
|
|
||||||
- Fast for testing
|
|
||||||
- No persistence
|
|
||||||
|
|
||||||
### S3CompatibleStorage (Cloud)
|
|
||||||
- 5,000+ lines
|
|
||||||
- AWS S3, Cloudflare R2, GCS (via S3 API)
|
|
||||||
- Adaptive batching, request coalescing
|
|
||||||
- High-volume mode, write buffers
|
|
||||||
|
|
||||||
### GcsStorage (Google Cloud)
|
|
||||||
- 1,835 lines
|
|
||||||
- Native @google-cloud/storage SDK
|
|
||||||
- ADC, service account, HMAC auth
|
|
||||||
- Cache managers, backpressure
|
|
||||||
|
|
||||||
### OPFSStorage (Browser)
|
|
||||||
- Browser Origin Private File System
|
|
||||||
- Persistent across sessions
|
|
||||||
- Modern browsers only
|
|
||||||
|
|
||||||
## Factory Integration
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// src/storage/storageFactory.ts
|
|
||||||
const storage = await createStorage({
|
|
||||||
type: 'filesystem', // auto, memory, filesystem, s3, gcs, gcs-native, opfs
|
|
||||||
path: './data',
|
|
||||||
s3Storage: { bucketName, region, ... },
|
|
||||||
gcsStorage: { bucketName, credentials, ... },
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
## Adding TypeAwareStorageAdapter
|
|
||||||
|
|
||||||
### Recommended Approach: Direct Implementation
|
|
||||||
```typescript
|
|
||||||
// src/storage/adapters/typeAwareStorageAdapter.ts
|
|
||||||
export class TypeAwareStorageAdapter extends BaseStorage {
|
|
||||||
// Implement 17 abstract methods
|
|
||||||
// Add type indexing logic
|
|
||||||
// Track noun/verb types in separate indexes
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Integration Steps
|
|
||||||
1. Create `/src/storage/adapters/typeAwareStorageAdapter.ts`
|
|
||||||
2. Add to factory in `/src/storage/storageFactory.ts`
|
|
||||||
3. Update StorageOptions interface with `type: 'type-aware'`
|
|
||||||
4. No changes to Brainy.ts or existing adapters needed
|
|
||||||
|
|
||||||
## Key Features Inherited from BaseStorageAdapter
|
|
||||||
|
|
||||||
- **Statistics Caching:** Batches updates for efficiency
|
|
||||||
- **Throttling Detection:** Handles 429/503 errors
|
|
||||||
- **Count Management:** O(1) operations with persistence
|
|
||||||
- **Service Tracking:** Per-service statistics
|
|
||||||
- **Field Name Tracking:** Metadata field discovery
|
|
||||||
|
|
||||||
## Performance Characteristics
|
|
||||||
|
|
||||||
### O(1) Operations
|
|
||||||
- `getNounCount()` - total noun count
|
|
||||||
- `getVerbCount()` - total verb count
|
|
||||||
|
|
||||||
### O(n) Operations
|
|
||||||
- `getNouns()` - paginated listing (n = page size)
|
|
||||||
- `getVerbs()` - paginated listing
|
|
||||||
- `getNounsByNounType()` - filter by type
|
|
||||||
- `getVerbsBySource()` - filter by source
|
|
||||||
|
|
||||||
### Cloud Storage Features (GCS, S3)
|
|
||||||
- High-volume mode detection
|
|
||||||
- Adaptive batching
|
|
||||||
- Request coalescing for deduplication
|
|
||||||
- Write buffers for bulk operations
|
|
||||||
- Backpressure management
|
|
||||||
- Socket pool management
|
|
||||||
|
|
||||||
## Testing Storage Adapters
|
|
||||||
|
|
||||||
All adapters implement the same interface, so:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Test with MemoryStorage (fastest)
|
|
||||||
const storage = new MemoryStorage()
|
|
||||||
|
|
||||||
// Test with FileSystemStorage (persistent)
|
|
||||||
const storage = new FileSystemStorage('./test-data')
|
|
||||||
|
|
||||||
// All adapters support the same operations
|
|
||||||
await storage.init()
|
|
||||||
await storage.saveNoun(noun)
|
|
||||||
const result = await storage.getNoun(id)
|
|
||||||
await storage.clear()
|
|
||||||
```
|
|
||||||
|
|
||||||
## Brainy Integration
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
export class Brainy {
|
|
||||||
private storage!: BaseStorage
|
|
||||||
|
|
||||||
async init(config: BrainyConfig): Promise<void> {
|
|
||||||
// Factory creates appropriate adapter
|
|
||||||
this.storage = await createStorage(config.storage) as BaseStorage
|
|
||||||
await this.storage.init()
|
|
||||||
|
|
||||||
// Pass to HNSW index
|
|
||||||
this.index = new HNSWIndex(this.storage, ...)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Brainy depends on `BaseStorage` interface, not specific adapters.
|
|
||||||
|
|
||||||
## Design Patterns Used
|
|
||||||
|
|
||||||
1. **Factory Pattern** - `createStorage()` selects adapter
|
|
||||||
2. **Strategy Pattern** - Adapters are interchangeable
|
|
||||||
3. **Template Method** - BaseStorage defines skeleton
|
|
||||||
4. **Decorator Pattern** - Can wrap adapters (e.g., TypeAware wrapper)
|
|
||||||
5. **Adapter Pattern** - Maps different storage backends to same interface
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
TypeAwareStorageAdapter can be added as a **new adapter alongside existing ones** without:
|
|
||||||
- Modifying Brainy.ts
|
|
||||||
- Replacing existing adapters
|
|
||||||
- Breaking the StorageAdapter interface
|
|
||||||
- Changing how storage is used throughout the codebase
|
|
||||||
|
|
||||||
Simply extend `BaseStorage`, implement 17 abstract methods, and register in `storageFactory.ts`.
|
|
||||||
|
|
@ -1,431 +0,0 @@
|
||||||
# Brainy Storage System - Complete File Reference
|
|
||||||
|
|
||||||
## Core Storage Files
|
|
||||||
|
|
||||||
### 1. Storage Interface Definition
|
|
||||||
**File:** `src/coreTypes.ts`
|
|
||||||
- **Lines:** ~250 (StorageAdapter interface)
|
|
||||||
- **Type:** Interface definition
|
|
||||||
- **Content:**
|
|
||||||
- `StorageAdapter` interface (27 methods)
|
|
||||||
- Supporting types: `HNSWNoun`, `HNSWVerb`, `GraphVerb`
|
|
||||||
- Statistics types: `StatisticsData`, `ServiceStatistics`
|
|
||||||
|
|
||||||
### 2. Base Storage Adapter (Abstract Class)
|
|
||||||
**File:** `src/storage/adapters/baseStorageAdapter.ts`
|
|
||||||
- **Lines:** 1,156
|
|
||||||
- **Type:** Abstract base class
|
|
||||||
- **Purpose:** Common functionality for all adapters
|
|
||||||
- **Key Features:**
|
|
||||||
- Statistics caching and batching
|
|
||||||
- Throttling detection and backoff
|
|
||||||
- Count management (O(1) operations)
|
|
||||||
- Service-level statistics tracking
|
|
||||||
- Field name discovery
|
|
||||||
- **Implements:** StorageAdapter interface
|
|
||||||
- **Key Methods:**
|
|
||||||
- `flushStatistics()` - Write statistics to storage
|
|
||||||
- `isThrottlingError()` - Detect cloud storage throttling
|
|
||||||
- `handleThrottling()` - Exponential backoff
|
|
||||||
- `trackThrottlingEvent()` - Record throttling events
|
|
||||||
- `incrementStatistic()` - Increment service stats
|
|
||||||
- `decrementStatistic()` - Decrement service stats
|
|
||||||
|
|
||||||
### 3. Base Storage Implementation
|
|
||||||
**File:** `src/storage/baseStorage.ts`
|
|
||||||
- **Lines:** 1,098
|
|
||||||
- **Type:** Abstract class extending BaseStorageAdapter
|
|
||||||
- **Purpose:** Core storage logic (routing, sharding, metadata)
|
|
||||||
- **Key Features:**
|
|
||||||
- 2-file system implementation (vectors + metadata)
|
|
||||||
- UUID-based sharding (256 directories)
|
|
||||||
- Metadata routing and separation
|
|
||||||
- Pagination support
|
|
||||||
- Backward compatibility with legacy paths
|
|
||||||
- **Implements:**
|
|
||||||
- Public API (saveNoun, getNoun, etc.)
|
|
||||||
- Metadata operations (saveNounMetadata, etc.)
|
|
||||||
- **Key Abstract Methods:**
|
|
||||||
- `saveNoun_internal()` - Adapter-specific noun save
|
|
||||||
- `getNoun_internal()` - Adapter-specific noun read
|
|
||||||
- `writeObjectToPath()` - Generic write
|
|
||||||
- `readObjectFromPath()` - Generic read
|
|
||||||
- `listObjectsUnderPath()` - Listing support
|
|
||||||
|
|
||||||
### 4. Storage Factory
|
|
||||||
**File:** `src/storage/storageFactory.ts`
|
|
||||||
- **Lines:** ~200
|
|
||||||
- **Purpose:** Factory function for adapter selection
|
|
||||||
- **Function:** `createStorage(options: StorageOptions): Promise<StorageAdapter>`
|
|
||||||
- **Selection Logic:**
|
|
||||||
1. Forced memory/filesystem (testing)
|
|
||||||
2. Explicit type selection
|
|
||||||
3. Auto-detection (browser vs Node.js)
|
|
||||||
- **Supported Types:**
|
|
||||||
- `'memory'` - MemoryStorage
|
|
||||||
- `'filesystem'` - FileSystemStorage
|
|
||||||
- `'s3'` - S3CompatibleStorage (AWS S3, R2)
|
|
||||||
- `'gcs'` - S3CompatibleStorage (GCS S3 API)
|
|
||||||
- `'gcs-native'` - GcsStorage (native SDK)
|
|
||||||
- `'opfs'` - OPFSStorage (browser)
|
|
||||||
|
|
||||||
### 5. Sharding Utilities
|
|
||||||
**File:** `src/storage/sharding.ts`
|
|
||||||
- **Purpose:** UUID-based sharding helpers
|
|
||||||
- **Key Functions:**
|
|
||||||
- `getShardIdFromUuid(id: string): string` - Get first 2 hex chars
|
|
||||||
- `getShardIdByIndex(index: number): string` - Get shard by index
|
|
||||||
- `getAllShardIds(): string[]` - All 256 shard IDs
|
|
||||||
- **Constants:**
|
|
||||||
- `TOTAL_SHARDS = 256`
|
|
||||||
- `MIN_SHARD_ID = '00'`
|
|
||||||
- `MAX_SHARD_ID = 'ff'`
|
|
||||||
|
|
||||||
### 6. Cache Manager
|
|
||||||
**File:** `src/storage/cacheManager.ts`
|
|
||||||
- **Purpose:** Generic LRU cache for storage
|
|
||||||
- **Type:** Generic class `CacheManager<T>`
|
|
||||||
- **Used By:** GcsStorage, S3CompatibleStorage
|
|
||||||
- **Features:**
|
|
||||||
- LRU eviction
|
|
||||||
- TTL support
|
|
||||||
- Max size limits
|
|
||||||
- Batch operations
|
|
||||||
|
|
||||||
## Storage Adapter Implementations
|
|
||||||
|
|
||||||
### FileSystemStorage (Node.js)
|
|
||||||
**File:** `src/storage/adapters/fileSystemStorage.ts`
|
|
||||||
- **Lines:** 2,677
|
|
||||||
- **Type:** Concrete implementation extending BaseStorage
|
|
||||||
- **Platform:** Node.js only
|
|
||||||
- **Storage Target:** Local file system
|
|
||||||
- **Key Features:**
|
|
||||||
- Sharding with automatic migration
|
|
||||||
- File-based locking (multi-process)
|
|
||||||
- O(1) count persistence
|
|
||||||
- HNSW index persistence
|
|
||||||
- Dual-write for migrations
|
|
||||||
- Path depth migration (0→1, 2→1)
|
|
||||||
|
|
||||||
**Key Methods:**
|
|
||||||
```typescript
|
|
||||||
private getNounPath(id: string, depth: number): string
|
|
||||||
private getVerbPath(id: string, depth: number): string
|
|
||||||
private getNounMetadataPath(id: string, depth: number): string
|
|
||||||
async migrateShardingStructure(fromDepth: number, toDepth: number): Promise<void>
|
|
||||||
async migrateFromOldStructure(): Promise<void>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Statistics Tracking:**
|
|
||||||
- Persists counts to `_system/counts.json`
|
|
||||||
- Tracks noun/verb type distributions
|
|
||||||
- Service-level activity timestamps
|
|
||||||
- Field name discovery
|
|
||||||
|
|
||||||
### MemoryStorage (Testing)
|
|
||||||
**File:** `src/storage/adapters/memoryStorage.ts`
|
|
||||||
- **Lines:** 822
|
|
||||||
- **Type:** Concrete implementation extending BaseStorage
|
|
||||||
- **Platform:** Browser & Node.js
|
|
||||||
- **Storage Target:** In-memory Maps
|
|
||||||
- **Key Features:**
|
|
||||||
- Fast for testing
|
|
||||||
- No persistence (ephemeral)
|
|
||||||
- Full pagination support
|
|
||||||
- Filtering capabilities
|
|
||||||
- 2-file system simulation
|
|
||||||
|
|
||||||
**Key Maps:**
|
|
||||||
```typescript
|
|
||||||
private nouns: Map<string, HNSWNoun>
|
|
||||||
private verbs: Map<string, HNSWVerb>
|
|
||||||
private objectStore: Map<string, any> // Unified metadata store
|
|
||||||
```
|
|
||||||
|
|
||||||
**Methods:**
|
|
||||||
- `getNouns()` - Paginated noun listing
|
|
||||||
- `getVerbs()` - Paginated verb listing
|
|
||||||
- `getMetadataBatch()` - Batch metadata loading
|
|
||||||
|
|
||||||
### S3CompatibleStorage (Cloud)
|
|
||||||
**File:** `src/storage/adapters/s3CompatibleStorage.ts`
|
|
||||||
- **Lines:** 5,000+
|
|
||||||
- **Type:** Concrete implementation extending BaseStorage
|
|
||||||
- **Platform:** Node.js (server-side)
|
|
||||||
- **Storage Targets:**
|
|
||||||
- Amazon S3
|
|
||||||
- Cloudflare R2 (via S3 API)
|
|
||||||
- Google Cloud Storage (via S3 API)
|
|
||||||
|
|
||||||
**Key Features:**
|
|
||||||
- Adaptive batching (10-1000 items)
|
|
||||||
- Request coalescing for deduplication
|
|
||||||
- High-volume mode detection
|
|
||||||
- Write buffers for bulk operations
|
|
||||||
- Socket pool management
|
|
||||||
- Backpressure system
|
|
||||||
- Change log tracking
|
|
||||||
- Cache management (nouns & verbs)
|
|
||||||
|
|
||||||
**Performance Optimization:**
|
|
||||||
- `nounWriteBuffer` - Batches noun writes
|
|
||||||
- `verbWriteBuffer` - Batches verb writes
|
|
||||||
- `requestCoalescer` - Deduplicates requests
|
|
||||||
- `highVolumeMode` - Activates at >20 pending ops
|
|
||||||
- `baseBatchSize` - Adaptive from 10 to 1000
|
|
||||||
|
|
||||||
### GcsStorage (Google Cloud Native)
|
|
||||||
**File:** `src/storage/adapters/gcsStorage.ts`
|
|
||||||
- **Lines:** 1,835
|
|
||||||
- **Type:** Concrete implementation extending BaseStorage
|
|
||||||
- **Platform:** Node.js (server-side)
|
|
||||||
- **Storage Target:** Google Cloud Storage
|
|
||||||
- **SDK:** `@google-cloud/storage` (native)
|
|
||||||
|
|
||||||
**Key Features:**
|
|
||||||
- Application Default Credentials (ADC)
|
|
||||||
- Service Account Key File support
|
|
||||||
- Service Account Credentials Object
|
|
||||||
- HMAC Keys (backward compatible)
|
|
||||||
- Multi-level cache managers
|
|
||||||
- Backpressure management
|
|
||||||
- High-volume mode
|
|
||||||
- Request coalescing
|
|
||||||
|
|
||||||
**Authentication Priority:**
|
|
||||||
1. ADC (Application Default Credentials)
|
|
||||||
2. Service Account Key File
|
|
||||||
3. Service Account Credentials
|
|
||||||
4. HMAC Keys
|
|
||||||
|
|
||||||
**Cache Managers:**
|
|
||||||
```typescript
|
|
||||||
private nounCacheManager: CacheManager<HNSWNode>
|
|
||||||
private verbCacheManager: CacheManager<Edge>
|
|
||||||
```
|
|
||||||
|
|
||||||
### OPFSStorage (Browser Storage)
|
|
||||||
**File:** `src/storage/adapters/opfsStorage.ts`
|
|
||||||
- **Type:** Concrete implementation extending BaseStorage
|
|
||||||
- **Platform:** Browser only
|
|
||||||
- **Storage Target:** Origin Private File System (OPFS)
|
|
||||||
- **Fallback:** MemoryStorage if OPFS unavailable
|
|
||||||
|
|
||||||
**Browser Compatibility:**
|
|
||||||
- Chrome 96+
|
|
||||||
- Edge 96+
|
|
||||||
- Safari 15.1+
|
|
||||||
|
|
||||||
## Storage Patterns and Utilities
|
|
||||||
|
|
||||||
### Backward Compatibility
|
|
||||||
**File:** `src/storage/backwardCompatibility.ts`
|
|
||||||
- **Purpose:** Handle legacy storage paths
|
|
||||||
- **Features:**
|
|
||||||
- Path migration detection
|
|
||||||
- Dual-write during transition
|
|
||||||
- Graceful fallback to old locations
|
|
||||||
- Read-from-new, fallback-to-old
|
|
||||||
|
|
||||||
### Metadata Index
|
|
||||||
**File:** `src/utils/metadataIndex.ts`
|
|
||||||
- **Purpose:** Build searchable indexes from metadata
|
|
||||||
- **Used By:** Brainy for fast metadata queries
|
|
||||||
- **Features:**
|
|
||||||
- Field name discovery
|
|
||||||
- Standard field mapping
|
|
||||||
- Service-level statistics
|
|
||||||
|
|
||||||
### Storage Discovery (Distributed)
|
|
||||||
**File:** `src/distributed/storageDiscovery.ts`
|
|
||||||
- **Purpose:** Discover storage config in distributed systems
|
|
||||||
- **Features:**
|
|
||||||
- Node coordination
|
|
||||||
- Storage synchronization
|
|
||||||
|
|
||||||
### Adaptive Backpressure
|
|
||||||
**File:** `src/utils/adaptiveBackpressure.ts`
|
|
||||||
- **Purpose:** Flow control for storage operations
|
|
||||||
- **Used By:** All cloud storage adapters
|
|
||||||
- **Features:**
|
|
||||||
- Request queuing
|
|
||||||
- Throttling detection
|
|
||||||
- Backoff scheduling
|
|
||||||
|
|
||||||
### Write Buffer
|
|
||||||
**File:** `src/utils/writeBuffer.ts`
|
|
||||||
- **Purpose:** Batch write operations
|
|
||||||
- **Used By:** S3, GCS adapters
|
|
||||||
- **Features:**
|
|
||||||
- Configurable batch size
|
|
||||||
- Flush on timeout
|
|
||||||
- Deduplication
|
|
||||||
|
|
||||||
### Request Coalescer
|
|
||||||
**File:** `src/utils/requestCoalescer.ts`
|
|
||||||
- **Purpose:** Deduplicate concurrent requests
|
|
||||||
- **Used By:** S3, GCS adapters
|
|
||||||
- **Features:**
|
|
||||||
- Request deduplication
|
|
||||||
- Batch processing
|
|
||||||
|
|
||||||
## Integration Points
|
|
||||||
|
|
||||||
### In Brainy.ts (Main Class)
|
|
||||||
```typescript
|
|
||||||
private storage!: BaseStorage
|
|
||||||
|
|
||||||
async init(): Promise<void> {
|
|
||||||
// Create storage from factory
|
|
||||||
const storageAdapter = await createStorage(this.config.storage)
|
|
||||||
this.storage = storageAdapter as BaseStorage
|
|
||||||
|
|
||||||
// Initialize
|
|
||||||
await this.storage.init()
|
|
||||||
|
|
||||||
// Pass to HNSW index
|
|
||||||
this.index = new HNSWIndex(this.storage, ...)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### In HNSW Index
|
|
||||||
```typescript
|
|
||||||
export class HNSWIndex {
|
|
||||||
constructor(private storage: StorageAdapter, ...)
|
|
||||||
|
|
||||||
// Uses storage for node/edge persistence
|
|
||||||
async saveNode(noun: HNSWNoun): Promise<void>
|
|
||||||
async getNode(id: string): Promise<HNSWNoun>
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### In Metadata Index
|
|
||||||
```typescript
|
|
||||||
export class MetadataIndexManager {
|
|
||||||
constructor(storage: StorageAdapter, ...)
|
|
||||||
|
|
||||||
// Uses storage for metadata queries
|
|
||||||
async getNounsByFilter(filter: Filter): Promise<HNSWNoun[]>
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Data Flow
|
|
||||||
|
|
||||||
### Saving a Noun
|
|
||||||
```
|
|
||||||
Brainy.add()
|
|
||||||
↓
|
|
||||||
HNSWIndex.insert()
|
|
||||||
↓
|
|
||||||
BaseStorage.saveNoun()
|
|
||||||
├─ saveNoun_internal() → adapter-specific
|
|
||||||
├─ saveNounMetadata() → path routing
|
|
||||||
└─ updateStatistics()
|
|
||||||
|
|
||||||
FileSystemStorage.saveNoun_internal()
|
|
||||||
├─ Create shard directory (ab/)
|
|
||||||
├─ Write JSON file
|
|
||||||
└─ Update counts
|
|
||||||
```
|
|
||||||
|
|
||||||
### Querying Nouns
|
|
||||||
```
|
|
||||||
Brainy.search()
|
|
||||||
↓
|
|
||||||
HNSW.search()
|
|
||||||
↓
|
|
||||||
BaseStorage.getNoun()
|
|
||||||
├─ getNoun_internal() → adapter-specific
|
|
||||||
└─ getNounMetadata() → path routing
|
|
||||||
|
|
||||||
S3CompatibleStorage.getNoun_internal()
|
|
||||||
├─ Check cache
|
|
||||||
├─ Download from S3
|
|
||||||
├─ Parse JSON
|
|
||||||
└─ Update cache
|
|
||||||
```
|
|
||||||
|
|
||||||
## Storage Statistics Tracking
|
|
||||||
|
|
||||||
### Stored in `_system/statistics.json`
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"nounCount": { "Person": 5, "Company": 2 },
|
|
||||||
"verbCount": { "knows": 10, "works_at": 3 },
|
|
||||||
"metadataCount": { "user-service": 50 },
|
|
||||||
"hnswIndexSize": 15,
|
|
||||||
"totalNodes": 7,
|
|
||||||
"totalEdges": 13,
|
|
||||||
"services": [
|
|
||||||
{ "name": "user-service", "totalNouns": 5, ... }
|
|
||||||
],
|
|
||||||
"lastUpdated": "2024-10-15T..."
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Stored in `_system/counts.json`
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"totalNounCount": 7,
|
|
||||||
"totalVerbCount": 13,
|
|
||||||
"entityCounts": { "Person": 5, "Company": 2 },
|
|
||||||
"verbCounts": { "knows": 10, "works_at": 3 }
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Type Definitions
|
|
||||||
|
|
||||||
### HNSWNoun (Vector + HNSW)
|
|
||||||
```typescript
|
|
||||||
interface HNSWNoun {
|
|
||||||
id: string
|
|
||||||
vector: number[]
|
|
||||||
connections: Map<number, Set<string>> // level → node IDs
|
|
||||||
level: number
|
|
||||||
metadata?: any // Optional in vector file, separate file system
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### GraphVerb (Relationship)
|
|
||||||
```typescript
|
|
||||||
interface GraphVerb {
|
|
||||||
id: string
|
|
||||||
sourceId: string
|
|
||||||
targetId: string
|
|
||||||
vector: number[]
|
|
||||||
type?: string
|
|
||||||
weight?: number
|
|
||||||
metadata?: any
|
|
||||||
// Plus aliases: source, target, verb, embedding
|
|
||||||
createdAt?: Timestamp
|
|
||||||
createdBy?: { augmentation: string; version: string }
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Summary Statistics
|
|
||||||
|
|
||||||
| Component | File | Lines | Purpose |
|
|
||||||
|-----------|------|-------|---------|
|
|
||||||
| StorageAdapter Interface | coreTypes.ts | 250 | Interface definition |
|
|
||||||
| BaseStorageAdapter | baseStorageAdapter.ts | 1,156 | Common functionality |
|
|
||||||
| BaseStorage | baseStorage.ts | 1,098 | Core routing & pagination |
|
|
||||||
| storageFactory | storageFactory.ts | 200 | Adapter selection |
|
|
||||||
| FileSystemStorage | fileSystemStorage.ts | 2,677 | Node.js FS |
|
|
||||||
| MemoryStorage | memoryStorage.ts | 822 | In-memory (test) |
|
|
||||||
| S3CompatibleStorage | s3CompatibleStorage.ts | 5,000+ | AWS S3 / R2 / GCS |
|
|
||||||
| GcsStorage | gcsStorage.ts | 1,835 | Google Cloud native |
|
|
||||||
| sharding.ts | sharding.ts | ~100 | UUID sharding |
|
|
||||||
| cacheManager.ts | cacheManager.ts | ~200 | LRU cache |
|
|
||||||
| **TOTAL** | | **~13,000+** | **Complete storage system** |
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
Brainy's storage architecture is:
|
|
||||||
- Well-layered (Interface → Abstract → Concrete)
|
|
||||||
- Extensible (factory pattern)
|
|
||||||
- Flexible (multiple backends)
|
|
||||||
- Scalable (sharding, caching, batching)
|
|
||||||
- Type-safe (full TypeScript support)
|
|
||||||
|
|
||||||
New adapters like TypeAwareStorageAdapter simply extend BaseStorage and implement 17 abstract methods.
|
|
||||||
194
scripts/diagnose-workshop-issue.ts
Normal file
194
scripts/diagnose-workshop-issue.ts
Normal file
|
|
@ -0,0 +1,194 @@
|
||||||
|
/**
|
||||||
|
* Diagnostic script for Workshop type filtering issue
|
||||||
|
*
|
||||||
|
* This script mimics Workshop's exact import pattern to see what's happening
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { Brainy, NounType } from '../src/index.js'
|
||||||
|
import * as fs from 'fs'
|
||||||
|
import * as path from 'path'
|
||||||
|
|
||||||
|
async function diagnose() {
|
||||||
|
console.log('\n🔬 Workshop Type Filtering Diagnostic\n')
|
||||||
|
console.log('='.repeat(70))
|
||||||
|
|
||||||
|
// Use temporary directory
|
||||||
|
const testDir = './test-workshop-data'
|
||||||
|
if (fs.existsSync(testDir)) {
|
||||||
|
fs.rmSync(testDir, { recursive: true })
|
||||||
|
}
|
||||||
|
|
||||||
|
const brain = new Brainy({
|
||||||
|
storage: {
|
||||||
|
type: 'filesystem',
|
||||||
|
path: testDir
|
||||||
|
}
|
||||||
|
})
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
|
console.log('\n1️⃣ Testing WITHOUT VFS (control group)...\n')
|
||||||
|
|
||||||
|
// Add entities without VFS
|
||||||
|
console.log('Adding 3 person entities directly...')
|
||||||
|
await brain.add({ data: 'Person 1', type: NounType.Person, metadata: { name: 'Person 1' } })
|
||||||
|
await brain.add({ data: 'Person 2', type: NounType.Person, metadata: { name: 'Person 2' } })
|
||||||
|
await brain.add({ data: 'Person 3', type: NounType.Person, metadata: { name: 'Person 3' } })
|
||||||
|
|
||||||
|
console.log('Adding 2 location entities directly...')
|
||||||
|
await brain.add({ data: 'Location 1', type: NounType.Location, metadata: { name: 'Location 1' } })
|
||||||
|
await brain.add({ data: 'Location 2', type: NounType.Location, metadata: { name: 'Location 2' } })
|
||||||
|
|
||||||
|
console.log('\n📊 Testing type filtering on direct entities...\n')
|
||||||
|
|
||||||
|
const allDirect = await brain.find({ limit: 100 })
|
||||||
|
console.log(` Total entities: ${allDirect.length}`)
|
||||||
|
|
||||||
|
const peopleDirect = await brain.find({ type: NounType.Person, limit: 100 })
|
||||||
|
console.log(` Person filter: ${peopleDirect.length} (expected: 3)`)
|
||||||
|
|
||||||
|
const locationsDirect = await brain.find({ type: NounType.Location, limit: 100 })
|
||||||
|
console.log(` Location filter: ${locationsDirect.length} (expected: 2)`)
|
||||||
|
|
||||||
|
if (peopleDirect.length === 3 && locationsDirect.length === 2) {
|
||||||
|
console.log('\n ✅ Type filtering works on direct entities!')
|
||||||
|
} else {
|
||||||
|
console.log('\n ❌ Type filtering BROKEN on direct entities!')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('\n' + '='.repeat(70))
|
||||||
|
console.log('\n2️⃣ Testing WITH VFS (Workshop pattern)...\n')
|
||||||
|
|
||||||
|
// Simulate Workshop's pattern: create VFS file entities
|
||||||
|
console.log('Creating VFS file wrappers (like import does)...')
|
||||||
|
|
||||||
|
const vfs = brain.vfs()
|
||||||
|
await vfs.init()
|
||||||
|
|
||||||
|
// Create VFS directory
|
||||||
|
await vfs.mkdir('/imports/test', { recursive: true })
|
||||||
|
|
||||||
|
// Create VFS files with embedded entity data (mimics import)
|
||||||
|
console.log('Creating VFS file for Person entity...')
|
||||||
|
const personEntityData = {
|
||||||
|
id: 'ent_person_test',
|
||||||
|
name: 'John Smith',
|
||||||
|
type: 'person',
|
||||||
|
metadata: {
|
||||||
|
source: 'excel',
|
||||||
|
originalData: {
|
||||||
|
Name: 'John Smith',
|
||||||
|
_sheet: 'Characters'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
await vfs.writeFile(
|
||||||
|
'/imports/test/john_smith.json',
|
||||||
|
Buffer.from(JSON.stringify(personEntityData, null, 2))
|
||||||
|
)
|
||||||
|
|
||||||
|
console.log('Creating VFS file for Location entity...')
|
||||||
|
const locationEntityData = {
|
||||||
|
id: 'ent_location_test',
|
||||||
|
name: 'New York',
|
||||||
|
type: 'location',
|
||||||
|
metadata: {
|
||||||
|
source: 'excel',
|
||||||
|
originalData: {
|
||||||
|
Name: 'New York',
|
||||||
|
_sheet: 'Places'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
await vfs.writeFile(
|
||||||
|
'/imports/test/new_york.json',
|
||||||
|
Buffer.from(JSON.stringify(locationEntityData, null, 2))
|
||||||
|
)
|
||||||
|
|
||||||
|
console.log('\n📊 Checking what entities exist now...\n')
|
||||||
|
|
||||||
|
const allWithVfs = await brain.find({ limit: 100 })
|
||||||
|
console.log(` Total entities: ${allWithVfs.length}`)
|
||||||
|
|
||||||
|
// Analyze entity types
|
||||||
|
const typeCounts: Record<string, number> = {}
|
||||||
|
for (const result of allWithVfs) {
|
||||||
|
const type = result.type || 'unknown'
|
||||||
|
typeCounts[type] = (typeCounts[type] || 0) + 1
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('\n Entity types breakdown:')
|
||||||
|
for (const [type, count] of Object.entries(typeCounts)) {
|
||||||
|
console.log(` - ${type}: ${count}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('\n📊 Testing type filtering with VFS entities...\n')
|
||||||
|
|
||||||
|
const peopleWithVfs = await brain.find({ type: NounType.Person, limit: 100 })
|
||||||
|
console.log(` Person filter: ${peopleWithVfs.length} (expected: 3 direct + maybe VFS?)`)
|
||||||
|
|
||||||
|
const locationsWithVfs = await brain.find({ type: NounType.Location, limit: 100 })
|
||||||
|
console.log(` Location filter: ${locationsWithVfs.length} (expected: 2 direct + maybe VFS?)`)
|
||||||
|
|
||||||
|
const documents = await brain.find({ type: NounType.Document, limit: 100 })
|
||||||
|
console.log(` Document filter: ${documents.length} (VFS wrappers?)`)
|
||||||
|
|
||||||
|
console.log('\n' + '='.repeat(70))
|
||||||
|
console.log('\n3️⃣ Analyzing VFS wrapper structure...\n')
|
||||||
|
|
||||||
|
// Get a VFS wrapper entity
|
||||||
|
const vfsWrapper = allWithVfs.find(e => e.metadata?.vfsType === 'file')
|
||||||
|
if (vfsWrapper) {
|
||||||
|
console.log('Found VFS wrapper entity:')
|
||||||
|
console.log(` - ID: ${vfsWrapper.id}`)
|
||||||
|
console.log(` - Type: ${vfsWrapper.type}`)
|
||||||
|
console.log(` - VFS Type: ${vfsWrapper.metadata?.vfsType}`)
|
||||||
|
console.log(` - Has rawData: ${!!vfsWrapper.metadata?.rawData}`)
|
||||||
|
console.log(` - Path: ${vfsWrapper.metadata?.path}`)
|
||||||
|
|
||||||
|
if (vfsWrapper.metadata?.rawData) {
|
||||||
|
console.log('\n Decoding rawData...')
|
||||||
|
try {
|
||||||
|
const decoded = Buffer.from(vfsWrapper.metadata.rawData, 'base64').toString()
|
||||||
|
const entity = JSON.parse(decoded)
|
||||||
|
console.log(` - Embedded entity name: ${entity.name}`)
|
||||||
|
console.log(` - Embedded entity type: ${entity.type}`)
|
||||||
|
console.log(` - Wrapper type: ${vfsWrapper.type}`)
|
||||||
|
console.log('\n 🔍 KEY INSIGHT:')
|
||||||
|
console.log(` Wrapper has type="${vfsWrapper.type}"`)
|
||||||
|
console.log(` But embedded entity has type="${entity.type}"`)
|
||||||
|
console.log(' When you filter by person, you get the WRAPPER type, not embedded type!')
|
||||||
|
} catch (err) {
|
||||||
|
console.log(' ❌ Failed to decode rawData')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
console.log('No VFS wrapper entities found')
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('\n' + '='.repeat(70))
|
||||||
|
console.log('\n4️⃣ DIAGNOSIS\n')
|
||||||
|
|
||||||
|
if (documents.length > 0 && peopleWithVfs.length === 3) {
|
||||||
|
console.log('❌ FOUND THE BUG!')
|
||||||
|
console.log('')
|
||||||
|
console.log('VFS creates document wrappers with type="document".')
|
||||||
|
console.log('The actual entity data is stored as base64 in metadata.rawData.')
|
||||||
|
console.log('When you filter by type="person", you\'re filtering the WRAPPER type.')
|
||||||
|
console.log('Since wrappers are type="document", you get 0 results.')
|
||||||
|
console.log('')
|
||||||
|
console.log('This is a DESIGN ISSUE in how VFS import works!')
|
||||||
|
} else if (peopleWithVfs.length > 3) {
|
||||||
|
console.log('✅ Type filtering works correctly!')
|
||||||
|
console.log('VFS entities are being created with proper types.')
|
||||||
|
} else {
|
||||||
|
console.log('🤔 Unclear - need more investigation')
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('\n' + '='.repeat(70) + '\n')
|
||||||
|
|
||||||
|
// Cleanup
|
||||||
|
fs.rmSync(testDir, { recursive: true })
|
||||||
|
}
|
||||||
|
|
||||||
|
diagnose().catch(console.error)
|
||||||
|
|
@ -1239,6 +1239,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
if (params.where) Object.assign(filter, params.where)
|
if (params.where) Object.assign(filter, params.where)
|
||||||
if (params.service) filter.service = params.service
|
if (params.service) filter.service = params.service
|
||||||
|
|
||||||
|
// v4.3.3: Exclude VFS entities by default (Option 3C architecture)
|
||||||
|
// Only include VFS if explicitly requested via includeVFS: true
|
||||||
|
// BUT: Don't add automatic exclusion if user explicitly queries isVFS in where clause
|
||||||
|
if (params.includeVFS !== true && !params.where?.hasOwnProperty('isVFS')) {
|
||||||
|
filter.isVFS = { notEquals: true }
|
||||||
|
}
|
||||||
|
|
||||||
if (params.type) {
|
if (params.type) {
|
||||||
const types = Array.isArray(params.type) ? params.type : [params.type]
|
const types = Array.isArray(params.type) ? params.type : [params.type]
|
||||||
if (types.length === 1) {
|
if (types.length === 1) {
|
||||||
|
|
@ -1275,15 +1282,35 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
const limit = params.limit || 20
|
const limit = params.limit || 20
|
||||||
const offset = params.offset || 0
|
const offset = params.offset || 0
|
||||||
|
|
||||||
const storageResults = await this.storage.getNouns({
|
// v4.3.3: Apply VFS filtering even for empty queries
|
||||||
pagination: { limit: limit + offset, offset: 0 }
|
let filter: any = {}
|
||||||
})
|
if (params.includeVFS !== true) {
|
||||||
|
filter.isVFS = { notEquals: true }
|
||||||
|
}
|
||||||
|
|
||||||
for (let i = offset; i < Math.min(offset + limit, storageResults.items.length); i++) {
|
// Use metadata index if we need to filter VFS
|
||||||
const noun = storageResults.items[i]
|
if (Object.keys(filter).length > 0) {
|
||||||
if (noun) {
|
const filteredIds = await this.metadataIndex.getIdsForFilter(filter)
|
||||||
const entity = await this.convertNounToEntity(noun)
|
const pageIds = filteredIds.slice(offset, offset + limit)
|
||||||
results.push(this.createResult(noun.id, 1.0, entity))
|
|
||||||
|
for (const id of pageIds) {
|
||||||
|
const entity = await this.get(id)
|
||||||
|
if (entity) {
|
||||||
|
results.push(this.createResult(id, 1.0, entity))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// No filtering needed, use direct storage query
|
||||||
|
const storageResults = await this.storage.getNouns({
|
||||||
|
pagination: { limit: limit + offset, offset: 0 }
|
||||||
|
})
|
||||||
|
|
||||||
|
for (let i = offset; i < Math.min(offset + limit, storageResults.items.length); i++) {
|
||||||
|
const noun = storageResults.items[i]
|
||||||
|
if (noun) {
|
||||||
|
const entity = await this.convertNounToEntity(noun)
|
||||||
|
results.push(this.createResult(noun.id, 1.0, entity))
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1324,14 +1351,20 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
}
|
}
|
||||||
|
|
||||||
// Apply O(log n) metadata filtering using core MetadataIndexManager
|
// Apply O(log n) metadata filtering using core MetadataIndexManager
|
||||||
if (params.where || params.type || params.service) {
|
if (params.where || params.type || params.service || params.includeVFS !== true) {
|
||||||
// Build filter object for metadata index
|
// Build filter object for metadata index
|
||||||
let filter: any = {}
|
let filter: any = {}
|
||||||
|
|
||||||
// Base filter from where and service
|
// Base filter from where and service
|
||||||
if (params.where) Object.assign(filter, params.where)
|
if (params.where) Object.assign(filter, params.where)
|
||||||
if (params.service) filter.service = params.service
|
if (params.service) filter.service = params.service
|
||||||
|
|
||||||
|
// v4.3.3: Exclude VFS entities by default (Option 3C architecture)
|
||||||
|
// BUT: Don't add automatic exclusion if user explicitly queries isVFS in where clause
|
||||||
|
if (params.includeVFS !== true && !params.where?.hasOwnProperty('isVFS')) {
|
||||||
|
filter.isVFS = { notEquals: true }
|
||||||
|
}
|
||||||
|
|
||||||
if (params.type) {
|
if (params.type) {
|
||||||
const types = Array.isArray(params.type) ? params.type : [params.type]
|
const types = Array.isArray(params.type) ? params.type : [params.type]
|
||||||
if (types.length === 1) {
|
if (types.length === 1) {
|
||||||
|
|
@ -1346,7 +1379,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const filteredIds = await this.metadataIndex.getIdsForFilter(filter)
|
const filteredIds = await this.metadataIndex.getIdsForFilter(filter)
|
||||||
|
|
||||||
// CRITICAL FIX: Handle both cases properly
|
// CRITICAL FIX: Handle both cases properly
|
||||||
|
|
|
||||||
|
|
@ -185,6 +185,7 @@ export interface FindParams<T = any> {
|
||||||
mode?: SearchMode // Search strategy
|
mode?: SearchMode // Search strategy
|
||||||
explain?: boolean // Return scoring explanation
|
explain?: boolean // Return scoring explanation
|
||||||
includeRelations?: boolean // Include entity relationships
|
includeRelations?: boolean // Include entity relationships
|
||||||
|
includeVFS?: boolean // v4.3.3: Include VFS entities (default: false for knowledge queries)
|
||||||
service?: string // Multi-tenancy filter
|
service?: string // Multi-tenancy filter
|
||||||
|
|
||||||
// Triple Intelligence Fusion
|
// Triple Intelligence Fusion
|
||||||
|
|
|
||||||
207
tests/integration/vfs-knowledge-separation.test.ts
Normal file
207
tests/integration/vfs-knowledge-separation.test.ts
Normal file
|
|
@ -0,0 +1,207 @@
|
||||||
|
/**
|
||||||
|
* VFS-Knowledge Separation Test (v4.3.3)
|
||||||
|
*
|
||||||
|
* Tests Option 3C Architecture:
|
||||||
|
* - VFS entities marked with isVFS: true
|
||||||
|
* - brain.find() excludes VFS by default
|
||||||
|
* - brain.find({ includeVFS: true }) includes VFS
|
||||||
|
* - Enables relationships between VFS and knowledge
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { describe, it, expect, beforeAll, afterAll } from 'vitest'
|
||||||
|
import { Brainy } from '../../src/brainy.js'
|
||||||
|
import { NounType } from '../../src/types/graphTypes.js'
|
||||||
|
import * as fs from 'fs'
|
||||||
|
import * as path from 'path'
|
||||||
|
|
||||||
|
describe('VFS-Knowledge Separation (Option 3C)', () => {
|
||||||
|
const testDir = path.join(process.cwd(), 'test-vfs-knowledge-separation')
|
||||||
|
let brain: Brainy
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
// Clean up
|
||||||
|
if (fs.existsSync(testDir)) {
|
||||||
|
fs.rmSync(testDir, { recursive: true, force: true })
|
||||||
|
}
|
||||||
|
fs.mkdirSync(testDir, { recursive: true })
|
||||||
|
|
||||||
|
brain = new Brainy({
|
||||||
|
storage: {
|
||||||
|
type: 'filesystem',
|
||||||
|
options: { path: testDir }
|
||||||
|
}
|
||||||
|
})
|
||||||
|
await brain.init()
|
||||||
|
})
|
||||||
|
|
||||||
|
afterAll(() => {
|
||||||
|
if (fs.existsSync(testDir)) {
|
||||||
|
fs.rmSync(testDir, { recursive: true, force: true })
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should exclude VFS entities from brain.find() by default', async () => {
|
||||||
|
// Create VFS file entity
|
||||||
|
const vfs = brain.vfs()
|
||||||
|
await vfs.init()
|
||||||
|
await vfs.mkdir('/docs', { recursive: true })
|
||||||
|
await vfs.writeFile('/docs/readme.md', '# Hello World')
|
||||||
|
|
||||||
|
// Create knowledge entity (no isVFS flag)
|
||||||
|
const knowledgeEntity = await brain.add({
|
||||||
|
data: 'This is a knowledge document about AI',
|
||||||
|
type: NounType.Document,
|
||||||
|
metadata: {
|
||||||
|
title: 'AI Research Paper',
|
||||||
|
category: 'research'
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
// Query for documents WITHOUT includeVFS
|
||||||
|
console.log('\n📋 Test 1: brain.find() excludes VFS by default')
|
||||||
|
const results = await brain.find({
|
||||||
|
type: NounType.Document,
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
|
||||||
|
console.log(` Total results: ${results.length}`)
|
||||||
|
const vfsResults = results.filter(r => r.metadata?.isVFS === true)
|
||||||
|
const knowledgeResults = results.filter(r => r.metadata?.isVFS !== true)
|
||||||
|
|
||||||
|
console.log(` VFS results: ${vfsResults.length}`)
|
||||||
|
console.log(` Knowledge results: ${knowledgeResults.length}`)
|
||||||
|
|
||||||
|
// Should only return knowledge entities (no VFS)
|
||||||
|
expect(vfsResults.length).toBe(0)
|
||||||
|
expect(knowledgeResults.length).toBeGreaterThan(0)
|
||||||
|
expect(results.some(r => r.id === knowledgeEntity.id)).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should include VFS entities when includeVFS: true', async () => {
|
||||||
|
console.log('\n📋 Test 2: brain.find({ includeVFS: true }) includes VFS')
|
||||||
|
|
||||||
|
// Query WITH includeVFS: true
|
||||||
|
const results = await brain.find({
|
||||||
|
type: NounType.Document,
|
||||||
|
includeVFS: true,
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
|
||||||
|
console.log(` Total results: ${results.length}`)
|
||||||
|
const vfsResults = results.filter(r => r.metadata?.isVFS === true)
|
||||||
|
const knowledgeResults = results.filter(r => r.metadata?.isVFS !== true)
|
||||||
|
|
||||||
|
console.log(` VFS results: ${vfsResults.length}`)
|
||||||
|
console.log(` Knowledge results: ${knowledgeResults.length}`)
|
||||||
|
|
||||||
|
// Should return BOTH VFS and knowledge entities
|
||||||
|
expect(vfsResults.length).toBeGreaterThan(0)
|
||||||
|
expect(knowledgeResults.length).toBeGreaterThan(0)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should allow relationships between VFS files and knowledge entities', async () => {
|
||||||
|
console.log('\n📋 Test 3: VFS-Knowledge relationships')
|
||||||
|
|
||||||
|
// Get VFS file entity (need includeVFS: true when querying VFS by path)
|
||||||
|
const vfsFile = await brain.find({
|
||||||
|
where: {
|
||||||
|
path: '/docs/readme.md'
|
||||||
|
},
|
||||||
|
includeVFS: true,
|
||||||
|
limit: 1
|
||||||
|
})
|
||||||
|
expect(vfsFile.length).toBe(1)
|
||||||
|
|
||||||
|
// Get knowledge entity
|
||||||
|
const knowledgeEntity = await brain.find({
|
||||||
|
type: NounType.Document,
|
||||||
|
where: {
|
||||||
|
title: 'AI Research Paper'
|
||||||
|
},
|
||||||
|
limit: 1
|
||||||
|
})
|
||||||
|
expect(knowledgeEntity.length).toBe(1)
|
||||||
|
|
||||||
|
// Create relationship: knowledge entity -> references -> VFS file
|
||||||
|
const relation = await brain.relate({
|
||||||
|
from: knowledgeEntity[0].id,
|
||||||
|
to: vfsFile[0].id,
|
||||||
|
type: 'references'
|
||||||
|
})
|
||||||
|
|
||||||
|
console.log(` Created relation: ${relation.id}`)
|
||||||
|
console.log(` From: ${knowledgeEntity[0].metadata?.title} (knowledge)`)
|
||||||
|
console.log(` To: ${vfsFile[0].metadata?.path} (VFS file)`)
|
||||||
|
|
||||||
|
// Verify relationship exists
|
||||||
|
const relations = await brain.getRelations({
|
||||||
|
from: knowledgeEntity[0].id,
|
||||||
|
to: vfsFile[0].id
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(relations.length).toBe(1)
|
||||||
|
expect(relations[0].type).toBe('references')
|
||||||
|
console.log(` ✅ Relationship verified: knowledge can reference VFS files`)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should filter VFS entities using where clause', async () => {
|
||||||
|
console.log('\n📋 Test 4: Where clause filtering with isVFS')
|
||||||
|
|
||||||
|
// Query for VFS files explicitly
|
||||||
|
const vfsFiles = await brain.find({
|
||||||
|
where: {
|
||||||
|
isVFS: true,
|
||||||
|
vfsType: 'file'
|
||||||
|
},
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
|
||||||
|
console.log(` VFS files found: ${vfsFiles.length}`)
|
||||||
|
expect(vfsFiles.length).toBeGreaterThan(0)
|
||||||
|
expect(vfsFiles.every(f => f.metadata?.isVFS === true)).toBe(true)
|
||||||
|
expect(vfsFiles.every(f => f.metadata?.vfsType === 'file')).toBe(true)
|
||||||
|
|
||||||
|
// Query for non-VFS entities explicitly
|
||||||
|
const nonVFS = await brain.find({
|
||||||
|
where: {
|
||||||
|
category: 'research'
|
||||||
|
},
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
|
||||||
|
console.log(` Non-VFS entities found: ${nonVFS.length}`)
|
||||||
|
expect(nonVFS.length).toBeGreaterThan(0)
|
||||||
|
expect(nonVFS.every(e => e.metadata?.isVFS !== true)).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should handle semantic search with VFS filtering', async () => {
|
||||||
|
console.log('\n📋 Test 5: Semantic search excludes VFS by default')
|
||||||
|
|
||||||
|
// Semantic search WITHOUT includeVFS (should exclude VFS files)
|
||||||
|
const results = await brain.find({
|
||||||
|
query: 'research paper artificial intelligence',
|
||||||
|
limit: 10
|
||||||
|
})
|
||||||
|
|
||||||
|
console.log(` Semantic results: ${results.length}`)
|
||||||
|
const hasVFS = results.some(r => r.metadata?.isVFS === true)
|
||||||
|
console.log(` Contains VFS entities: ${hasVFS}`)
|
||||||
|
|
||||||
|
// Should NOT include VFS files in semantic search by default
|
||||||
|
expect(hasVFS).toBe(false)
|
||||||
|
|
||||||
|
// Semantic search WITH includeVFS: true
|
||||||
|
const resultsWithVFS = await brain.find({
|
||||||
|
query: 'hello world markdown',
|
||||||
|
includeVFS: true,
|
||||||
|
limit: 10
|
||||||
|
})
|
||||||
|
|
||||||
|
console.log(` Semantic results (with VFS): ${resultsWithVFS.length}`)
|
||||||
|
const hasVFSWithFlag = resultsWithVFS.some(r => r.metadata?.isVFS === true)
|
||||||
|
console.log(` Contains VFS entities: ${hasVFSWithFlag}`)
|
||||||
|
|
||||||
|
// Should include VFS files when explicitly requested
|
||||||
|
expect(hasVFSWithFlag).toBe(true)
|
||||||
|
})
|
||||||
|
})
|
||||||
86
tests/manual/metadata-index-debug.test.ts
Normal file
86
tests/manual/metadata-index-debug.test.ts
Normal file
|
|
@ -0,0 +1,86 @@
|
||||||
|
/**
|
||||||
|
* Debug: Check metadata index for isVFS field
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { describe, it, expect, beforeAll, afterAll } from 'vitest'
|
||||||
|
import { Brainy } from '../../src/brainy.js'
|
||||||
|
import { NounType } from '../../src/types/graphTypes.js'
|
||||||
|
import * as fs from 'fs'
|
||||||
|
import * as path from 'path'
|
||||||
|
|
||||||
|
describe('Metadata Index Debug', () => {
|
||||||
|
const testDir = path.join(process.cwd(), 'test-metadata-index-debug')
|
||||||
|
let brain: Brainy
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
if (fs.existsSync(testDir)) {
|
||||||
|
fs.rmSync(testDir, { recursive: true, force: true })
|
||||||
|
}
|
||||||
|
fs.mkdirSync(testDir, { recursive: true })
|
||||||
|
|
||||||
|
brain = new Brainy({
|
||||||
|
storage: {
|
||||||
|
type: 'filesystem',
|
||||||
|
options: { path: testDir }
|
||||||
|
}
|
||||||
|
})
|
||||||
|
await brain.init()
|
||||||
|
})
|
||||||
|
|
||||||
|
afterAll(() => {
|
||||||
|
if (fs.existsSync(testDir)) {
|
||||||
|
fs.rmSync(testDir, { recursive: true, force: true })
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should index custom metadata fields', async () => {
|
||||||
|
// Add entity with custom metadata field
|
||||||
|
const entity1 = await brain.add({
|
||||||
|
data: 'Test entity with custom field',
|
||||||
|
type: NounType.Document,
|
||||||
|
metadata: {
|
||||||
|
customFlag: true,
|
||||||
|
customString: 'hello',
|
||||||
|
customNumber: 42
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
console.log('\n📋 Added entity:', entity1.id)
|
||||||
|
console.log(' Metadata:', entity1.metadata)
|
||||||
|
|
||||||
|
// Try to find by custom fields
|
||||||
|
console.log('\n📋 Query 1: where: { customFlag: true }')
|
||||||
|
const result1 = await brain.find({
|
||||||
|
where: { customFlag: true },
|
||||||
|
limit: 10
|
||||||
|
})
|
||||||
|
console.log(` Results: ${result1.length}`)
|
||||||
|
if (result1.length > 0) {
|
||||||
|
console.log(` Found: ${result1[0].id}`)
|
||||||
|
console.log(` Metadata.customFlag: ${result1[0].metadata?.customFlag}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('\n📋 Query 2: where: { customString: "hello" }')
|
||||||
|
const result2 = await brain.find({
|
||||||
|
where: { customString: 'hello' },
|
||||||
|
limit: 10
|
||||||
|
})
|
||||||
|
console.log(` Results: ${result2.length}`)
|
||||||
|
|
||||||
|
console.log('\n📋 Query 3: where: { customNumber: 42 }')
|
||||||
|
const result3 = await brain.find({
|
||||||
|
where: { customNumber: 42 },
|
||||||
|
limit: 10
|
||||||
|
})
|
||||||
|
console.log(` Results: ${result3.length}`)
|
||||||
|
|
||||||
|
// Check what fields are indexed
|
||||||
|
console.log('\n📋 Indexed fields:')
|
||||||
|
const fields = await brain.getFilterFields()
|
||||||
|
console.log(` ${fields.join(', ')}`)
|
||||||
|
|
||||||
|
expect(result1.length).toBeGreaterThan(0)
|
||||||
|
expect(result2.length).toBeGreaterThan(0)
|
||||||
|
expect(result3.length).toBeGreaterThan(0)
|
||||||
|
})
|
||||||
|
})
|
||||||
90
tests/manual/vfs-isvfs-diagnostic.test.ts
Normal file
90
tests/manual/vfs-isvfs-diagnostic.test.ts
Normal file
|
|
@ -0,0 +1,90 @@
|
||||||
|
/**
|
||||||
|
* Diagnostic: Check if isVFS flag is actually being stored
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { describe, it, expect, beforeAll, afterAll } from 'vitest'
|
||||||
|
import { Brainy } from '../../src/brainy.js'
|
||||||
|
import { NounType } from '../../src/types/graphTypes.js'
|
||||||
|
import * as fs from 'fs'
|
||||||
|
import * as path from 'path'
|
||||||
|
|
||||||
|
describe('isVFS Flag Diagnostic', () => {
|
||||||
|
const testDir = path.join(process.cwd(), 'test-isvfs-diagnostic')
|
||||||
|
let brain: Brainy
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
if (fs.existsSync(testDir)) {
|
||||||
|
fs.rmSync(testDir, { recursive: true, force: true })
|
||||||
|
}
|
||||||
|
fs.mkdirSync(testDir, { recursive: true })
|
||||||
|
|
||||||
|
brain = new Brainy({
|
||||||
|
storage: {
|
||||||
|
type: 'filesystem',
|
||||||
|
options: { path: testDir }
|
||||||
|
}
|
||||||
|
})
|
||||||
|
await brain.init()
|
||||||
|
})
|
||||||
|
|
||||||
|
afterAll(() => {
|
||||||
|
if (fs.existsSync(testDir)) {
|
||||||
|
fs.rmSync(testDir, { recursive: true, force: true })
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should show what entities are created', async () => {
|
||||||
|
// Create VFS file
|
||||||
|
const vfs = brain.vfs()
|
||||||
|
await vfs.init()
|
||||||
|
await vfs.writeFile('/test.txt', 'Hello World')
|
||||||
|
|
||||||
|
// Get ALL entities
|
||||||
|
console.log('\n📋 All entities in database:')
|
||||||
|
const allEntities = await brain.find({ limit: 100, includeVFS: true })
|
||||||
|
console.log(` Total: ${allEntities.length}`)
|
||||||
|
|
||||||
|
for (const entity of allEntities) {
|
||||||
|
console.log(`\n Entity: ${entity.id}`)
|
||||||
|
console.log(` Type: ${entity.type}`)
|
||||||
|
console.log(` Metadata keys: ${Object.keys(entity.metadata || {}).join(', ')}`)
|
||||||
|
if (entity.metadata?.path) {
|
||||||
|
console.log(` Path: ${entity.metadata.path}`)
|
||||||
|
}
|
||||||
|
if (entity.metadata?.vfsType) {
|
||||||
|
console.log(` VfsType: ${entity.metadata.vfsType}`)
|
||||||
|
}
|
||||||
|
if (entity.metadata?.isVFS !== undefined) {
|
||||||
|
console.log(` isVFS: ${entity.metadata.isVFS}`)
|
||||||
|
}
|
||||||
|
if (entity.metadata?.name) {
|
||||||
|
console.log(` Name: ${entity.metadata.name}`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try different queries
|
||||||
|
console.log('\n\n📋 Query 1: brain.find({ limit: 100 }) [default, no includeVFS]')
|
||||||
|
const query1 = await brain.find({ limit: 100 })
|
||||||
|
console.log(` Results: ${query1.length}`)
|
||||||
|
|
||||||
|
console.log('\n📋 Query 2: brain.find({ limit: 100, includeVFS: true })')
|
||||||
|
const query2 = await brain.find({ limit: 100, includeVFS: true })
|
||||||
|
console.log(` Results: ${query2.length}`)
|
||||||
|
|
||||||
|
console.log('\n📋 Query 3: brain.find({ where: { isVFS: true } })')
|
||||||
|
const query3 = await brain.find({ where: { isVFS: true }, limit: 100 })
|
||||||
|
console.log(` Results: ${query3.length}`)
|
||||||
|
|
||||||
|
console.log('\n📋 Query 4: brain.find({ where: { path: "/test.txt" } })')
|
||||||
|
const query4 = await brain.find({ where: { path: '/test.txt' }, limit: 100 })
|
||||||
|
console.log(` Results: ${query4.length}`)
|
||||||
|
|
||||||
|
console.log('\n📋 Query 5: brain.find({ type: NounType.Document })')
|
||||||
|
const query5 = await brain.find({ type: NounType.Document, limit: 100 })
|
||||||
|
console.log(` Results: ${query5.length}`)
|
||||||
|
|
||||||
|
console.log('\n📋 Query 6: brain.find({ type: NounType.Document, includeVFS: true })')
|
||||||
|
const query6 = await brain.find({ type: NounType.Document, includeVFS: true, limit: 100 })
|
||||||
|
console.log(` Results: ${query6.length}`)
|
||||||
|
})
|
||||||
|
})
|
||||||
207
tests/manual/vfs-multiple-init-diagnostic.test.ts
Normal file
207
tests/manual/vfs-multiple-init-diagnostic.test.ts
Normal file
|
|
@ -0,0 +1,207 @@
|
||||||
|
/**
|
||||||
|
* VFS Multiple Init() Diagnostic Test
|
||||||
|
*
|
||||||
|
* Tests Workshop team's issue: Does calling vfs.init() multiple times
|
||||||
|
* create duplicate root entities?
|
||||||
|
*
|
||||||
|
* Scenario:
|
||||||
|
* - Create brain instance
|
||||||
|
* - Call vfs.init() multiple times (simulating multiple requests)
|
||||||
|
* - Check if multiple root entities are created
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { describe, it, expect, beforeAll, afterAll } from 'vitest'
|
||||||
|
import { Brainy } from '../../src/brainy.js'
|
||||||
|
import * as fs from 'fs'
|
||||||
|
import * as path from 'path'
|
||||||
|
|
||||||
|
describe('VFS Multiple Init Diagnostic', () => {
|
||||||
|
const testDir = path.join(process.cwd(), 'test-vfs-multi-init')
|
||||||
|
let brain: Brainy
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
// Clean up test directory
|
||||||
|
if (fs.existsSync(testDir)) {
|
||||||
|
fs.rmSync(testDir, { recursive: true, force: true })
|
||||||
|
}
|
||||||
|
fs.mkdirSync(testDir, { recursive: true })
|
||||||
|
|
||||||
|
// Create brain with filesystem storage
|
||||||
|
brain = new Brainy({
|
||||||
|
storage: {
|
||||||
|
type: 'filesystem',
|
||||||
|
path: testDir
|
||||||
|
}
|
||||||
|
})
|
||||||
|
await brain.init()
|
||||||
|
})
|
||||||
|
|
||||||
|
afterAll(() => {
|
||||||
|
// Clean up
|
||||||
|
if (fs.existsSync(testDir)) {
|
||||||
|
fs.rmSync(testDir, { recursive: true, force: true })
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should not create duplicate roots when calling vfs.init() multiple times on SAME instance', async () => {
|
||||||
|
const vfs = brain.vfs()
|
||||||
|
|
||||||
|
// Call init multiple times
|
||||||
|
await vfs.init()
|
||||||
|
await vfs.init()
|
||||||
|
await vfs.init()
|
||||||
|
|
||||||
|
// Query for root entities using workaround (where clause is broken)
|
||||||
|
const collections = await brain.find({
|
||||||
|
type: 'collection',
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
|
||||||
|
const roots = collections.filter(e =>
|
||||||
|
e.metadata?.path === '/' &&
|
||||||
|
e.metadata?.vfsType === 'directory'
|
||||||
|
)
|
||||||
|
|
||||||
|
console.log(`\n✅ Test 1: Same VFS instance`)
|
||||||
|
console.log(` Total collections: ${collections.length}`)
|
||||||
|
console.log(` Roots found: ${roots.length}`)
|
||||||
|
roots.forEach((root, i) => {
|
||||||
|
console.log(` Root ${i + 1}: ${root.id}`)
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(roots.length).toBe(1)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should not create duplicate roots when creating MULTIPLE VFS instances (Workshop scenario)', async () => {
|
||||||
|
// Simulate Workshop's scenario: Getting VFS on multiple requests
|
||||||
|
// brain.vfs() returns cached instance, so this should be safe
|
||||||
|
const vfs1 = brain.vfs()
|
||||||
|
const vfs2 = brain.vfs()
|
||||||
|
const vfs3 = brain.vfs()
|
||||||
|
|
||||||
|
await vfs1.init()
|
||||||
|
await vfs2.init()
|
||||||
|
await vfs3.init()
|
||||||
|
|
||||||
|
// Query for root entities using workaround
|
||||||
|
const collections = await brain.find({
|
||||||
|
type: 'collection',
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
|
||||||
|
const roots = collections.filter(e =>
|
||||||
|
e.metadata?.path === '/' &&
|
||||||
|
e.metadata?.vfsType === 'directory'
|
||||||
|
)
|
||||||
|
|
||||||
|
console.log(`\n✅ Test 2: Multiple VFS references (cached)`)
|
||||||
|
console.log(` VFS instances are same? ${vfs1 === vfs2 && vfs2 === vfs3}`)
|
||||||
|
console.log(` Roots found: ${roots.length}`)
|
||||||
|
roots.forEach((root, i) => {
|
||||||
|
console.log(` Root ${i + 1}: ${root.id}`)
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(vfs1).toBe(vfs2) // Should be same instance (cached)
|
||||||
|
expect(roots.length).toBe(1) // Should still be 1 root
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should handle NEW brain instances gracefully (per-user scenario)', async () => {
|
||||||
|
// Simulate creating separate brain instances per user
|
||||||
|
// This is closer to Workshop's getUserBrainy() pattern
|
||||||
|
const brain2 = new Brainy({
|
||||||
|
storage: {
|
||||||
|
type: 'filesystem',
|
||||||
|
path: testDir // Same storage!
|
||||||
|
}
|
||||||
|
})
|
||||||
|
await brain2.init()
|
||||||
|
|
||||||
|
const vfs2 = brain2.vfs()
|
||||||
|
await vfs2.init()
|
||||||
|
|
||||||
|
// Query for roots using workaround
|
||||||
|
const collections = await brain.find({
|
||||||
|
type: 'collection',
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
|
||||||
|
const roots = collections.filter(e =>
|
||||||
|
e.metadata?.path === '/' &&
|
||||||
|
e.metadata?.vfsType === 'directory'
|
||||||
|
)
|
||||||
|
|
||||||
|
console.log(`\n✅ Test 3: New Brainy instance (same storage)`)
|
||||||
|
console.log(` Roots found: ${roots.length}`)
|
||||||
|
roots.forEach((root, i) => {
|
||||||
|
console.log(` Root ${i + 1}: ${root.id} (created: ${root.metadata?.createdAt})`)
|
||||||
|
})
|
||||||
|
|
||||||
|
// This is the CRITICAL test - should find existing root, not create new one
|
||||||
|
expect(roots.length).toBe(1)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should verify readdir works after multiple inits', async () => {
|
||||||
|
// Create a test directory
|
||||||
|
const vfs = brain.vfs()
|
||||||
|
await vfs.mkdir('/test-dir', { recursive: true })
|
||||||
|
await vfs.writeFile('/test-dir/test.txt', 'Hello')
|
||||||
|
|
||||||
|
// Call init again (simulating new request)
|
||||||
|
const brain3 = new Brainy({
|
||||||
|
storage: {
|
||||||
|
type: 'filesystem',
|
||||||
|
path: testDir
|
||||||
|
}
|
||||||
|
})
|
||||||
|
await brain3.init()
|
||||||
|
const vfs3 = brain3.vfs()
|
||||||
|
await vfs3.init()
|
||||||
|
|
||||||
|
// Try to read root directory
|
||||||
|
const entries = await vfs3.readdir('/', { withFileTypes: true })
|
||||||
|
|
||||||
|
console.log(`\n✅ Test 4: readdir after new brain instance`)
|
||||||
|
console.log(` Entries found: ${entries.length}`)
|
||||||
|
entries.forEach((entry: any) => {
|
||||||
|
console.log(` - ${entry.name} (${entry.type})`)
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(entries.length).toBeGreaterThan(0)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should show if Contains relationships are created', async () => {
|
||||||
|
const vfs = brain.vfs()
|
||||||
|
|
||||||
|
// Get root entity ID using workaround
|
||||||
|
const collections = await brain.find({
|
||||||
|
type: 'collection',
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
|
||||||
|
const roots = collections.filter(e =>
|
||||||
|
e.metadata?.path === '/' &&
|
||||||
|
e.metadata?.vfsType === 'directory'
|
||||||
|
)
|
||||||
|
|
||||||
|
expect(roots.length).toBeGreaterThanOrEqual(1)
|
||||||
|
const rootId = roots[0].id
|
||||||
|
|
||||||
|
// Check for Contains relationships FROM root
|
||||||
|
const relations = await brain.getRelations({
|
||||||
|
from: rootId
|
||||||
|
})
|
||||||
|
|
||||||
|
console.log(`\n✅ Test 5: Contains relationships`)
|
||||||
|
console.log(` Root ID: ${rootId}`)
|
||||||
|
console.log(` Relationships from root: ${relations.length}`)
|
||||||
|
relations.forEach((rel) => {
|
||||||
|
console.log(` - ${rel.from} -> ${rel.to} (${rel.type})`)
|
||||||
|
})
|
||||||
|
|
||||||
|
// Root should have at least one Contains relationship (to /test-dir)
|
||||||
|
const containsRels = relations.filter(r => r.type === 'contains')
|
||||||
|
console.log(` Contains relationships: ${containsRels.length}`)
|
||||||
|
|
||||||
|
expect(containsRels.length).toBeGreaterThan(0)
|
||||||
|
})
|
||||||
|
})
|
||||||
171
tests/manual/workshop-type-filtering-test.ts
Normal file
171
tests/manual/workshop-type-filtering-test.ts
Normal file
|
|
@ -0,0 +1,171 @@
|
||||||
|
/**
|
||||||
|
* Manual test to reproduce Workshop team's type filtering issue
|
||||||
|
*
|
||||||
|
* This test verifies that brain.find({ type: NounType.Person }) actually works
|
||||||
|
* as documented in the API.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { Brainy, NounType } from '../../src/index.js'
|
||||||
|
|
||||||
|
async function testTypeFiltering() {
|
||||||
|
console.log('\n🔬 Testing Type Filtering Issue from Workshop Team\n')
|
||||||
|
console.log('=' .repeat(60))
|
||||||
|
|
||||||
|
// Create in-memory instance for testing
|
||||||
|
const brain = new Brainy({
|
||||||
|
storage: { type: 'memory' }
|
||||||
|
})
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
|
console.log('\n1️⃣ Adding test entities with different types...\n')
|
||||||
|
|
||||||
|
// Add 5 person entities
|
||||||
|
const people = []
|
||||||
|
for (let i = 0; i < 5; i++) {
|
||||||
|
const id = await brain.add({
|
||||||
|
data: `Person ${i + 1}`,
|
||||||
|
type: NounType.Person,
|
||||||
|
metadata: { name: `Person ${i + 1}` }
|
||||||
|
})
|
||||||
|
people.push(id)
|
||||||
|
}
|
||||||
|
console.log(`✅ Added ${people.length} person entities`)
|
||||||
|
|
||||||
|
// Add 3 location entities
|
||||||
|
const locations = []
|
||||||
|
for (let i = 0; i < 3; i++) {
|
||||||
|
const id = await brain.add({
|
||||||
|
data: `Location ${i + 1}`,
|
||||||
|
type: NounType.Location,
|
||||||
|
metadata: { name: `Location ${i + 1}` }
|
||||||
|
})
|
||||||
|
locations.push(id)
|
||||||
|
}
|
||||||
|
console.log(`✅ Added ${locations.length} location entities`)
|
||||||
|
|
||||||
|
// Add 2 concept entities
|
||||||
|
const concepts = []
|
||||||
|
for (let i = 0; i < 2; i++) {
|
||||||
|
const id = await brain.add({
|
||||||
|
data: `Concept ${i + 1}`,
|
||||||
|
type: NounType.Concept,
|
||||||
|
metadata: { name: `Concept ${i + 1}` }
|
||||||
|
})
|
||||||
|
concepts.push(id)
|
||||||
|
}
|
||||||
|
console.log(`✅ Added ${concepts.length} concept entities`)
|
||||||
|
|
||||||
|
console.log(`\n📊 Total: ${people.length + locations.length + concepts.length} entities`)
|
||||||
|
|
||||||
|
console.log('\n' + '='.repeat(60))
|
||||||
|
console.log('\n2️⃣ Testing find() without type filter...\n')
|
||||||
|
|
||||||
|
const allResults = await brain.find({ limit: 100 })
|
||||||
|
console.log(`✅ brain.find({}) returned ${allResults.length} entities`)
|
||||||
|
|
||||||
|
// Check types in results
|
||||||
|
const typeCounts = allResults.reduce((acc, r) => {
|
||||||
|
acc[r.type || 'unknown'] = (acc[r.type || 'unknown'] || 0) + 1
|
||||||
|
return acc
|
||||||
|
}, {} as Record<string, number>)
|
||||||
|
|
||||||
|
console.log('\nTypes in results:')
|
||||||
|
for (const [type, count] of Object.entries(typeCounts)) {
|
||||||
|
console.log(` - ${type}: ${count}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('\n' + '='.repeat(60))
|
||||||
|
console.log('\n3️⃣ Testing find() WITH type filter (using enum)...\n')
|
||||||
|
|
||||||
|
// Test 1: Filter by NounType.Person (enum value)
|
||||||
|
console.log('Test 1: brain.find({ type: NounType.Person })')
|
||||||
|
const personResults = await brain.find({ type: NounType.Person, limit: 100 })
|
||||||
|
console.log(` Result: ${personResults.length} entities`)
|
||||||
|
console.log(` Expected: ${people.length} entities`)
|
||||||
|
if (personResults.length === people.length) {
|
||||||
|
console.log(' ✅ PASS')
|
||||||
|
} else {
|
||||||
|
console.log(' ❌ FAIL - Type filtering not working!')
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test 2: Filter by NounType.Location
|
||||||
|
console.log('\nTest 2: brain.find({ type: NounType.Location })')
|
||||||
|
const locationResults = await brain.find({ type: NounType.Location, limit: 100 })
|
||||||
|
console.log(` Result: ${locationResults.length} entities`)
|
||||||
|
console.log(` Expected: ${locations.length} entities`)
|
||||||
|
if (locationResults.length === locations.length) {
|
||||||
|
console.log(' ✅ PASS')
|
||||||
|
} else {
|
||||||
|
console.log(' ❌ FAIL - Type filtering not working!')
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test 3: Filter by NounType.Concept
|
||||||
|
console.log('\nTest 3: brain.find({ type: NounType.Concept })')
|
||||||
|
const conceptResults = await brain.find({ type: NounType.Concept, limit: 100 })
|
||||||
|
console.log(` Result: ${conceptResults.length} entities`)
|
||||||
|
console.log(` Expected: ${concepts.length} entities`)
|
||||||
|
if (conceptResults.length === concepts.length) {
|
||||||
|
console.log(' ✅ PASS')
|
||||||
|
} else {
|
||||||
|
console.log(' ❌ FAIL - Type filtering not working!')
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('\n' + '='.repeat(60))
|
||||||
|
console.log('\n4️⃣ Testing find() WITH type filter (using string)...\n')
|
||||||
|
|
||||||
|
// Test 4: Filter by string 'person'
|
||||||
|
console.log('Test 4: brain.find({ type: "person" })')
|
||||||
|
const personResults2 = await brain.find({ type: 'person' as any, limit: 100 })
|
||||||
|
console.log(` Result: ${personResults2.length} entities`)
|
||||||
|
console.log(` Expected: ${people.length} entities`)
|
||||||
|
if (personResults2.length === people.length) {
|
||||||
|
console.log(' ✅ PASS')
|
||||||
|
} else {
|
||||||
|
console.log(' ❌ FAIL - String type filtering not working!')
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('\n' + '='.repeat(60))
|
||||||
|
console.log('\n5️⃣ Checking entity storage (metadata.noun)...\n')
|
||||||
|
|
||||||
|
// Get a person entity and check its storage
|
||||||
|
const personEntity = await brain.get(people[0])
|
||||||
|
console.log('Person entity structure:')
|
||||||
|
console.log(' - id:', personEntity?.id)
|
||||||
|
console.log(' - type:', personEntity?.type)
|
||||||
|
console.log(' - metadata.noun:', (personEntity as any)?.metadata?.noun)
|
||||||
|
console.log(' - metadata.name:', personEntity?.metadata?.name)
|
||||||
|
|
||||||
|
console.log('\n' + '='.repeat(60))
|
||||||
|
console.log('\n📋 Summary\n')
|
||||||
|
|
||||||
|
const tests = [
|
||||||
|
{ name: 'Filter by NounType.Person', passed: personResults.length === people.length },
|
||||||
|
{ name: 'Filter by NounType.Location', passed: locationResults.length === locations.length },
|
||||||
|
{ name: 'Filter by NounType.Concept', passed: conceptResults.length === concepts.length },
|
||||||
|
{ name: 'Filter by string "person"', passed: personResults2.length === people.length }
|
||||||
|
]
|
||||||
|
|
||||||
|
const passedCount = tests.filter(t => t.passed).length
|
||||||
|
const totalCount = tests.length
|
||||||
|
|
||||||
|
console.log(`Tests passed: ${passedCount}/${totalCount}\n`)
|
||||||
|
|
||||||
|
for (const test of tests) {
|
||||||
|
console.log(`${test.passed ? '✅' : '❌'} ${test.name}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
if (passedCount === totalCount) {
|
||||||
|
console.log('\n🎉 All tests passed! Type filtering works correctly.')
|
||||||
|
console.log('\n💡 The Workshop team might be experiencing a different issue.')
|
||||||
|
console.log(' Check: storage persistence, Brainy instance reuse, or data migration')
|
||||||
|
} else {
|
||||||
|
console.log('\n❌ Type filtering is BROKEN in Brainy!')
|
||||||
|
console.log('\n🐛 This is a bug that needs to be fixed.')
|
||||||
|
console.log(' Workshop team was right - it\'s not user error.')
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('\n' + '='.repeat(60) + '\n')
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run the test
|
||||||
|
testTypeFiltering().catch(console.error)
|
||||||
171
tests/unit/workshop-vfs-diagnostic.test.ts
Normal file
171
tests/unit/workshop-vfs-diagnostic.test.ts
Normal file
|
|
@ -0,0 +1,171 @@
|
||||||
|
/**
|
||||||
|
* Workshop VFS Diagnostic Test
|
||||||
|
*
|
||||||
|
* Tests to verify VFS import behavior and identify if VFS creates only wrappers or also graph entities
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { describe, it, expect, beforeEach } from 'vitest'
|
||||||
|
import { Brainy, NounType } from '../../src/index.js'
|
||||||
|
|
||||||
|
describe('Workshop VFS Diagnostic', () => {
|
||||||
|
let brain: Brainy
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
brain = new Brainy({
|
||||||
|
storage: { type: 'memory' }
|
||||||
|
})
|
||||||
|
await brain.init()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should verify VFS creates document wrappers AND allows entity filtering', async () => {
|
||||||
|
console.log('\n🔬 Workshop VFS Diagnostic Test\n')
|
||||||
|
console.log('='.repeat(70))
|
||||||
|
|
||||||
|
// Step 1: Add entities directly (control group)
|
||||||
|
console.log('\n1️⃣ Adding entities directly (without VFS)...\n')
|
||||||
|
|
||||||
|
await brain.add({ data: 'Person 1', type: NounType.Person, metadata: { name: 'Person 1' } })
|
||||||
|
await brain.add({ data: 'Person 2', type: NounType.Person, metadata: { name: 'Person 2' } })
|
||||||
|
await brain.add({ data: 'Location 1', type: NounType.Location, metadata: { name: 'Location 1' } })
|
||||||
|
|
||||||
|
const beforeVfs = await brain.find({ limit: 100 })
|
||||||
|
console.log(` Total entities: ${beforeVfs.length}`)
|
||||||
|
|
||||||
|
const peopleBefore = await brain.find({ type: NounType.Person, limit: 100 })
|
||||||
|
console.log(` Person filter: ${peopleBefore.length} (expected: 2)`)
|
||||||
|
expect(peopleBefore.length).toBe(2)
|
||||||
|
console.log(' ✅ Type filtering works on direct entities\n')
|
||||||
|
|
||||||
|
// Step 2: Use VFS to create files
|
||||||
|
console.log('2️⃣ Creating VFS files...\n')
|
||||||
|
|
||||||
|
const vfs = brain.vfs()
|
||||||
|
await vfs.init()
|
||||||
|
await vfs.mkdir('/test', { recursive: true })
|
||||||
|
|
||||||
|
// Create a VFS file with entity data
|
||||||
|
const personData = {
|
||||||
|
id: 'ent_person_test',
|
||||||
|
name: 'John Smith',
|
||||||
|
type: 'person',
|
||||||
|
metadata: { source: 'test' }
|
||||||
|
}
|
||||||
|
|
||||||
|
await vfs.writeFile('/test/john.json', Buffer.from(JSON.stringify(personData, null, 2)))
|
||||||
|
console.log(' Created VFS file: /test/john.json')
|
||||||
|
|
||||||
|
// Step 3: Check what entities exist now
|
||||||
|
console.log('\n3️⃣ Analyzing entities after VFS...\n')
|
||||||
|
|
||||||
|
const afterVfs = await brain.find({ limit: 100 })
|
||||||
|
console.log(` Total entities: ${afterVfs.length}`)
|
||||||
|
|
||||||
|
// Count by type
|
||||||
|
const typeCounts: Record<string, number> = {}
|
||||||
|
for (const result of afterVfs) {
|
||||||
|
const type = result.type || 'unknown'
|
||||||
|
typeCounts[type] = (typeCounts[type] || 0) + 1
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('\n Entity type breakdown:')
|
||||||
|
for (const [type, count] of Object.entries(typeCounts)) {
|
||||||
|
console.log(` - ${type}: ${count}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Count VFS wrappers vs regular entities
|
||||||
|
const vfsWrappers = afterVfs.filter(e => e.metadata?.vfsType === 'file')
|
||||||
|
const regularEntities = afterVfs.filter(e => !e.metadata?.vfsType)
|
||||||
|
|
||||||
|
console.log(`\n VFS wrappers: ${vfsWrappers.length}`)
|
||||||
|
console.log(` Regular entities: ${regularEntities.length}`)
|
||||||
|
|
||||||
|
// Step 4: Test type filtering after VFS
|
||||||
|
console.log('\n4️⃣ Testing type filtering after VFS...\n')
|
||||||
|
|
||||||
|
const peopleAfter = await brain.find({ type: NounType.Person, limit: 100 })
|
||||||
|
console.log(` Person filter: ${peopleAfter.length} (expected: 2 - same as before)`)
|
||||||
|
|
||||||
|
const documents = await brain.find({ type: NounType.Document, limit: 100 })
|
||||||
|
console.log(` Document filter: ${documents.length} (expected: ${vfsWrappers.length})`)
|
||||||
|
|
||||||
|
// Step 5: Analyze VFS wrapper structure
|
||||||
|
console.log('\n5️⃣ Analyzing VFS wrapper structure...\n')
|
||||||
|
|
||||||
|
const wrapper = vfsWrappers[0]
|
||||||
|
if (wrapper) {
|
||||||
|
console.log(' VFS Wrapper Entity:')
|
||||||
|
console.log(` - ID: ${wrapper.id}`)
|
||||||
|
console.log(` - Type: ${wrapper.type}`)
|
||||||
|
console.log(` - VFS Type: ${wrapper.metadata?.vfsType}`)
|
||||||
|
console.log(` - Path: ${wrapper.metadata?.path}`)
|
||||||
|
console.log(` - Has rawData: ${!!wrapper.metadata?.rawData}`)
|
||||||
|
|
||||||
|
if (wrapper.metadata?.rawData) {
|
||||||
|
const decoded = Buffer.from(wrapper.metadata.rawData, 'base64').toString()
|
||||||
|
const embedded = JSON.parse(decoded)
|
||||||
|
console.log(`\n Embedded Entity Data:`)
|
||||||
|
console.log(` - Name: ${embedded.name}`)
|
||||||
|
console.log(` - Type: ${embedded.type}`)
|
||||||
|
console.log(`\n 🔍 KEY FINDING:`)
|
||||||
|
console.log(` Wrapper type: "${wrapper.type}"`)
|
||||||
|
console.log(` Embedded type: "${embedded.type}"`)
|
||||||
|
console.log(` Filtering by type="${embedded.type}" searches wrapper type, not embedded!`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 6: Diagnosis
|
||||||
|
console.log('\n' + '='.repeat(70))
|
||||||
|
console.log('📋 DIAGNOSIS\n')
|
||||||
|
|
||||||
|
if (peopleAfter.length === peopleBefore.length) {
|
||||||
|
console.log('✅ VFS does NOT create duplicate graph entities')
|
||||||
|
console.log('✅ VFS only creates document wrappers')
|
||||||
|
console.log('✅ Type filtering works on original entities, ignores VFS wrappers')
|
||||||
|
console.log('\nThis means:')
|
||||||
|
console.log(' - VFS files are type="document" wrappers')
|
||||||
|
console.log(' - Original entities keep their types')
|
||||||
|
console.log(' - filter({ type: "person" }) returns original entities only')
|
||||||
|
} else {
|
||||||
|
console.log('❌ Unexpected behavior - VFS may have created additional entities')
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('\n' + '='.repeat(70) + '\n')
|
||||||
|
|
||||||
|
// Assertions
|
||||||
|
expect(peopleAfter.length).toBe(2) // Should still be 2, VFS doesn't create person entities
|
||||||
|
expect(documents.length).toBeGreaterThan(0) // VFS creates document wrappers
|
||||||
|
expect(vfsWrappers.length).toBeGreaterThan(0) // Should have VFS wrappers
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should verify import creates BOTH VFS wrappers AND graph entities', async () => {
|
||||||
|
// This test would require creating a test Excel file and running import
|
||||||
|
// For now, we'll document the expected behavior based on code analysis
|
||||||
|
|
||||||
|
console.log('\n📚 Expected Import Behavior (from code analysis):\n')
|
||||||
|
console.log('When you run brain.import("file.xlsx", { vfsPath: "/imports" }):')
|
||||||
|
console.log('\n1. ImportCoordinator.execute() calls:')
|
||||||
|
console.log(' a) vfsGenerator.generate() - creates VFS file wrappers')
|
||||||
|
console.log(' - Each entity → JSON file in VFS')
|
||||||
|
console.log(' - Wrapper entity with type="document"')
|
||||||
|
console.log(' - Entity data stored in metadata.rawData (base64)')
|
||||||
|
console.log('')
|
||||||
|
console.log(' b) createGraphEntities() - creates graph entities')
|
||||||
|
console.log(' - Each entity → graph entity with proper type')
|
||||||
|
console.log(' - type="person", "location", "concept", etc.')
|
||||||
|
console.log(' - metadata.vfsPath points to VFS file')
|
||||||
|
console.log('')
|
||||||
|
console.log('2. Result: Database contains BOTH:')
|
||||||
|
console.log(' - VFS wrappers (type="document", vfsType="file")')
|
||||||
|
console.log(' - Graph entities (type="person", etc., vfsPath set)')
|
||||||
|
console.log('')
|
||||||
|
console.log('3. Type filtering:')
|
||||||
|
console.log(' - filter({ type: "person" }) → returns graph entities')
|
||||||
|
console.log(' - filter({ type: "document" }) → returns VFS wrappers')
|
||||||
|
console.log('')
|
||||||
|
console.log('If Workshop gets 0 results, likely causes:')
|
||||||
|
console.log(' ❌ Only VFS wrappers created (createEntities: false)')
|
||||||
|
console.log(' ❌ Import not completing before query')
|
||||||
|
console.log(' ❌ Querying different Brainy instance')
|
||||||
|
console.log('')
|
||||||
|
})
|
||||||
|
})
|
||||||
Loading…
Add table
Add a link
Reference in a new issue