From 014b8104da44118628881932d731d1adb9776678 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Fri, 24 Oct 2025 11:42:47 -0700 Subject: [PATCH] feat: brain.find() excludes VFS by default (Option 3C) Added includeVFS parameter to FindParams: - brain.find() excludes VFS entities by default (clean knowledge graph) - Opt-in with brain.find({ includeVFS: true }) - Automatically excludes VFS in all query paths (empty, metadata, vector) - Respects explicit where: { isVFS: ... } queries Implementation: - Empty query path: Apply VFS filtering even with no criteria - Metadata query path: Filter out isVFS: true by default - Vector search path: Apply VFS filter after search - Skip auto-exclusion if where clause explicitly queries isVFS Architecture (Option 3C): - VFS entities are first-class graph entities - Marked with isVFS: true flag - Separated via filtering, not storage - Enables VFS-knowledge relationships Moved internal docs to .strategy/: - README_STORAGE_EXPLORATION.md - EXPLORATION_SUMMARY.md - STORAGE_FILES_REFERENCE.md - STORAGE_ADAPTER_QUICK_REFERENCE.md - SECURITY.md --- EXPLORATION_SUMMARY.md | 393 ---------------- README_STORAGE_EXPLORATION.md | 291 ------------ SECURITY.md | 389 ---------------- STORAGE_ADAPTER_QUICK_REFERENCE.md | 291 ------------ STORAGE_FILES_REFERENCE.md | 431 ------------------ scripts/diagnose-workshop-issue.ts | 194 ++++++++ src/brainy.ts | 57 ++- src/types/brainy.types.ts | 1 + .../vfs-knowledge-separation.test.ts | 207 +++++++++ tests/manual/metadata-index-debug.test.ts | 86 ++++ tests/manual/vfs-isvfs-diagnostic.test.ts | 90 ++++ .../vfs-multiple-init-diagnostic.test.ts | 207 +++++++++ tests/manual/workshop-type-filtering-test.ts | 171 +++++++ tests/unit/workshop-vfs-diagnostic.test.ts | 171 +++++++ 14 files changed, 1172 insertions(+), 1807 deletions(-) delete mode 100644 EXPLORATION_SUMMARY.md delete mode 100644 README_STORAGE_EXPLORATION.md delete mode 100644 SECURITY.md delete mode 100644 STORAGE_ADAPTER_QUICK_REFERENCE.md delete mode 100644 STORAGE_FILES_REFERENCE.md create mode 100644 scripts/diagnose-workshop-issue.ts create mode 100644 tests/integration/vfs-knowledge-separation.test.ts create mode 100644 tests/manual/metadata-index-debug.test.ts create mode 100644 tests/manual/vfs-isvfs-diagnostic.test.ts create mode 100644 tests/manual/vfs-multiple-init-diagnostic.test.ts create mode 100644 tests/manual/workshop-type-filtering-test.ts create mode 100644 tests/unit/workshop-vfs-diagnostic.test.ts diff --git a/EXPLORATION_SUMMARY.md b/EXPLORATION_SUMMARY.md deleted file mode 100644 index 0b7176e1..00000000 --- a/EXPLORATION_SUMMARY.md +++ /dev/null @@ -1,393 +0,0 @@ -# Brainy Storage Adapter Architecture - Exploration Summary - -## Overview - -This exploration analyzed the complete storage adapter architecture in Brainy to understand how it works and determine whether a TypeAwareStorageAdapter can be added alongside existing adapters. - -## Key Findings - -### 1. Architecture is Clean and Extensible - -Brainy implements a **well-designed, modular storage adapter architecture** using: -- **Interface-based design** (StorageAdapter interface in coreTypes.ts) -- **Abstract base classes** for common functionality -- **Concrete implementations** for specific backends -- **Factory pattern** for runtime adapter selection - -### 2. Six Storage Adapters Currently Exist - -| Adapter | Platform | Backend | File | Lines | -|---------|----------|---------|------|-------| -| FileSystemStorage | Node.js | Local filesystem | fileSystemStorage.ts | 2,677 | -| MemoryStorage | Browser/Node.js | In-memory Maps | memoryStorage.ts | 822 | -| S3CompatibleStorage | Node.js | AWS S3, Cloudflare R2, GCS (S3 API) | s3CompatibleStorage.ts | 5,000+ | -| GcsStorage | Node.js | Google Cloud Storage (native SDK) | gcsStorage.ts | 1,835 | -| OPFSStorage | Browser | Origin Private File System | opfsStorage.ts | - | -| R2Storage | Node.js | Alias for S3CompatibleStorage | (alias) | - | - -### 3. Inheritance Hierarchy is Clean - -``` -StorageAdapter (interface - 27 methods) - ↓ -BaseStorageAdapter (abstract - 1,156 lines) - ├─ Statistics management - ├─ Throttling detection - ├─ Count management (O(1)) - └─ Service tracking - ↓ -BaseStorage (abstract - 1,098 lines) - ├─ 2-file system (vectors + metadata) - ├─ UUID-based sharding (256 shards) - ├─ Pagination support - └─ Metadata routing - ↓ -Concrete Adapters (FileSystem, Memory, S3, GCS, OPFS) -``` - -### 4. Core Components - -**Storage System Files (~13,000+ lines total):** -- `src/coreTypes.ts` - StorageAdapter interface -- `src/storage/baseStorageAdapter.ts` - Abstract base (1,156 lines) -- `src/storage/baseStorage.ts` - Core layer (1,098 lines) -- `src/storage/storageFactory.ts` - Factory for selection -- `src/storage/adapters/*.ts` - Concrete implementations -- `src/storage/sharding.ts` - UUID sharding utilities -- `src/storage/cacheManager.ts` - LRU cache - -**Supporting Utilities:** -- `src/utils/writeBuffer.ts` - Batch operations -- `src/utils/adaptiveBackpressure.ts` - Flow control -- `src/utils/requestCoalescer.ts` - Request deduplication -- `src/storage/backwardCompatibility.ts` - Migration support - -### 5. Storage Path Structure - -**Modern Entity-Based Structure:** -``` -entities/ -├── nouns/vectors/{shard}/{id}.json (vector data) -├── nouns/metadata/{shard}/{id}.json (flexible metadata) -├── nouns/hnsw/{shard}/{id}.json (HNSW graph) -├── verbs/vectors/{shard}/{id}.json -├── verbs/metadata/{shard}/{id}.json -└── verbs/hnsw/{shard}/{id}.json - -_system/ -├── statistics.json (aggregate counts) -├── counts.json (O(1) totals) -└── hnsw-system.json (HNSW metadata) -``` - -**Sharding:** UUID first 2 hex chars = 256 shard directories (00-ff) - -### 6. 2-File System Design - -Brainy separates **vector data** from **metadata** for scalability: -- **File 1:** `vectors/{id}.json` - Vector, HNSW connections (lightweight) -- **File 2:** `metadata/{id}.json` - Flexible metadata (any schema) - -**Benefits:** -- Decouple vector operations from metadata queries -- Enable type-aware queries without loading vectors -- Independent scaling of vector vs metadata storage -- Support for metadata-only updates - -### 7. Brainy Integration - -How Brainy uses storage: - -```typescript -// In brainy.ts -class Brainy { - private storage!: BaseStorage - - async init(config: BrainyConfig): Promise { - // Factory creates appropriate adapter - this.storage = await createStorage(config.storage) as BaseStorage - await this.storage.init() - - // Pass to HNSW index - this.index = new HNSWIndex(this.storage, ...) - } -} -``` - -Key insight: **Brainy only knows about `BaseStorage` interface, not specific adapters** - -### 8. Design Patterns Used - -1. **Factory Pattern** - `createStorage()` selects adapter at runtime -2. **Strategy Pattern** - Adapters are interchangeable -3. **Template Method** - BaseStorage defines skeleton, adapters fill details -4. **Adapter Pattern** - Maps different backends to same interface -5. **Decorator Pattern** - Could wrap adapters (e.g., TypeAware wrapper) - ---- - -## Answer: Can TypeAwareStorageAdapter Be Added? - -### YES - DEFINITIVELY - -**TypeAwareStorageAdapter can be added as a new adapter alongside existing ones WITHOUT replacing them.** - -### Reasons - -1. **Factory Pattern:** Multiple adapters coexist via factory function -2. **No Coupling:** Brainy depends on `BaseStorage` interface, not specific adapters -3. **Clean Inheritance:** Just extend `BaseStorage` like all other adapters -4. **Isolated:** Type awareness doesn't affect other adapters -5. **Backward Compatible:** Existing code continues to work unchanged - -### Implementation Path - -**3 Simple Steps:** - -**Step 1: Create new adapter file** -```typescript -// src/storage/adapters/typeAwareStorageAdapter.ts -export class TypeAwareStorageAdapter extends BaseStorage { - // Implement 17 abstract methods - // Add type indexing logic -} -``` - -**Step 2: Update factory** -```typescript -// src/storage/storageFactory.ts -if (options.type === 'type-aware') { - return new TypeAwareStorageAdapter(options) -} -``` - -**Step 3: Update options interface** -```typescript -// src/storage/storageFactory.ts -export interface StorageOptions { - type?: 'auto' | 'memory' | 'filesystem' | 's3' | 'gcs' | 'type-aware' - typeAwareStorage?: { ... } -} -``` - -**No changes needed to:** -- Brainy.ts -- coreTypes.ts (unless adding new methods) -- Existing adapters -- HNSW index -- Any other components - -### Abstract Methods to Implement - -When creating TypeAwareStorageAdapter, implement these 17 methods: - -**Noun/Verb Operations (6):** -- `saveNoun_internal()` -- `getNoun_internal()` -- `deleteNoun_internal()` -- `saveVerb_internal()` -- `getVerb_internal()` -- `deleteVerb_internal()` - -**Path Operations (4):** -- `writeObjectToPath()` -- `readObjectFromPath()` -- `deleteObjectFromPath()` -- `listObjectsUnderPath()` - -**Count Management (2):** -- `initializeCounts()` -- `persistCounts()` - -**Statistics (2):** -- `saveStatisticsData()` -- `getStatisticsData()` - -**Lifecycle (3):** -- `init()` -- `clear()` -- `getStorageStatus()` - -### Recommended Design Approach - -**Option A: Direct Implementation (Recommended)** -``` -TypeAwareStorageAdapter -├─ Extends BaseStorage -├─ Implements all 17 abstract methods -├─ Adds type indexing logic -└─ Can back any storage engine -``` - -**Option B: Wrapper/Decorator Pattern** -``` -TypeAwareStorageAdapter (wrapper) -├─ Wraps any BaseStorage adapter -├─ Intercepts saveNoun/saveVerb -├─ Tracks types in separate index -└─ Delegates all operations -``` - ---- - -## Key Insights - -### Storage Architecture Strengths - -✅ **Well-organized:** Clear separation of concerns -✅ **Extensible:** Factory pattern makes adding adapters simple -✅ **Scalable:** Sharding, caching, batching, backpressure -✅ **Flexible:** Multiple backends coexist without conflicts -✅ **Type-safe:** Full TypeScript with proper interfaces -✅ **Production-ready:** Used in real deployments - -### What Makes This Possible - -1. **Interface-based design** - Adapters implement same contract -2. **Factory pattern** - Runtime selection without coupling -3. **No hardcoded dependencies** - Brainy uses `BaseStorage` type -4. **Common base class** - Shared logic prevents duplication -5. **Metadata separation** - 2-file system enables type indexing - -### Storage Adapter Evolution Path - -``` -Current State (v3.44.0): -├─ FileSystemStorage ✅ -├─ MemoryStorage ✅ -├─ S3CompatibleStorage ✅ -├─ GcsStorage ✅ -└─ OPFSStorage ✅ - -Future State (proposed): -├─ FileSystemStorage ✅ -├─ MemoryStorage ✅ -├─ S3CompatibleStorage ✅ -├─ GcsStorage ✅ -├─ OPFSStorage ✅ -└─ TypeAwareStorageAdapter ✅ (new) - -All coexist without conflicts -``` - ---- - -## Documents Created - -This exploration generated three comprehensive documents: - -### 1. STORAGE_ARCHITECTURE_ANALYSIS.md (28 KB) -Complete analysis covering: -- Current storage architecture overview -- All existing storage adapters -- StorageAdapter interface specification -- How Brainy uses storage -- Storage paths and patterns -- Storage adapter pattern analysis -- Detailed implementation recommendations -- Design patterns and best practices - -### 2. STORAGE_ADAPTER_QUICK_REFERENCE.md (8.6 KB) -Quick reference guide with: -- File locations -- Storage adapter hierarchy -- Abstract methods checklist (17 methods) -- Storage path structure -- 2-file system design -- Existing adapters overview -- Factory integration -- Performance characteristics -- Design patterns summary - -### 3. STORAGE_FILES_REFERENCE.md (13 KB) -Complete file reference with: -- All core storage files -- Line counts and purposes -- Each adapter's features -- Integration points -- Data flow diagrams -- Statistics tracking -- Type definitions -- Summary statistics table - ---- - -## Recommendations - -### For TypeAwareStorageAdapter Implementation - -1. **Use Direct Implementation approach** (not wrapper) - - Simpler to maintain - - Better performance - - Easier to debug - - Can back any storage engine - -2. **Implement as new entry in factory** - - `type: 'type-aware'` with storage config - - Auto-detection can select it - - No changes to existing code - -3. **Leverage 2-file system** - - Store type index in metadata files - - Queries don't require loading vectors - - Aligns with existing patterns - -4. **Inherit common functionality** - - Throttling detection - - Statistics tracking - - Caching and batching - - Count management (O(1)) - -5. **Follow existing patterns** - - Sharding strategy (first 2 hex chars) - - Path structure (entities/{noun|verb}/{vectors|metadata}/{shard}/{id}.json) - - Pagination support - - Metadata separation - -### For Integration - -1. Add new file: `/src/storage/adapters/typeAwareStorageAdapter.ts` -2. Modify: `/src/storage/storageFactory.ts` (add case + interface) -3. Optional: `/src/coreTypes.ts` (if extending StorageAdapter interface) -4. No changes needed elsewhere - -### For Testing - -1. Test with MemoryStorage first (fastest) -2. Test with FileSystemStorage (persistent) -3. Ensure all existing tests still pass -4. Add type-aware specific tests - ---- - -## Conclusion - -Brainy's storage adapter architecture is **professionally designed and inherently extensible**. Adding a TypeAwareStorageAdapter is straightforward because: - -- The architecture supports multiple concurrent adapters -- Brainy uses interface-based dependency injection -- The factory pattern enables runtime selection -- No breaking changes required anywhere - -**The answer is unambiguous: TypeAwareStorageAdapter can be added alongside existing adapters with minimal integration effort.** - ---- - -## Files Analyzed - -- `/src/coreTypes.ts` - Interface definition -- `/src/storage/baseStorageAdapter.ts` - Abstract base -- `/src/storage/baseStorage.ts` - Core layer -- `/src/storage/storageFactory.ts` - Factory -- `/src/storage/adapters/fileSystemStorage.ts` - FileSystem -- `/src/storage/adapters/memoryStorage.ts` - Memory -- `/src/storage/adapters/s3CompatibleStorage.ts` - S3/R2 -- `/src/storage/adapters/gcsStorage.ts` - GCS native -- `/src/storage/adapters/opfsStorage.ts` - Browser OPFS -- `/src/brainy.ts` - Main class -- Plus all supporting utilities and type definitions - -**Total files analyzed:** 50+ -**Total lines examined:** 13,000+ -**Analysis coverage:** Complete storage system - diff --git a/README_STORAGE_EXPLORATION.md b/README_STORAGE_EXPLORATION.md deleted file mode 100644 index de3d8894..00000000 --- a/README_STORAGE_EXPLORATION.md +++ /dev/null @@ -1,291 +0,0 @@ -# Storage Adapter Architecture Exploration - Documentation Index - -## Quick Answer - -**Can TypeAwareStorageAdapter be added alongside existing adapters?** - -**YES - ABSOLUTELY.** It can be added as a new adapter without replacing any existing ones. See **EXPLORATION_SUMMARY.md** for details. - ---- - -## Documentation Files - -### 1. EXPLORATION_SUMMARY.md (12 KB) -**START HERE** - Overview of the entire exploration - -**Contains:** -- Executive summary of findings -- Definitive answer to the main question -- Implementation roadmap (3 simple steps) -- List of 17 abstract methods to implement -- Key insights about architecture -- Recommended design approaches - -**Read this when:** You want a quick understanding of what we discovered - ---- - -### 2. STORAGE_ARCHITECTURE_ANALYSIS.md (28 KB) -**COMPREHENSIVE DEEP DIVE** - Complete technical analysis - -**Sections:** -1. Current storage architecture overview -2. Existing storage adapters (5 detailed profiles) -3. StorageAdapter interface specification (27 methods) -4. How Brainy uses storage -5. Storage factory pattern -6. Current storage paths and patterns -7. Storage adapter pattern analysis -8. Detailed recommendations for TypeAwareStorageAdapter -9. Storage directory structure details -10. Key design patterns -11. Summary and recommendations - -**Read this when:** You need comprehensive technical understanding - ---- - -### 3. STORAGE_ADAPTER_QUICK_REFERENCE.md (8.6 KB) -**QUICK LOOKUP GUIDE** - Fast reference for developers - -**Sections:** -- File locations (all storage files) -- Storage adapter hierarchy (visual tree) -- Abstract methods checklist (17 methods) -- Storage path structure (modern format) -- 2-file system design explanation -- Existing adapters overview (quick stats) -- Factory integration example -- Key inherited features -- Performance characteristics -- Design patterns used -- Conclusion and next steps - -**Read this when:** You're implementing TypeAwareStorageAdapter - ---- - -### 4. STORAGE_FILES_REFERENCE.md (13 KB) -**COMPLETE FILE REFERENCE** - Detailed information about each file - -**Sections:** -1. Core storage files (6 files described) -2. Storage adapter implementations (5 adapters detailed) -3. Storage patterns and utilities (5 utilities listed) -4. Integration points (3 main integration points) -5. Data flow examples (saving and querying) -6. Storage statistics tracking (JSON examples) -7. Type definitions (HNSWNoun, GraphVerb) -8. Summary statistics table (13,000+ lines) - -**Read this when:** You need details about specific storage files - ---- - -## Reading Guide by Use Case - -### I want a quick answer -1. Read this README (you're here!) -2. Read: EXPLORATION_SUMMARY.md - first 30% (Key Findings + Answer) - -### I'm implementing TypeAwareStorageAdapter -1. Read: EXPLORATION_SUMMARY.md (full) -2. Read: STORAGE_ADAPTER_QUICK_REFERENCE.md (Implementation section) -3. Reference: STORAGE_FILES_REFERENCE.md (for specific files) - -### I need comprehensive understanding -1. Read: EXPLORATION_SUMMARY.md -2. Read: STORAGE_ARCHITECTURE_ANALYSIS.md (sections 1-7) -3. Reference: STORAGE_ADAPTER_QUICK_REFERENCE.md - -### I'm debugging storage issues -1. Check: STORAGE_FILES_REFERENCE.md (which file handles what) -2. Check: STORAGE_ARCHITECTURE_ANALYSIS.md (data flow sections) -3. Check: STORAGE_ADAPTER_QUICK_REFERENCE.md (path structure) - -### I'm integrating with storage system -1. Read: STORAGE_ARCHITECTURE_ANALYSIS.md (sections 3-4) -2. Read: STORAGE_FILES_REFERENCE.md (integration points) -3. Reference: STORAGE_ADAPTER_QUICK_REFERENCE.md (as needed) - ---- - -## Key Findings Summary - -### Current State -- 5 storage adapters exist (FileSystem, Memory, S3, GCS, OPFS) -- 27-method StorageAdapter interface -- 17 abstract methods to implement for new adapters -- 13,000+ lines of storage code -- Clean inheritance hierarchy -- Factory pattern for runtime selection - -### Architecture Strengths -- Well-organized and modular -- Factory pattern enables multiple backends -- Interface-based design (no coupling) -- Common base class (code reuse) -- 2-file system (separation of concerns) - -### For TypeAwareStorageAdapter -- Can extend BaseStorage class -- Must implement 17 abstract methods -- Simple factory integration (1 case + interface update) -- No changes to existing code -- Inherits statistics, throttling, caching - ---- - -## Core Facts - -### Storage Layers -``` -StorageAdapter interface (27 methods) - ↓ -BaseStorageAdapter (1,156 lines - common functionality) - ↓ -BaseStorage (1,098 lines - routing & pagination) - ↓ -Concrete Adapters (FileSystem, Memory, S3, GCS, OPFS) -``` - -### Storage Paths -``` -entities/nouns/vectors/{shard}/{id}.json ← vector data -entities/nouns/metadata/{shard}/{id}.json ← flexible metadata -entities/verbs/vectors/{shard}/{id}.json -entities/verbs/metadata/{shard}/{id}.json -_system/statistics.json ← aggregate stats -_system/counts.json ← O(1) totals -``` - -### Sharding -- UUID first 2 hex characters (00-ff) -- 256 shard directories -- Handles 2.5M+ entities efficiently - -### 2-File System -- File 1: Vectors (lightweight, always loaded) -- File 2: Metadata (flexible schema, separately loaded) -- Enables type-aware queries without loading vectors - ---- - -## Implementation Checklist - -For adding TypeAwareStorageAdapter: - -- [ ] Create `/src/storage/adapters/typeAwareStorageAdapter.ts` -- [ ] Extend BaseStorage class -- [ ] Implement 17 abstract methods: - - [ ] saveNoun_internal() - - [ ] getNoun_internal() - - [ ] deleteNoun_internal() - - [ ] saveVerb_internal() - - [ ] getVerb_internal() - - [ ] deleteVerb_internal() - - [ ] writeObjectToPath() - - [ ] readObjectFromPath() - - [ ] deleteObjectFromPath() - - [ ] listObjectsUnderPath() - - [ ] initializeCounts() - - [ ] persistCounts() - - [ ] saveStatisticsData() - - [ ] getStatisticsData() - - [ ] init() - - [ ] clear() - - [ ] getStorageStatus() -- [ ] Update `/src/storage/storageFactory.ts`: - - [ ] Add case for 'type-aware' - - [ ] Update StorageOptions interface -- [ ] Test with MemoryStorage -- [ ] Test with FileSystemStorage -- [ ] Verify existing tests still pass - ---- - -## Quick Reference - -### Files to Analyze -- `/src/coreTypes.ts` - StorageAdapter interface -- `/src/storage/baseStorageAdapter.ts` - Abstract base -- `/src/storage/baseStorage.ts` - Core layer -- `/src/storage/storageFactory.ts` - Factory -- `/src/storage/adapters/memoryStorage.ts` - Simple example - -### Key Classes -- `StorageAdapter` - Interface (27 methods) -- `BaseStorageAdapter` - Abstract base (1,156 lines) -- `BaseStorage` - Abstract impl (1,098 lines) -- `FileSystemStorage` - Concrete impl (2,677 lines) -- `MemoryStorage` - Simple impl (822 lines) - -### Key Methods to Implement -- Node/Verb: saveNoun_internal, getNoun_internal, etc. -- Path: writeObjectToPath, readObjectFromPath, etc. -- Counts: initializeCounts, persistCounts -- Stats: saveStatisticsData, getStatisticsData -- Lifecycle: init, clear, getStorageStatus - -### Design Patterns -1. Factory - `createStorage()` for adapter selection -2. Strategy - Adapters are interchangeable -3. Template Method - BaseStorage defines skeleton -4. Adapter - Maps different backends to same interface -5. Decorator - Can wrap adapters if needed - ---- - -## Analysis Statistics - -| Metric | Count | -|--------|-------| -| Files analyzed | 50+ | -| Lines of code examined | 13,000+ | -| Storage adapters found | 5 | -| Abstract methods to implement | 17 | -| Interface methods | 27 | -| Storage backends supported | 6 (FS, Memory, S3, GCS, OPFS, R2) | -| Documentation pages created | 4 | - ---- - -## Contact & Questions - -For questions about: -- **Architecture:** See STORAGE_ARCHITECTURE_ANALYSIS.md -- **Specific files:** See STORAGE_FILES_REFERENCE.md -- **Quick lookup:** See STORAGE_ADAPTER_QUICK_REFERENCE.md -- **Overall findings:** See EXPLORATION_SUMMARY.md - ---- - -## Conclusion - -Brainy's storage architecture is **professionally designed** and **inherently extensible**. TypeAwareStorageAdapter can be added as a new adapter in just a few minutes by: - -1. Creating a new class extending BaseStorage -2. Implementing 17 abstract methods -3. Registering in the factory - -**No breaking changes required. No existing code needs modification.** - -The architecture supports multiple backends coexisting peacefully through proper use of: -- Interface-based design -- Factory pattern -- Dependency injection -- Abstract base classes - -This is a textbook example of good software architecture. - ---- - -## Document Metadata - -**Created:** October 15, 2025 -**Repository:** Brainy (Neural Database) -**Version:** Analysis of v3.44.0 -**Scope:** Complete storage adapter architecture -**Coverage:** 100% of storage system - -Generated with thorough code analysis and deep understanding of the system. diff --git a/SECURITY.md b/SECURITY.md deleted file mode 100644 index 9d54e26f..00000000 --- a/SECURITY.md +++ /dev/null @@ -1,389 +0,0 @@ -# Security Best Practices for Brainy - -## 🔒 Data Security - -### Encryption at Rest -```typescript -const brain = new Brainy({ - storage: { - type: 's3', - options: { - encryption: 'AES256', // Server-side encryption - kmsKeyId: process.env.KMS_KEY_ID // Optional KMS key - } - } -}) -``` - -### Encryption in Transit -- Always use HTTPS/TLS for API endpoints -- Enable SSL for database connections -- Use VPN or private networks for internal communication - -## 🔑 Authentication & Authorization - -### API Key Management -```typescript -// Middleware example -app.use('/api/brainy', (req, res, next) => { - const apiKey = req.headers['x-api-key'] - - if (!apiKey || !isValidApiKey(apiKey)) { - return res.status(401).json({ error: 'Unauthorized' }) - } - - // Rate limit by API key - const limit = getRateLimitForKey(apiKey) - if (exceedsRateLimit(apiKey, limit)) { - return res.status(429).json({ error: 'Rate limit exceeded' }) - } - - next() -}) -``` - -### JWT Authentication -```typescript -import jwt from 'jsonwebtoken' - -// Verify JWT token -app.use('/api/brainy', (req, res, next) => { - const token = req.headers.authorization?.split(' ')[1] - - try { - const decoded = jwt.verify(token, process.env.JWT_SECRET) - req.user = decoded - next() - } catch (error) { - return res.status(401).json({ error: 'Invalid token' }) - } -}) -``` - -## 🛡️ Input Validation & Sanitization - -### Query Validation -```typescript -import { z } from 'zod' - -const SearchSchema = z.object({ - query: z.string().min(1).max(1000), - limit: z.number().min(1).max(100).default(10), - metadata: z.record(z.unknown()).optional() -}) - -app.post('/api/search', async (req, res) => { - try { - const params = SearchSchema.parse(req.body) - const results = await brain.find(params) - res.json(results) - } catch (error) { - if (error instanceof z.ZodError) { - return res.status(400).json({ error: 'Invalid input', details: error.errors }) - } - throw error - } -}) -``` - -### Metadata Sanitization -```typescript -function sanitizeMetadata(metadata: any): any { - // Remove potential XSS vectors - const sanitized = {} - - for (const [key, value] of Object.entries(metadata)) { - // Sanitize keys - const cleanKey = key.replace(/[<>'"]/g, '') - - // Sanitize values - if (typeof value === 'string') { - sanitized[cleanKey] = value.replace(/[<>'"]/g, '') - } else if (typeof value === 'object' && value !== null) { - sanitized[cleanKey] = sanitizeMetadata(value) - } else { - sanitized[cleanKey] = value - } - } - - return sanitized -} - -// Use before adding to brain -const sanitizedData = { - text: sanitizeText(input.text), - metadata: sanitizeMetadata(input.metadata) -} -await brain.add(sanitizedData) -``` - -## 🚦 Rate Limiting - -### Per-IP Rate Limiting -```typescript -import rateLimit from 'express-rate-limit' - -const limiter = rateLimit({ - windowMs: 15 * 60 * 1000, // 15 minutes - max: 100, // Limit each IP to 100 requests per windowMs - message: 'Too many requests from this IP' -}) - -app.use('/api/brainy', limiter) -``` - -### Per-User Rate Limiting -```typescript -const userLimits = new Map() - -function checkUserRateLimit(userId: string, limit = 1000): boolean { - const now = Date.now() - const userRequests = userLimits.get(userId) || [] - - // Remove old requests (older than 1 hour) - const recentRequests = userRequests.filter((time: number) => - now - time < 3600000 - ) - - if (recentRequests.length >= limit) { - return false - } - - recentRequests.push(now) - userLimits.set(userId, recentRequests) - return true -} -``` - -## 🔍 Audit Logging - -### Comprehensive Audit Trail -```typescript -interface AuditLog { - timestamp: Date - userId: string - action: string - resource: string - details: any - ip: string - userAgent: string -} - -class AuditLogger { - async log(entry: AuditLog): Promise { - // Log to secure storage - await this.storage.append('audit.log', JSON.stringify(entry) + '\n') - - // Alert on suspicious activity - if (this.isSuspicious(entry)) { - await this.alertSecurityTeam(entry) - } - } - - private isSuspicious(entry: AuditLog): boolean { - // Check for patterns like: - // - Multiple failed auth attempts - // - Unusual data access patterns - // - Bulk data exports - // - Access from new locations - return false // Implement your logic - } -} - -// Use in your API -app.use(async (req, res, next) => { - const entry: AuditLog = { - timestamp: new Date(), - userId: req.user?.id || 'anonymous', - action: req.method, - resource: req.path, - details: req.body, - ip: req.ip, - userAgent: req.headers['user-agent'] - } - - await auditLogger.log(entry) - next() -}) -``` - -## 🗑️ Data Privacy & GDPR Compliance - -### Right to Deletion -```typescript -async function deleteUserData(userId: string): Promise { - // Find all items belonging to user - const userItems = await brain.find({ - metadata: { userId } - }) - - // Delete each item - for (const item of userItems) { - await brain.delete(item.id) - } - - // Log the deletion - await auditLogger.log({ - timestamp: new Date(), - userId, - action: 'DELETE_USER_DATA', - resource: 'user_data', - details: { itemCount: userItems.length }, - ip: 'system', - userAgent: 'gdpr-compliance' - }) -} -``` - -### Data Export -```typescript -async function exportUserData(userId: string): Promise { - // Get all user data - const items = await brain.find({ - metadata: { userId } - }) - - // Get all relationships - const relationships = [] - for (const item of items) { - const relations = await brain.getRelations(item.id) - relationships.push(...relations) - } - - return { - exportDate: new Date().toISOString(), - userId, - items, - relationships, - metadata: { - itemCount: items.length, - relationshipCount: relationships.length - } - } -} -``` - -## 🚨 Security Headers - -### Express.js Security Headers -```typescript -import helmet from 'helmet' - -app.use(helmet({ - contentSecurityPolicy: { - directives: { - defaultSrc: ["'self'"], - styleSrc: ["'self'", "'unsafe-inline'"], - scriptSrc: ["'self'"], - imgSrc: ["'self'", "data:", "https:"], - }, - }, - hsts: { - maxAge: 31536000, - includeSubDomains: true, - preload: true - } -})) -``` - -## 🔐 Environment Variables - -### Secure Configuration -```bash -# .env.production -NODE_ENV=production -JWT_SECRET= -DATABASE_URL= -AWS_ACCESS_KEY_ID= -AWS_SECRET_ACCESS_KEY= -REDIS_PASSWORD= -ENCRYPTION_KEY=<32-byte-random-key> -``` - -### Runtime Validation -```typescript -import { z } from 'zod' - -const EnvSchema = z.object({ - NODE_ENV: z.enum(['development', 'production', 'test']), - JWT_SECRET: z.string().min(32), - DATABASE_URL: z.string().url(), - AWS_REGION: z.string(), - REDIS_HOST: z.string(), - REDIS_PORT: z.string().transform(Number), - ENCRYPTION_KEY: z.string().length(64) // Hex encoded 32 bytes -}) - -// Validate on startup -try { - const env = EnvSchema.parse(process.env) - console.log('✅ Environment configuration valid') -} catch (error) { - console.error('❌ Invalid environment configuration:', error) - process.exit(1) -} -``` - -## 🛠️ Security Checklist - -### Development -- [ ] Use `.env` files for secrets (never commit) -- [ ] Enable TypeScript strict mode -- [ ] Run security linting (eslint-plugin-security) -- [ ] Use dependency scanning (npm audit) -- [ ] Implement unit tests for auth logic - -### Staging -- [ ] Penetration testing -- [ ] Load testing with security scenarios -- [ ] Review audit logs -- [ ] Test rate limiting -- [ ] Verify encryption working - -### Production -- [ ] Enable all security headers -- [ ] Configure WAF (Web Application Firewall) -- [ ] Set up intrusion detection -- [ ] Enable DDoS protection -- [ ] Configure automated backups -- [ ] Set up security alerts -- [ ] Regular security audits -- [ ] Incident response plan - -## 📊 Monitoring & Alerts - -### Security Metrics -```typescript -// Track and alert on: -const securityMetrics = { - failedAuthAttempts: 0, - rateLimitHits: 0, - suspiciousQueries: 0, - largeDataExports: 0, - unusualAccessPatterns: 0 -} - -// Alert thresholds -const alertThresholds = { - failedAuthAttempts: 10, // per minute - rateLimitHits: 100, // per minute - suspiciousQueries: 5, // per minute - largeDataExports: 10, // per hour -} -``` - -## 🚪 Incident Response - -### Response Plan -1. **Detect** - Monitoring alerts trigger -2. **Contain** - Isolate affected systems -3. **Investigate** - Review audit logs -4. **Remediate** - Fix vulnerability -5. **Recover** - Restore normal operations -6. **Review** - Post-incident analysis - -### Emergency Contacts -- Security Team: security@yourcompany.com -- On-call Engineer: Use PagerDuty -- Legal Team: legal@yourcompany.com -- PR Team: pr@yourcompany.com \ No newline at end of file diff --git a/STORAGE_ADAPTER_QUICK_REFERENCE.md b/STORAGE_ADAPTER_QUICK_REFERENCE.md deleted file mode 100644 index 62ed62fe..00000000 --- a/STORAGE_ADAPTER_QUICK_REFERENCE.md +++ /dev/null @@ -1,291 +0,0 @@ -# Brainy Storage Adapter - Quick Reference Guide - -## File Locations - -``` -src/storage/ -├── baseStorageAdapter.ts # Abstract base class (1,156 lines) -├── baseStorage.ts # Implementation layer (1,098 lines) -├── storageFactory.ts # Factory for adapter selection -├── sharding.ts # UUID-based sharding utilities -├── cacheManager.ts # Cache implementation -└── adapters/ - ├── fileSystemStorage.ts # Node.js file system (2,677 lines) - ├── memoryStorage.ts # In-memory storage (822 lines) - ├── s3CompatibleStorage.ts # AWS S3 / R2 / GCS compat (5000+ lines) - ├── gcsStorage.ts # Google Cloud Storage native (1,835 lines) - ├── opfsStorage.ts # Browser OPFS storage - └── baseStorageAdapter.ts # Base class - -src/coreTypes.ts -└── StorageAdapter interface (27 methods) -``` - -## Storage Adapter Hierarchy - -``` -┌─ StorageAdapter (interface) -│ ├─ StorageAdapter.init() -│ ├─ StorageAdapter.saveNoun() -│ ├─ StorageAdapter.getNouns() -│ └─ ... (23 more methods) -│ -└─ BaseStorageAdapter (abstract class) - ├─ Statistics management - ├─ Throttling detection - ├─ Count tracking (O(1)) - ├─ Service-level statistics - └─ Abstract methods for subclasses - - └─ BaseStorage (abstract class) - ├─ 2-file system routing - ├─ Pagination support - ├─ Metadata handling - ├─ Public API (saveNoun, getNoun, etc.) - └─ Abstract internal methods - - └─ Concrete Adapters - ├─ FileSystemStorage - ├─ MemoryStorage - ├─ S3CompatibleStorage - ├─ GcsStorage - └─ OPFSStorage -``` - -## Abstract Methods to Implement - -When creating a new adapter, extend `BaseStorage` and implement: - -### Noun/Verb Operations (6 methods) -```typescript -protected abstract saveNoun_internal(noun: HNSWNoun): Promise -protected abstract getNoun_internal(id: string): Promise -protected abstract deleteNoun_internal(id: string): Promise -protected abstract saveVerb_internal(verb: HNSWVerb): Promise -protected abstract getVerb_internal(id: string): Promise -protected abstract deleteVerb_internal(id: string): Promise -``` - -### Path Operations (4 methods) -```typescript -protected abstract writeObjectToPath(path: string, data: any): Promise -protected abstract readObjectFromPath(path: string): Promise -protected abstract deleteObjectFromPath(path: string): Promise -protected abstract listObjectsUnderPath(prefix: string): Promise -``` - -### Count Management (2 methods) -```typescript -protected abstract initializeCounts(): Promise -protected abstract persistCounts(): Promise -``` - -### Statistics (2 methods) -```typescript -protected abstract saveStatisticsData(statistics: StatisticsData): Promise -protected abstract getStatisticsData(): Promise -``` - -### Lifecycle (3 methods) -```typescript -abstract init(): Promise -abstract clear(): Promise -abstract getStorageStatus(): Promise -``` - -**Total: 17 abstract methods to implement** - -## Storage Path Structure - -### Modern Entity-Based Structure -``` -storage-root/ -├── entities/ -│ ├── nouns/vectors/{shard}/{id}.json ← Vector data -│ ├── nouns/metadata/{shard}/{id}.json ← Metadata -│ ├── nouns/hnsw/{shard}/{id}.json ← HNSW graph -│ ├── verbs/vectors/{shard}/{id}.json -│ ├── verbs/metadata/{shard}/{id}.json -│ └── verbs/hnsw/{shard}/{id}.json -├── indexes/ -│ ├── metadata/... ← Search indexes -│ └── graph/... -└── _system/ - ├── statistics.json ← Aggregate stats - ├── counts.json ← O(1) totals - └── hnsw-system.json ← HNSW metadata -``` - -### Shard Format -- First 2 hex chars of UUID (00-ff) = 256 shards -- Example: `ab123456-...` → stored in `ab/` directory -- Enables 2.5M+ entities with consistent performance - -## 2-File System Design - -### Vector File (always loaded with HNSW) -```json -{ - "id": "ab123456-...", - "vector": [0.1, 0.2, ...], - "connections": { "0": [...], "1": [...] }, - "level": 2 -} -``` - -### Metadata File (loaded separately) -```json -{ - "noun": "Person", - "name": "Alice", - "email": "alice@example.com", - "createdAt": "...", - "service": "user-service" -} -``` - -**Benefit:** Decouple vector operations from flexible metadata queries - -## Existing Adapters Overview - -### FileSystemStorage (Node.js) -- 2,677 lines -- Sharding with migration support -- File-based locking for multi-process -- Production-ready - -### MemoryStorage (Testing) -- 822 lines -- In-memory Maps -- Fast for testing -- No persistence - -### S3CompatibleStorage (Cloud) -- 5,000+ lines -- AWS S3, Cloudflare R2, GCS (via S3 API) -- Adaptive batching, request coalescing -- High-volume mode, write buffers - -### GcsStorage (Google Cloud) -- 1,835 lines -- Native @google-cloud/storage SDK -- ADC, service account, HMAC auth -- Cache managers, backpressure - -### OPFSStorage (Browser) -- Browser Origin Private File System -- Persistent across sessions -- Modern browsers only - -## Factory Integration - -```typescript -// src/storage/storageFactory.ts -const storage = await createStorage({ - type: 'filesystem', // auto, memory, filesystem, s3, gcs, gcs-native, opfs - path: './data', - s3Storage: { bucketName, region, ... }, - gcsStorage: { bucketName, credentials, ... }, -}) -``` - -## Adding TypeAwareStorageAdapter - -### Recommended Approach: Direct Implementation -```typescript -// src/storage/adapters/typeAwareStorageAdapter.ts -export class TypeAwareStorageAdapter extends BaseStorage { - // Implement 17 abstract methods - // Add type indexing logic - // Track noun/verb types in separate indexes -} -``` - -### Integration Steps -1. Create `/src/storage/adapters/typeAwareStorageAdapter.ts` -2. Add to factory in `/src/storage/storageFactory.ts` -3. Update StorageOptions interface with `type: 'type-aware'` -4. No changes to Brainy.ts or existing adapters needed - -## Key Features Inherited from BaseStorageAdapter - -- **Statistics Caching:** Batches updates for efficiency -- **Throttling Detection:** Handles 429/503 errors -- **Count Management:** O(1) operations with persistence -- **Service Tracking:** Per-service statistics -- **Field Name Tracking:** Metadata field discovery - -## Performance Characteristics - -### O(1) Operations -- `getNounCount()` - total noun count -- `getVerbCount()` - total verb count - -### O(n) Operations -- `getNouns()` - paginated listing (n = page size) -- `getVerbs()` - paginated listing -- `getNounsByNounType()` - filter by type -- `getVerbsBySource()` - filter by source - -### Cloud Storage Features (GCS, S3) -- High-volume mode detection -- Adaptive batching -- Request coalescing for deduplication -- Write buffers for bulk operations -- Backpressure management -- Socket pool management - -## Testing Storage Adapters - -All adapters implement the same interface, so: - -```typescript -// Test with MemoryStorage (fastest) -const storage = new MemoryStorage() - -// Test with FileSystemStorage (persistent) -const storage = new FileSystemStorage('./test-data') - -// All adapters support the same operations -await storage.init() -await storage.saveNoun(noun) -const result = await storage.getNoun(id) -await storage.clear() -``` - -## Brainy Integration - -```typescript -export class Brainy { - private storage!: BaseStorage - - async init(config: BrainyConfig): Promise { - // Factory creates appropriate adapter - this.storage = await createStorage(config.storage) as BaseStorage - await this.storage.init() - - // Pass to HNSW index - this.index = new HNSWIndex(this.storage, ...) - } -} -``` - -Brainy depends on `BaseStorage` interface, not specific adapters. - -## Design Patterns Used - -1. **Factory Pattern** - `createStorage()` selects adapter -2. **Strategy Pattern** - Adapters are interchangeable -3. **Template Method** - BaseStorage defines skeleton -4. **Decorator Pattern** - Can wrap adapters (e.g., TypeAware wrapper) -5. **Adapter Pattern** - Maps different storage backends to same interface - -## Conclusion - -TypeAwareStorageAdapter can be added as a **new adapter alongside existing ones** without: -- Modifying Brainy.ts -- Replacing existing adapters -- Breaking the StorageAdapter interface -- Changing how storage is used throughout the codebase - -Simply extend `BaseStorage`, implement 17 abstract methods, and register in `storageFactory.ts`. diff --git a/STORAGE_FILES_REFERENCE.md b/STORAGE_FILES_REFERENCE.md deleted file mode 100644 index 8ddc531d..00000000 --- a/STORAGE_FILES_REFERENCE.md +++ /dev/null @@ -1,431 +0,0 @@ -# Brainy Storage System - Complete File Reference - -## Core Storage Files - -### 1. Storage Interface Definition -**File:** `src/coreTypes.ts` -- **Lines:** ~250 (StorageAdapter interface) -- **Type:** Interface definition -- **Content:** - - `StorageAdapter` interface (27 methods) - - Supporting types: `HNSWNoun`, `HNSWVerb`, `GraphVerb` - - Statistics types: `StatisticsData`, `ServiceStatistics` - -### 2. Base Storage Adapter (Abstract Class) -**File:** `src/storage/adapters/baseStorageAdapter.ts` -- **Lines:** 1,156 -- **Type:** Abstract base class -- **Purpose:** Common functionality for all adapters -- **Key Features:** - - Statistics caching and batching - - Throttling detection and backoff - - Count management (O(1) operations) - - Service-level statistics tracking - - Field name discovery -- **Implements:** StorageAdapter interface -- **Key Methods:** - - `flushStatistics()` - Write statistics to storage - - `isThrottlingError()` - Detect cloud storage throttling - - `handleThrottling()` - Exponential backoff - - `trackThrottlingEvent()` - Record throttling events - - `incrementStatistic()` - Increment service stats - - `decrementStatistic()` - Decrement service stats - -### 3. Base Storage Implementation -**File:** `src/storage/baseStorage.ts` -- **Lines:** 1,098 -- **Type:** Abstract class extending BaseStorageAdapter -- **Purpose:** Core storage logic (routing, sharding, metadata) -- **Key Features:** - - 2-file system implementation (vectors + metadata) - - UUID-based sharding (256 directories) - - Metadata routing and separation - - Pagination support - - Backward compatibility with legacy paths -- **Implements:** - - Public API (saveNoun, getNoun, etc.) - - Metadata operations (saveNounMetadata, etc.) -- **Key Abstract Methods:** - - `saveNoun_internal()` - Adapter-specific noun save - - `getNoun_internal()` - Adapter-specific noun read - - `writeObjectToPath()` - Generic write - - `readObjectFromPath()` - Generic read - - `listObjectsUnderPath()` - Listing support - -### 4. Storage Factory -**File:** `src/storage/storageFactory.ts` -- **Lines:** ~200 -- **Purpose:** Factory function for adapter selection -- **Function:** `createStorage(options: StorageOptions): Promise` -- **Selection Logic:** - 1. Forced memory/filesystem (testing) - 2. Explicit type selection - 3. Auto-detection (browser vs Node.js) -- **Supported Types:** - - `'memory'` - MemoryStorage - - `'filesystem'` - FileSystemStorage - - `'s3'` - S3CompatibleStorage (AWS S3, R2) - - `'gcs'` - S3CompatibleStorage (GCS S3 API) - - `'gcs-native'` - GcsStorage (native SDK) - - `'opfs'` - OPFSStorage (browser) - -### 5. Sharding Utilities -**File:** `src/storage/sharding.ts` -- **Purpose:** UUID-based sharding helpers -- **Key Functions:** - - `getShardIdFromUuid(id: string): string` - Get first 2 hex chars - - `getShardIdByIndex(index: number): string` - Get shard by index - - `getAllShardIds(): string[]` - All 256 shard IDs -- **Constants:** - - `TOTAL_SHARDS = 256` - - `MIN_SHARD_ID = '00'` - - `MAX_SHARD_ID = 'ff'` - -### 6. Cache Manager -**File:** `src/storage/cacheManager.ts` -- **Purpose:** Generic LRU cache for storage -- **Type:** Generic class `CacheManager` -- **Used By:** GcsStorage, S3CompatibleStorage -- **Features:** - - LRU eviction - - TTL support - - Max size limits - - Batch operations - -## Storage Adapter Implementations - -### FileSystemStorage (Node.js) -**File:** `src/storage/adapters/fileSystemStorage.ts` -- **Lines:** 2,677 -- **Type:** Concrete implementation extending BaseStorage -- **Platform:** Node.js only -- **Storage Target:** Local file system -- **Key Features:** - - Sharding with automatic migration - - File-based locking (multi-process) - - O(1) count persistence - - HNSW index persistence - - Dual-write for migrations - - Path depth migration (0→1, 2→1) - -**Key Methods:** -```typescript -private getNounPath(id: string, depth: number): string -private getVerbPath(id: string, depth: number): string -private getNounMetadataPath(id: string, depth: number): string -async migrateShardingStructure(fromDepth: number, toDepth: number): Promise -async migrateFromOldStructure(): Promise -``` - -**Statistics Tracking:** -- Persists counts to `_system/counts.json` -- Tracks noun/verb type distributions -- Service-level activity timestamps -- Field name discovery - -### MemoryStorage (Testing) -**File:** `src/storage/adapters/memoryStorage.ts` -- **Lines:** 822 -- **Type:** Concrete implementation extending BaseStorage -- **Platform:** Browser & Node.js -- **Storage Target:** In-memory Maps -- **Key Features:** - - Fast for testing - - No persistence (ephemeral) - - Full pagination support - - Filtering capabilities - - 2-file system simulation - -**Key Maps:** -```typescript -private nouns: Map -private verbs: Map -private objectStore: Map // Unified metadata store -``` - -**Methods:** -- `getNouns()` - Paginated noun listing -- `getVerbs()` - Paginated verb listing -- `getMetadataBatch()` - Batch metadata loading - -### S3CompatibleStorage (Cloud) -**File:** `src/storage/adapters/s3CompatibleStorage.ts` -- **Lines:** 5,000+ -- **Type:** Concrete implementation extending BaseStorage -- **Platform:** Node.js (server-side) -- **Storage Targets:** - - Amazon S3 - - Cloudflare R2 (via S3 API) - - Google Cloud Storage (via S3 API) - -**Key Features:** -- Adaptive batching (10-1000 items) -- Request coalescing for deduplication -- High-volume mode detection -- Write buffers for bulk operations -- Socket pool management -- Backpressure system -- Change log tracking -- Cache management (nouns & verbs) - -**Performance Optimization:** -- `nounWriteBuffer` - Batches noun writes -- `verbWriteBuffer` - Batches verb writes -- `requestCoalescer` - Deduplicates requests -- `highVolumeMode` - Activates at >20 pending ops -- `baseBatchSize` - Adaptive from 10 to 1000 - -### GcsStorage (Google Cloud Native) -**File:** `src/storage/adapters/gcsStorage.ts` -- **Lines:** 1,835 -- **Type:** Concrete implementation extending BaseStorage -- **Platform:** Node.js (server-side) -- **Storage Target:** Google Cloud Storage -- **SDK:** `@google-cloud/storage` (native) - -**Key Features:** -- Application Default Credentials (ADC) -- Service Account Key File support -- Service Account Credentials Object -- HMAC Keys (backward compatible) -- Multi-level cache managers -- Backpressure management -- High-volume mode -- Request coalescing - -**Authentication Priority:** -1. ADC (Application Default Credentials) -2. Service Account Key File -3. Service Account Credentials -4. HMAC Keys - -**Cache Managers:** -```typescript -private nounCacheManager: CacheManager -private verbCacheManager: CacheManager -``` - -### OPFSStorage (Browser Storage) -**File:** `src/storage/adapters/opfsStorage.ts` -- **Type:** Concrete implementation extending BaseStorage -- **Platform:** Browser only -- **Storage Target:** Origin Private File System (OPFS) -- **Fallback:** MemoryStorage if OPFS unavailable - -**Browser Compatibility:** -- Chrome 96+ -- Edge 96+ -- Safari 15.1+ - -## Storage Patterns and Utilities - -### Backward Compatibility -**File:** `src/storage/backwardCompatibility.ts` -- **Purpose:** Handle legacy storage paths -- **Features:** - - Path migration detection - - Dual-write during transition - - Graceful fallback to old locations - - Read-from-new, fallback-to-old - -### Metadata Index -**File:** `src/utils/metadataIndex.ts` -- **Purpose:** Build searchable indexes from metadata -- **Used By:** Brainy for fast metadata queries -- **Features:** - - Field name discovery - - Standard field mapping - - Service-level statistics - -### Storage Discovery (Distributed) -**File:** `src/distributed/storageDiscovery.ts` -- **Purpose:** Discover storage config in distributed systems -- **Features:** - - Node coordination - - Storage synchronization - -### Adaptive Backpressure -**File:** `src/utils/adaptiveBackpressure.ts` -- **Purpose:** Flow control for storage operations -- **Used By:** All cloud storage adapters -- **Features:** - - Request queuing - - Throttling detection - - Backoff scheduling - -### Write Buffer -**File:** `src/utils/writeBuffer.ts` -- **Purpose:** Batch write operations -- **Used By:** S3, GCS adapters -- **Features:** - - Configurable batch size - - Flush on timeout - - Deduplication - -### Request Coalescer -**File:** `src/utils/requestCoalescer.ts` -- **Purpose:** Deduplicate concurrent requests -- **Used By:** S3, GCS adapters -- **Features:** - - Request deduplication - - Batch processing - -## Integration Points - -### In Brainy.ts (Main Class) -```typescript -private storage!: BaseStorage - -async init(): Promise { - // Create storage from factory - const storageAdapter = await createStorage(this.config.storage) - this.storage = storageAdapter as BaseStorage - - // Initialize - await this.storage.init() - - // Pass to HNSW index - this.index = new HNSWIndex(this.storage, ...) -} -``` - -### In HNSW Index -```typescript -export class HNSWIndex { - constructor(private storage: StorageAdapter, ...) - - // Uses storage for node/edge persistence - async saveNode(noun: HNSWNoun): Promise - async getNode(id: string): Promise -} -``` - -### In Metadata Index -```typescript -export class MetadataIndexManager { - constructor(storage: StorageAdapter, ...) - - // Uses storage for metadata queries - async getNounsByFilter(filter: Filter): Promise -} -``` - -## Data Flow - -### Saving a Noun -``` -Brainy.add() - ↓ -HNSWIndex.insert() - ↓ -BaseStorage.saveNoun() - ├─ saveNoun_internal() → adapter-specific - ├─ saveNounMetadata() → path routing - └─ updateStatistics() - -FileSystemStorage.saveNoun_internal() - ├─ Create shard directory (ab/) - ├─ Write JSON file - └─ Update counts -``` - -### Querying Nouns -``` -Brainy.search() - ↓ -HNSW.search() - ↓ -BaseStorage.getNoun() - ├─ getNoun_internal() → adapter-specific - └─ getNounMetadata() → path routing - -S3CompatibleStorage.getNoun_internal() - ├─ Check cache - ├─ Download from S3 - ├─ Parse JSON - └─ Update cache -``` - -## Storage Statistics Tracking - -### Stored in `_system/statistics.json` -```json -{ - "nounCount": { "Person": 5, "Company": 2 }, - "verbCount": { "knows": 10, "works_at": 3 }, - "metadataCount": { "user-service": 50 }, - "hnswIndexSize": 15, - "totalNodes": 7, - "totalEdges": 13, - "services": [ - { "name": "user-service", "totalNouns": 5, ... } - ], - "lastUpdated": "2024-10-15T..." -} -``` - -### Stored in `_system/counts.json` -```json -{ - "totalNounCount": 7, - "totalVerbCount": 13, - "entityCounts": { "Person": 5, "Company": 2 }, - "verbCounts": { "knows": 10, "works_at": 3 } -} -``` - -## Type Definitions - -### HNSWNoun (Vector + HNSW) -```typescript -interface HNSWNoun { - id: string - vector: number[] - connections: Map> // level → node IDs - level: number - metadata?: any // Optional in vector file, separate file system -} -``` - -### GraphVerb (Relationship) -```typescript -interface GraphVerb { - id: string - sourceId: string - targetId: string - vector: number[] - type?: string - weight?: number - metadata?: any - // Plus aliases: source, target, verb, embedding - createdAt?: Timestamp - createdBy?: { augmentation: string; version: string } -} -``` - -## Summary Statistics - -| Component | File | Lines | Purpose | -|-----------|------|-------|---------| -| StorageAdapter Interface | coreTypes.ts | 250 | Interface definition | -| BaseStorageAdapter | baseStorageAdapter.ts | 1,156 | Common functionality | -| BaseStorage | baseStorage.ts | 1,098 | Core routing & pagination | -| storageFactory | storageFactory.ts | 200 | Adapter selection | -| FileSystemStorage | fileSystemStorage.ts | 2,677 | Node.js FS | -| MemoryStorage | memoryStorage.ts | 822 | In-memory (test) | -| S3CompatibleStorage | s3CompatibleStorage.ts | 5,000+ | AWS S3 / R2 / GCS | -| GcsStorage | gcsStorage.ts | 1,835 | Google Cloud native | -| sharding.ts | sharding.ts | ~100 | UUID sharding | -| cacheManager.ts | cacheManager.ts | ~200 | LRU cache | -| **TOTAL** | | **~13,000+** | **Complete storage system** | - -## Conclusion - -Brainy's storage architecture is: -- Well-layered (Interface → Abstract → Concrete) -- Extensible (factory pattern) -- Flexible (multiple backends) -- Scalable (sharding, caching, batching) -- Type-safe (full TypeScript support) - -New adapters like TypeAwareStorageAdapter simply extend BaseStorage and implement 17 abstract methods. diff --git a/scripts/diagnose-workshop-issue.ts b/scripts/diagnose-workshop-issue.ts new file mode 100644 index 00000000..332fb8a5 --- /dev/null +++ b/scripts/diagnose-workshop-issue.ts @@ -0,0 +1,194 @@ +/** + * Diagnostic script for Workshop type filtering issue + * + * This script mimics Workshop's exact import pattern to see what's happening + */ + +import { Brainy, NounType } from '../src/index.js' +import * as fs from 'fs' +import * as path from 'path' + +async function diagnose() { + console.log('\n🔬 Workshop Type Filtering Diagnostic\n') + console.log('='.repeat(70)) + + // Use temporary directory + const testDir = './test-workshop-data' + if (fs.existsSync(testDir)) { + fs.rmSync(testDir, { recursive: true }) + } + + const brain = new Brainy({ + storage: { + type: 'filesystem', + path: testDir + } + }) + await brain.init() + + console.log('\n1️⃣ Testing WITHOUT VFS (control group)...\n') + + // Add entities without VFS + console.log('Adding 3 person entities directly...') + await brain.add({ data: 'Person 1', type: NounType.Person, metadata: { name: 'Person 1' } }) + await brain.add({ data: 'Person 2', type: NounType.Person, metadata: { name: 'Person 2' } }) + await brain.add({ data: 'Person 3', type: NounType.Person, metadata: { name: 'Person 3' } }) + + console.log('Adding 2 location entities directly...') + await brain.add({ data: 'Location 1', type: NounType.Location, metadata: { name: 'Location 1' } }) + await brain.add({ data: 'Location 2', type: NounType.Location, metadata: { name: 'Location 2' } }) + + console.log('\n📊 Testing type filtering on direct entities...\n') + + const allDirect = await brain.find({ limit: 100 }) + console.log(` Total entities: ${allDirect.length}`) + + const peopleDirect = await brain.find({ type: NounType.Person, limit: 100 }) + console.log(` Person filter: ${peopleDirect.length} (expected: 3)`) + + const locationsDirect = await brain.find({ type: NounType.Location, limit: 100 }) + console.log(` Location filter: ${locationsDirect.length} (expected: 2)`) + + if (peopleDirect.length === 3 && locationsDirect.length === 2) { + console.log('\n ✅ Type filtering works on direct entities!') + } else { + console.log('\n ❌ Type filtering BROKEN on direct entities!') + return + } + + console.log('\n' + '='.repeat(70)) + console.log('\n2️⃣ Testing WITH VFS (Workshop pattern)...\n') + + // Simulate Workshop's pattern: create VFS file entities + console.log('Creating VFS file wrappers (like import does)...') + + const vfs = brain.vfs() + await vfs.init() + + // Create VFS directory + await vfs.mkdir('/imports/test', { recursive: true }) + + // Create VFS files with embedded entity data (mimics import) + console.log('Creating VFS file for Person entity...') + const personEntityData = { + id: 'ent_person_test', + name: 'John Smith', + type: 'person', + metadata: { + source: 'excel', + originalData: { + Name: 'John Smith', + _sheet: 'Characters' + } + } + } + await vfs.writeFile( + '/imports/test/john_smith.json', + Buffer.from(JSON.stringify(personEntityData, null, 2)) + ) + + console.log('Creating VFS file for Location entity...') + const locationEntityData = { + id: 'ent_location_test', + name: 'New York', + type: 'location', + metadata: { + source: 'excel', + originalData: { + Name: 'New York', + _sheet: 'Places' + } + } + } + await vfs.writeFile( + '/imports/test/new_york.json', + Buffer.from(JSON.stringify(locationEntityData, null, 2)) + ) + + console.log('\n📊 Checking what entities exist now...\n') + + const allWithVfs = await brain.find({ limit: 100 }) + console.log(` Total entities: ${allWithVfs.length}`) + + // Analyze entity types + const typeCounts: Record = {} + for (const result of allWithVfs) { + const type = result.type || 'unknown' + typeCounts[type] = (typeCounts[type] || 0) + 1 + } + + console.log('\n Entity types breakdown:') + for (const [type, count] of Object.entries(typeCounts)) { + console.log(` - ${type}: ${count}`) + } + + console.log('\n📊 Testing type filtering with VFS entities...\n') + + const peopleWithVfs = await brain.find({ type: NounType.Person, limit: 100 }) + console.log(` Person filter: ${peopleWithVfs.length} (expected: 3 direct + maybe VFS?)`) + + const locationsWithVfs = await brain.find({ type: NounType.Location, limit: 100 }) + console.log(` Location filter: ${locationsWithVfs.length} (expected: 2 direct + maybe VFS?)`) + + const documents = await brain.find({ type: NounType.Document, limit: 100 }) + console.log(` Document filter: ${documents.length} (VFS wrappers?)`) + + console.log('\n' + '='.repeat(70)) + console.log('\n3️⃣ Analyzing VFS wrapper structure...\n') + + // Get a VFS wrapper entity + const vfsWrapper = allWithVfs.find(e => e.metadata?.vfsType === 'file') + if (vfsWrapper) { + console.log('Found VFS wrapper entity:') + console.log(` - ID: ${vfsWrapper.id}`) + console.log(` - Type: ${vfsWrapper.type}`) + console.log(` - VFS Type: ${vfsWrapper.metadata?.vfsType}`) + console.log(` - Has rawData: ${!!vfsWrapper.metadata?.rawData}`) + console.log(` - Path: ${vfsWrapper.metadata?.path}`) + + if (vfsWrapper.metadata?.rawData) { + console.log('\n Decoding rawData...') + try { + const decoded = Buffer.from(vfsWrapper.metadata.rawData, 'base64').toString() + const entity = JSON.parse(decoded) + console.log(` - Embedded entity name: ${entity.name}`) + console.log(` - Embedded entity type: ${entity.type}`) + console.log(` - Wrapper type: ${vfsWrapper.type}`) + console.log('\n 🔍 KEY INSIGHT:') + console.log(` Wrapper has type="${vfsWrapper.type}"`) + console.log(` But embedded entity has type="${entity.type}"`) + console.log(' When you filter by person, you get the WRAPPER type, not embedded type!') + } catch (err) { + console.log(' ❌ Failed to decode rawData') + } + } + } else { + console.log('No VFS wrapper entities found') + } + + console.log('\n' + '='.repeat(70)) + console.log('\n4️⃣ DIAGNOSIS\n') + + if (documents.length > 0 && peopleWithVfs.length === 3) { + console.log('❌ FOUND THE BUG!') + console.log('') + console.log('VFS creates document wrappers with type="document".') + console.log('The actual entity data is stored as base64 in metadata.rawData.') + console.log('When you filter by type="person", you\'re filtering the WRAPPER type.') + console.log('Since wrappers are type="document", you get 0 results.') + console.log('') + console.log('This is a DESIGN ISSUE in how VFS import works!') + } else if (peopleWithVfs.length > 3) { + console.log('✅ Type filtering works correctly!') + console.log('VFS entities are being created with proper types.') + } else { + console.log('🤔 Unclear - need more investigation') + } + + console.log('\n' + '='.repeat(70) + '\n') + + // Cleanup + fs.rmSync(testDir, { recursive: true }) +} + +diagnose().catch(console.error) diff --git a/src/brainy.ts b/src/brainy.ts index e9b56e65..b09aa189 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -1239,6 +1239,13 @@ export class Brainy implements BrainyInterface { if (params.where) Object.assign(filter, params.where) if (params.service) filter.service = params.service + // v4.3.3: Exclude VFS entities by default (Option 3C architecture) + // Only include VFS if explicitly requested via includeVFS: true + // BUT: Don't add automatic exclusion if user explicitly queries isVFS in where clause + if (params.includeVFS !== true && !params.where?.hasOwnProperty('isVFS')) { + filter.isVFS = { notEquals: true } + } + if (params.type) { const types = Array.isArray(params.type) ? params.type : [params.type] if (types.length === 1) { @@ -1275,15 +1282,35 @@ export class Brainy implements BrainyInterface { const limit = params.limit || 20 const offset = params.offset || 0 - const storageResults = await this.storage.getNouns({ - pagination: { limit: limit + offset, offset: 0 } - }) + // v4.3.3: Apply VFS filtering even for empty queries + let filter: any = {} + if (params.includeVFS !== true) { + filter.isVFS = { notEquals: true } + } - for (let i = offset; i < Math.min(offset + limit, storageResults.items.length); i++) { - const noun = storageResults.items[i] - if (noun) { - const entity = await this.convertNounToEntity(noun) - results.push(this.createResult(noun.id, 1.0, entity)) + // Use metadata index if we need to filter VFS + if (Object.keys(filter).length > 0) { + const filteredIds = await this.metadataIndex.getIdsForFilter(filter) + const pageIds = filteredIds.slice(offset, offset + limit) + + for (const id of pageIds) { + const entity = await this.get(id) + if (entity) { + results.push(this.createResult(id, 1.0, entity)) + } + } + } else { + // No filtering needed, use direct storage query + const storageResults = await this.storage.getNouns({ + pagination: { limit: limit + offset, offset: 0 } + }) + + for (let i = offset; i < Math.min(offset + limit, storageResults.items.length); i++) { + const noun = storageResults.items[i] + if (noun) { + const entity = await this.convertNounToEntity(noun) + results.push(this.createResult(noun.id, 1.0, entity)) + } } } @@ -1324,14 +1351,20 @@ export class Brainy implements BrainyInterface { } // Apply O(log n) metadata filtering using core MetadataIndexManager - if (params.where || params.type || params.service) { + if (params.where || params.type || params.service || params.includeVFS !== true) { // Build filter object for metadata index let filter: any = {} - + // Base filter from where and service if (params.where) Object.assign(filter, params.where) if (params.service) filter.service = params.service - + + // v4.3.3: Exclude VFS entities by default (Option 3C architecture) + // BUT: Don't add automatic exclusion if user explicitly queries isVFS in where clause + if (params.includeVFS !== true && !params.where?.hasOwnProperty('isVFS')) { + filter.isVFS = { notEquals: true } + } + if (params.type) { const types = Array.isArray(params.type) ? params.type : [params.type] if (types.length === 1) { @@ -1346,7 +1379,7 @@ export class Brainy implements BrainyInterface { } } } - + const filteredIds = await this.metadataIndex.getIdsForFilter(filter) // CRITICAL FIX: Handle both cases properly diff --git a/src/types/brainy.types.ts b/src/types/brainy.types.ts index fccdeb80..260be1e4 100644 --- a/src/types/brainy.types.ts +++ b/src/types/brainy.types.ts @@ -185,6 +185,7 @@ export interface FindParams { mode?: SearchMode // Search strategy explain?: boolean // Return scoring explanation includeRelations?: boolean // Include entity relationships + includeVFS?: boolean // v4.3.3: Include VFS entities (default: false for knowledge queries) service?: string // Multi-tenancy filter // Triple Intelligence Fusion diff --git a/tests/integration/vfs-knowledge-separation.test.ts b/tests/integration/vfs-knowledge-separation.test.ts new file mode 100644 index 00000000..d5e1a85d --- /dev/null +++ b/tests/integration/vfs-knowledge-separation.test.ts @@ -0,0 +1,207 @@ +/** + * VFS-Knowledge Separation Test (v4.3.3) + * + * Tests Option 3C Architecture: + * - VFS entities marked with isVFS: true + * - brain.find() excludes VFS by default + * - brain.find({ includeVFS: true }) includes VFS + * - Enables relationships between VFS and knowledge + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest' +import { Brainy } from '../../src/brainy.js' +import { NounType } from '../../src/types/graphTypes.js' +import * as fs from 'fs' +import * as path from 'path' + +describe('VFS-Knowledge Separation (Option 3C)', () => { + const testDir = path.join(process.cwd(), 'test-vfs-knowledge-separation') + let brain: Brainy + + beforeAll(async () => { + // Clean up + if (fs.existsSync(testDir)) { + fs.rmSync(testDir, { recursive: true, force: true }) + } + fs.mkdirSync(testDir, { recursive: true }) + + brain = new Brainy({ + storage: { + type: 'filesystem', + options: { path: testDir } + } + }) + await brain.init() + }) + + afterAll(() => { + if (fs.existsSync(testDir)) { + fs.rmSync(testDir, { recursive: true, force: true }) + } + }) + + it('should exclude VFS entities from brain.find() by default', async () => { + // Create VFS file entity + const vfs = brain.vfs() + await vfs.init() + await vfs.mkdir('/docs', { recursive: true }) + await vfs.writeFile('/docs/readme.md', '# Hello World') + + // Create knowledge entity (no isVFS flag) + const knowledgeEntity = await brain.add({ + data: 'This is a knowledge document about AI', + type: NounType.Document, + metadata: { + title: 'AI Research Paper', + category: 'research' + } + }) + + // Query for documents WITHOUT includeVFS + console.log('\n📋 Test 1: brain.find() excludes VFS by default') + const results = await brain.find({ + type: NounType.Document, + limit: 100 + }) + + console.log(` Total results: ${results.length}`) + const vfsResults = results.filter(r => r.metadata?.isVFS === true) + const knowledgeResults = results.filter(r => r.metadata?.isVFS !== true) + + console.log(` VFS results: ${vfsResults.length}`) + console.log(` Knowledge results: ${knowledgeResults.length}`) + + // Should only return knowledge entities (no VFS) + expect(vfsResults.length).toBe(0) + expect(knowledgeResults.length).toBeGreaterThan(0) + expect(results.some(r => r.id === knowledgeEntity.id)).toBe(true) + }) + + it('should include VFS entities when includeVFS: true', async () => { + console.log('\n📋 Test 2: brain.find({ includeVFS: true }) includes VFS') + + // Query WITH includeVFS: true + const results = await brain.find({ + type: NounType.Document, + includeVFS: true, + limit: 100 + }) + + console.log(` Total results: ${results.length}`) + const vfsResults = results.filter(r => r.metadata?.isVFS === true) + const knowledgeResults = results.filter(r => r.metadata?.isVFS !== true) + + console.log(` VFS results: ${vfsResults.length}`) + console.log(` Knowledge results: ${knowledgeResults.length}`) + + // Should return BOTH VFS and knowledge entities + expect(vfsResults.length).toBeGreaterThan(0) + expect(knowledgeResults.length).toBeGreaterThan(0) + }) + + it('should allow relationships between VFS files and knowledge entities', async () => { + console.log('\n📋 Test 3: VFS-Knowledge relationships') + + // Get VFS file entity (need includeVFS: true when querying VFS by path) + const vfsFile = await brain.find({ + where: { + path: '/docs/readme.md' + }, + includeVFS: true, + limit: 1 + }) + expect(vfsFile.length).toBe(1) + + // Get knowledge entity + const knowledgeEntity = await brain.find({ + type: NounType.Document, + where: { + title: 'AI Research Paper' + }, + limit: 1 + }) + expect(knowledgeEntity.length).toBe(1) + + // Create relationship: knowledge entity -> references -> VFS file + const relation = await brain.relate({ + from: knowledgeEntity[0].id, + to: vfsFile[0].id, + type: 'references' + }) + + console.log(` Created relation: ${relation.id}`) + console.log(` From: ${knowledgeEntity[0].metadata?.title} (knowledge)`) + console.log(` To: ${vfsFile[0].metadata?.path} (VFS file)`) + + // Verify relationship exists + const relations = await brain.getRelations({ + from: knowledgeEntity[0].id, + to: vfsFile[0].id + }) + + expect(relations.length).toBe(1) + expect(relations[0].type).toBe('references') + console.log(` ✅ Relationship verified: knowledge can reference VFS files`) + }) + + it('should filter VFS entities using where clause', async () => { + console.log('\n📋 Test 4: Where clause filtering with isVFS') + + // Query for VFS files explicitly + const vfsFiles = await brain.find({ + where: { + isVFS: true, + vfsType: 'file' + }, + limit: 100 + }) + + console.log(` VFS files found: ${vfsFiles.length}`) + expect(vfsFiles.length).toBeGreaterThan(0) + expect(vfsFiles.every(f => f.metadata?.isVFS === true)).toBe(true) + expect(vfsFiles.every(f => f.metadata?.vfsType === 'file')).toBe(true) + + // Query for non-VFS entities explicitly + const nonVFS = await brain.find({ + where: { + category: 'research' + }, + limit: 100 + }) + + console.log(` Non-VFS entities found: ${nonVFS.length}`) + expect(nonVFS.length).toBeGreaterThan(0) + expect(nonVFS.every(e => e.metadata?.isVFS !== true)).toBe(true) + }) + + it('should handle semantic search with VFS filtering', async () => { + console.log('\n📋 Test 5: Semantic search excludes VFS by default') + + // Semantic search WITHOUT includeVFS (should exclude VFS files) + const results = await brain.find({ + query: 'research paper artificial intelligence', + limit: 10 + }) + + console.log(` Semantic results: ${results.length}`) + const hasVFS = results.some(r => r.metadata?.isVFS === true) + console.log(` Contains VFS entities: ${hasVFS}`) + + // Should NOT include VFS files in semantic search by default + expect(hasVFS).toBe(false) + + // Semantic search WITH includeVFS: true + const resultsWithVFS = await brain.find({ + query: 'hello world markdown', + includeVFS: true, + limit: 10 + }) + + console.log(` Semantic results (with VFS): ${resultsWithVFS.length}`) + const hasVFSWithFlag = resultsWithVFS.some(r => r.metadata?.isVFS === true) + console.log(` Contains VFS entities: ${hasVFSWithFlag}`) + + // Should include VFS files when explicitly requested + expect(hasVFSWithFlag).toBe(true) + }) +}) diff --git a/tests/manual/metadata-index-debug.test.ts b/tests/manual/metadata-index-debug.test.ts new file mode 100644 index 00000000..765c3a50 --- /dev/null +++ b/tests/manual/metadata-index-debug.test.ts @@ -0,0 +1,86 @@ +/** + * Debug: Check metadata index for isVFS field + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest' +import { Brainy } from '../../src/brainy.js' +import { NounType } from '../../src/types/graphTypes.js' +import * as fs from 'fs' +import * as path from 'path' + +describe('Metadata Index Debug', () => { + const testDir = path.join(process.cwd(), 'test-metadata-index-debug') + let brain: Brainy + + beforeAll(async () => { + if (fs.existsSync(testDir)) { + fs.rmSync(testDir, { recursive: true, force: true }) + } + fs.mkdirSync(testDir, { recursive: true }) + + brain = new Brainy({ + storage: { + type: 'filesystem', + options: { path: testDir } + } + }) + await brain.init() + }) + + afterAll(() => { + if (fs.existsSync(testDir)) { + fs.rmSync(testDir, { recursive: true, force: true }) + } + }) + + it('should index custom metadata fields', async () => { + // Add entity with custom metadata field + const entity1 = await brain.add({ + data: 'Test entity with custom field', + type: NounType.Document, + metadata: { + customFlag: true, + customString: 'hello', + customNumber: 42 + } + }) + + console.log('\n📋 Added entity:', entity1.id) + console.log(' Metadata:', entity1.metadata) + + // Try to find by custom fields + console.log('\n📋 Query 1: where: { customFlag: true }') + const result1 = await brain.find({ + where: { customFlag: true }, + limit: 10 + }) + console.log(` Results: ${result1.length}`) + if (result1.length > 0) { + console.log(` Found: ${result1[0].id}`) + console.log(` Metadata.customFlag: ${result1[0].metadata?.customFlag}`) + } + + console.log('\n📋 Query 2: where: { customString: "hello" }') + const result2 = await brain.find({ + where: { customString: 'hello' }, + limit: 10 + }) + console.log(` Results: ${result2.length}`) + + console.log('\n📋 Query 3: where: { customNumber: 42 }') + const result3 = await brain.find({ + where: { customNumber: 42 }, + limit: 10 + }) + console.log(` Results: ${result3.length}`) + + // Check what fields are indexed + console.log('\n📋 Indexed fields:') + const fields = await brain.getFilterFields() + console.log(` ${fields.join(', ')}`) + + expect(result1.length).toBeGreaterThan(0) + expect(result2.length).toBeGreaterThan(0) + expect(result3.length).toBeGreaterThan(0) + }) +}) diff --git a/tests/manual/vfs-isvfs-diagnostic.test.ts b/tests/manual/vfs-isvfs-diagnostic.test.ts new file mode 100644 index 00000000..b90667b9 --- /dev/null +++ b/tests/manual/vfs-isvfs-diagnostic.test.ts @@ -0,0 +1,90 @@ +/** + * Diagnostic: Check if isVFS flag is actually being stored + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest' +import { Brainy } from '../../src/brainy.js' +import { NounType } from '../../src/types/graphTypes.js' +import * as fs from 'fs' +import * as path from 'path' + +describe('isVFS Flag Diagnostic', () => { + const testDir = path.join(process.cwd(), 'test-isvfs-diagnostic') + let brain: Brainy + + beforeAll(async () => { + if (fs.existsSync(testDir)) { + fs.rmSync(testDir, { recursive: true, force: true }) + } + fs.mkdirSync(testDir, { recursive: true }) + + brain = new Brainy({ + storage: { + type: 'filesystem', + options: { path: testDir } + } + }) + await brain.init() + }) + + afterAll(() => { + if (fs.existsSync(testDir)) { + fs.rmSync(testDir, { recursive: true, force: true }) + } + }) + + it('should show what entities are created', async () => { + // Create VFS file + const vfs = brain.vfs() + await vfs.init() + await vfs.writeFile('/test.txt', 'Hello World') + + // Get ALL entities + console.log('\n📋 All entities in database:') + const allEntities = await brain.find({ limit: 100, includeVFS: true }) + console.log(` Total: ${allEntities.length}`) + + for (const entity of allEntities) { + console.log(`\n Entity: ${entity.id}`) + console.log(` Type: ${entity.type}`) + console.log(` Metadata keys: ${Object.keys(entity.metadata || {}).join(', ')}`) + if (entity.metadata?.path) { + console.log(` Path: ${entity.metadata.path}`) + } + if (entity.metadata?.vfsType) { + console.log(` VfsType: ${entity.metadata.vfsType}`) + } + if (entity.metadata?.isVFS !== undefined) { + console.log(` isVFS: ${entity.metadata.isVFS}`) + } + if (entity.metadata?.name) { + console.log(` Name: ${entity.metadata.name}`) + } + } + + // Try different queries + console.log('\n\n📋 Query 1: brain.find({ limit: 100 }) [default, no includeVFS]') + const query1 = await brain.find({ limit: 100 }) + console.log(` Results: ${query1.length}`) + + console.log('\n📋 Query 2: brain.find({ limit: 100, includeVFS: true })') + const query2 = await brain.find({ limit: 100, includeVFS: true }) + console.log(` Results: ${query2.length}`) + + console.log('\n📋 Query 3: brain.find({ where: { isVFS: true } })') + const query3 = await brain.find({ where: { isVFS: true }, limit: 100 }) + console.log(` Results: ${query3.length}`) + + console.log('\n📋 Query 4: brain.find({ where: { path: "/test.txt" } })') + const query4 = await brain.find({ where: { path: '/test.txt' }, limit: 100 }) + console.log(` Results: ${query4.length}`) + + console.log('\n📋 Query 5: brain.find({ type: NounType.Document })') + const query5 = await brain.find({ type: NounType.Document, limit: 100 }) + console.log(` Results: ${query5.length}`) + + console.log('\n📋 Query 6: brain.find({ type: NounType.Document, includeVFS: true })') + const query6 = await brain.find({ type: NounType.Document, includeVFS: true, limit: 100 }) + console.log(` Results: ${query6.length}`) + }) +}) diff --git a/tests/manual/vfs-multiple-init-diagnostic.test.ts b/tests/manual/vfs-multiple-init-diagnostic.test.ts new file mode 100644 index 00000000..e07b8c5d --- /dev/null +++ b/tests/manual/vfs-multiple-init-diagnostic.test.ts @@ -0,0 +1,207 @@ +/** + * VFS Multiple Init() Diagnostic Test + * + * Tests Workshop team's issue: Does calling vfs.init() multiple times + * create duplicate root entities? + * + * Scenario: + * - Create brain instance + * - Call vfs.init() multiple times (simulating multiple requests) + * - Check if multiple root entities are created + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest' +import { Brainy } from '../../src/brainy.js' +import * as fs from 'fs' +import * as path from 'path' + +describe('VFS Multiple Init Diagnostic', () => { + const testDir = path.join(process.cwd(), 'test-vfs-multi-init') + let brain: Brainy + + beforeAll(async () => { + // Clean up test directory + if (fs.existsSync(testDir)) { + fs.rmSync(testDir, { recursive: true, force: true }) + } + fs.mkdirSync(testDir, { recursive: true }) + + // Create brain with filesystem storage + brain = new Brainy({ + storage: { + type: 'filesystem', + path: testDir + } + }) + await brain.init() + }) + + afterAll(() => { + // Clean up + if (fs.existsSync(testDir)) { + fs.rmSync(testDir, { recursive: true, force: true }) + } + }) + + it('should not create duplicate roots when calling vfs.init() multiple times on SAME instance', async () => { + const vfs = brain.vfs() + + // Call init multiple times + await vfs.init() + await vfs.init() + await vfs.init() + + // Query for root entities using workaround (where clause is broken) + const collections = await brain.find({ + type: 'collection', + limit: 100 + }) + + const roots = collections.filter(e => + e.metadata?.path === '/' && + e.metadata?.vfsType === 'directory' + ) + + console.log(`\n✅ Test 1: Same VFS instance`) + console.log(` Total collections: ${collections.length}`) + console.log(` Roots found: ${roots.length}`) + roots.forEach((root, i) => { + console.log(` Root ${i + 1}: ${root.id}`) + }) + + expect(roots.length).toBe(1) + }) + + it('should not create duplicate roots when creating MULTIPLE VFS instances (Workshop scenario)', async () => { + // Simulate Workshop's scenario: Getting VFS on multiple requests + // brain.vfs() returns cached instance, so this should be safe + const vfs1 = brain.vfs() + const vfs2 = brain.vfs() + const vfs3 = brain.vfs() + + await vfs1.init() + await vfs2.init() + await vfs3.init() + + // Query for root entities using workaround + const collections = await brain.find({ + type: 'collection', + limit: 100 + }) + + const roots = collections.filter(e => + e.metadata?.path === '/' && + e.metadata?.vfsType === 'directory' + ) + + console.log(`\n✅ Test 2: Multiple VFS references (cached)`) + console.log(` VFS instances are same? ${vfs1 === vfs2 && vfs2 === vfs3}`) + console.log(` Roots found: ${roots.length}`) + roots.forEach((root, i) => { + console.log(` Root ${i + 1}: ${root.id}`) + }) + + expect(vfs1).toBe(vfs2) // Should be same instance (cached) + expect(roots.length).toBe(1) // Should still be 1 root + }) + + it('should handle NEW brain instances gracefully (per-user scenario)', async () => { + // Simulate creating separate brain instances per user + // This is closer to Workshop's getUserBrainy() pattern + const brain2 = new Brainy({ + storage: { + type: 'filesystem', + path: testDir // Same storage! + } + }) + await brain2.init() + + const vfs2 = brain2.vfs() + await vfs2.init() + + // Query for roots using workaround + const collections = await brain.find({ + type: 'collection', + limit: 100 + }) + + const roots = collections.filter(e => + e.metadata?.path === '/' && + e.metadata?.vfsType === 'directory' + ) + + console.log(`\n✅ Test 3: New Brainy instance (same storage)`) + console.log(` Roots found: ${roots.length}`) + roots.forEach((root, i) => { + console.log(` Root ${i + 1}: ${root.id} (created: ${root.metadata?.createdAt})`) + }) + + // This is the CRITICAL test - should find existing root, not create new one + expect(roots.length).toBe(1) + }) + + it('should verify readdir works after multiple inits', async () => { + // Create a test directory + const vfs = brain.vfs() + await vfs.mkdir('/test-dir', { recursive: true }) + await vfs.writeFile('/test-dir/test.txt', 'Hello') + + // Call init again (simulating new request) + const brain3 = new Brainy({ + storage: { + type: 'filesystem', + path: testDir + } + }) + await brain3.init() + const vfs3 = brain3.vfs() + await vfs3.init() + + // Try to read root directory + const entries = await vfs3.readdir('/', { withFileTypes: true }) + + console.log(`\n✅ Test 4: readdir after new brain instance`) + console.log(` Entries found: ${entries.length}`) + entries.forEach((entry: any) => { + console.log(` - ${entry.name} (${entry.type})`) + }) + + expect(entries.length).toBeGreaterThan(0) + }) + + it('should show if Contains relationships are created', async () => { + const vfs = brain.vfs() + + // Get root entity ID using workaround + const collections = await brain.find({ + type: 'collection', + limit: 100 + }) + + const roots = collections.filter(e => + e.metadata?.path === '/' && + e.metadata?.vfsType === 'directory' + ) + + expect(roots.length).toBeGreaterThanOrEqual(1) + const rootId = roots[0].id + + // Check for Contains relationships FROM root + const relations = await brain.getRelations({ + from: rootId + }) + + console.log(`\n✅ Test 5: Contains relationships`) + console.log(` Root ID: ${rootId}`) + console.log(` Relationships from root: ${relations.length}`) + relations.forEach((rel) => { + console.log(` - ${rel.from} -> ${rel.to} (${rel.type})`) + }) + + // Root should have at least one Contains relationship (to /test-dir) + const containsRels = relations.filter(r => r.type === 'contains') + console.log(` Contains relationships: ${containsRels.length}`) + + expect(containsRels.length).toBeGreaterThan(0) + }) +}) diff --git a/tests/manual/workshop-type-filtering-test.ts b/tests/manual/workshop-type-filtering-test.ts new file mode 100644 index 00000000..8c319171 --- /dev/null +++ b/tests/manual/workshop-type-filtering-test.ts @@ -0,0 +1,171 @@ +/** + * Manual test to reproduce Workshop team's type filtering issue + * + * This test verifies that brain.find({ type: NounType.Person }) actually works + * as documented in the API. + */ + +import { Brainy, NounType } from '../../src/index.js' + +async function testTypeFiltering() { + console.log('\n🔬 Testing Type Filtering Issue from Workshop Team\n') + console.log('=' .repeat(60)) + + // Create in-memory instance for testing + const brain = new Brainy({ + storage: { type: 'memory' } + }) + await brain.init() + + console.log('\n1️⃣ Adding test entities with different types...\n') + + // Add 5 person entities + const people = [] + for (let i = 0; i < 5; i++) { + const id = await brain.add({ + data: `Person ${i + 1}`, + type: NounType.Person, + metadata: { name: `Person ${i + 1}` } + }) + people.push(id) + } + console.log(`✅ Added ${people.length} person entities`) + + // Add 3 location entities + const locations = [] + for (let i = 0; i < 3; i++) { + const id = await brain.add({ + data: `Location ${i + 1}`, + type: NounType.Location, + metadata: { name: `Location ${i + 1}` } + }) + locations.push(id) + } + console.log(`✅ Added ${locations.length} location entities`) + + // Add 2 concept entities + const concepts = [] + for (let i = 0; i < 2; i++) { + const id = await brain.add({ + data: `Concept ${i + 1}`, + type: NounType.Concept, + metadata: { name: `Concept ${i + 1}` } + }) + concepts.push(id) + } + console.log(`✅ Added ${concepts.length} concept entities`) + + console.log(`\n📊 Total: ${people.length + locations.length + concepts.length} entities`) + + console.log('\n' + '='.repeat(60)) + console.log('\n2️⃣ Testing find() without type filter...\n') + + const allResults = await brain.find({ limit: 100 }) + console.log(`✅ brain.find({}) returned ${allResults.length} entities`) + + // Check types in results + const typeCounts = allResults.reduce((acc, r) => { + acc[r.type || 'unknown'] = (acc[r.type || 'unknown'] || 0) + 1 + return acc + }, {} as Record) + + console.log('\nTypes in results:') + for (const [type, count] of Object.entries(typeCounts)) { + console.log(` - ${type}: ${count}`) + } + + console.log('\n' + '='.repeat(60)) + console.log('\n3️⃣ Testing find() WITH type filter (using enum)...\n') + + // Test 1: Filter by NounType.Person (enum value) + console.log('Test 1: brain.find({ type: NounType.Person })') + const personResults = await brain.find({ type: NounType.Person, limit: 100 }) + console.log(` Result: ${personResults.length} entities`) + console.log(` Expected: ${people.length} entities`) + if (personResults.length === people.length) { + console.log(' ✅ PASS') + } else { + console.log(' ❌ FAIL - Type filtering not working!') + } + + // Test 2: Filter by NounType.Location + console.log('\nTest 2: brain.find({ type: NounType.Location })') + const locationResults = await brain.find({ type: NounType.Location, limit: 100 }) + console.log(` Result: ${locationResults.length} entities`) + console.log(` Expected: ${locations.length} entities`) + if (locationResults.length === locations.length) { + console.log(' ✅ PASS') + } else { + console.log(' ❌ FAIL - Type filtering not working!') + } + + // Test 3: Filter by NounType.Concept + console.log('\nTest 3: brain.find({ type: NounType.Concept })') + const conceptResults = await brain.find({ type: NounType.Concept, limit: 100 }) + console.log(` Result: ${conceptResults.length} entities`) + console.log(` Expected: ${concepts.length} entities`) + if (conceptResults.length === concepts.length) { + console.log(' ✅ PASS') + } else { + console.log(' ❌ FAIL - Type filtering not working!') + } + + console.log('\n' + '='.repeat(60)) + console.log('\n4️⃣ Testing find() WITH type filter (using string)...\n') + + // Test 4: Filter by string 'person' + console.log('Test 4: brain.find({ type: "person" })') + const personResults2 = await brain.find({ type: 'person' as any, limit: 100 }) + console.log(` Result: ${personResults2.length} entities`) + console.log(` Expected: ${people.length} entities`) + if (personResults2.length === people.length) { + console.log(' ✅ PASS') + } else { + console.log(' ❌ FAIL - String type filtering not working!') + } + + console.log('\n' + '='.repeat(60)) + console.log('\n5️⃣ Checking entity storage (metadata.noun)...\n') + + // Get a person entity and check its storage + const personEntity = await brain.get(people[0]) + console.log('Person entity structure:') + console.log(' - id:', personEntity?.id) + console.log(' - type:', personEntity?.type) + console.log(' - metadata.noun:', (personEntity as any)?.metadata?.noun) + console.log(' - metadata.name:', personEntity?.metadata?.name) + + console.log('\n' + '='.repeat(60)) + console.log('\n📋 Summary\n') + + const tests = [ + { name: 'Filter by NounType.Person', passed: personResults.length === people.length }, + { name: 'Filter by NounType.Location', passed: locationResults.length === locations.length }, + { name: 'Filter by NounType.Concept', passed: conceptResults.length === concepts.length }, + { name: 'Filter by string "person"', passed: personResults2.length === people.length } + ] + + const passedCount = tests.filter(t => t.passed).length + const totalCount = tests.length + + console.log(`Tests passed: ${passedCount}/${totalCount}\n`) + + for (const test of tests) { + console.log(`${test.passed ? '✅' : '❌'} ${test.name}`) + } + + if (passedCount === totalCount) { + console.log('\n🎉 All tests passed! Type filtering works correctly.') + console.log('\n💡 The Workshop team might be experiencing a different issue.') + console.log(' Check: storage persistence, Brainy instance reuse, or data migration') + } else { + console.log('\n❌ Type filtering is BROKEN in Brainy!') + console.log('\n🐛 This is a bug that needs to be fixed.') + console.log(' Workshop team was right - it\'s not user error.') + } + + console.log('\n' + '='.repeat(60) + '\n') +} + +// Run the test +testTypeFiltering().catch(console.error) diff --git a/tests/unit/workshop-vfs-diagnostic.test.ts b/tests/unit/workshop-vfs-diagnostic.test.ts new file mode 100644 index 00000000..f7171177 --- /dev/null +++ b/tests/unit/workshop-vfs-diagnostic.test.ts @@ -0,0 +1,171 @@ +/** + * Workshop VFS Diagnostic Test + * + * Tests to verify VFS import behavior and identify if VFS creates only wrappers or also graph entities + */ + +import { describe, it, expect, beforeEach } from 'vitest' +import { Brainy, NounType } from '../../src/index.js' + +describe('Workshop VFS Diagnostic', () => { + let brain: Brainy + + beforeEach(async () => { + brain = new Brainy({ + storage: { type: 'memory' } + }) + await brain.init() + }) + + it('should verify VFS creates document wrappers AND allows entity filtering', async () => { + console.log('\n🔬 Workshop VFS Diagnostic Test\n') + console.log('='.repeat(70)) + + // Step 1: Add entities directly (control group) + console.log('\n1️⃣ Adding entities directly (without VFS)...\n') + + await brain.add({ data: 'Person 1', type: NounType.Person, metadata: { name: 'Person 1' } }) + await brain.add({ data: 'Person 2', type: NounType.Person, metadata: { name: 'Person 2' } }) + await brain.add({ data: 'Location 1', type: NounType.Location, metadata: { name: 'Location 1' } }) + + const beforeVfs = await brain.find({ limit: 100 }) + console.log(` Total entities: ${beforeVfs.length}`) + + const peopleBefore = await brain.find({ type: NounType.Person, limit: 100 }) + console.log(` Person filter: ${peopleBefore.length} (expected: 2)`) + expect(peopleBefore.length).toBe(2) + console.log(' ✅ Type filtering works on direct entities\n') + + // Step 2: Use VFS to create files + console.log('2️⃣ Creating VFS files...\n') + + const vfs = brain.vfs() + await vfs.init() + await vfs.mkdir('/test', { recursive: true }) + + // Create a VFS file with entity data + const personData = { + id: 'ent_person_test', + name: 'John Smith', + type: 'person', + metadata: { source: 'test' } + } + + await vfs.writeFile('/test/john.json', Buffer.from(JSON.stringify(personData, null, 2))) + console.log(' Created VFS file: /test/john.json') + + // Step 3: Check what entities exist now + console.log('\n3️⃣ Analyzing entities after VFS...\n') + + const afterVfs = await brain.find({ limit: 100 }) + console.log(` Total entities: ${afterVfs.length}`) + + // Count by type + const typeCounts: Record = {} + for (const result of afterVfs) { + const type = result.type || 'unknown' + typeCounts[type] = (typeCounts[type] || 0) + 1 + } + + console.log('\n Entity type breakdown:') + for (const [type, count] of Object.entries(typeCounts)) { + console.log(` - ${type}: ${count}`) + } + + // Count VFS wrappers vs regular entities + const vfsWrappers = afterVfs.filter(e => e.metadata?.vfsType === 'file') + const regularEntities = afterVfs.filter(e => !e.metadata?.vfsType) + + console.log(`\n VFS wrappers: ${vfsWrappers.length}`) + console.log(` Regular entities: ${regularEntities.length}`) + + // Step 4: Test type filtering after VFS + console.log('\n4️⃣ Testing type filtering after VFS...\n') + + const peopleAfter = await brain.find({ type: NounType.Person, limit: 100 }) + console.log(` Person filter: ${peopleAfter.length} (expected: 2 - same as before)`) + + const documents = await brain.find({ type: NounType.Document, limit: 100 }) + console.log(` Document filter: ${documents.length} (expected: ${vfsWrappers.length})`) + + // Step 5: Analyze VFS wrapper structure + console.log('\n5️⃣ Analyzing VFS wrapper structure...\n') + + const wrapper = vfsWrappers[0] + if (wrapper) { + console.log(' VFS Wrapper Entity:') + console.log(` - ID: ${wrapper.id}`) + console.log(` - Type: ${wrapper.type}`) + console.log(` - VFS Type: ${wrapper.metadata?.vfsType}`) + console.log(` - Path: ${wrapper.metadata?.path}`) + console.log(` - Has rawData: ${!!wrapper.metadata?.rawData}`) + + if (wrapper.metadata?.rawData) { + const decoded = Buffer.from(wrapper.metadata.rawData, 'base64').toString() + const embedded = JSON.parse(decoded) + console.log(`\n Embedded Entity Data:`) + console.log(` - Name: ${embedded.name}`) + console.log(` - Type: ${embedded.type}`) + console.log(`\n 🔍 KEY FINDING:`) + console.log(` Wrapper type: "${wrapper.type}"`) + console.log(` Embedded type: "${embedded.type}"`) + console.log(` Filtering by type="${embedded.type}" searches wrapper type, not embedded!`) + } + } + + // Step 6: Diagnosis + console.log('\n' + '='.repeat(70)) + console.log('📋 DIAGNOSIS\n') + + if (peopleAfter.length === peopleBefore.length) { + console.log('✅ VFS does NOT create duplicate graph entities') + console.log('✅ VFS only creates document wrappers') + console.log('✅ Type filtering works on original entities, ignores VFS wrappers') + console.log('\nThis means:') + console.log(' - VFS files are type="document" wrappers') + console.log(' - Original entities keep their types') + console.log(' - filter({ type: "person" }) returns original entities only') + } else { + console.log('❌ Unexpected behavior - VFS may have created additional entities') + } + + console.log('\n' + '='.repeat(70) + '\n') + + // Assertions + expect(peopleAfter.length).toBe(2) // Should still be 2, VFS doesn't create person entities + expect(documents.length).toBeGreaterThan(0) // VFS creates document wrappers + expect(vfsWrappers.length).toBeGreaterThan(0) // Should have VFS wrappers + }) + + it('should verify import creates BOTH VFS wrappers AND graph entities', async () => { + // This test would require creating a test Excel file and running import + // For now, we'll document the expected behavior based on code analysis + + console.log('\n📚 Expected Import Behavior (from code analysis):\n') + console.log('When you run brain.import("file.xlsx", { vfsPath: "/imports" }):') + console.log('\n1. ImportCoordinator.execute() calls:') + console.log(' a) vfsGenerator.generate() - creates VFS file wrappers') + console.log(' - Each entity → JSON file in VFS') + console.log(' - Wrapper entity with type="document"') + console.log(' - Entity data stored in metadata.rawData (base64)') + console.log('') + console.log(' b) createGraphEntities() - creates graph entities') + console.log(' - Each entity → graph entity with proper type') + console.log(' - type="person", "location", "concept", etc.') + console.log(' - metadata.vfsPath points to VFS file') + console.log('') + console.log('2. Result: Database contains BOTH:') + console.log(' - VFS wrappers (type="document", vfsType="file")') + console.log(' - Graph entities (type="person", etc., vfsPath set)') + console.log('') + console.log('3. Type filtering:') + console.log(' - filter({ type: "person" }) → returns graph entities') + console.log(' - filter({ type: "document" }) → returns VFS wrappers') + console.log('') + console.log('If Workshop gets 0 results, likely causes:') + console.log(' ❌ Only VFS wrappers created (createEntities: false)') + console.log(' ❌ Import not completing before query') + console.log(' ❌ Querying different Brainy instance') + console.log('') + }) +})