brainy/CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file. See standard-version for commit guidelines.

3.28.0 (2025-10-08)

• feat: add unified import system with auto-detection and dual storage (a06e877)

3.27.1 (2025-10-08)

• docs: clarify GCS storage type and config object pairing (dcbd0fd)

3.27.0 (2025-10-08)

• test: skip incomplete clusterByDomain tests pending implementation (19aa4af)
• feat: add native Google Cloud Storage adapter with ADC support (e2aa8e3)

3.26.0 (2025-10-08)

⚠ BREAKING CHANGES

• Requires data migration for existing S3/GCS/R2/OpFS deployments. See .strategy/UNIFIED-UUID-SHARDING.md for migration guidance.

🐛 Bug Fixes

• implement unified UUID-based sharding for metadata across all storage adapters (2f33571)

3.25.2 (2025-10-08)

🐛 Bug Fixes

• export ImportManager and add getStats() convenience method (06b3bc7)

3.25.1 (2025-10-07)

🐛 Bug Fixes

• implement stub methods in Neural API clustering (1d2da82)

✅ Tests

• use memory storage for domain-time clustering tests (34fb6e0)

3.25.0 (2025-10-07)

• test: skip GitBridge Integration test (empty suite) (8939f59)
• test: skip batch-operations-fixed tests (flaky order test) (d582069)
• test: skip comprehensive VFS tests (pre-existing failures) (1d786f6)
• feat: add resolvePathToId() method and fix test issues (2931aa2)

3.24.0 (2025-10-07)

• feat: simplify sharding to fixed depth-1 for reliability and performance (87515b9)

3.23.0 (2025-10-04)

• refactor: streamline core API surface

3.22.0 (2025-10-01)

• feat: add intelligent import for CSV, Excel, and PDF files (814cbb4)

3.21.0 (2025-10-01)

• feat: add progress tracking, entity caching, and relationship confidence (2f9d512)

3.21.0 (2025-10-01)

Features

📊 Standardized Progress Tracking

• progress types: Add unified BrainyProgress<T> interface for all long-running operations
• progress tracker: Implement ProgressTracker class with automatic time estimation
• throughput: Calculate items/second for real-time performance monitoring
• formatting: Add formatProgress() and formatDuration() utilities

⚡ Entity Extraction Caching

• cache system: Implement LRU cache with TTL expiration (default: 7 days)
• invalidation: Support file mtime and content hash-based cache invalidation
• performance: 10-100x speedup on repeated entity extraction
• statistics: Comprehensive cache hit/miss tracking and reporting
• management: Full cache control (invalidate, cleanup, clear)

🔗 Relationship Confidence Scoring

• confidence: Multi-factor confidence scoring for detected relationships (0-1 scale)
• evidence: Track source text, position, detection method, and reasoning
• scoring: Proximity-based, pattern-based, and structural analysis
• filtering: Filter relationships by confidence threshold
• backward compatible: Confidence and evidence are optional fields

API Enhancements

// Progress Tracking
import { ProgressTracker, formatProgress } from '@soulcraft/brainy/types'
const tracker = ProgressTracker.create(1000)
tracker.start()
tracker.update(500, 'current-item.txt')

// Entity Extraction with Caching
const entities = await brain.neural.extractor.extract(text, {
  path: '/path/to/file.txt',
  cache: {
    enabled: true,
    ttl: 7 * 24 * 60 * 60 * 1000,
    invalidateOn: 'mtime',
    mtime: fileMtime
  }
})

// Relationship Confidence
import { detectRelationshipsWithConfidence } from '@soulcraft/brainy/neural'
const relationships = detectRelationshipsWithConfidence(entities, text, {
  minConfidence: 0.7
})

await brain.relate({
  from: sourceId,
  to: targetId,
  type: VerbType.Creates,
  confidence: 0.85,
  evidence: {
    sourceText: 'John created the database',
    method: 'pattern',
    reasoning: 'Matches creation pattern; entities in same sentence'
  }
})

Performance

• Cache Hit Rate: Expected >80% for typical workloads
• Cache Speedup: 10-100x faster on cache hits
• Memory Overhead: <20% increase with default settings
• Scoring Speed: <1ms per relationship

Documentation

• Add comprehensive example: examples/directory-import-with-caching.ts
• Add implementation summary: .strategy/IMPLEMENTATION_SUMMARY.md
• Add API documentation for all new features
• Update README with new features section

BREAKING CHANGES

• None - All new features are backward compatible and opt-in

---

3.20.5 (2025-10-01)

• feat: add --skip-tests flag to release script (0614171)
• fix: resolve critical bugs in delete operations and fix flaky tests (8476047)
• feat: implement simpler, more reliable release workflow (386fd2c)

3.20.2 (2025-09-30)

Bug Fixes

• vfs: resolve VFS race conditions and decompression errors (1a2661f)
  • Fixes duplicate directory nodes caused by concurrent writes
  • Fixes file read decompression errors caused by rawData compression state mismatch
  • Adds mutex-based concurrency control for mkdir operations
  • Adds explicit compression tracking for file reads

BREAKING CHANGES (Deprecated API Removal)

• removed BrainyData: The deprecated BrainyData class has been completely removed
  • BrainyData was never part of the official Brainy 3.0 API
  • All users should migrate to the Brainy class
  • Migration is simple: Replace new BrainyData() with new Brainy() and add await brain.init()
  • See .strategy/NEURAL_API_RESPONSE.md for complete migration guide
  • Renamed brainyDataInterface.ts to brainyInterface.ts for clarity

3.19.1 (2025-09-29)

3.19.0 (2025-09-29)

3.17.0 (2025-09-27)

3.15.0 (2025-09-26)

Bug Fixes

• vfs: Ensure Contains relationships are maintained when updating files
• vfs: Fix root directory metadata handling to prevent "Not a directory" errors
• vfs: Add entity metadata compatibility layer for proper VFS operations
• vfs: Fix resolvePath() to return entity IDs instead of path strings
• vfs: Improve error handling in ensureDirectory() method

Features

• vfs: Add comprehensive tests for Contains relationship integrity
• vfs: Ensure all VFS entities use standard Brainy NounType and VerbType enums
• vfs: Add metadata validation and repair for existing entities

3.0.1 (2025-09-15)

Brainy 3.0 Production Release - World's first Triple Intelligence™ database unifying vector, graph, and document search

Features

• new api: Complete API redesign with add(), find(), update(), delete(), relate() methods
• triple intelligence: Unified vector, graph, and document search in one API
• comprehensive validation: Zero-config validation system with production-ready type safety
• neural clustering: Advanced clustering with clusterFast(), clusterLarge(), and hierarchical algorithms
• augmentation system: Built-in cache, display, and metrics augmentations
• extensive testing: 100+ comprehensive tests covering all APIs and edge cases

BREAKING CHANGES

• All previous APIs (addNoun, findNoun, etc.) have been replaced with new 3.0 APIs
• See README.md for complete migration guide from 2.x to 3.0

2.14.0 (2025-09-02)

Features

• implement clean embedding architecture with Q8/FP32 precision control (b55c454)

2.13.0 (2025-09-02)

Features

• implement comprehensive neural clustering system (7345e53)
• implement comprehensive type safety system with BrainyTypes API (0f4ab52)

2.10.0 (2025-08-29)

2.8.0 (2025-08-29)

[2.7.4] - 2025-08-29

Fixed

• Use fp32 models consistently everywhere to ensure compatibility
• Changed default dtype from q8 to fp32 across all embedding implementations
• Ensures the exact same model (model.onnx) is used everywhere
• Prevents 404 errors when looking for quantized models that don't exist on CDN
• Maintains data compatibility across all Brainy instances

[2.7.3] - 2025-08-29

Fixed

• Allow automatic model downloads without requiring BRAINY_ALLOW_REMOTE_MODELS environment variable
• Models now download automatically when not present locally
• Fixed environment variable check to only block downloads when explicitly set to 'false'

[2.0.0] - 2025-08-26

🎉 Major Release - Triple Intelligence™ Engine

This release represents a complete evolution of Brainy with groundbreaking features and performance improvements.

Added

• Triple Intelligence™ Engine: Unified Vector + Metadata + Graph search in one API
• Natural Language Processing: 220+ pre-computed NLP patterns for instant understanding
• Universal Memory Manager: Worker-based embeddings with automatic memory management
• Zero Configuration: Everything works instantly with no setup required
• Brain Cloud Integration: Connect to soulcraft.com for team sync and persistent memory
• Augmentation System: 19 production-ready augmentations for extended capabilities
• CLI Enhancements: Complete command-line interface with all API methods
• New find() API: Natural language queries with context understanding
• OPFS Storage: Browser-native storage support
• S3 Storage: Production-ready cloud storage adapter
• Graph Relationships: Navigate connected knowledge with addVerb()
• Cursor Pagination: Efficient handling of large result sets
• Automatic Caching: Intelligent result and embedding caching

Changed

• API Consolidation: 15+ search methods → 2 clean APIs (search() and find())
• Search Signature: From search(query, limit, options) to search(query, options)
• Result Format: Now returns full objects with id, score, content, and metadata
• Storage Configuration: Moved under storage option with type-specific settings
• Performance: O(log n) metadata filtering with binary search
• Memory Usage: Reduced from 200MB to 24MB baseline
• Search Latency: Improved from 50ms to 3ms average

Fixed

• Circular dependency in Triple Intelligence system
• Memory leaks in embedding generation
• Worker thread communication timeouts
• Metadata index performance bottlenecks
• TypeScript compilation errors (153 → 0)
• Storage adapter consistency issues

Deprecated

• Individual search methods (searchByVector, searchByNounTypes, etc.)
• Three-parameter search signature
• Direct storage type configuration

Removed

• Legacy delegation pattern
• Redundant search method implementations
• Unused dependencies

Security

• Improved input sanitization
• Safe metadata filtering
• Secure storage adapter implementations

---

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[2.0.0] - 2024-08-22

🚀 Major Features

Triple Intelligence Engine

• NEW: Unified query system combining vector similarity, graph relationships, and field filtering
• NEW: Cross-intelligence optimization - queries automatically use the most efficient combination
• NEW: Natural language query processing with intent recognition

Advanced Indexing Systems

• NEW: HNSW indexing for sub-millisecond vector search
• NEW: Field indexing with O(1) metadata lookups
• NEW: Graph pathfinding with multiple algorithms (Dijkstra, PageRank, BFS/DFS)
• NEW: Metadata index manager for intelligent query optimization

Storage & Performance

• NEW: Universal storage adapters (FileSystem, S3, OPFS, Memory)
• NEW: Smart caching with LRU and intelligent cache invalidation
• NEW: Streaming data processing for large datasets
• NEW: Write-Ahead Logging (WAL) for data integrity

Developer Experience

• NEW: Comprehensive CLI with interactive mode
• NEW: Brain Patterns Query Language (MongoDB-compatible syntax)
• NEW: 220 embedded natural language patterns for query understanding
• NEW: Full TypeScript support with advanced type definitions

🔧 API Changes

Breaking Changes

• CHANGED: search() now returns {id, score, content, metadata} objects instead of arrays
• CHANGED: Storage configuration moved to storage option in constructor
• CHANGED: Vector search results include similarity scores as objects
• CHANGED: Metadata filtering uses new optimized field indexes

New APIs

• ADDED: brain.find() - MongoDB-style queries with semantic extensions
• ADDED: brain.cluster() - Semantic clustering functionality
• ADDED: brain.findRelated() - Relationship discovery and traversal
• ADDED: brain.statistics() - Performance and usage analytics

🏗️ Architecture

Core Systems

• NEW: Triple Intelligence architecture unifying three search paradigms
• NEW: Augmentation system for extensible functionality
• NEW: Entity registry for intelligent data deduplication
• NEW: Pipeline processing for complex data transformations

Performance Optimizations

• IMPROVED: 10x faster metadata filtering using specialized indexes
• IMPROVED: Memory usage optimization with embedded patterns
• IMPROVED: Query optimization with smart execution planning
• IMPROVED: Batch processing for high-throughput scenarios

📚 Documentation & Testing

• NEW: Comprehensive test suite with 50+ tests covering all features
• NEW: Professional documentation with clear examples
• NEW: Migration guide for 1.x users
• NEW: API reference with TypeScript signatures

🐛 Bug Fixes

• FIXED: Memory leaks in pattern matching system
• FIXED: Vector dimension mismatches in multi-model scenarios
• FIXED: Infinite recursion in graph traversal edge cases
• FIXED: Race conditions in concurrent access scenarios
• FIXED: Edge cases in field filtering with complex nested queries

💔 Removed

• REMOVED: Legacy query history (replaced with LRU cache)
• REMOVED: Deprecated 1.x storage format (auto-migration provided)
• REMOVED: Debug logging in production builds

---

[1.6.0] - 2024-08-15

Added

• Enhanced vector operations with better similarity scoring
• Improved metadata filtering capabilities
• Basic graph relationship support
• CLI improvements for better user experience

Fixed

• Vector search accuracy improvements
• Storage stability enhancements
• Memory usage optimizations

---

[1.5.0] - 2024-07-20

Added

• OPFS (Origin Private File System) support for browsers
• Enhanced TypeScript definitions
• Better error handling and reporting

Changed

• Improved API consistency across storage adapters
• Enhanced test coverage

---

[1.0.0] - 2024-06-01

Added

• Initial stable release
• Core vector database functionality
• File system storage adapter
• Basic CLI interface
• TypeScript support

---

Migration Guides

Migrating from 1.x to 2.0

See MIGRATION.md for detailed migration instructions including:

• API changes and new patterns
• Storage format updates
• Configuration changes
• New features and capabilities