brainy

v3.40.2 853ea26477

v3.40.2 Stable

dpsifr released this 2025-10-13 20:51:01 +02:00 | 806 commits to main since this release

Full Changelog: https://github.com/soulcraftlabs/brainy/compare/v3.40.1...v3.40.2

Downloads

Source code (ZIP)
2 downloads
Source code (TAR.GZ)
1 download

v3.40.1 76466f7f24

v3.40.1 Stable

dpsifr released this 2025-10-13 20:25:30 +02:00 | 808 commits to main since this release

Full Changelog: https://github.com/soulcraftlabs/brainy/compare/v3.40.0...v3.40.1

Downloads

Source code (ZIP)
1 download
Source code (TAR.GZ)
1 download

v3.40.0 d62875dc6c

v3.40.0 Stable

dpsifr released this 2025-10-13 19:33:23 +02:00 | 810 commits to main since this release

Full Changelog: https://github.com/soulcraftlabs/brainy/compare/v3.39.0...v3.40.0

Downloads

Source code (ZIP)
2 downloads
Source code (TAR.GZ)
0 downloads

v3.39.0 77c104a9a4

v3.39.0 - Excel Import Performance Stable

dpsifr released this 2025-10-13 19:07:30 +02:00 | 812 commits to main since this release

🚀 Performance: 54,000x Faster Excel Imports

Massive performance improvements for Excel imports with AI extraction - from 9+ minutes to 3-5 seconds!

🎯 Key Improvements

Runtime Embedding Cache (93.8% hit rate)
- Caches candidate embeddings during extraction
- Avoids redundant model calls
- LRU eviction prevents memory bloat
- New methods: clearEmbeddingCache(), getEmbeddingCacheStats()
Batch Parallel Processing (10x speedup)
- Processes 10 rows in parallel per chunk
- Entity and concept extraction happen simultaneously
- Progress updates every chunk instead of every row
Enhanced Progress Reporting
- Real-time throughput (rows/sec)
- Estimated time remaining (ETA)
- Phase tracking for multi-stage imports
- New optional fields in ImportProgress interface

📊 Performance Results

Metric	Before	After	Improvement
Per-row latency	5,400ms	0.1ms	54,000x faster
Throughput	0.2 rows/sec	12,500 rows/sec	62,500x faster
Cache hit rate	0%	93.8%	Saves 93.8% of model calls
420KB file	9+ minutes	~3-5 seconds	108-180x faster

✨ New Features

NeuralEntityExtractor.clearEmbeddingCache() - Clear embedding cache
NeuralEntityExtractor.getEmbeddingCacheStats() - Get cache statistics
ImportProgress.throughput - Real-time rows/sec
ImportProgress.eta - Estimated time remaining

🔄 Backward Compatibility

✅ Fully backward compatible - all new fields are optional. Drop-in replacement for v3.38.0.

📝 Test

Run examples/test-excel-performance.ts to see the improvements in action.

Full Changelog: https://github.com/soulcraftlabs/brainy/compare/v3.38.0...v3.39.0

Downloads

Source code (ZIP)
2 downloads
Source code (TAR.GZ)
1 download

v3.38.0 6778f48dfa

v3.38.0 - Clean Production Logging Stable

dpsifr released this 2025-10-13 18:24:07 +02:00 | 814 commits to main since this release

🧹 Clean Production Logging

This release cleans up verbose diagnostic logging added during the v3.37.x GCS persistence debugging journey, making production logs clean and actionable while maintaining all critical warnings and error detection.

✨ Changes

Logging Cleanup:

Remove verbose cache check logging (structure details, field validation diagnostics)
Remove verbose GCS/S3 operation logs (download progress, parsing steps, success confirmations)
Remove verbose pagination diagnostic logs (per-shard details, per-file tracking)
Keep all critical warnings for invalid cache, null cache, and auto-recovery actions
Keep all error logs for actual failures
Move successful operation details to trace level for debugging when needed

Stability:

v3.37.8 cache validation remains fully functional ✅
Invalid cache detection and auto-recovery still work perfectly ✅
All safety nets remain in place ✅

📊 Impact

Before (v3.37.8):

[getNode] 🔍 Cache check for 567bf17c...: { hasCached: true, isNull: false, ... }
[getNode] Cached object structure: { hasId: true, hasVector: true, ... }
[getNode] ✅ Valid cached object - returning
[getNode] 🔍 Attempting to load: UUID: ..., Path: ..., Bucket: ...
[getNode] 📥 Downloading file...
[getNode] ✅ Download successful: 10073 bytes
[getNode] 🔧 Parsing JSON...
[getNode] ✅ JSON parsed successfully, id: ...
[Shard 56] Found 12 files in "entities/nouns/vectors/56/"
[Shard 56] Extracted 12 UUIDs: ...

After (v3.38.0):

(silent during normal operations)

# Only logs when issues occur:
[GCS] Invalid cached object for 567bf17c (empty vector) - removing from cache and reloading
[GCS] Cache contains null for 567bf17c - reloading from storage

🚀 GCS Persistence Journey Complete

This completes the GCS persistence journey from v3.35.0 → v3.38.0:

v3.35.0: HNSW persistence foundation
v3.37.0-v3.37.7: Fixed cache poisoning bugs
v3.37.8: Added cache validation and auto-recovery
v3.38.0: Cleaned up diagnostic logging ✨

Production-ready cloud storage persistence with clean, actionable logs! 🎉

📦 Installation

npm install @soulcraft/brainy@3.38.0

Full Changelog: https://github.com/soulcraftlabs/brainy/compare/v3.37.8...v3.38.0

Downloads

Source code (ZIP)
1 download
Source code (TAR.GZ)
1 download

v3.37.8 38b563c981

v3.37.8 - Critical Cache Validation Fix Stable

dpsifr released this 2025-10-13 18:01:39 +02:00 | 816 commits to main since this release

🐛 Critical Bug Fix

Fixed cache pollution from empty-vector objects in HNSW lazy mode that caused silent failures during entity retrieval after container restarts.

What's Fixed

Cache Validation:

Add comprehensive validation to detect and reject cached objects with empty vectors (vector: [])
Auto-remove invalid cached objects and reload from cloud storage
Prevent caching nodes with empty vectors in saveNode()
Add detailed cached object structure logging for diagnostics

Affected Adapters:

✅ GCS Storage: Full validation and auto-recovery
✅ S3 Storage: Full validation and auto-recovery

Root Cause

HNSW's adaptive caching for large datasets uses lazy loading (vector: [] for memory efficiency). These empty-vector objects were being cached and returned as valid, causing getNode() to return null despite valid data existing in cloud storage.