soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

335 releases 524 tags

RSS feed
  • v3.39.0 77c104a9a4

    v3.39.0 - Excel Import Performance Stable

    dpsifr released this 2025-10-13 19:07:30 +02:00 | 812 commits to main since this release

    🚀 Performance: 54,000x Faster Excel Imports

    Massive performance improvements for Excel imports with AI extraction - from 9+ minutes to 3-5 seconds!

    🎯 Key Improvements

    1. Runtime Embedding Cache (93.8% hit rate)

      • Caches candidate embeddings during extraction
      • Avoids redundant model calls
      • LRU eviction prevents memory bloat
      • New methods: clearEmbeddingCache(), getEmbeddingCacheStats()

    2. Batch Parallel Processing (10x speedup)

      • Processes 10 rows in parallel per chunk
      • Entity and concept extraction happen simultaneously
      • Progress updates every chunk instead of every row

    3. Enhanced Progress Reporting

      • Real-time throughput (rows/sec)
      • Estimated time remaining (ETA)
      • Phase tracking for multi-stage imports
      • New optional fields in ImportProgress interface

    📊 Performance Results

    Metric Before After Improvement
    Per-row latency 5,400ms 0.1ms 54,000x faster
    Throughput 0.2 rows/sec 12,500 rows/sec 62,500x faster
    Cache hit rate 0% 93.8% Saves 93.8% of model calls
    420KB file 9+ minutes ~3-5 seconds 108-180x faster

    ✨ New Features

    • NeuralEntityExtractor.clearEmbeddingCache() - Clear embedding cache
    • NeuralEntityExtractor.getEmbeddingCacheStats() - Get cache statistics
    • ImportProgress.throughput - Real-time rows/sec
    • ImportProgress.eta - Estimated time remaining

    🔄 Backward Compatibility

    ✅ Fully backward compatible - all new fields are optional. Drop-in replacement for v3.38.0.

    📝 Test

    Run examples/test-excel-performance.ts to see the improvements in action.

    Full Changelog: https://github.com/soulcraftlabs/brainy/compare/v3.38.0...v3.39.0

    Downloads