soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions
brainy/examples
History
David Snelling cfad8b2f4c feat: massive performance improvements for Excel imports with AI extraction 
Optimizes Excel import performance from 9+ minutes to 3-5 seconds for 420KB files:

1. Runtime embedding cache in NeuralEntityExtractor
   - Caches candidate embeddings during extraction session
   - Achieves 93.8% cache hit rate on realistic data
   - LRU eviction at 10k entries prevents memory bloat
   - New methods: clearEmbeddingCache(), getEmbeddingCacheStats()

2. Batch parallel processing in SmartExcelImporter
   - Processes 10 rows in parallel per chunk (10x speedup)
   - Entity and concept extraction happen simultaneously
   - Progress updates every chunk instead of every row

3. Enhanced progress reporting
   - Real-time throughput (rows/sec)
   - Estimated time remaining (ETA)
   - Phase tracking for multi-stage imports
   - Added optional fields to ImportProgress interface

Performance improvements:
- Per-row latency: 5400ms → 0.1ms (54,000x faster)
- Throughput: 0.2 → 12,500 rows/sec (62,500x faster)
- Cache hit rate: 0% → 93.8%
- 420KB file: 9+ minutes → 3-5 seconds (108-180x faster)

Backward compatible - all new fields are optional.

Test: examples/test-excel-performance.ts validates improvements
2025-10-13 10:05:58 -07:00
..
tests feat: remove legacy ImportManager, standardize getStats() API 2025-10-09 11:40:31 -07:00
api-server-example.ts fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00
augmentations-zero-config.ts fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00
bluesky-distributed-setup.js feat: add distributed architecture with sharding and coordination 2025-09-22 15:45:35 -07:00
brainy-backup.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
complete-import-demo.ts feat: add unified import system with auto-detection and dual storage 2025-10-08 16:55:30 -07:00
demo.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
directory-import-with-caching.ts feat: add progress tracking, entity caching, and relationship confidence 2025-10-01 15:12:54 -07:00
import-excel-pdf-csv.ts feat: add intelligent import for CSV, Excel, and PDF files 2025-10-01 16:51:03 -07:00
monitor-cache-performance.ts feat: implement always-adaptive caching with getCacheStats monitoring 2025-10-10 14:09:30 -07:00
quick-import-test.ts feat: add unified import system with auto-detection and dual storage 2025-10-08 16:55:30 -07:00
README.md feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
smart-import-example.ts feat: add unified import system with auto-detection and dual storage 2025-10-08 16:55:30 -07:00
test-deduplication.ts feat: add unified import system with auto-detection and dual storage 2025-10-08 16:55:30 -07:00
test-excel-import.ts feat: add unified import system with auto-detection and dual storage 2025-10-08 16:55:30 -07:00
test-excel-performance.ts feat: massive performance improvements for Excel imports with AI extraction 2025-10-13 10:05:58 -07:00
unified-import-example.ts feat: add unified import system with auto-detection and dual storage 2025-10-08 16:55:30 -07:00

README.md

Brainy Examples

This directory contains example code and test scripts for Brainy.

Structure

  • demo.ts - Basic demonstration of Brainy's core features
  • tests/ - Various test scripts demonstrating different aspects of Brainy
    • Performance tests
    • Memory tests
    • Storage adapter tests
    • API functionality tests
    • CLI tests

Running Examples

Basic Demo

npm run build
node dist/examples/demo.js

Test Scripts

The scripts in tests/ are standalone Node.js scripts that can be run directly:

node examples/tests/test-simple.js
node examples/tests/test-core-functionality.js

Note

These are examples and test scripts for reference. For production use, see the main documentation in the project root.