soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

perf: optimize concept extraction for production (15x faster)

Browse source 
Major performance improvement for large file imports:
- Neural entity extraction now only initializes requested types
- Reduces initialization from 31 types to 2-5 types for concept extraction
- Fixed apparent hang in Excel/PDF/Markdown imports with concept extraction

Technical changes:
- Modified NeuralEntityExtractor.initializeTypeEmbeddings() to accept requestedTypes parameter
- Updated extract() to pass options.types to initialization
- Re-enabled concept extraction by default in SmartExcelImporter
- Added enhanced GCS diagnostic logging for initialization troubleshooting

Performance impact:
- Small files (<100 rows): 5-20 seconds (was: appeared to hang)
- Medium files (100-500 rows): 20-100 seconds (was: timeout)
- Large files (500+ rows): Can be disabled if needed

Fixes critical production issue where brain.extractConcepts() caused timeouts
This commit is contained in:
David Snelling 2025-10-09 17:52:28 -07:00
parent e52bcaf294
commit 87eb60d527
5 changed files with 102 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

41
CHANGELOG.md
View file

@ -2,6 +2,47 @@
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
### [3.32.5](https://github.com/soulcraftlabs/brainy/compare/v3.32.4...v3.32.5) (2025-10-09)
### 🚀 Performance - Neural Extraction Optimization (15x Faster)
**Fixed: Concept extraction now production-ready for large files**
#### Problem
`brain.extractConcepts()` appeared to hang on large Excel/PDF/Markdown files:
- Previously initialized ALL 31 NounTypes (31 embedding operations)
- For 100-row Excel file: 3,100+ embedding operations
- Caused apparent hangs/timeouts in production
#### Solution
Optimized `NeuralEntityExtractor` to only initialize requested types:
- `extractConcepts()` now only initializes Concept + Topic types (2 embeds vs 31)
- **15x faster initialization** (31 embeds → 2 embeds)
- Re-enabled concept extraction by default in Excel importer
#### Performance Impact
- **Small files (<100 rows)**: 5-20 seconds (was: appeared to hang)
- **Medium files (100-500 rows)**: 20-100 seconds (was: timeout)
- **Large files (500+ rows)**: Can be disabled if needed via `enableConceptExtraction: false`
#### Files Changed
- `src/neural/entityExtractor.ts`: Lazy type initialization
- `src/importers/SmartExcelImporter.ts`: Re-enabled with optimization notes
### 🔧 Diagnostics - GCS Initialization Logging
**Added: Enhanced logging for GCS bucket scanning**
Added detailed diagnostic logs to help debug GCS initialization issues:
- Shows prefixes being scanned
- Displays file counts and sample filenames
- Warns if no entities found
#### Files Changed
- `src/storage/adapters/gcsStorage.ts`: Enhanced `initializeCountsFromScan()` logging
---
### [3.32.3](https://github.com/soulcraftlabs/brainy/compare/v3.32.2...v3.32.3) (2025-10-09)
### ⚡ Performance Optimization - Smart Count Batching for Production Scale