perf: optimize concept extraction for production (15x faster)

Major performance improvement for large file imports:
- Neural entity extraction now only initializes requested types
- Reduces initialization from 31 types to 2-5 types for concept extraction
- Fixed apparent hang in Excel/PDF/Markdown imports with concept extraction

Technical changes:
- Modified NeuralEntityExtractor.initializeTypeEmbeddings() to accept requestedTypes parameter
- Updated extract() to pass options.types to initialization
- Re-enabled concept extraction by default in SmartExcelImporter
- Added enhanced GCS diagnostic logging for initialization troubleshooting

Performance impact:
- Small files (<100 rows): 5-20 seconds (was: appeared to hang)
- Medium files (100-500 rows): 20-100 seconds (was: timeout)
- Large files (500+ rows): Can be disabled if needed

Fixes critical production issue where brain.extractConcepts() caused timeouts
This commit is contained in:
David Snelling 2025-10-09 17:52:28 -07:00
parent e52bcaf294
commit 87eb60d527
5 changed files with 102 additions and 15 deletions

View file

@ -141,6 +141,18 @@ export class SmartExcelImporter {
const opts = {
enableNeuralExtraction: true,
enableRelationshipInference: true,
// CONCEPT EXTRACTION NOW PRODUCTION-READY (v3.32.5+):
// Performance optimization: Only initializes needed types (Concept + Topic = 2 embeds)
// Previously initialized all 31 types (31 embeds) which caused apparent hangs
//
// Performance profile:
// - Model loading: ~2-5 seconds (one-time, cached after first use)
// - Type init: 2 embeds for Concept + Topic (vs 31 previously)
// - Per-row extraction: ~50-200ms depending on definition length
// - 100 rows: ~5-20 seconds total (acceptable for production)
// - 1000 rows: ~50-200 seconds (may want to disable for very large files)
//
// Enabled by default for production use. Disable for files >500 rows if needed.
enableConceptExtraction: true,
confidenceThreshold: 0.6,
termColumn: 'term|name|title|concept',