soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

perf: optimize concept extraction for production (15x faster)

Browse source 
Major performance improvement for large file imports:
- Neural entity extraction now only initializes requested types
- Reduces initialization from 31 types to 2-5 types for concept extraction
- Fixed apparent hang in Excel/PDF/Markdown imports with concept extraction

Technical changes:
- Modified NeuralEntityExtractor.initializeTypeEmbeddings() to accept requestedTypes parameter
- Updated extract() to pass options.types to initialization
- Re-enabled concept extraction by default in SmartExcelImporter
- Added enhanced GCS diagnostic logging for initialization troubleshooting

Performance impact:
- Small files (<100 rows): 5-20 seconds (was: appeared to hang)
- Medium files (100-500 rows): 20-100 seconds (was: timeout)
- Large files (500+ rows): Can be disabled if needed

Fixes critical production issue where brain.extractConcepts() caused timeouts
This commit is contained in:
David Snelling 2025-10-09 17:52:28 -07:00
parent e52bcaf294
commit 87eb60d527
5 changed files with 102 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

29
src/storage/adapters/gcsStorage.ts
View file

@ -1484,19 +1484,38 @@ export class GcsStorage extends BaseStorage {
private async initializeCountsFromScan(): Promise<void> {
try {
prodLog.info('📊 Scanning GCS bucket to initialize counts...')
prodLog.info(`🔍 Noun prefix: ${this.nounPrefix}`)
prodLog.info(`🔍 Verb prefix: ${this.verbPrefix}`)
// Count nouns
const [nounFiles] = await this.bucket!.getFiles({ prefix: this.nounPrefix })
this.totalNounCount = nounFiles?.filter((f: any) => f.name?.endsWith('.json')).length || 0
prodLog.info(`🔍 Found ${nounFiles?.length || 0} total files under noun prefix`)
const jsonNounFiles = nounFiles?.filter((f: any) => f.name?.endsWith('.json')) || []
this.totalNounCount = jsonNounFiles.length
if (jsonNounFiles.length > 0 && jsonNounFiles.length <= 5) {
prodLog.info(`📄 Sample noun files: ${jsonNounFiles.slice(0, 5).map((f: any) => f.name).join(', ')}`)
}
// Count verbs
const [verbFiles] = await this.bucket!.getFiles({ prefix: this.verbPrefix })
this.totalVerbCount = verbFiles?.filter((f: any) => f.name?.endsWith('.json')).length || 0
prodLog.info(`🔍 Found ${verbFiles?.length || 0} total files under verb prefix`)
const jsonVerbFiles = verbFiles?.filter((f: any) => f.name?.endsWith('.json')) || []
this.totalVerbCount = jsonVerbFiles.length
if (jsonVerbFiles.length > 0 && jsonVerbFiles.length <= 5) {
prodLog.info(`📄 Sample verb files: ${jsonVerbFiles.slice(0, 5).map((f: any) => f.name).join(', ')}`)
}
// Save initial counts
await this.persistCounts()
prodLog.info(`✅ Initialized counts from scan: ${this.totalNounCount} nouns, ${this.totalVerbCount} verbs`)
if (this.totalNounCount > 0 || this.totalVerbCount > 0) {
await this.persistCounts()
prodLog.info(`✅ Initialized counts from scan: ${this.totalNounCount} nouns, ${this.totalVerbCount} verbs`)
} else {
prodLog.warn(`⚠️ No entities found during bucket scan. Check that entities exist and prefixes are correct.`)
}
} catch (error) {
// CRITICAL FIX: Don't silently fail - this prevents data loss scenarios
this.logger.error('❌ CRITICAL: Failed to initialize counts from GCS bucket scan:', error)