perf: pre-compute type embeddings at build time (zero runtime cost)

Major optimization - all type embeddings now built into package:

Build-time generation:
- Created scripts/buildTypeEmbeddings.ts to generate all type embeddings
- Generates embeddings for 31 NounTypes + 40 VerbTypes at build time
- Stores as base64-encoded binary data in embeddedTypeEmbeddings.ts
- Added check script to rebuild only when needed

Updated all consumers:
- NeuralEntityExtractor: loads pre-computed embeddings (instant)
- BrainyTypes: loads pre-computed embeddings (instant init)
- NaturalLanguageProcessor: loads pre-computed embeddings (instant init)

Build process:
- Added npm run build:types to generate embeddings
- Added npm run build:types:if-needed for conditional rebuild
- Integrated into main build pipeline
- Auto-rebuilds only when types or build script change

Benefits:
- Zero runtime cost - embeddings loaded instantly
- Survives all container restarts
- All 71 types always available (31 nouns + 40 verbs)
- ~100KB memory overhead for permanent performance gain
- Eliminates 5-10 second initialization delay

This completes the type embedding optimization started in v3.32.5
This commit is contained in:
David Snelling 2025-10-09 18:08:57 -07:00
parent 87eb60d527
commit 0d649b8a79
9 changed files with 643 additions and 111 deletions

View file

@ -141,18 +141,18 @@ export class SmartExcelImporter {
const opts = {
enableNeuralExtraction: true,
enableRelationshipInference: true,
// CONCEPT EXTRACTION NOW PRODUCTION-READY (v3.32.5+):
// Performance optimization: Only initializes needed types (Concept + Topic = 2 embeds)
// Previously initialized all 31 types (31 embeds) which caused apparent hangs
// CONCEPT EXTRACTION PRODUCTION-READY (v3.33.0+):
// Type embeddings are now pre-computed at build time - zero runtime cost!
// All 31 noun types + 40 verb types instantly available
//
// Performance profile:
// - Type embeddings: INSTANT (pre-computed at build time, ~100KB in-memory)
// - Model loading: ~2-5 seconds (one-time, cached after first use)
// - Type init: 2 embeds for Concept + Topic (vs 31 previously)
// - Per-row extraction: ~50-200ms depending on definition length
// - 100 rows: ~5-20 seconds total (acceptable for production)
// - 1000 rows: ~50-200 seconds (may want to disable for very large files)
// - 100 rows: ~5-20 seconds total (production ready)
// - 1000 rows: ~50-200 seconds (disable if needed via enableConceptExtraction: false)
//
// Enabled by default for production use. Disable for files >500 rows if needed.
// Enabled by default for production use.
enableConceptExtraction: true,
confidenceThreshold: 0.6,
termColumn: 'term|name|title|concept',