perf: pre-compute type embeddings at build time (zero runtime cost)
Major optimization - all type embeddings now built into package: Build-time generation: - Created scripts/buildTypeEmbeddings.ts to generate all type embeddings - Generates embeddings for 31 NounTypes + 40 VerbTypes at build time - Stores as base64-encoded binary data in embeddedTypeEmbeddings.ts - Added check script to rebuild only when needed Updated all consumers: - NeuralEntityExtractor: loads pre-computed embeddings (instant) - BrainyTypes: loads pre-computed embeddings (instant init) - NaturalLanguageProcessor: loads pre-computed embeddings (instant init) Build process: - Added npm run build:types to generate embeddings - Added npm run build:types:if-needed for conditional rebuild - Integrated into main build pipeline - Auto-rebuilds only when types or build script change Benefits: - Zero runtime cost - embeddings loaded instantly - Survives all container restarts - All 71 types always available (31 nouns + 40 verbs) - ~100KB memory overhead for permanent performance gain - Eliminates 5-10 second initialization delay This completes the type embedding optimization started in v3.32.5
This commit is contained in:
parent
87eb60d527
commit
0d649b8a79
9 changed files with 643 additions and 111 deletions
|
|
@ -141,18 +141,18 @@ export class SmartExcelImporter {
|
|||
const opts = {
|
||||
enableNeuralExtraction: true,
|
||||
enableRelationshipInference: true,
|
||||
// CONCEPT EXTRACTION NOW PRODUCTION-READY (v3.32.5+):
|
||||
// Performance optimization: Only initializes needed types (Concept + Topic = 2 embeds)
|
||||
// Previously initialized all 31 types (31 embeds) which caused apparent hangs
|
||||
// CONCEPT EXTRACTION PRODUCTION-READY (v3.33.0+):
|
||||
// Type embeddings are now pre-computed at build time - zero runtime cost!
|
||||
// All 31 noun types + 40 verb types instantly available
|
||||
//
|
||||
// Performance profile:
|
||||
// - Type embeddings: INSTANT (pre-computed at build time, ~100KB in-memory)
|
||||
// - Model loading: ~2-5 seconds (one-time, cached after first use)
|
||||
// - Type init: 2 embeds for Concept + Topic (vs 31 previously)
|
||||
// - Per-row extraction: ~50-200ms depending on definition length
|
||||
// - 100 rows: ~5-20 seconds total (acceptable for production)
|
||||
// - 1000 rows: ~50-200 seconds (may want to disable for very large files)
|
||||
// - 100 rows: ~5-20 seconds total (production ready)
|
||||
// - 1000 rows: ~50-200 seconds (disable if needed via enableConceptExtraction: false)
|
||||
//
|
||||
// Enabled by default for production use. Disable for files >500 rows if needed.
|
||||
// Enabled by default for production use.
|
||||
enableConceptExtraction: true,
|
||||
confidenceThreshold: 0.6,
|
||||
termColumn: 'term|name|title|concept',
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue