feat: migrate embeddings to Candle WASM + remove semantic type inference
Major architectural changes: 1. EMBEDDINGS ENGINE (ONNX → Candle WASM): - Replace ONNX Runtime with Rust Candle compiled to WASM - Embedded model in WASM binary (no external downloads) - Quantized Q8 precision with <50MB memory footprint - Zero-download, offline-first operation - Same embedding quality (all-MiniLM-L6-v2) 2. REMOVE SEMANTIC TYPE INFERENCE: - Delete embeddedKeywordEmbeddings.ts (14MB of pre-computed embeddings) - Remove typeAwareQueryPlanner.ts and semanticTypeInference.ts - Remove VerbExactMatchSignal (uses keyword embeddings) - Update SmartRelationshipExtractor to 3 signals (55%/30%/15% weights) API CHANGES (requires v7.0.0): - Removed: inferTypes(), inferNouns(), inferVerbs(), inferIntent() - Removed: getSemanticTypeInference(), SemanticTypeInference class - Removed: TypeInference, SemanticTypeInferenceOptions types Users can still use natural language queries in find() - they just need to specify type explicitly for type-optimized searches. PACKAGE SIZE IMPACT: - Compressed: 90.1 MB → 86.2 MB (-4.3%) - Uncompressed: 114.4 MB → 100.3 MB (-12%) - ~448K lines of code removed 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
81cd16e41b
commit
da7d2ed29d
60 changed files with 3887 additions and 448557 deletions
132
scripts/build-candle-wasm.sh
Executable file
132
scripts/build-candle-wasm.sh
Executable file
|
|
@ -0,0 +1,132 @@
|
|||
#!/bin/bash
|
||||
# Build script for Candle WASM embedding engine
|
||||
#
|
||||
# Requirements:
|
||||
# - Rust toolchain (rustup)
|
||||
# - wasm-pack (cargo install wasm-pack)
|
||||
# - Build tools (build-essential on Ubuntu/Debian)
|
||||
#
|
||||
# Usage:
|
||||
# ./scripts/build-candle-wasm.sh
|
||||
# ./scripts/build-candle-wasm.sh --release
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
CANDLE_DIR="$PROJECT_ROOT/src/embeddings/candle-wasm"
|
||||
OUTPUT_DIR="$PROJECT_ROOT/src/embeddings/wasm/pkg"
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
echo -e "${GREEN}Building Candle WASM embedding engine...${NC}"
|
||||
|
||||
# Check prerequisites
|
||||
check_prerequisites() {
|
||||
local missing=()
|
||||
|
||||
if ! command -v rustc &> /dev/null; then
|
||||
missing+=("rust (install via: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh)")
|
||||
fi
|
||||
|
||||
if ! command -v wasm-pack &> /dev/null; then
|
||||
missing+=("wasm-pack (install via: cargo install wasm-pack)")
|
||||
fi
|
||||
|
||||
if ! command -v cc &> /dev/null && ! command -v gcc &> /dev/null; then
|
||||
missing+=("C compiler (install via: sudo apt-get install build-essential)")
|
||||
fi
|
||||
|
||||
if [ ${#missing[@]} -gt 0 ]; then
|
||||
echo -e "${RED}Missing prerequisites:${NC}"
|
||||
for prereq in "${missing[@]}"; do
|
||||
echo " - $prereq"
|
||||
done
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}All prerequisites found.${NC}"
|
||||
}
|
||||
|
||||
# Download model files if not present
|
||||
download_model() {
|
||||
local MODEL_DIR="$PROJECT_ROOT/assets/models/all-MiniLM-L6-v2"
|
||||
local SAFETENSORS="$MODEL_DIR/model.safetensors"
|
||||
local TOKENIZER="$MODEL_DIR/tokenizer.json"
|
||||
local CONFIG="$MODEL_DIR/config.json"
|
||||
|
||||
if [ -f "$SAFETENSORS" ] && [ -f "$TOKENIZER" ] && [ -f "$CONFIG" ]; then
|
||||
echo -e "${GREEN}Model files already present.${NC}"
|
||||
return
|
||||
fi
|
||||
|
||||
echo -e "${YELLOW}Downloading model files...${NC}"
|
||||
mkdir -p "$MODEL_DIR"
|
||||
|
||||
# Download from HuggingFace Hub
|
||||
local HF_URL="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main"
|
||||
|
||||
if [ ! -f "$SAFETENSORS" ]; then
|
||||
curl -L "$HF_URL/model.safetensors" -o "$SAFETENSORS"
|
||||
fi
|
||||
|
||||
if [ ! -f "$TOKENIZER" ]; then
|
||||
curl -L "$HF_URL/tokenizer.json" -o "$TOKENIZER"
|
||||
fi
|
||||
|
||||
if [ ! -f "$CONFIG" ]; then
|
||||
curl -L "$HF_URL/config.json" -o "$CONFIG"
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}Model files downloaded.${NC}"
|
||||
}
|
||||
|
||||
# Build WASM
|
||||
build_wasm() {
|
||||
local BUILD_MODE="${1:-release}"
|
||||
|
||||
echo -e "${GREEN}Building WASM (${BUILD_MODE})...${NC}"
|
||||
cd "$CANDLE_DIR"
|
||||
|
||||
if [ "$BUILD_MODE" = "release" ]; then
|
||||
wasm-pack build --target web --release --out-dir "$OUTPUT_DIR"
|
||||
else
|
||||
wasm-pack build --target web --dev --out-dir "$OUTPUT_DIR"
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}WASM build complete. Output: $OUTPUT_DIR${NC}"
|
||||
}
|
||||
|
||||
# Main
|
||||
main() {
|
||||
local mode="release"
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--dev|--debug)
|
||||
mode="dev"
|
||||
shift
|
||||
;;
|
||||
--release)
|
||||
mode="release"
|
||||
shift
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
check_prerequisites
|
||||
download_model
|
||||
build_wasm "$mode"
|
||||
|
||||
echo -e "${GREEN}Done! WASM package ready at: $OUTPUT_DIR${NC}"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
|
|
@ -1,571 +0,0 @@
|
|||
/**
|
||||
* Build Keyword Embeddings - Generate pre-computed embeddings for all keywords
|
||||
*
|
||||
* Extracts keywords from TypeInferenceSystem, adds strategic synonyms,
|
||||
* and generates semantic embeddings for fast type inference.
|
||||
*
|
||||
* Output: src/neural/embeddedKeywordEmbeddings.ts (~2-3MB)
|
||||
* Runtime: ~60-90 seconds (one-time build cost)
|
||||
*/
|
||||
|
||||
import { TransformerEmbedding } from '../src/utils/embedding.js'
|
||||
import { NounType, VerbType } from '../src/types/graphTypes.js'
|
||||
import { writeFileSync } from 'fs'
|
||||
import { prodLog } from '../src/utils/logger.js'
|
||||
|
||||
interface KeywordDefinition {
|
||||
keyword: string
|
||||
type: NounType | VerbType
|
||||
typeCategory: 'noun' | 'verb'
|
||||
confidence: number
|
||||
isCanonical: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract and expand keywords with synonyms (NOUNS + VERBS)
|
||||
*/
|
||||
function buildExpandedKeywordList(): KeywordDefinition[] {
|
||||
const keywords: KeywordDefinition[] = []
|
||||
|
||||
// Helper to add noun keywords
|
||||
const addNoun = (words: string[], type: NounType, confidence: number, isCanonical = true) => {
|
||||
for (const word of words) {
|
||||
keywords.push({ keyword: word, type, typeCategory: 'noun', confidence, isCanonical })
|
||||
}
|
||||
}
|
||||
|
||||
// Helper to add verb keywords
|
||||
const addVerb = (words: string[], type: VerbType, confidence: number, isCanonical = true) => {
|
||||
for (const word of words) {
|
||||
keywords.push({ keyword: word, type, typeCategory: 'verb', confidence, isCanonical })
|
||||
}
|
||||
}
|
||||
|
||||
// Legacy alias for noun keywords
|
||||
const add = addNoun
|
||||
|
||||
// ========== Person Type ==========
|
||||
// Core professional roles (canonical)
|
||||
add(['person', 'people', 'individual', 'human'], NounType.Person, 0.95, true)
|
||||
add(['employee', 'worker', 'staff', 'personnel'], NounType.Person, 0.90, true)
|
||||
|
||||
// Engineering & Tech
|
||||
add(['engineer', 'developer', 'programmer', 'architect', 'designer', 'technician'], NounType.Person, 0.95, true)
|
||||
add(['coder', 'techie'], NounType.Person, 0.85, false) // Synonyms
|
||||
|
||||
// Medical
|
||||
add(['doctor', 'physician', 'surgeon', 'nurse', 'therapist'], NounType.Person, 0.95, true)
|
||||
add(['cardiologist', 'oncologist', 'neurologist', 'psychiatrist', 'psychologist'], NounType.Person, 0.90, true)
|
||||
add(['radiologist', 'pathologist', 'anesthesiologist', 'dermatologist'], NounType.Person, 0.90, true)
|
||||
add(['pediatrician', 'obstetrician', 'gynecologist', 'ophthalmologist'], NounType.Person, 0.90, true)
|
||||
add(['dentist', 'orthodontist', 'pharmacist', 'paramedic', 'emt'], NounType.Person, 0.90, true)
|
||||
add(['medic', 'practitioner', 'clinician'], NounType.Person, 0.85, false) // Medical synonyms
|
||||
|
||||
// Management & Leadership
|
||||
add(['manager', 'director', 'executive', 'leader', 'supervisor', 'coordinator'], NounType.Person, 0.95, true)
|
||||
add(['ceo', 'cto', 'cfo', 'coo', 'vp', 'president', 'founder', 'owner'], NounType.Person, 0.95, true)
|
||||
|
||||
// Professional services
|
||||
add(['analyst', 'consultant', 'specialist', 'expert', 'professional'], NounType.Person, 0.90, true)
|
||||
add(['lawyer', 'attorney', 'judge', 'paralegal'], NounType.Person, 0.95, true)
|
||||
add(['accountant', 'auditor', 'banker', 'trader', 'broker'], NounType.Person, 0.90, true)
|
||||
add(['advisor', 'counselor'], NounType.Person, 0.85, false)
|
||||
|
||||
// Education & Research
|
||||
add(['teacher', 'professor', 'instructor', 'educator', 'tutor'], NounType.Person, 0.95, true)
|
||||
add(['student', 'pupil', 'learner', 'trainee', 'intern'], NounType.Person, 0.90, true)
|
||||
add(['researcher', 'scientist', 'scholar', 'academic'], NounType.Person, 0.95, true)
|
||||
|
||||
// Creative professions
|
||||
add(['artist', 'musician', 'painter', 'sculptor', 'performer'], NounType.Person, 0.90, true)
|
||||
add(['author', 'writer', 'journalist', 'editor', 'reporter'], NounType.Person, 0.90, true)
|
||||
|
||||
// Sales & Marketing
|
||||
add(['salesperson', 'marketer', 'recruiter', 'agent'], NounType.Person, 0.85, true)
|
||||
|
||||
// Social relationships
|
||||
add(['friend', 'colleague', 'coworker', 'teammate', 'partner'], NounType.Person, 0.85, true)
|
||||
add(['customer', 'client', 'vendor', 'supplier', 'contractor'], NounType.Person, 0.85, true)
|
||||
add(['mentor', 'mentee', 'coach', 'volunteer', 'activist', 'advocate', 'supporter'], NounType.Person, 0.80, true)
|
||||
|
||||
// Demographics
|
||||
add(['male', 'female', 'adult', 'child', 'teen', 'senior', 'junior'], NounType.Person, 0.75, true)
|
||||
|
||||
// Multi-word professions (important for semantic matching)
|
||||
add(['software engineer', 'software developer', 'web developer'], NounType.Person, 0.95, true)
|
||||
add(['data scientist', 'data engineer', 'machine learning engineer', 'ml engineer'], NounType.Person, 0.95, true)
|
||||
add(['product manager', 'project manager', 'engineering manager'], NounType.Person, 0.95, true)
|
||||
add(['ux designer', 'ui designer', 'graphic designer'], NounType.Person, 0.90, true)
|
||||
add(['medical doctor', 'registered nurse', 'healthcare worker'], NounType.Person, 0.90, false)
|
||||
|
||||
// ========== Organization Type ==========
|
||||
add(['organization', 'company', 'business', 'corporation', 'enterprise'], NounType.Organization, 0.95, true)
|
||||
add(['firm', 'agency', 'bureau', 'office', 'department'], NounType.Organization, 0.90, true)
|
||||
add(['startup', 'venture', 'subsidiary', 'branch', 'division'], NounType.Organization, 0.90, true)
|
||||
add(['institution', 'foundation', 'association', 'society', 'club'], NounType.Organization, 0.90, true)
|
||||
add(['nonprofit', 'ngo', 'charity', 'trust', 'federation'], NounType.Organization, 0.90, true)
|
||||
add(['government', 'ministry', 'administration', 'authority'], NounType.Organization, 0.90, true)
|
||||
add(['university', 'college', 'school', 'academy', 'institute'], NounType.Organization, 0.90, true)
|
||||
add(['hospital', 'clinic', 'medical center'], NounType.Organization, 0.90, true)
|
||||
add(['bank', 'credit union'], NounType.Organization, 0.90, true)
|
||||
add(['manufacturer', 'factory', 'plant', 'facility'], NounType.Organization, 0.85, true)
|
||||
add(['retailer', 'store', 'shop', 'outlet', 'chain'], NounType.Organization, 0.85, true)
|
||||
add(['restaurant', 'hotel', 'resort', 'casino'], NounType.Organization, 0.85, true)
|
||||
add(['publisher', 'studio', 'gallery', 'museum', 'library'], NounType.Organization, 0.85, true)
|
||||
add(['lab', 'laboratory', 'research center'], NounType.Organization, 0.85, true)
|
||||
add(['team', 'squad', 'crew', 'group', 'committee'], NounType.Organization, 0.80, true)
|
||||
|
||||
// Organization synonyms
|
||||
add(['corp', 'inc', 'llc', 'ltd'], NounType.Organization, 0.85, false)
|
||||
|
||||
// ========== Location Type ==========
|
||||
add(['location', 'place', 'area', 'region', 'zone', 'district'], NounType.Location, 0.90, true)
|
||||
add(['city', 'town', 'village', 'municipality', 'metro'], NounType.Location, 0.95, true)
|
||||
add(['country', 'nation', 'state', 'province', 'territory'], NounType.Location, 0.95, true)
|
||||
add(['county', 'parish', 'prefecture', 'canton'], NounType.Location, 0.85, true)
|
||||
add(['continent', 'island', 'peninsula', 'archipelago'], NounType.Location, 0.90, true)
|
||||
add(['street', 'road', 'avenue', 'boulevard', 'lane', 'drive'], NounType.Location, 0.85, true)
|
||||
add(['address', 'building', 'structure', 'tower', 'complex'], NounType.Location, 0.85, true)
|
||||
add(['headquarters', 'hq', 'campus', 'site'], NounType.Location, 0.85, true)
|
||||
add(['center', 'venue', 'space', 'room'], NounType.Location, 0.80, true)
|
||||
add(['warehouse', 'depot', 'terminal', 'station', 'port'], NounType.Location, 0.85, true)
|
||||
add(['park', 'garden', 'plaza', 'square', 'mall'], NounType.Location, 0.85, true)
|
||||
add(['neighborhood', 'suburb', 'downtown', 'uptown'], NounType.Location, 0.80, true)
|
||||
add(['north', 'south', 'east', 'west', 'central'], NounType.Location, 0.70, true)
|
||||
add(['coastal', 'inland', 'urban', 'rural', 'remote'], NounType.Location, 0.70, true)
|
||||
|
||||
// Common cities (for better semantic matching)
|
||||
add(['san francisco', 'new york', 'los angeles', 'chicago', 'boston'], NounType.Location, 0.95, true)
|
||||
add(['seattle', 'austin', 'denver', 'portland', 'miami'], NounType.Location, 0.95, true)
|
||||
add(['london', 'paris', 'berlin', 'tokyo', 'beijing'], NounType.Location, 0.95, true)
|
||||
add(['silicon valley', 'bay area', 'new york city', 'washington dc'], NounType.Location, 0.95, true)
|
||||
|
||||
// ========== Document Type ==========
|
||||
add(['document', 'file', 'text', 'writing', 'manuscript'], NounType.Document, 0.95, true)
|
||||
add(['report', 'summary', 'brief', 'overview', 'analysis'], NounType.Document, 0.90, true)
|
||||
add(['article', 'essay', 'paper', 'publication', 'journal'], NounType.Document, 0.90, true)
|
||||
add(['book', 'ebook', 'novel', 'chapter', 'volume'], NounType.Document, 0.90, true)
|
||||
add(['manual', 'guide', 'handbook', 'reference', 'documentation'], NounType.Document, 0.90, true)
|
||||
add(['tutorial', 'walkthrough', 'instructions'], NounType.Document, 0.85, true)
|
||||
add(['specification', 'spec', 'standard', 'protocol'], NounType.Document, 0.85, true)
|
||||
add(['proposal', 'pitch', 'presentation', 'slide', 'deck'], NounType.Document, 0.85, true)
|
||||
add(['contract', 'agreement', 'license', 'terms', 'policy'], NounType.Document, 0.90, true)
|
||||
add(['invoice', 'receipt', 'statement', 'bill', 'voucher'], NounType.Document, 0.85, true)
|
||||
add(['form', 'application', 'survey', 'questionnaire'], NounType.Document, 0.85, true)
|
||||
add(['transcript', 'minutes', 'record', 'log', 'entry'], NounType.Document, 0.85, true)
|
||||
add(['note', 'memo', 'message', 'email', 'letter'], NounType.Document, 0.85, true)
|
||||
add(['whitepaper', 'thesis', 'dissertation', 'abstract'], NounType.Document, 0.90, true)
|
||||
add(['readme', 'changelog', 'wiki'], NounType.Document, 0.85, true)
|
||||
add(['cv', 'resume', 'portfolio', 'profile'], NounType.Document, 0.85, true)
|
||||
|
||||
// Document synonyms
|
||||
add(['doc', 'docs', 'howto'], NounType.Document, 0.80, false)
|
||||
|
||||
// ========== Media Type ==========
|
||||
add(['media', 'multimedia', 'content'], NounType.Media, 0.90, true)
|
||||
add(['image', 'photo', 'picture', 'photograph', 'illustration'], NounType.Media, 0.90, true)
|
||||
add(['graphic', 'icon', 'logo', 'banner', 'thumbnail'], NounType.Media, 0.85, true)
|
||||
add(['video', 'movie', 'film', 'clip', 'recording'], NounType.Media, 0.90, true)
|
||||
add(['animation', 'gif', 'stream', 'broadcast'], NounType.Media, 0.85, true)
|
||||
add(['audio', 'sound', 'music', 'song', 'track', 'album'], NounType.Media, 0.90, true)
|
||||
add(['podcast', 'episode', 'audiobook'], NounType.Media, 0.85, true)
|
||||
add(['screenshot', 'asset', 'resource', 'attachment'], NounType.Media, 0.80, true)
|
||||
|
||||
// ========== Concept Type ==========
|
||||
add(['concept', 'idea', 'notion', 'theory', 'principle'], NounType.Concept, 0.90, true)
|
||||
add(['philosophy', 'ideology', 'belief', 'doctrine'], NounType.Concept, 0.85, true)
|
||||
add(['topic', 'subject', 'theme', 'matter', 'issue'], NounType.Concept, 0.85, true)
|
||||
add(['category', 'classification', 'taxonomy', 'domain'], NounType.Concept, 0.80, true)
|
||||
add(['field', 'discipline', 'specialty'], NounType.Concept, 0.85, true)
|
||||
add(['technology', 'tech', 'innovation', 'invention'], NounType.Concept, 0.90, true)
|
||||
add(['science', 'scientific', 'research'], NounType.Concept, 0.90, true)
|
||||
|
||||
// Scientific domains
|
||||
add(['mathematics', 'math', 'statistics', 'algebra', 'calculus'], NounType.Concept, 0.85, true)
|
||||
add(['physics', 'quantum', 'mechanics', 'thermodynamics'], NounType.Concept, 0.85, true)
|
||||
add(['chemistry', 'biology', 'genetics', 'neuroscience'], NounType.Concept, 0.85, true)
|
||||
add(['engineering', 'architecture', 'design'], NounType.Concept, 0.85, true)
|
||||
|
||||
// Computer science
|
||||
add(['computer science', 'programming', 'algorithm'], NounType.Concept, 0.90, true)
|
||||
add(['artificial intelligence', 'machine learning', 'deep learning'], NounType.Concept, 0.95, true)
|
||||
add(['ai', 'ml'], NounType.Concept, 0.90, true)
|
||||
add(['data science', 'analytics', 'big data'], NounType.Concept, 0.90, true)
|
||||
add(['natural language processing', 'computer vision'], NounType.Concept, 0.90, true)
|
||||
|
||||
// Humanities
|
||||
add(['history', 'literature', 'poetry', 'fiction'], NounType.Concept, 0.85, true)
|
||||
add(['art', 'music', 'sports'], NounType.Concept, 0.85, true)
|
||||
add(['politics', 'economics', 'psychology', 'sociology'], NounType.Concept, 0.85, true)
|
||||
add(['religion', 'spiritual', 'philosophy'], NounType.Concept, 0.85, true)
|
||||
|
||||
// ========== Event Type ==========
|
||||
add(['event', 'occasion', 'happening', 'occurrence'], NounType.Event, 0.90, true)
|
||||
add(['meeting', 'conference', 'summit', 'convention'], NounType.Event, 0.90, true)
|
||||
add(['seminar', 'symposium', 'forum', 'workshop'], NounType.Event, 0.90, true)
|
||||
add(['training', 'bootcamp', 'course', 'webinar'], NounType.Event, 0.85, true)
|
||||
add(['presentation', 'talk', 'lecture', 'session', 'class'], NounType.Event, 0.85, true)
|
||||
add(['party', 'celebration', 'gathering', 'ceremony'], NounType.Event, 0.85, true)
|
||||
add(['festival', 'carnival', 'fair', 'exhibition'], NounType.Event, 0.85, true)
|
||||
add(['concert', 'performance', 'show'], NounType.Event, 0.85, true)
|
||||
add(['game', 'match', 'tournament', 'championship', 'race'], NounType.Event, 0.85, true)
|
||||
add(['launch', 'release', 'premiere', 'debut', 'announcement'], NounType.Event, 0.85, true)
|
||||
|
||||
// ========== Product Type ==========
|
||||
add(['product', 'item', 'goods', 'merchandise', 'commodity'], NounType.Product, 0.90, true)
|
||||
add(['offering', 'solution', 'package', 'bundle'], NounType.Product, 0.85, true)
|
||||
add(['software', 'app', 'application', 'program', 'tool'], NounType.Product, 0.90, true)
|
||||
add(['platform', 'system', 'framework', 'library'], NounType.Product, 0.85, true)
|
||||
add(['device', 'gadget', 'machine', 'equipment'], NounType.Product, 0.85, true)
|
||||
add(['hardware', 'component', 'part', 'accessory'], NounType.Product, 0.85, true)
|
||||
add(['vehicle', 'car', 'automobile', 'truck', 'bike'], NounType.Product, 0.85, true)
|
||||
add(['phone', 'smartphone', 'mobile', 'tablet'], NounType.Product, 0.90, true)
|
||||
add(['computer', 'laptop', 'desktop', 'pc', 'mac'], NounType.Product, 0.90, true)
|
||||
add(['watch', 'wearable', 'tracker', 'monitor'], NounType.Product, 0.85, true)
|
||||
add(['camera', 'lens', 'sensor', 'scanner'], NounType.Product, 0.85, true)
|
||||
|
||||
// ========== Service Type ==========
|
||||
add(['service', 'support', 'assistance'], NounType.Service, 0.90, true)
|
||||
add(['consulting', 'advisory', 'guidance'], NounType.Service, 0.85, true)
|
||||
add(['maintenance', 'repair', 'installation', 'setup'], NounType.Service, 0.85, true)
|
||||
add(['hosting', 'cloud', 'saas', 'paas', 'iaas'], NounType.Service, 0.85, true)
|
||||
add(['delivery', 'shipping', 'logistics', 'transport'], NounType.Service, 0.85, true)
|
||||
add(['subscription', 'membership', 'plan'], NounType.Service, 0.85, true)
|
||||
add(['training', 'education', 'coaching', 'mentoring'], NounType.Service, 0.85, true)
|
||||
add(['healthcare', 'medical', 'dental', 'therapy'], NounType.Service, 0.85, true)
|
||||
add(['legal', 'accounting', 'financial', 'insurance'], NounType.Service, 0.85, true)
|
||||
add(['marketing', 'advertising', 'promotion'], NounType.Service, 0.85, true)
|
||||
|
||||
// ========== User Type ==========
|
||||
add(['user', 'account', 'profile', 'identity'], NounType.Person, 0.90, true)
|
||||
add(['username', 'login', 'credential'], NounType.Person, 0.85, true)
|
||||
add(['subscriber', 'follower', 'fan', 'supporter'], NounType.Person, 0.85, true)
|
||||
add(['member', 'participant', 'contributor', 'author'], NounType.Person, 0.85, true)
|
||||
add(['viewer', 'reader', 'listener', 'watcher'], NounType.Person, 0.80, true)
|
||||
add(['player', 'gamer', 'competitor'], NounType.Person, 0.80, true)
|
||||
add(['guest', 'visitor', 'attendee'], NounType.Person, 0.80, true)
|
||||
|
||||
// ========== Task & Project Types ==========
|
||||
add(['task', 'todo', 'action', 'activity', 'job'], NounType.Task, 0.85, true)
|
||||
add(['assignment', 'duty', 'work'], NounType.Task, 0.85, true)
|
||||
add(['ticket', 'issue', 'bug', 'defect', 'problem'], NounType.Task, 0.85, true)
|
||||
add(['feature', 'enhancement', 'improvement', 'request'], NounType.Task, 0.85, true)
|
||||
add(['project', 'program', 'initiative'], NounType.Project, 0.90, true)
|
||||
add(['campaign', 'drive', 'venture'], NounType.Project, 0.85, true)
|
||||
add(['plan', 'strategy', 'roadmap'], NounType.Project, 0.85, true)
|
||||
add(['milestone', 'deliverable', 'objective', 'goal'], NounType.Project, 0.85, true)
|
||||
add(['sprint', 'iteration', 'cycle', 'phase'], NounType.Project, 0.80, true)
|
||||
|
||||
// ========== Other Types ==========
|
||||
add(['process', 'procedure', 'method', 'approach'], NounType.Process, 0.80, true)
|
||||
add(['workflow', 'pipeline', 'sequence'], NounType.Process, 0.80, true)
|
||||
add(['algorithm', 'logic', 'routine', 'operation'], NounType.Process, 0.75, true)
|
||||
|
||||
add(['collection', 'set', 'group', 'batch'], NounType.Collection, 0.80, true)
|
||||
add(['list', 'array', 'series'], NounType.Collection, 0.80, true)
|
||||
add(['dataset', 'data', 'database'], NounType.Collection, 0.85, true)
|
||||
add(['repository', 'archive', 'library'], NounType.Collection, 0.80, true)
|
||||
|
||||
add(['state', 'status', 'condition'], NounType.State, 0.75, true)
|
||||
add(['active', 'inactive', 'pending', 'completed'], NounType.State, 0.70, true)
|
||||
|
||||
add(['role', 'position', 'title'], NounType.Role, 0.80, true)
|
||||
add(['permission', 'access', 'privilege'], NounType.Role, 0.75, true)
|
||||
|
||||
add(['hypothesis', 'theory', 'conjecture'], NounType.Hypothesis, 0.85, true)
|
||||
add(['experiment', 'study', 'trial', 'test'], NounType.Experiment, 0.85, true)
|
||||
add(['regulation', 'rule', 'law', 'statute'], NounType.Regulation, 0.85, true)
|
||||
add(['interface', 'api', 'endpoint'], NounType.Interface, 0.85, true)
|
||||
add(['resource', 'asset', 'capacity'], NounType.Resource, 0.80, true)
|
||||
|
||||
// ==================== VERB TYPES ====================
|
||||
// Now add all 127 VerbTypes with keywords and synonyms
|
||||
|
||||
console.log('\n Adding verb keywords...')
|
||||
|
||||
// ========== Core Relationship Types ==========
|
||||
addVerb(['related to', 'related', 'connected to', 'associated with', 'linked to'], VerbType.RelatedTo, 0.90, true)
|
||||
addVerb(['connection', 'association', 'link', 'relationship'], VerbType.RelatedTo, 0.85, false)
|
||||
|
||||
addVerb(['contains', 'includes', 'has', 'comprises', 'encompasses'], VerbType.Contains, 0.95, true)
|
||||
addVerb(['holding', 'containing', 'including'], VerbType.Contains, 0.85, false)
|
||||
|
||||
addVerb(['part of', 'belongs to', 'component of', 'element of', 'member of'], VerbType.PartOf, 0.95, true)
|
||||
addVerb(['within', 'inside', 'subset of'], VerbType.PartOf, 0.85, false)
|
||||
|
||||
addVerb(['located at', 'positioned at', 'situated at', 'found at', 'based at'], VerbType.LocatedAt, 0.95, true)
|
||||
addVerb(['location', 'position', 'whereabouts'], VerbType.LocatedAt, 0.85, false)
|
||||
|
||||
addVerb(['references', 'cites', 'refers to', 'mentions', 'points to'], VerbType.References, 0.95, true)
|
||||
addVerb(['citation', 'reference', 'pointer'], VerbType.References, 0.85, false)
|
||||
|
||||
// ========== Temporal/Causal Types ==========
|
||||
addVerb(['precedes', 'comes before', 'happens before', 'leads to', 'prior to'], VerbType.Precedes, 0.90, true)
|
||||
addVerb(['preceding', 'earlier than', 'before'], VerbType.Precedes, 0.85, false)
|
||||
|
||||
addVerb(['succeeds', 'comes after', 'follows', 'happens after', 'subsequent to'], VerbType.Precedes, 0.90, true)
|
||||
addVerb(['succeeding', 'later than', 'after'], VerbType.Precedes, 0.85, false)
|
||||
|
||||
addVerb(['causes', 'results in', 'leads to', 'brings about', 'triggers'], VerbType.Causes, 0.95, true)
|
||||
addVerb(['influences', 'affects', 'impacts', 'produces'], VerbType.Causes, 0.90, true)
|
||||
addVerb(['causation', 'consequence', 'effect'], VerbType.Causes, 0.85, false)
|
||||
|
||||
addVerb(['depends on', 'relies on', 'contingent on', 'conditional on'], VerbType.DependsOn, 0.95, true)
|
||||
addVerb(['dependency', 'reliance', 'dependence'], VerbType.DependsOn, 0.85, false)
|
||||
|
||||
addVerb(['requires', 'needs', 'necessitates', 'demands', 'calls for'], VerbType.Requires, 0.95, true)
|
||||
addVerb(['requirement', 'necessity', 'prerequisite'], VerbType.Requires, 0.85, false)
|
||||
|
||||
// ========== Creation/Transformation Types ==========
|
||||
addVerb(['creates', 'makes', 'builds', 'produces', 'generates'], VerbType.Creates, 0.95, true)
|
||||
addVerb(['constructs', 'develops', 'crafts', 'forms'], VerbType.Creates, 0.90, true)
|
||||
addVerb(['creation', 'production', 'generation'], VerbType.Creates, 0.85, false)
|
||||
|
||||
addVerb(['transforms', 'converts', 'changes', 'morphs', 'alters'], VerbType.Transforms, 0.95, true)
|
||||
addVerb(['transformation', 'conversion', 'metamorphosis'], VerbType.Transforms, 0.85, false)
|
||||
|
||||
addVerb(['becomes', 'turns into', 'evolves into', 'transitions to'], VerbType.Becomes, 0.95, true)
|
||||
addVerb(['becoming', 'transition', 'evolution'], VerbType.Becomes, 0.85, false)
|
||||
|
||||
addVerb(['modifies', 'updates', 'changes', 'edits', 'adjusts'], VerbType.Modifies, 0.95, true)
|
||||
addVerb(['alters', 'amends', 'revises', 'tweaks'], VerbType.Modifies, 0.90, true)
|
||||
addVerb(['modification', 'update', 'change'], VerbType.Modifies, 0.85, false)
|
||||
|
||||
addVerb(['consumes', 'uses up', 'depletes', 'exhausts', 'drains'], VerbType.Consumes, 0.95, true)
|
||||
addVerb(['consumption', 'usage', 'depletion'], VerbType.Consumes, 0.85, false)
|
||||
|
||||
// ========== Ownership/Attribution Types ==========
|
||||
addVerb(['owns', 'possesses', 'holds', 'controls', 'has'], VerbType.Owns, 0.95, true)
|
||||
addVerb(['ownership', 'possession', 'control'], VerbType.Owns, 0.85, false)
|
||||
|
||||
addVerb(['attributed to', 'credited to', 'ascribed to', 'assigned to'], VerbType.AttributedTo, 0.95, true)
|
||||
addVerb(['attribution', 'credit', 'acknowledgment'], VerbType.AttributedTo, 0.85, false)
|
||||
|
||||
addVerb(['created by', 'made by', 'built by', 'authored by', 'developed by'], VerbType.Creates, 0.95, true)
|
||||
addVerb(['creator', 'author', 'maker'], VerbType.Creates, 0.85, false)
|
||||
|
||||
addVerb(['belongs to', 'owned by', 'property of', 'part of'], VerbType.Owns, 0.95, true)
|
||||
addVerb(['belonging', 'membership'], VerbType.Owns, 0.85, false)
|
||||
|
||||
// ========== Social/Organizational Types ==========
|
||||
addVerb(['member of', 'belongs to', 'affiliated with', 'part of'], VerbType.MemberOf, 0.95, true)
|
||||
addVerb(['works for', 'employed by', 'serves'], VerbType.MemberOf, 0.90, true)
|
||||
addVerb(['membership', 'affiliation'], VerbType.MemberOf, 0.85, false)
|
||||
|
||||
addVerb(['works with', 'collaborates with', 'partners with', 'cooperates with'], VerbType.WorksWith, 0.95, true)
|
||||
addVerb(['teams with', 'joins forces with'], VerbType.WorksWith, 0.90, true)
|
||||
addVerb(['collaboration', 'partnership', 'cooperation'], VerbType.WorksWith, 0.85, false)
|
||||
|
||||
addVerb(['friend of', 'friends with', 'befriends', 'friendly with'], VerbType.FriendOf, 0.95, true)
|
||||
addVerb(['friendship', 'companionship'], VerbType.FriendOf, 0.85, false)
|
||||
|
||||
addVerb(['follows', 'tracks', 'monitors', 'subscribes to', 'watches'], VerbType.Follows, 0.95, true)
|
||||
addVerb(['following', 'follower', 'subscriber'], VerbType.Follows, 0.85, false)
|
||||
|
||||
addVerb(['likes', 'enjoys', 'prefers', 'favors', 'appreciates'], VerbType.Likes, 0.95, true)
|
||||
addVerb(['fond of', 'partial to'], VerbType.Likes, 0.90, true)
|
||||
|
||||
addVerb(['reports to', 'answers to', 'subordinate to', 'under'], VerbType.ReportsTo, 0.95, true)
|
||||
addVerb(['reporting', 'subordination'], VerbType.ReportsTo, 0.85, false)
|
||||
|
||||
addVerb(['supervises', 'manages', 'oversees', 'directs', 'leads'], VerbType.ReportsTo, 0.95, true)
|
||||
addVerb(['supervision', 'management', 'oversight'], VerbType.ReportsTo, 0.85, false)
|
||||
|
||||
addVerb(['mentors', 'coaches', 'guides', 'advises', 'teaches'], VerbType.Mentors, 0.95, true)
|
||||
addVerb(['mentorship', 'coaching', 'guidance'], VerbType.Mentors, 0.85, false)
|
||||
|
||||
addVerb(['communicates with', 'talks to', 'corresponds with', 'exchanges with'], VerbType.Communicates, 0.95, true)
|
||||
addVerb(['speaks with', 'chats with', 'discusses with'], VerbType.Communicates, 0.90, true)
|
||||
addVerb(['communication', 'correspondence', 'dialogue'], VerbType.Communicates, 0.85, false)
|
||||
|
||||
// ========== Descriptive/Functional Types ==========
|
||||
addVerb(['describes', 'explains', 'details', 'characterizes', 'portrays'], VerbType.Describes, 0.95, true)
|
||||
addVerb(['depicts', 'illustrates', 'outlines'], VerbType.Describes, 0.90, true)
|
||||
addVerb(['description', 'explanation', 'account'], VerbType.Describes, 0.85, false)
|
||||
|
||||
addVerb(['defines', 'specifies', 'determines', 'establishes', 'sets'], VerbType.Defines, 0.95, true)
|
||||
addVerb(['definition', 'specification', 'determination'], VerbType.Defines, 0.85, false)
|
||||
|
||||
addVerb(['categorizes', 'classifies', 'groups', 'sorts', 'organizes'], VerbType.Categorizes, 0.95, true)
|
||||
addVerb(['categorization', 'classification', 'taxonomy'], VerbType.Categorizes, 0.85, false)
|
||||
|
||||
addVerb(['measures', 'quantifies', 'gauges', 'assesses', 'evaluates'], VerbType.Measures, 0.95, true)
|
||||
addVerb(['measurement', 'quantification', 'assessment'], VerbType.Measures, 0.85, false)
|
||||
|
||||
addVerb(['evaluates', 'assesses', 'judges', 'appraises', 'reviews'], VerbType.Evaluates, 0.95, true)
|
||||
addVerb(['rates', 'scores', 'critiques'], VerbType.Evaluates, 0.90, true)
|
||||
addVerb(['evaluation', 'assessment', 'appraisal'], VerbType.Evaluates, 0.85, false)
|
||||
|
||||
addVerb(['uses', 'utilizes', 'employs', 'applies', 'leverages'], VerbType.Uses, 0.95, true)
|
||||
addVerb(['usage', 'utilization', 'application'], VerbType.Uses, 0.85, false)
|
||||
|
||||
addVerb(['implements', 'executes', 'realizes', 'enacts', 'carries out'], VerbType.Implements, 0.95, true)
|
||||
addVerb(['implementation', 'execution', 'realization'], VerbType.Implements, 0.85, false)
|
||||
|
||||
addVerb(['extends', 'expands', 'broadens', 'enlarges', 'builds on'], VerbType.Extends, 0.95, true)
|
||||
addVerb(['enhances', 'augments', 'amplifies'], VerbType.Extends, 0.90, true)
|
||||
addVerb(['extension', 'expansion', 'enhancement'], VerbType.Extends, 0.85, false)
|
||||
|
||||
// ========== Enhanced Relationship Types ==========
|
||||
addVerb(['inherits', 'derives from', 'inherits from', 'descended from'], VerbType.Inherits, 0.95, true)
|
||||
addVerb(['inheritance', 'derivation', 'legacy'], VerbType.Inherits, 0.85, false)
|
||||
|
||||
addVerb(['conflicts with', 'contradicts', 'opposes', 'clashes with'], VerbType.Conflicts, 0.95, true)
|
||||
addVerb(['disagrees with', 'incompatible with'], VerbType.Conflicts, 0.90, true)
|
||||
addVerb(['conflict', 'contradiction', 'opposition'], VerbType.Conflicts, 0.85, false)
|
||||
|
||||
addVerb(['synchronizes with', 'coordinates with', 'syncs with', 'aligns with'], VerbType.Synchronizes, 0.95, true)
|
||||
addVerb(['synchronization', 'coordination', 'alignment'], VerbType.Synchronizes, 0.85, false)
|
||||
|
||||
addVerb(['competes with', 'rivals', 'contests', 'vies with'], VerbType.Competes, 0.95, true)
|
||||
addVerb(['competition', 'rivalry', 'contest'], VerbType.Competes, 0.85, false)
|
||||
|
||||
return keywords
|
||||
}
|
||||
|
||||
/**
|
||||
* Main build function
|
||||
*/
|
||||
async function buildKeywordEmbeddings() {
|
||||
console.log('🔨 Building Keyword Embeddings for Semantic Type Inference\n')
|
||||
console.log('='.repeat(70))
|
||||
|
||||
// Step 1: Build keyword list
|
||||
console.log('\n📝 Step 1: Building expanded keyword dictionary...')
|
||||
const keywords = buildExpandedKeywordList()
|
||||
|
||||
const canonical = keywords.filter(k => k.isCanonical).length
|
||||
const synonyms = keywords.filter(k => !k.isCanonical).length
|
||||
|
||||
console.log(`✅ Generated ${keywords.length} keywords (${canonical} canonical, ${synonyms} synonyms)`)
|
||||
|
||||
// Step 2: Initialize embedder
|
||||
console.log('\n🎯 Step 2: Initializing TransformerEmbedding model...')
|
||||
const embedder = new TransformerEmbedding({ verbose: true })
|
||||
await embedder.init()
|
||||
console.log('✅ Embedder initialized')
|
||||
|
||||
// Step 3: Generate embeddings
|
||||
console.log(`\n🚀 Step 3: Generating embeddings for ${keywords.length} keywords...`)
|
||||
console.log('(This may take 60-90 seconds)\n')
|
||||
|
||||
const embeddings = []
|
||||
let processed = 0
|
||||
const startTime = Date.now()
|
||||
|
||||
for (const def of keywords) {
|
||||
const embedding = await embedder.embed(def.keyword)
|
||||
|
||||
embeddings.push({
|
||||
keyword: def.keyword,
|
||||
type: def.type,
|
||||
typeCategory: def.typeCategory,
|
||||
confidence: def.confidence,
|
||||
isCanonical: def.isCanonical,
|
||||
embedding: Array.from(embedding)
|
||||
})
|
||||
|
||||
processed++
|
||||
if (processed % 50 === 0) {
|
||||
const elapsed = ((Date.now() - startTime) / 1000).toFixed(1)
|
||||
const rate = (processed / (Date.now() - startTime) * 1000).toFixed(1)
|
||||
const eta = ((keywords.length - processed) / parseFloat(rate)).toFixed(0)
|
||||
console.log(` Progress: ${processed}/${keywords.length} (${(processed/keywords.length*100).toFixed(1)}%) - ${rate}/sec - ETA: ${eta}s`)
|
||||
}
|
||||
}
|
||||
|
||||
const totalTime = ((Date.now() - startTime) / 1000).toFixed(1)
|
||||
console.log(`\n✅ All embeddings generated in ${totalTime}s`)
|
||||
|
||||
// Step 4: Generate TypeScript file
|
||||
console.log('\n📄 Step 4: Writing embeddedKeywordEmbeddings.ts...')
|
||||
|
||||
const sizeKB = (embeddings.length * 384 * 4 / 1024).toFixed(1)
|
||||
const sizeMB = (parseFloat(sizeKB) / 1024).toFixed(2)
|
||||
|
||||
// Calculate stats
|
||||
const nounKeywords = embeddings.filter(e => e.typeCategory === 'noun').length
|
||||
const verbKeywords = embeddings.filter(e => e.typeCategory === 'verb').length
|
||||
const canonicalKeywords = embeddings.filter(e => e.isCanonical).length
|
||||
const synonymKeywords = embeddings.filter(e => !e.isCanonical).length
|
||||
|
||||
const output = `/**
|
||||
* Pre-computed Keyword Embeddings for Unified Semantic Type Inference
|
||||
*
|
||||
* Generated by: scripts/buildKeywordEmbeddings.ts
|
||||
* Generated on: ${new Date().toISOString()}
|
||||
* Total keywords: ${embeddings.length} (${nounKeywords} nouns + ${verbKeywords} verbs)
|
||||
* Canonical: ${canonicalKeywords}, Synonyms: ${synonymKeywords}
|
||||
* Embedding dimension: 384
|
||||
* Total size: ${sizeMB}MB
|
||||
*
|
||||
* This file contains pre-computed semantic embeddings for ALL type inference keywords.
|
||||
* Supports unified noun + verb semantic inference via SemanticTypeInference.
|
||||
* Used for O(log n) semantic matching via HNSW index.
|
||||
*/
|
||||
|
||||
import { NounType, VerbType } from '../types/graphTypes.js'
|
||||
import { Vector } from '../coreTypes.js'
|
||||
|
||||
export interface KeywordEmbedding {
|
||||
keyword: string
|
||||
type: NounType | VerbType
|
||||
typeCategory: 'noun' | 'verb'
|
||||
confidence: number
|
||||
isCanonical: boolean
|
||||
embedding: Vector
|
||||
}
|
||||
|
||||
// Use 'any' type to avoid TypeScript union complexity issues with 1050+ literal types
|
||||
const KEYWORD_EMBEDDINGS: any = ${JSON.stringify(embeddings, null, 2)}
|
||||
|
||||
export function getKeywordEmbeddings(): KeywordEmbedding[] {
|
||||
return KEYWORD_EMBEDDINGS
|
||||
}
|
||||
|
||||
export function getKeywordCount(): number {
|
||||
return KEYWORD_EMBEDDINGS.length
|
||||
}
|
||||
|
||||
export function getNounKeywordCount(): number {
|
||||
return ${nounKeywords}
|
||||
}
|
||||
|
||||
export function getVerbKeywordCount(): number {
|
||||
return ${verbKeywords}
|
||||
}
|
||||
|
||||
export function getEmbeddingDimension(): number {
|
||||
return 384
|
||||
}
|
||||
`
|
||||
|
||||
writeFileSync('src/neural/embeddedKeywordEmbeddings.ts', output, 'utf-8')
|
||||
|
||||
console.log(`✅ Generated src/neural/embeddedKeywordEmbeddings.ts`)
|
||||
console.log(` Total keywords: ${embeddings.length} (${nounKeywords} nouns + ${verbKeywords} verbs)`)
|
||||
console.log(` Canonical: ${canonicalKeywords}, Synonyms: ${synonymKeywords}`)
|
||||
console.log(` Size: ${sizeMB}MB (${sizeKB}KB)`)
|
||||
|
||||
console.log('\n' + '='.repeat(70))
|
||||
console.log('✅ Keyword embeddings build complete!')
|
||||
console.log('='.repeat(70))
|
||||
|
||||
return embeddings.length
|
||||
}
|
||||
|
||||
// Run if called directly
|
||||
if (import.meta.url === `file://${process.argv[1]}`) {
|
||||
buildKeywordEmbeddings()
|
||||
.then(count => {
|
||||
console.log(`\n🎉 Success! Generated embeddings for ${count} keywords.`)
|
||||
process.exit(0)
|
||||
})
|
||||
.catch(error => {
|
||||
console.error('\n❌ Build failed:', error.message)
|
||||
console.error(error.stack)
|
||||
process.exit(1)
|
||||
})
|
||||
}
|
||||
|
||||
export { buildKeywordEmbeddings }
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
/**
|
||||
* Check if keyword embeddings need to be rebuilt
|
||||
* Exits with code 1 if rebuild needed, 0 if up-to-date
|
||||
*/
|
||||
|
||||
const fs = require('fs')
|
||||
const path = require('path')
|
||||
|
||||
const EMBEDDED_FILE = 'src/neural/embeddedKeywordEmbeddings.ts'
|
||||
const BUILD_SCRIPT = 'scripts/buildKeywordEmbeddings.ts'
|
||||
|
||||
const embeddedPath = path.join(process.cwd(), EMBEDDED_FILE)
|
||||
const buildScriptPath = path.join(process.cwd(), BUILD_SCRIPT)
|
||||
|
||||
// Check if embedded file exists
|
||||
if (!fs.existsSync(embeddedPath)) {
|
||||
console.log('⚠️ Keyword embeddings file not found. Build required.')
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
// Check if build script is newer than embedded file
|
||||
const embeddedStat = fs.statSync(embeddedPath)
|
||||
const buildScriptStat = fs.statSync(buildScriptPath)
|
||||
|
||||
if (buildScriptStat.mtime > embeddedStat.mtime) {
|
||||
console.log('⚠️ Build script is newer than keyword embeddings. Rebuild required.')
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
console.log('✅ Embedded keyword embeddings are up-to-date. Skipping rebuild.')
|
||||
process.exit(0)
|
||||
|
|
@ -1,175 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
/**
|
||||
* Download Model Assets
|
||||
*
|
||||
* Downloads the all-MiniLM-L6-v2 Q8 model from Hugging Face.
|
||||
* Run: node scripts/download-model.cjs
|
||||
*/
|
||||
|
||||
const fs = require('node:fs')
|
||||
const path = require('node:path')
|
||||
const https = require('node:https')
|
||||
|
||||
const MODEL_DIR = path.join(__dirname, '..', 'assets', 'models', 'all-MiniLM-L6-v2-q8')
|
||||
const BASE_URL = 'https://huggingface.co/Xenova/all-MiniLM-L6-v2/resolve/main/onnx'
|
||||
|
||||
const FILES = [
|
||||
{
|
||||
name: 'model_quantized.onnx',
|
||||
url: `${BASE_URL}/model_quantized.onnx`,
|
||||
dest: 'model.onnx',
|
||||
},
|
||||
{
|
||||
name: 'tokenizer.json',
|
||||
url: 'https://huggingface.co/Xenova/all-MiniLM-L6-v2/resolve/main/tokenizer.json',
|
||||
dest: 'tokenizer.json',
|
||||
},
|
||||
{
|
||||
name: 'config.json',
|
||||
url: 'https://huggingface.co/Xenova/all-MiniLM-L6-v2/resolve/main/config.json',
|
||||
dest: 'config.json',
|
||||
},
|
||||
{
|
||||
name: 'vocab.txt',
|
||||
url: 'https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/vocab.txt',
|
||||
dest: 'vocab.txt',
|
||||
},
|
||||
]
|
||||
|
||||
/**
|
||||
* Follow redirects and download file
|
||||
*/
|
||||
function downloadFile(url, destPath, maxRedirects = 5) {
|
||||
return new Promise((resolve, reject) => {
|
||||
if (maxRedirects === 0) {
|
||||
reject(new Error('Too many redirects'))
|
||||
return
|
||||
}
|
||||
|
||||
const doRequest = (reqUrl) => {
|
||||
const parsedUrl = new URL(reqUrl)
|
||||
const options = {
|
||||
hostname: parsedUrl.hostname,
|
||||
path: parsedUrl.pathname + parsedUrl.search,
|
||||
headers: {
|
||||
'User-Agent': 'Brainy-Model-Downloader/1.0',
|
||||
},
|
||||
}
|
||||
|
||||
https.get(options, (response) => {
|
||||
// Handle redirects
|
||||
if (response.statusCode >= 300 && response.statusCode < 400 && response.headers.location) {
|
||||
response.resume() // Consume response data to free memory
|
||||
const redirectUrl = response.headers.location.startsWith('http')
|
||||
? response.headers.location
|
||||
: new URL(response.headers.location, reqUrl).toString()
|
||||
console.log(` ↳ Redirecting to: ${redirectUrl.slice(0, 80)}...`)
|
||||
downloadFile(redirectUrl, destPath, maxRedirects - 1)
|
||||
.then(resolve)
|
||||
.catch(reject)
|
||||
return
|
||||
}
|
||||
|
||||
if (response.statusCode !== 200) {
|
||||
reject(new Error(`HTTP ${response.statusCode}`))
|
||||
return
|
||||
}
|
||||
|
||||
const fileStream = fs.createWriteStream(destPath)
|
||||
let downloadedBytes = 0
|
||||
const totalBytes = parseInt(response.headers['content-length'] || '0', 10)
|
||||
|
||||
response.on('data', (chunk) => {
|
||||
downloadedBytes += chunk.length
|
||||
if (totalBytes > 0) {
|
||||
const percent = Math.round((downloadedBytes / totalBytes) * 100)
|
||||
process.stdout.write(`\r Progress: ${percent}% (${Math.round(downloadedBytes / 1024 / 1024)}MB)`)
|
||||
}
|
||||
})
|
||||
|
||||
response.pipe(fileStream)
|
||||
|
||||
fileStream.on('finish', () => {
|
||||
fileStream.close()
|
||||
console.log(`\n ✅ Downloaded: ${path.basename(destPath)} (${Math.round(downloadedBytes / 1024 / 1024)}MB)`)
|
||||
resolve()
|
||||
})
|
||||
|
||||
fileStream.on('error', (err) => {
|
||||
fs.unlink(destPath, () => {}) // Delete partial file
|
||||
reject(err)
|
||||
})
|
||||
}).on('error', reject)
|
||||
}
|
||||
|
||||
doRequest(url)
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert vocab.txt to vocab.json
|
||||
*/
|
||||
function convertVocabToJson(vocabTxtPath, vocabJsonPath) {
|
||||
console.log('📝 Converting vocab.txt to vocab.json...')
|
||||
const content = fs.readFileSync(vocabTxtPath, 'utf-8')
|
||||
const lines = content.split('\n').filter(line => line.trim())
|
||||
|
||||
const vocab = {}
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
vocab[lines[i]] = i
|
||||
}
|
||||
|
||||
fs.writeFileSync(vocabJsonPath, JSON.stringify(vocab))
|
||||
console.log(` ✅ Created vocab.json with ${Object.keys(vocab).length} tokens`)
|
||||
|
||||
// Remove vocab.txt since we have vocab.json
|
||||
fs.unlinkSync(vocabTxtPath)
|
||||
}
|
||||
|
||||
async function main() {
|
||||
console.log('🔽 Downloading all-MiniLM-L6-v2 Q8 model assets...\n')
|
||||
|
||||
// Create model directory
|
||||
fs.mkdirSync(MODEL_DIR, { recursive: true })
|
||||
console.log(`📁 Model directory: ${MODEL_DIR}\n`)
|
||||
|
||||
// Download each file
|
||||
for (const file of FILES) {
|
||||
const destPath = path.join(MODEL_DIR, file.dest)
|
||||
|
||||
// Check if already exists
|
||||
if (fs.existsSync(destPath)) {
|
||||
const stats = fs.statSync(destPath)
|
||||
if (stats.size > 0) {
|
||||
console.log(`⏭️ Skipping ${file.name} (already exists)`)
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`📥 Downloading ${file.name}...`)
|
||||
try {
|
||||
await downloadFile(file.url, destPath)
|
||||
} catch (error) {
|
||||
console.error(` ❌ Failed to download ${file.name}: ${error.message}`)
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// Convert vocab.txt to vocab.json
|
||||
const vocabTxtPath = path.join(MODEL_DIR, 'vocab.txt')
|
||||
const vocabJsonPath = path.join(MODEL_DIR, 'vocab.json')
|
||||
if (fs.existsSync(vocabTxtPath) && !fs.existsSync(vocabJsonPath)) {
|
||||
convertVocabToJson(vocabTxtPath, vocabJsonPath)
|
||||
}
|
||||
|
||||
console.log('\n✅ All model assets downloaded successfully!')
|
||||
console.log('\nModel files:')
|
||||
const files = fs.readdirSync(MODEL_DIR)
|
||||
for (const file of files) {
|
||||
const stats = fs.statSync(path.join(MODEL_DIR, file))
|
||||
const sizeMB = (stats.size / 1024 / 1024).toFixed(2)
|
||||
console.log(` - ${file}: ${sizeMB}MB`)
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(console.error)
|
||||
|
|
@ -1,264 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
/**
|
||||
* Download and bundle models for offline usage
|
||||
*/
|
||||
|
||||
const fs = require('fs').promises
|
||||
const path = require('path')
|
||||
|
||||
const MODEL_NAME = 'Xenova/all-MiniLM-L6-v2'
|
||||
const OUTPUT_DIR = './models'
|
||||
|
||||
// Always download Q8 model only
|
||||
const downloadType = 'q8'
|
||||
|
||||
async function downloadModels() {
|
||||
// Use dynamic import for ES modules in CommonJS
|
||||
const { pipeline, env } = await import('@huggingface/transformers')
|
||||
|
||||
// Configure transformers.js to use local cache
|
||||
env.cacheDir = './models-cache'
|
||||
env.allowRemoteModels = true
|
||||
|
||||
try {
|
||||
console.log('🧠 Brainy Model Downloader v2.8.0')
|
||||
console.log('===================================')
|
||||
console.log(` Model: ${MODEL_NAME}`)
|
||||
console.log(` Type: Q8 (optimized, 99% accuracy)`)
|
||||
console.log(` Cache: ${env.cacheDir}`)
|
||||
console.log('')
|
||||
|
||||
// Create output directory
|
||||
await fs.mkdir(OUTPUT_DIR, { recursive: true })
|
||||
|
||||
// Download Q8 model only
|
||||
console.log('📥 Downloading Q8 model (quantized, 33MB, 99% accuracy)...')
|
||||
await downloadModelVariant('q8')
|
||||
|
||||
// Copy ALL model files from cache to our models directory
|
||||
console.log('📋 Copying model files to bundle directory...')
|
||||
|
||||
const cacheDir = path.resolve(env.cacheDir)
|
||||
const outputDir = path.resolve(OUTPUT_DIR)
|
||||
|
||||
console.log(` From: ${cacheDir}`)
|
||||
console.log(` To: ${outputDir}`)
|
||||
|
||||
// Copy the entire cache directory structure to ensure we get ALL files
|
||||
// including tokenizer.json, config.json, and all ONNX model files
|
||||
const modelCacheDir = path.join(cacheDir, 'Xenova', 'all-MiniLM-L6-v2')
|
||||
|
||||
if (await dirExists(modelCacheDir)) {
|
||||
const targetModelDir = path.join(outputDir, 'Xenova', 'all-MiniLM-L6-v2')
|
||||
console.log(` Copying complete model: Xenova/all-MiniLM-L6-v2`)
|
||||
await copyDirectory(modelCacheDir, targetModelDir)
|
||||
} else {
|
||||
throw new Error(`Model cache directory not found: ${modelCacheDir}`)
|
||||
}
|
||||
|
||||
console.log('✅ Model bundling complete!')
|
||||
console.log(` Total size: ${await calculateDirectorySize(outputDir)} MB`)
|
||||
console.log(` Location: ${outputDir}`)
|
||||
|
||||
// Create a marker file with downloaded model info
|
||||
const markerData = {
|
||||
model: MODEL_NAME,
|
||||
bundledAt: new Date().toISOString(),
|
||||
version: '2.8.0',
|
||||
downloadType: downloadType,
|
||||
models: {}
|
||||
}
|
||||
|
||||
// Check which models were downloaded
|
||||
const fp32Path = path.join(outputDir, 'Xenova/all-MiniLM-L6-v2/onnx/model.onnx')
|
||||
const q8Path = path.join(outputDir, 'Xenova/all-MiniLM-L6-v2/onnx/model_quantized.onnx')
|
||||
|
||||
if (await fileExists(fp32Path)) {
|
||||
const stats = await fs.stat(fp32Path)
|
||||
markerData.models.fp32 = {
|
||||
file: 'onnx/model.onnx',
|
||||
size: stats.size,
|
||||
sizeFormatted: `${Math.round(stats.size / (1024 * 1024))}MB`
|
||||
}
|
||||
}
|
||||
|
||||
if (await fileExists(q8Path)) {
|
||||
const stats = await fs.stat(q8Path)
|
||||
markerData.models.q8 = {
|
||||
file: 'onnx/model_quantized.onnx',
|
||||
size: stats.size,
|
||||
sizeFormatted: `${Math.round(stats.size / (1024 * 1024))}MB`
|
||||
}
|
||||
}
|
||||
|
||||
await fs.writeFile(
|
||||
path.join(outputDir, '.brainy-models-bundled'),
|
||||
JSON.stringify(markerData, null, 2)
|
||||
)
|
||||
|
||||
console.log('')
|
||||
console.log('✅ Download complete! Available models:')
|
||||
if (markerData.models.fp32) {
|
||||
console.log(` • FP32: ${markerData.models.fp32.sizeFormatted} (full precision)`)
|
||||
}
|
||||
if (markerData.models.q8) {
|
||||
console.log(` • Q8: ${markerData.models.q8.sizeFormatted} (quantized, 75% smaller)`)
|
||||
}
|
||||
console.log('')
|
||||
console.log('Air-gap deployment ready! 🚀')
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Error downloading models:', error)
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// Download a specific model variant
|
||||
async function downloadModelVariant(dtype) {
|
||||
const { pipeline } = await import('@huggingface/transformers')
|
||||
|
||||
try {
|
||||
// Load the model to force download
|
||||
const extractor = await pipeline('feature-extraction', MODEL_NAME, {
|
||||
dtype: dtype,
|
||||
cache_dir: './models-cache'
|
||||
})
|
||||
|
||||
// Test the model
|
||||
const testResult = await extractor(['Hello world!'], {
|
||||
pooling: 'mean',
|
||||
normalize: true
|
||||
})
|
||||
|
||||
console.log(` ✅ ${dtype.toUpperCase()} model downloaded and tested (${testResult.data.length} dimensions)`)
|
||||
|
||||
// Dispose to free memory
|
||||
if (extractor.dispose) {
|
||||
await extractor.dispose()
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error(` ❌ Failed to download ${dtype} model:`, error)
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
async function findModelDirectories(baseDir, modelName) {
|
||||
const dirs = []
|
||||
|
||||
try {
|
||||
// Convert model name to expected directory structure
|
||||
const modelPath = modelName.replace('/', '--')
|
||||
|
||||
async function searchDirectory(currentDir) {
|
||||
try {
|
||||
const entries = await fs.readdir(currentDir, { withFileTypes: true })
|
||||
|
||||
for (const entry of entries) {
|
||||
if (entry.isDirectory()) {
|
||||
const fullPath = path.join(currentDir, entry.name)
|
||||
|
||||
// Check if this directory contains model files
|
||||
if (entry.name.includes(modelPath) || entry.name === 'onnx') {
|
||||
const hasModelFiles = await containsModelFiles(fullPath)
|
||||
if (hasModelFiles) {
|
||||
dirs.push(fullPath)
|
||||
}
|
||||
}
|
||||
|
||||
// Recursively search subdirectories
|
||||
await searchDirectory(fullPath)
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
// Ignore access errors
|
||||
}
|
||||
}
|
||||
|
||||
await searchDirectory(baseDir)
|
||||
} catch (error) {
|
||||
console.warn('Warning: Error searching for model directories:', error)
|
||||
}
|
||||
|
||||
return dirs
|
||||
}
|
||||
|
||||
async function containsModelFiles(dir) {
|
||||
try {
|
||||
const files = await fs.readdir(dir)
|
||||
return files.some(file =>
|
||||
file.endsWith('.onnx') ||
|
||||
file.endsWith('.json') ||
|
||||
file === 'config.json' ||
|
||||
file === 'tokenizer.json'
|
||||
)
|
||||
} catch (error) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
async function dirExists(dir) {
|
||||
try {
|
||||
const stats = await fs.stat(dir)
|
||||
return stats.isDirectory()
|
||||
} catch (error) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
async function fileExists(file) {
|
||||
try {
|
||||
const stats = await fs.stat(file)
|
||||
return stats.isFile()
|
||||
} catch (error) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
async function copyDirectory(src, dest) {
|
||||
await fs.mkdir(dest, { recursive: true })
|
||||
const entries = await fs.readdir(src, { withFileTypes: true })
|
||||
|
||||
for (const entry of entries) {
|
||||
const srcPath = path.join(src, entry.name)
|
||||
const destPath = path.join(dest, entry.name)
|
||||
|
||||
if (entry.isDirectory()) {
|
||||
await copyDirectory(srcPath, destPath)
|
||||
} else {
|
||||
await fs.copyFile(srcPath, destPath)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function calculateDirectorySize(dir) {
|
||||
let size = 0
|
||||
|
||||
async function calculateSize(currentDir) {
|
||||
try {
|
||||
const entries = await fs.readdir(currentDir, { withFileTypes: true })
|
||||
|
||||
for (const entry of entries) {
|
||||
const fullPath = path.join(currentDir, entry.name)
|
||||
|
||||
if (entry.isDirectory()) {
|
||||
await calculateSize(fullPath)
|
||||
} else {
|
||||
const stats = await fs.stat(fullPath)
|
||||
size += stats.size
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
// Ignore access errors
|
||||
}
|
||||
}
|
||||
|
||||
await calculateSize(dir)
|
||||
return Math.round(size / (1024 * 1024))
|
||||
}
|
||||
|
||||
// Run the download
|
||||
downloadModels().catch(error => {
|
||||
console.error('Fatal error:', error)
|
||||
process.exit(1)
|
||||
})
|
||||
|
|
@ -1,108 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
/**
|
||||
* Ensures transformer models are available for production
|
||||
* This script handles model availability in multiple ways:
|
||||
* 1. Check if models exist locally
|
||||
* 2. Download from CDN if needed
|
||||
* 3. Verify model integrity
|
||||
*/
|
||||
|
||||
import { existsSync } from 'fs'
|
||||
import { readFile, mkdir, writeFile } from 'fs/promises'
|
||||
import { join, dirname } from 'path'
|
||||
import { createHash } from 'crypto'
|
||||
import { fileURLToPath } from 'url'
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url))
|
||||
const PROJECT_ROOT = join(__dirname, '..')
|
||||
|
||||
// Model configuration
|
||||
const MODEL_CONFIG = {
|
||||
name: 'Xenova/all-MiniLM-L6-v2',
|
||||
files: {
|
||||
'onnx/model.onnx': {
|
||||
size: 90555481, // 86.3 MB
|
||||
sha256: 'expected_hash_here' // We'd compute this from actual model
|
||||
},
|
||||
'tokenizer.json': {
|
||||
size: 711661,
|
||||
sha256: 'expected_hash_here'
|
||||
},
|
||||
'tokenizer_config.json': {
|
||||
size: 366,
|
||||
sha256: 'expected_hash_here'
|
||||
},
|
||||
'config.json': {
|
||||
size: 650,
|
||||
sha256: 'expected_hash_here'
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// CDN URLs for model files (would be your own CDN in production)
|
||||
const CDN_BASE = 'https://cdn.soulcraft.com/models'
|
||||
|
||||
async function ensureModels() {
|
||||
const modelsDir = join(PROJECT_ROOT, 'models', 'Xenova', 'all-MiniLM-L6-v2')
|
||||
|
||||
console.log('🔍 Checking for transformer models...')
|
||||
|
||||
// Check if all model files exist
|
||||
let missingFiles = []
|
||||
for (const [filePath, info] of Object.entries(MODEL_CONFIG.files)) {
|
||||
const fullPath = join(modelsDir, filePath)
|
||||
if (!existsSync(fullPath)) {
|
||||
missingFiles.push(filePath)
|
||||
}
|
||||
}
|
||||
|
||||
if (missingFiles.length === 0) {
|
||||
console.log('✅ All model files present')
|
||||
|
||||
// Optionally verify integrity
|
||||
if (process.env.VERIFY_MODELS === 'true') {
|
||||
console.log('🔐 Verifying model integrity...')
|
||||
// Add hash verification here
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
console.log(`⚠️ Missing ${missingFiles.length} model files`)
|
||||
|
||||
// In production, models should be pre-bundled
|
||||
if (process.env.NODE_ENV === 'production' && !process.env.ALLOW_MODEL_DOWNLOAD) {
|
||||
throw new Error(
|
||||
'Critical: Transformer models not found in production. ' +
|
||||
'Run "npm run download-models" during build stage.'
|
||||
)
|
||||
}
|
||||
|
||||
// Development: offer to download
|
||||
if (process.env.CI !== 'true') {
|
||||
console.log('📥 Would download models from CDN in development')
|
||||
console.log(' Run: npm run download-models')
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// Export for use in main code
|
||||
export async function verifyModelsAvailable() {
|
||||
try {
|
||||
return await ensureModels()
|
||||
} catch (error) {
|
||||
console.error('❌ Model verification failed:', error.message)
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// Run if called directly
|
||||
if (import.meta.url === `file://${process.argv[1]}`) {
|
||||
ensureModels()
|
||||
.then(success => process.exit(success ? 0 : 1))
|
||||
.catch(error => {
|
||||
console.error(error)
|
||||
process.exit(1)
|
||||
})
|
||||
}
|
||||
|
|
@ -1,387 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
/**
|
||||
* Prepare Models Script
|
||||
*
|
||||
* Intelligently handles model preparation for different deployment scenarios:
|
||||
* 1. Development: Models download automatically on first use
|
||||
* 2. Docker/CI: Pre-download during build stage
|
||||
* 3. Serverless: Bundle with deployment package
|
||||
* 4. Production: Verify models exist, fail fast if missing
|
||||
*/
|
||||
|
||||
import { existsSync } from 'fs'
|
||||
import { readFile, mkdir, writeFile, stat } from 'fs/promises'
|
||||
import { join, dirname } from 'path'
|
||||
import { fileURLToPath } from 'url'
|
||||
import { pipeline, env } from '@huggingface/transformers'
|
||||
import { execSync } from 'child_process'
|
||||
import https from 'https'
|
||||
import { createWriteStream } from 'fs'
|
||||
import { promisify } from 'util'
|
||||
import { finished } from 'stream'
|
||||
|
||||
const streamFinished = promisify(finished)
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url))
|
||||
|
||||
// Model configuration
|
||||
const MODEL_CONFIG = {
|
||||
name: 'Xenova/all-MiniLM-L6-v2',
|
||||
expectedFiles: [
|
||||
'config.json',
|
||||
'tokenizer.json',
|
||||
'tokenizer_config.json',
|
||||
'onnx/model.onnx'
|
||||
],
|
||||
fallbackUrls: {
|
||||
// GitHub Releases (our backup)
|
||||
github: 'https://github.com/soulcraftlabs/brainy-models/releases/download/v1.0/all-MiniLM-L6-v2.tar.gz',
|
||||
// Future CDN
|
||||
cdn: 'https://models.soulcraft.com/brainy/all-MiniLM-L6-v2.tar.gz'
|
||||
}
|
||||
}
|
||||
|
||||
class ModelPreparer {
|
||||
constructor() {
|
||||
this.modelsDir = join(__dirname, '..', 'models')
|
||||
this.modelPath = join(this.modelsDir, ...MODEL_CONFIG.name.split('/'))
|
||||
}
|
||||
|
||||
/**
|
||||
* Main entry point - intelligently prepares models based on context
|
||||
*/
|
||||
async prepare() {
|
||||
console.log('🧠 Brainy Model Preparation')
|
||||
console.log('===========================')
|
||||
|
||||
// Detect deployment context
|
||||
const context = this.detectContext()
|
||||
console.log(`📍 Context: ${context}`)
|
||||
|
||||
switch (context) {
|
||||
case 'production':
|
||||
return await this.prepareProduction()
|
||||
case 'docker':
|
||||
return await this.prepareDocker()
|
||||
case 'ci':
|
||||
return await this.prepareCI()
|
||||
case 'development':
|
||||
return await this.prepareDevelopment()
|
||||
default:
|
||||
return await this.prepareDefault()
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect the deployment context
|
||||
*/
|
||||
detectContext() {
|
||||
// Check environment variables
|
||||
if (process.env.NODE_ENV === 'production') return 'production'
|
||||
if (process.env.DOCKER_BUILD === 'true') return 'docker'
|
||||
if (process.env.CI === 'true') return 'ci'
|
||||
if (process.env.NODE_ENV === 'development') return 'development'
|
||||
|
||||
// Check for Docker build context
|
||||
if (existsSync('/.dockerenv')) return 'docker'
|
||||
|
||||
// Check for common CI indicators
|
||||
if (process.env.GITHUB_ACTIONS || process.env.GITLAB_CI) return 'ci'
|
||||
|
||||
// Default to development
|
||||
return 'development'
|
||||
}
|
||||
|
||||
/**
|
||||
* Production: Models MUST exist, fail fast if not
|
||||
*/
|
||||
async prepareProduction() {
|
||||
console.log('🏭 Production mode - verifying models...')
|
||||
|
||||
const modelExists = await this.verifyModels()
|
||||
|
||||
if (!modelExists) {
|
||||
console.error('❌ CRITICAL: Models not found in production!')
|
||||
console.error(' Models must be pre-downloaded during build stage.')
|
||||
console.error(' Run: npm run download-models')
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
console.log('✅ Models verified for production')
|
||||
return true
|
||||
}
|
||||
|
||||
/**
|
||||
* Docker: Download models during build stage
|
||||
*/
|
||||
async prepareDocker() {
|
||||
console.log('🐳 Docker build - downloading models...')
|
||||
|
||||
// Check if already exists
|
||||
if (await this.verifyModels()) {
|
||||
console.log('✅ Models already present')
|
||||
return true
|
||||
}
|
||||
|
||||
// Download models
|
||||
return await this.downloadModels()
|
||||
}
|
||||
|
||||
/**
|
||||
* CI: Download models for testing
|
||||
*/
|
||||
async prepareCI() {
|
||||
console.log('🔧 CI environment - downloading models for tests...')
|
||||
|
||||
// Check cache first
|
||||
if (await this.checkCICache()) {
|
||||
console.log('✅ Using cached models')
|
||||
return true
|
||||
}
|
||||
|
||||
// Download and cache
|
||||
const success = await this.downloadModels()
|
||||
if (success) {
|
||||
await this.saveCICache()
|
||||
}
|
||||
return success
|
||||
}
|
||||
|
||||
/**
|
||||
* Development: Optional download, will auto-download on first use
|
||||
*/
|
||||
async prepareDevelopment() {
|
||||
console.log('💻 Development mode')
|
||||
|
||||
if (await this.verifyModels()) {
|
||||
console.log('✅ Models already downloaded')
|
||||
return true
|
||||
}
|
||||
|
||||
console.log('ℹ️ Models will download automatically on first use')
|
||||
console.log(' To pre-download now: npm run download-models')
|
||||
|
||||
// Ask if they want to download now
|
||||
if (process.stdout.isTTY && !process.env.SKIP_PROMPT) {
|
||||
const readline = await import('readline')
|
||||
const rl = readline.createInterface({
|
||||
input: process.stdin,
|
||||
output: process.stdout
|
||||
})
|
||||
|
||||
return new Promise((resolve) => {
|
||||
rl.question('Download models now? (y/N): ', async (answer) => {
|
||||
rl.close()
|
||||
if (answer.toLowerCase() === 'y') {
|
||||
resolve(await this.downloadModels())
|
||||
} else {
|
||||
resolve(true)
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
/**
|
||||
* Default: Try to be smart about it
|
||||
*/
|
||||
async prepareDefault() {
|
||||
console.log('🤖 Auto-detecting best approach...')
|
||||
|
||||
if (await this.verifyModels()) {
|
||||
console.log('✅ Models found')
|
||||
return true
|
||||
}
|
||||
|
||||
// If running as part of install, don't download
|
||||
if (process.env.npm_lifecycle_event === 'postinstall') {
|
||||
console.log('ℹ️ Skipping download during install (will download on first use)')
|
||||
return true
|
||||
}
|
||||
|
||||
// Otherwise download
|
||||
return await this.downloadModels()
|
||||
}
|
||||
|
||||
/**
|
||||
* Verify all required model files exist
|
||||
*/
|
||||
async verifyModels() {
|
||||
for (const file of MODEL_CONFIG.expectedFiles) {
|
||||
const filePath = join(this.modelPath, file)
|
||||
if (!existsSync(filePath)) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// Verify model.onnx size (should be ~87MB)
|
||||
const modelOnnxPath = join(this.modelPath, 'onnx', 'model.onnx')
|
||||
if (existsSync(modelOnnxPath)) {
|
||||
const stats = await stat(modelOnnxPath)
|
||||
const sizeMB = Math.round(stats.size / (1024 * 1024))
|
||||
if (sizeMB < 80 || sizeMB > 100) {
|
||||
console.warn(`⚠️ Model size unexpected: ${sizeMB}MB (expected ~87MB)`)
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
/**
|
||||
* Download models with fallback sources
|
||||
*/
|
||||
async downloadModels() {
|
||||
console.log('📥 Downloading transformer models...')
|
||||
|
||||
// Try transformers.js first (Hugging Face)
|
||||
try {
|
||||
await this.downloadFromTransformers()
|
||||
console.log('✅ Downloaded from Hugging Face')
|
||||
return true
|
||||
} catch (error) {
|
||||
console.warn('⚠️ Hugging Face download failed:', error.message)
|
||||
}
|
||||
|
||||
// Try GitHub releases
|
||||
try {
|
||||
await this.downloadFromGitHub()
|
||||
console.log('✅ Downloaded from GitHub')
|
||||
return true
|
||||
} catch (error) {
|
||||
console.warn('⚠️ GitHub download failed:', error.message)
|
||||
}
|
||||
|
||||
// Try CDN
|
||||
try {
|
||||
await this.downloadFromCDN()
|
||||
console.log('✅ Downloaded from CDN')
|
||||
return true
|
||||
} catch (error) {
|
||||
console.warn('⚠️ CDN download failed:', error.message)
|
||||
}
|
||||
|
||||
console.error('❌ All download sources failed')
|
||||
return false
|
||||
}
|
||||
|
||||
/**
|
||||
* Download using transformers.js (official Hugging Face)
|
||||
*/
|
||||
async downloadFromTransformers() {
|
||||
env.cacheDir = this.modelsDir
|
||||
env.allowRemoteModels = true
|
||||
|
||||
console.log(' Source: Hugging Face')
|
||||
console.log(' Model:', MODEL_CONFIG.name)
|
||||
|
||||
// Load pipeline to trigger download
|
||||
const extractor = await pipeline('feature-extraction', MODEL_CONFIG.name)
|
||||
|
||||
// Test it works
|
||||
const test = await extractor('test', { pooling: 'mean', normalize: true })
|
||||
console.log(` ✓ Model test passed (dims: ${test.data.length})`)
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
/**
|
||||
* Download from GitHub releases (our backup)
|
||||
*/
|
||||
async downloadFromGitHub() {
|
||||
const url = MODEL_CONFIG.fallbackUrls.github
|
||||
console.log(' Source: GitHub Releases')
|
||||
|
||||
// Download tar.gz
|
||||
const tempFile = join(this.modelsDir, 'temp-model.tar.gz')
|
||||
await this.downloadFile(url, tempFile)
|
||||
|
||||
// Extract
|
||||
await mkdir(this.modelPath, { recursive: true })
|
||||
execSync(`tar -xzf ${tempFile} -C ${this.modelPath}`, { stdio: 'inherit' })
|
||||
|
||||
// Cleanup
|
||||
await unlink(tempFile)
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
/**
|
||||
* Download from CDN (future)
|
||||
*/
|
||||
async downloadFromCDN() {
|
||||
const url = MODEL_CONFIG.fallbackUrls.cdn
|
||||
console.log(' Source: Soulcraft CDN')
|
||||
|
||||
// Similar to GitHub approach
|
||||
throw new Error('CDN not yet available')
|
||||
}
|
||||
|
||||
/**
|
||||
* Download a file from URL
|
||||
*/
|
||||
async downloadFile(url, destination) {
|
||||
await mkdir(dirname(destination), { recursive: true })
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const file = createWriteStream(destination)
|
||||
|
||||
https.get(url, (response) => {
|
||||
if (response.statusCode !== 200) {
|
||||
reject(new Error(`HTTP ${response.statusCode}`))
|
||||
return
|
||||
}
|
||||
|
||||
response.pipe(file)
|
||||
|
||||
file.on('finish', () => {
|
||||
file.close()
|
||||
resolve()
|
||||
})
|
||||
}).on('error', reject)
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Check CI cache for models
|
||||
*/
|
||||
async checkCICache() {
|
||||
// GitHub Actions cache
|
||||
if (process.env.GITHUB_ACTIONS) {
|
||||
const cachePath = process.env.RUNNER_TEMP + '/brainy-models'
|
||||
if (existsSync(cachePath)) {
|
||||
// Copy from cache
|
||||
execSync(`cp -r ${cachePath}/* ${this.modelsDir}/`, { stdio: 'inherit' })
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
/**
|
||||
* Save models to CI cache
|
||||
*/
|
||||
async saveCICache() {
|
||||
// GitHub Actions cache
|
||||
if (process.env.GITHUB_ACTIONS) {
|
||||
const cachePath = process.env.RUNNER_TEMP + '/brainy-models'
|
||||
await mkdir(cachePath, { recursive: true })
|
||||
execSync(`cp -r ${this.modelsDir}/* ${cachePath}/`, { stdio: 'inherit' })
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Run the preparer
|
||||
const preparer = new ModelPreparer()
|
||||
preparer.prepare()
|
||||
.then(success => {
|
||||
if (!success) {
|
||||
process.exit(1)
|
||||
}
|
||||
})
|
||||
.catch(error => {
|
||||
console.error('❌ Fatal error:', error)
|
||||
process.exit(1)
|
||||
})
|
||||
224
scripts/setup-dev.sh
Executable file
224
scripts/setup-dev.sh
Executable file
|
|
@ -0,0 +1,224 @@
|
|||
#!/bin/bash
|
||||
# Development environment setup script for Brainy
|
||||
#
|
||||
# This script installs all dependencies needed to build Brainy,
|
||||
# including the Candle WASM embedding engine.
|
||||
#
|
||||
# Usage:
|
||||
# ./scripts/setup-dev.sh
|
||||
#
|
||||
# Requirements:
|
||||
# - sudo access (for system packages)
|
||||
# - Internet connection
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m'
|
||||
|
||||
echo -e "${BLUE}======================================${NC}"
|
||||
echo -e "${BLUE} Brainy Development Setup${NC}"
|
||||
echo -e "${BLUE}======================================${NC}"
|
||||
echo ""
|
||||
|
||||
# Detect OS
|
||||
detect_os() {
|
||||
if [[ "$OSTYPE" == "linux-gnu"* ]]; then
|
||||
if command -v apt-get &> /dev/null; then
|
||||
echo "debian"
|
||||
elif command -v dnf &> /dev/null; then
|
||||
echo "fedora"
|
||||
elif command -v pacman &> /dev/null; then
|
||||
echo "arch"
|
||||
else
|
||||
echo "linux-unknown"
|
||||
fi
|
||||
elif [[ "$OSTYPE" == "darwin"* ]]; then
|
||||
echo "macos"
|
||||
else
|
||||
echo "unknown"
|
||||
fi
|
||||
}
|
||||
|
||||
OS=$(detect_os)
|
||||
echo -e "${GREEN}Detected OS: ${OS}${NC}"
|
||||
|
||||
# Install system dependencies
|
||||
install_system_deps() {
|
||||
echo -e "\n${YELLOW}Installing system dependencies...${NC}"
|
||||
|
||||
case $OS in
|
||||
debian)
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y build-essential pkg-config libssl-dev curl
|
||||
;;
|
||||
fedora)
|
||||
sudo dnf install -y gcc gcc-c++ make openssl-devel pkgconfig curl
|
||||
;;
|
||||
arch)
|
||||
sudo pacman -S --needed base-devel openssl pkg-config curl
|
||||
;;
|
||||
macos)
|
||||
if ! command -v gcc &> /dev/null; then
|
||||
echo -e "${YELLOW}Installing Xcode command line tools...${NC}"
|
||||
xcode-select --install 2>/dev/null || true
|
||||
fi
|
||||
;;
|
||||
*)
|
||||
echo -e "${RED}Unknown OS. Please install build tools manually:${NC}"
|
||||
echo " - C compiler (gcc or clang)"
|
||||
echo " - pkg-config"
|
||||
echo " - OpenSSL development headers"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
echo -e "${GREEN}System dependencies installed.${NC}"
|
||||
}
|
||||
|
||||
# Install Rust
|
||||
install_rust() {
|
||||
echo -e "\n${YELLOW}Checking Rust installation...${NC}"
|
||||
|
||||
if command -v rustc &> /dev/null; then
|
||||
RUST_VERSION=$(rustc --version)
|
||||
echo -e "${GREEN}Rust already installed: ${RUST_VERSION}${NC}"
|
||||
else
|
||||
echo -e "${YELLOW}Installing Rust...${NC}"
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
|
||||
source "$HOME/.cargo/env"
|
||||
echo -e "${GREEN}Rust installed: $(rustc --version)${NC}"
|
||||
fi
|
||||
|
||||
# Ensure cargo env is loaded
|
||||
if [ -f "$HOME/.cargo/env" ]; then
|
||||
source "$HOME/.cargo/env"
|
||||
fi
|
||||
}
|
||||
|
||||
# Install Rust WASM tools
|
||||
install_wasm_tools() {
|
||||
echo -e "\n${YELLOW}Installing WASM build tools...${NC}"
|
||||
|
||||
# Add WASM target
|
||||
if ! rustup target list --installed | grep -q wasm32-unknown-unknown; then
|
||||
echo "Adding wasm32-unknown-unknown target..."
|
||||
rustup target add wasm32-unknown-unknown
|
||||
else
|
||||
echo -e "${GREEN}WASM target already installed.${NC}"
|
||||
fi
|
||||
|
||||
# Install wasm-pack
|
||||
if ! command -v wasm-pack &> /dev/null; then
|
||||
echo "Installing wasm-pack..."
|
||||
cargo install wasm-pack
|
||||
else
|
||||
echo -e "${GREEN}wasm-pack already installed: $(wasm-pack --version)${NC}"
|
||||
fi
|
||||
}
|
||||
|
||||
# Install Node.js dependencies
|
||||
install_node_deps() {
|
||||
echo -e "\n${YELLOW}Checking Node.js...${NC}"
|
||||
|
||||
if ! command -v node &> /dev/null; then
|
||||
echo -e "${RED}Node.js not found. Please install Node.js 20+ first.${NC}"
|
||||
echo "Visit: https://nodejs.org/"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
NODE_VERSION=$(node --version)
|
||||
echo -e "${GREEN}Node.js: ${NODE_VERSION}${NC}"
|
||||
|
||||
echo -e "\n${YELLOW}Installing npm dependencies...${NC}"
|
||||
npm install
|
||||
echo -e "${GREEN}npm dependencies installed.${NC}"
|
||||
}
|
||||
|
||||
# Download model files
|
||||
download_models() {
|
||||
echo -e "\n${YELLOW}Downloading model files...${NC}"
|
||||
|
||||
MODEL_DIR="assets/models/all-MiniLM-L6-v2"
|
||||
|
||||
if [ -f "$MODEL_DIR/model.safetensors" ] && [ -f "$MODEL_DIR/tokenizer.json" ]; then
|
||||
echo -e "${GREEN}Model files already present.${NC}"
|
||||
return
|
||||
fi
|
||||
|
||||
mkdir -p "$MODEL_DIR"
|
||||
|
||||
HF_URL="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main"
|
||||
|
||||
echo "Downloading model.safetensors..."
|
||||
curl -L "$HF_URL/model.safetensors" -o "$MODEL_DIR/model.safetensors"
|
||||
|
||||
echo "Downloading tokenizer.json..."
|
||||
curl -L "$HF_URL/tokenizer.json" -o "$MODEL_DIR/tokenizer.json"
|
||||
|
||||
echo "Downloading config.json..."
|
||||
curl -L "$HF_URL/config.json" -o "$MODEL_DIR/config.json"
|
||||
|
||||
echo -e "${GREEN}Model files downloaded.${NC}"
|
||||
}
|
||||
|
||||
# Build Candle WASM
|
||||
build_candle() {
|
||||
echo -e "\n${YELLOW}Building Candle WASM...${NC}"
|
||||
|
||||
if [ -f "src/embeddings/wasm/pkg/candle_embeddings_bg.wasm" ]; then
|
||||
echo -e "${GREEN}Candle WASM already built. Use 'npm run build:candle' to rebuild.${NC}"
|
||||
return
|
||||
fi
|
||||
|
||||
npm run build:candle
|
||||
echo -e "${GREEN}Candle WASM built.${NC}"
|
||||
}
|
||||
|
||||
# Build TypeScript
|
||||
build_typescript() {
|
||||
echo -e "\n${YELLOW}Building TypeScript...${NC}"
|
||||
npm run build
|
||||
echo -e "${GREEN}TypeScript built.${NC}"
|
||||
}
|
||||
|
||||
# Run tests
|
||||
run_tests() {
|
||||
echo -e "\n${YELLOW}Running tests...${NC}"
|
||||
npm run test:unit
|
||||
echo -e "${GREEN}Tests passed.${NC}"
|
||||
}
|
||||
|
||||
# Main
|
||||
main() {
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
cd "$PROJECT_ROOT"
|
||||
|
||||
install_system_deps
|
||||
install_rust
|
||||
install_wasm_tools
|
||||
install_node_deps
|
||||
download_models
|
||||
build_candle
|
||||
build_typescript
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}======================================${NC}"
|
||||
echo -e "${GREEN} Setup Complete!${NC}"
|
||||
echo -e "${GREEN}======================================${NC}"
|
||||
echo ""
|
||||
echo "You can now:"
|
||||
echo " npm run build # Build TypeScript"
|
||||
echo " npm run build:candle # Rebuild Candle WASM"
|
||||
echo " npm run test:all # Run all tests"
|
||||
echo " npm run test:wasm # Test WASM embeddings"
|
||||
echo " npm run test:bun:compile # Test Bun compile"
|
||||
echo ""
|
||||
}
|
||||
|
||||
main "$@"
|
||||
Loading…
Add table
Add a link
Reference in a new issue