The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:
1. validateConsistency() to falsely detect corruption on every startup,
triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
and report inflated totalEntries/totalIds stats
Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
New public APIs:
- embedBatch(texts): Batch embed multiple texts efficiently
- similarity(textA, textB): Calculate semantic similarity (0-1 score)
- indexStats(): Get comprehensive index statistics with memory usage
- neighbors(entityId, options): Get graph neighbors with direction/depth/filter
- findDuplicates(options): Find semantic duplicates by embedding similarity
- cluster(options): Cluster entities by semantic similarity with centroids
All APIs:
- Added to BrainyInterface for type safety
- Documented in docs/API_REFERENCE.md and docs/api/README.md
- Include JSDoc examples and parameter descriptions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add confidence and weight properties to Entity interface and flatten Result fields to top level for improved developer experience and API consistency.
Breaking Changes: None (all changes are backward compatible)
Phase 2 - Entity Confidence & Weight:
- Add confidence (type classification certainty) and weight (entity importance) to Entity interface
- Add confidence/weight parameters to AddParams and UpdateParams
- Update convertNounToEntity() to extract confidence/weight from storage
- Update add() and update() methods to preserve confidence/weight in metadata
- Enable developers to specify and access entity confidence/weight scores
Phase 3 - Result Field Flattening:
- Flatten commonly-used entity fields (type, metadata, data, confidence, weight) to Result top level
- Add createResult() helper for consistent Result construction
- Update all find() code paths to use createResult()
- Enable direct access: result.metadata instead of result.entity.metadata
- Preserve full entity in result.entity for backward compatibility
VFS Fix (from previous work):
- Fix VFSStructureGenerator to use brain.vfs() cached instance instead of creating separate instance
- Improve VFS error messages with step-by-step guidance
- Update examples to show correct vfs.init() usage
- Add comprehensive VFS import verification tests
Documentation Updates:
- Update API_REFERENCE.md with confidence/weight examples and flattened Result documentation
- Enhance JSDoc for add(), get(), find(), similar() with v4.3.0 examples
- Document Result structure changes and backward compatibility
- Add migration examples showing both old and new access patterns
Tests:
- Add 16 comprehensive tests for Entity confidence/weight exposure
- Add tests for Result field flattening
- Add tests for backward compatibility
- All tests passing (16/16)
API Consistency:
- Entity: direct access to confidence/weight
- Result: flattened fields + nested entity (both work)
- Relation: already had confidence/weight (consistent)
- VFS: inherits from Entity (automatic)
Files Changed:
- src/types/brainy.types.ts - Updated Entity, AddParams, UpdateParams, Result interfaces
- src/brainy.ts - Updated implementation and JSDoc for all affected methods
- tests/integration/entity-confidence-weight.test.ts - 16 comprehensive tests
- docs/API_REFERENCE.md - Updated with v4.3.0 examples
- src/importers/VFSStructureGenerator.ts - VFS fix
- src/vfs/VirtualFileSystem.ts - Improved error messages
- examples/unified-import-example.ts - Added vfs.init() example
- tests/integration/vfs-*-verification.test.ts - VFS verification tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Removed ImportManager class and exports (use brain.import() instead)
- Fixed all documentation: getStatistics() → getStats()
- Updated 41 files across codebase for consistency
- Removed ImportManager section from API docs
- Added v3.30.0 migration guide to CHANGELOG
Co-Authored-By: Claude <noreply@anthropic.com>
Improve documentation to explicitly show that:
- type: 'gcs' requires gcsStorage config (S3-compatible, HMAC)
- type: 'gcs-native' requires gcsNativeStorage config (native SDK, ADC)
- Mixing type and config objects will fall back to memory storage
- Auto-detection works when type is omitted
Added 'Common Mistakes' section with examples of incorrect usage.
Enhanced migration guide with step-by-step instructions.
Fixes critical scalability bottleneck where metadata was stored in non-sharded
directories, causing performance degradation at scale (1M+ entities).
Changes:
- Add UUID-based sharding to metadata operations in S3Compatible, FileSystem, and OpFS storage
- Implement complete UUID-based sharding for OpFS storage (nouns, verbs, metadata)
- Update pagination methods to iterate through all 256 UUID-based shards
- Add integration tests verifying sharding behavior across storage adapters
Impact:
- Metadata now scales to millions of entities without directory bottlenecks
- All storage adapters now use consistent UUID-based sharding (256 buckets: 00-ff)
- Improves GCS/S3/R2/OpFS performance at scale
- Path format: entities/{type}/{subtype}/{shard}/{id}.json
Breaking change: Requires data migration for existing S3/GCS/R2/OpFS deployments.
See .strategy/UNIFIED-UUID-SHARDING.md for migration guidance.
Resolves API accessibility issues reported by Brain Cloud Studio team:
1. Export ImportManager and createImportManager
- ImportManager class was fully implemented but not exported
- Consumers can now access AI-powered import features
- Includes ImportOptions and ImportResult types
2. Add brain.getStats() convenience method
- Documentation showed getStats() as top-level method
- Implementation had it nested under brain.counts.getStats()
- Added convenience method that delegates to counts API
- Maintains backward compatibility
3. Update API documentation
- Corrected getStats() signature and return type
- Added comprehensive ImportManager documentation
- Included examples for all import patterns
These changes expose existing, tested functionality without
modifying core behavior. All tests pass.
Add IntelligentImportAugmentation with support for:
- CSV: auto-detection of encoding, delimiters, and field types
- Excel: multi-sheet extraction with metadata preservation
- PDF: text extraction, table detection, and metadata extraction
Features:
- Automatic format detection from file extension or content
- Intelligent type inference (string, number, boolean, date)
- Seamless integration with neural entity extraction
- Production-ready with 69 comprehensive tests
Dependencies added:
- xlsx@^0.18.5 for Excel parsing
- pdfjs-dist@^4.0.379 for PDF parsing
- csv-parse@^6.1.0 for CSV parsing
- chardet@^2.0.0 for encoding detection
Documentation:
- Updated README with import examples
- Updated API_REFERENCE with comprehensive import() docs
- Updated import-anything.md guide
- Added working example: import-excel-pdf-csv.ts
- Updated augmentations README
- Add @deprecated JSDoc tags to TypeScript definitions
- Update all documentation examples to use modern add() and relate() API
- Preserve batch operations (addNouns, addVerbs) as they remain current
- Mark deprecated methods in both source and compiled definitions
Migration guide:
- addNoun(data, type, metadata) → add(data, { nounType: type, ...metadata })
- addVerb(source, target, type, metadata) → relate(source, target, type, metadata)
- Implement self-configuring validation that adapts to system resources
- Add validation for all CRUD operations (add, update, delete, find, relate)
- Auto-configure limits based on available memory (1GB = 10K limit, 8GB = 80K)
- Monitor and auto-tune performance based on query response times
- Fix multiple type filtering with proper anyOf structure
- Enhance type safety by requiring NounType/VerbType enums
- Fix tests to validate correct behavior (no fake implementations)
- Add comprehensive VALIDATION.md documentation
- Update API_REFERENCE.md with validation rules and examples
- Clarify metadata update behavior (null keeps existing, {} clears)
BREAKING CHANGE: getFieldsForType() now requires NounType enum instead of string
Co-Authored-By: Claude <noreply@anthropic.com>
