soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions
brainy/docs/architecture
History
Exact
David Snelling 92c96246fb feat(v4.0.0): Complete metadata/vector separation architecture with Azure support 
This commit completes the core v4.0.0 architecture changes for billion-scale
performance with metadata/vector separation. NO RELEASE YET - remaining optimizations
and testing required before production release.

## Core v4.0.0 Architecture Changes

### Type System Updates
- Fixed all TypeScript compilation errors (zero errors achieved)
- Updated HNSWNoun/HNSWVerb to separate core fields from metadata
- Implemented HNSWNounWithMetadata/HNSWVerbWithMetadata for API boundaries
- Added required 'noun' field to NounMetadata for semantic structure
- Renamed verb.type to verb.verb for consistency

### Storage Adapter Updates
**All adapters updated for v4.0.0 two-file storage pattern:**
- memoryStorage: Proper metadata/vector separation
- fileSystemStorage: Two-file pattern with sharding
- opfsStorage: Browser persistent storage updated
- s3CompatibleStorage: AWS/MinIO/DigitalOcean support
- r2Storage: Cloudflare R2 optimization
- gcsStorage: Google Cloud with ADC support
- **azureBlobStorage: NEW - Full Azure Blob Storage support**

### Storage Features
- BaseStorage: Internal vs public method separation (_getNoun vs getNoun)
- Two-file storage: Vectors in one file, metadata in another
- Change tracking: getChangesSince return type updated
- Pagination: getNounsWithPagination returns WithMetadata types

### Azure Blob Storage Integration (NEW)
- Native @azure/storage-blob SDK integration
- Four authentication methods:
  * DefaultAzureCredential (Managed Identity) - recommended
  * Connection String - simplest setup
  * Account Name + Key - traditional auth
  * SAS Token - delegated access
- High-volume mode with write buffering
- Adaptive backpressure for throttling
- UUID-based sharding for billion-scale
- Full HNSW support with graph persistence

### Utility Updates
- EmbeddingManager: Updated to accept Record<string, unknown>
- LSMTree: Wrapped data in NounMetadata structure with 'noun' field
- EntityIdMapper: Fixed nested metadata.data structure access
- MetadataIndex: Fixed field type inference integration
- PeriodicCleanup: Updated for new metadata structure

### Core API Updates
- Brainy: Updated verb property access from v.type to v.verb
- ConfigAPI: Fixed NounMetadata access patterns
- DataAPI: Updated metadata handling

### Documentation Updates
- CREATING-AUGMENTATIONS.md: v4.0.0 breaking changes guide
- DEVELOPER-GUIDE.md: Migration checklist and examples
- COMPLETE-REFERENCE.md: v4.0.0 architecture improvements
- **finite-type-system.md: NEW - Revolutionary type system benefits**

### Build & Dependencies
- Zero TypeScript compilation errors
- Added @azure/storage-blob and @azure/identity
- 591 tests passing (23 timeout in long-running neural tests)

## What's NOT in This Release
This is a work-in-progress commit. Before v4.0.0 release we need:
- Storage adapter optimizations (batch operations, compression)
- Azure blob tier management (Hot/Cool/Archive)
- Cost optimization implementations
- Additional performance testing at billion-scale
- Migration guides for v3.x users

## Testing
- Clean build: ✅
- Type checking: ✅ (zero errors)
- Test suite: ✅ (591/614 passing, timeouts in neural tests only)

🔐 Generated with Claude Code
https://claude.com/claude-code

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-17 12:29:27 -07:00
..
API_SURFACE_DESIGN.md fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00
augmentation-system-audit.md fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00
augmentations-actual.md feat: remove legacy ImportManager, standardize getStats() API 2025-10-09 11:40:31 -07:00
augmentations.md fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00
CLUSTERING_ALGORITHMS_ANALYSIS.md docs: add deprecation warnings for addNoun and addVerb methods 2025-09-17 10:48:41 -07:00
data-storage-architecture.md feat: implement always-adaptive caching with getCacheStats monitoring 2025-10-10 14:09:30 -07:00
distributed-storage.md docs: add comprehensive scaling and storage architecture documentation 2025-09-08 14:49:25 -07:00
finite-type-system.md feat(v4.0.0): Complete metadata/vector separation architecture with Azure support 2025-10-17 12:29:27 -07:00
index-architecture.md fix: migrate from roaring (native C++) to roaring-wasm for universal compatibility 2025-10-14 10:24:59 -07:00
initialization-and-rebuild.md fix: 6000x speedup for TypeAwareHNSWIndex rebuild - enables billion-scale operations 2025-10-15 17:48:26 -07:00
METADATA_ARCHITECTURE.md feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
noun-verb-taxonomy.md fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00
overview.md docs: add comprehensive index architecture documentation 2025-10-13 13:42:04 -07:00
PERFORMANCE_ANALYSIS.md feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
storage-architecture.md feat: implement always-adaptive caching with getCacheStats monitoring 2025-10-10 14:09:30 -07:00
triple-intelligence.md 🧠 Brainy 2.0.0 - Zero-Configuration AI Database with Triple Intelligence™ 2025-08-26 12:32:21 -07:00
zero-config.md fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00