feat(v4.0.0): Complete metadata/vector separation architecture with Azure support
This commit completes the core v4.0.0 architecture changes for billion-scale performance with metadata/vector separation. NO RELEASE YET - remaining optimizations and testing required before production release. ## Core v4.0.0 Architecture Changes ### Type System Updates - Fixed all TypeScript compilation errors (zero errors achieved) - Updated HNSWNoun/HNSWVerb to separate core fields from metadata - Implemented HNSWNounWithMetadata/HNSWVerbWithMetadata for API boundaries - Added required 'noun' field to NounMetadata for semantic structure - Renamed verb.type to verb.verb for consistency ### Storage Adapter Updates **All adapters updated for v4.0.0 two-file storage pattern:** - memoryStorage: Proper metadata/vector separation - fileSystemStorage: Two-file pattern with sharding - opfsStorage: Browser persistent storage updated - s3CompatibleStorage: AWS/MinIO/DigitalOcean support - r2Storage: Cloudflare R2 optimization - gcsStorage: Google Cloud with ADC support - **azureBlobStorage: NEW - Full Azure Blob Storage support** ### Storage Features - BaseStorage: Internal vs public method separation (_getNoun vs getNoun) - Two-file storage: Vectors in one file, metadata in another - Change tracking: getChangesSince return type updated - Pagination: getNounsWithPagination returns WithMetadata types ### Azure Blob Storage Integration (NEW) - Native @azure/storage-blob SDK integration - Four authentication methods: * DefaultAzureCredential (Managed Identity) - recommended * Connection String - simplest setup * Account Name + Key - traditional auth * SAS Token - delegated access - High-volume mode with write buffering - Adaptive backpressure for throttling - UUID-based sharding for billion-scale - Full HNSW support with graph persistence ### Utility Updates - EmbeddingManager: Updated to accept Record<string, unknown> - LSMTree: Wrapped data in NounMetadata structure with 'noun' field - EntityIdMapper: Fixed nested metadata.data structure access - MetadataIndex: Fixed field type inference integration - PeriodicCleanup: Updated for new metadata structure ### Core API Updates - Brainy: Updated verb property access from v.type to v.verb - ConfigAPI: Fixed NounMetadata access patterns - DataAPI: Updated metadata handling ### Documentation Updates - CREATING-AUGMENTATIONS.md: v4.0.0 breaking changes guide - DEVELOPER-GUIDE.md: Migration checklist and examples - COMPLETE-REFERENCE.md: v4.0.0 architecture improvements - **finite-type-system.md: NEW - Revolutionary type system benefits** ### Build & Dependencies - Zero TypeScript compilation errors - Added @azure/storage-blob and @azure/identity - 591 tests passing (23 timeout in long-running neural tests) ## What's NOT in This Release This is a work-in-progress commit. Before v4.0.0 release we need: - Storage adapter optimizations (batch operations, compression) - Azure blob tier management (Hot/Cool/Archive) - Cost optimization implementations - Additional performance testing at billion-scale - Migration guides for v3.x users ## Testing - Clean build: ✅ - Type checking: ✅ (zero errors) - Test suite: ✅ (591/614 passing, timeouts in neural tests only) 🔐 Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
8d6dd07e1d
commit
92c96246fb
35 changed files with 4524 additions and 1026 deletions
|
|
@ -1,6 +1,49 @@
|
|||
# 🛠️ Brainy Augmentation Developer Guide
|
||||
|
||||
> **How to create, test, and use augmentations in Brainy 2.0**
|
||||
> **How to create, test, and use augmentations in Brainy v4.0.0**
|
||||
>
|
||||
> **⚠️ v4.0.0 Update**: This guide has been updated with breaking changes for metadata structure and type system improvements.
|
||||
|
||||
## v4.0.0 Migration Guide
|
||||
|
||||
### What Changed?
|
||||
|
||||
1. **Metadata Structure**: All metadata now requires type fields (`noun` or `verb`)
|
||||
2. **Property Rename**: `verb.type` → `verb.verb` for relationships
|
||||
3. **Two-File Storage**: Vectors and metadata stored separately for performance
|
||||
4. **Return Types**: Storage methods distinguish between internal (pure) and public (WithMetadata) returns
|
||||
|
||||
### Migration Checklist
|
||||
|
||||
- [ ] Update metadata creation to include required `noun` field
|
||||
- [ ] Change `verb.type` to `verb.verb` in all relationship code
|
||||
- [ ] Update storage adapter methods to follow internal/public pattern
|
||||
- [ ] Ensure metadata access uses correct structure
|
||||
|
||||
### Quick Migration Example
|
||||
|
||||
```typescript
|
||||
// ❌ v3.x
|
||||
const verb = {
|
||||
type: 'relatedTo',
|
||||
sourceId: 'a',
|
||||
targetId: 'b'
|
||||
}
|
||||
if (verb.type === 'relatedTo') { ... }
|
||||
|
||||
// ✅ v4.0.0
|
||||
const verb = {
|
||||
verb: 'relatedTo',
|
||||
sourceId: 'a',
|
||||
targetId: 'b'
|
||||
}
|
||||
const metadata: VerbMetadata = {
|
||||
verb: 'relatedTo',
|
||||
sourceId: 'a',
|
||||
targetId: 'b'
|
||||
}
|
||||
if (verb.verb === 'relatedTo') { ... }
|
||||
```
|
||||
|
||||
## Quick Start: Your First Augmentation
|
||||
|
||||
|
|
@ -12,12 +55,12 @@ export class MyFirstAugmentation extends BaseAugmentation {
|
|||
readonly timing = 'after' as const // When to run: before | after | both
|
||||
readonly operations = ['add'] as const // Which operations to hook
|
||||
readonly priority = 10 // Execution order (lower = first)
|
||||
|
||||
|
||||
protected async onInit(): Promise<void> {
|
||||
// Initialize your augmentation
|
||||
console.log('MyFirstAugmentation initialized!')
|
||||
}
|
||||
|
||||
|
||||
async execute<T = any>(
|
||||
operation: string,
|
||||
params: any,
|
||||
|
|
@ -26,12 +69,18 @@ export class MyFirstAugmentation extends BaseAugmentation {
|
|||
// Your augmentation logic
|
||||
if (operation === 'add') {
|
||||
console.log('Noun added:', params.noun)
|
||||
|
||||
// v4.0.0: Access metadata correctly
|
||||
if (params.noun?.metadata) {
|
||||
console.log('Noun type:', params.noun.metadata.noun) // Required field
|
||||
}
|
||||
|
||||
// You can access the brain instance
|
||||
const stats = await context?.brain.getStats()
|
||||
console.log('Total nouns:', stats.totalNouns)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
protected async onShutdown(): Promise<void> {
|
||||
// Cleanup
|
||||
console.log('MyFirstAugmentation shutting down')
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue