feat(v4.0.0): Complete metadata/vector separation architecture with Azure support

This commit completes the core v4.0.0 architecture changes for billion-scale
performance with metadata/vector separation. NO RELEASE YET - remaining optimizations
and testing required before production release.

## Core v4.0.0 Architecture Changes

### Type System Updates
- Fixed all TypeScript compilation errors (zero errors achieved)
- Updated HNSWNoun/HNSWVerb to separate core fields from metadata
- Implemented HNSWNounWithMetadata/HNSWVerbWithMetadata for API boundaries
- Added required 'noun' field to NounMetadata for semantic structure
- Renamed verb.type to verb.verb for consistency

### Storage Adapter Updates
**All adapters updated for v4.0.0 two-file storage pattern:**
- memoryStorage: Proper metadata/vector separation
- fileSystemStorage: Two-file pattern with sharding
- opfsStorage: Browser persistent storage updated
- s3CompatibleStorage: AWS/MinIO/DigitalOcean support
- r2Storage: Cloudflare R2 optimization
- gcsStorage: Google Cloud with ADC support
- **azureBlobStorage: NEW - Full Azure Blob Storage support**

### Storage Features
- BaseStorage: Internal vs public method separation (_getNoun vs getNoun)
- Two-file storage: Vectors in one file, metadata in another
- Change tracking: getChangesSince return type updated
- Pagination: getNounsWithPagination returns WithMetadata types

### Azure Blob Storage Integration (NEW)
- Native @azure/storage-blob SDK integration
- Four authentication methods:
  * DefaultAzureCredential (Managed Identity) - recommended
  * Connection String - simplest setup
  * Account Name + Key - traditional auth
  * SAS Token - delegated access
- High-volume mode with write buffering
- Adaptive backpressure for throttling
- UUID-based sharding for billion-scale
- Full HNSW support with graph persistence

### Utility Updates
- EmbeddingManager: Updated to accept Record<string, unknown>
- LSMTree: Wrapped data in NounMetadata structure with 'noun' field
- EntityIdMapper: Fixed nested metadata.data structure access
- MetadataIndex: Fixed field type inference integration
- PeriodicCleanup: Updated for new metadata structure

### Core API Updates
- Brainy: Updated verb property access from v.type to v.verb
- ConfigAPI: Fixed NounMetadata access patterns
- DataAPI: Updated metadata handling

### Documentation Updates
- CREATING-AUGMENTATIONS.md: v4.0.0 breaking changes guide
- DEVELOPER-GUIDE.md: Migration checklist and examples
- COMPLETE-REFERENCE.md: v4.0.0 architecture improvements
- **finite-type-system.md: NEW - Revolutionary type system benefits**

### Build & Dependencies
- Zero TypeScript compilation errors
- Added @azure/storage-blob and @azure/identity
- 591 tests passing (23 timeout in long-running neural tests)

## What's NOT in This Release
This is a work-in-progress commit. Before v4.0.0 release we need:
- Storage adapter optimizations (batch operations, compression)
- Azure blob tier management (Hot/Cool/Archive)
- Cost optimization implementations
- Additional performance testing at billion-scale
- Migration guides for v3.x users

## Testing
- Clean build: 
- Type checking:  (zero errors)
- Test suite:  (591/614 passing, timeouts in neural tests only)

🔐 Generated with Claude Code
https://claude.com/claude-code

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
David Snelling 2025-10-17 12:29:27 -07:00
parent 8d6dd07e1d
commit 92c96246fb
35 changed files with 4524 additions and 1026 deletions

View file

@ -362,8 +362,11 @@ export class LSMTree {
const storageKey = `${this.config.storagePrefix}-${sstable.metadata.id}`
await this.storage.saveMetadata(storageKey, {
type: 'lsm-sstable',
data: Array.from(data) // Convert Uint8Array to number[] for JSON storage
noun: 'thing', // Required for NounMetadata
data: {
type: 'lsm-sstable',
data: Array.from(data) // Convert Uint8Array to number[] for JSON storage
}
})
// Add to L0 SSTables
@ -424,8 +427,11 @@ export class LSMTree {
const storageKey = `${this.config.storagePrefix}-${merged.metadata.id}`
await this.storage.saveMetadata(storageKey, {
type: 'lsm-sstable',
data: Array.from(data)
noun: 'thing', // Required for NounMetadata
data: {
type: 'lsm-sstable',
data: Array.from(data)
}
})
// Delete old SSTables from storage
@ -500,12 +506,13 @@ export class LSMTree {
*/
private async loadManifest(): Promise<void> {
try {
const data = await this.storage.getMetadata(`${this.config.storagePrefix}-manifest`)
const metadata = await this.storage.getMetadata(`${this.config.storagePrefix}-manifest`)
if (data) {
if (metadata && metadata.data) {
const data = metadata.data as any
this.manifest.sstables = new Map(Object.entries(data.sstables || {}))
this.manifest.lastCompaction = data.lastCompaction || Date.now()
this.manifest.totalRelationships = data.totalRelationships || 0
this.manifest.lastCompaction = (data.lastCompaction as number) || Date.now()
this.manifest.totalRelationships = (data.totalRelationships as number) || 0
// Load SSTables from storage
await this.loadSSTables()
@ -525,17 +532,20 @@ export class LSMTree {
const loadPromise = (async () => {
try {
const storageKey = `${this.config.storagePrefix}-${sstableId}`
const data = await this.storage.getMetadata(storageKey)
const metadata = await this.storage.getMetadata(storageKey)
if (data && data.type === 'lsm-sstable') {
// Convert number[] back to Uint8Array
const uint8Data = new Uint8Array(data.data)
const sstable = SSTable.deserialize(uint8Data)
if (metadata && metadata.data) {
const data = metadata.data as any
if (data.type === 'lsm-sstable') {
// Convert number[] back to Uint8Array
const uint8Data = new Uint8Array(data.data)
const sstable = SSTable.deserialize(uint8Data)
if (!this.sstablesByLevel.has(level)) {
this.sstablesByLevel.set(level, [])
if (!this.sstablesByLevel.has(level)) {
this.sstablesByLevel.set(level, [])
}
this.sstablesByLevel.get(level)!.push(sstable)
}
this.sstablesByLevel.get(level)!.push(sstable)
}
} catch (error) {
prodLog.warn(`LSMTree: Failed to load SSTable ${sstableId}`, error)
@ -554,15 +564,16 @@ export class LSMTree {
*/
private async saveManifest(): Promise<void> {
try {
const manifestData = {
sstables: Object.fromEntries(this.manifest.sstables),
lastCompaction: this.manifest.lastCompaction,
totalRelationships: this.manifest.totalRelationships
}
await this.storage.saveMetadata(
`${this.config.storagePrefix}-manifest`,
manifestData
{
noun: 'thing', // Required for NounMetadata
data: {
sstables: Object.fromEntries(this.manifest.sstables),
lastCompaction: this.manifest.lastCompaction,
totalRelationships: this.manifest.totalRelationships
}
}
)
} catch (error) {
prodLog.error('LSMTree: Failed to save manifest', error)