feat(v4.0.0): Complete metadata/vector separation architecture with Azure support
This commit completes the core v4.0.0 architecture changes for billion-scale performance with metadata/vector separation. NO RELEASE YET - remaining optimizations and testing required before production release. ## Core v4.0.0 Architecture Changes ### Type System Updates - Fixed all TypeScript compilation errors (zero errors achieved) - Updated HNSWNoun/HNSWVerb to separate core fields from metadata - Implemented HNSWNounWithMetadata/HNSWVerbWithMetadata for API boundaries - Added required 'noun' field to NounMetadata for semantic structure - Renamed verb.type to verb.verb for consistency ### Storage Adapter Updates **All adapters updated for v4.0.0 two-file storage pattern:** - memoryStorage: Proper metadata/vector separation - fileSystemStorage: Two-file pattern with sharding - opfsStorage: Browser persistent storage updated - s3CompatibleStorage: AWS/MinIO/DigitalOcean support - r2Storage: Cloudflare R2 optimization - gcsStorage: Google Cloud with ADC support - **azureBlobStorage: NEW - Full Azure Blob Storage support** ### Storage Features - BaseStorage: Internal vs public method separation (_getNoun vs getNoun) - Two-file storage: Vectors in one file, metadata in another - Change tracking: getChangesSince return type updated - Pagination: getNounsWithPagination returns WithMetadata types ### Azure Blob Storage Integration (NEW) - Native @azure/storage-blob SDK integration - Four authentication methods: * DefaultAzureCredential (Managed Identity) - recommended * Connection String - simplest setup * Account Name + Key - traditional auth * SAS Token - delegated access - High-volume mode with write buffering - Adaptive backpressure for throttling - UUID-based sharding for billion-scale - Full HNSW support with graph persistence ### Utility Updates - EmbeddingManager: Updated to accept Record<string, unknown> - LSMTree: Wrapped data in NounMetadata structure with 'noun' field - EntityIdMapper: Fixed nested metadata.data structure access - MetadataIndex: Fixed field type inference integration - PeriodicCleanup: Updated for new metadata structure ### Core API Updates - Brainy: Updated verb property access from v.type to v.verb - ConfigAPI: Fixed NounMetadata access patterns - DataAPI: Updated metadata handling ### Documentation Updates - CREATING-AUGMENTATIONS.md: v4.0.0 breaking changes guide - DEVELOPER-GUIDE.md: Migration checklist and examples - COMPLETE-REFERENCE.md: v4.0.0 architecture improvements - **finite-type-system.md: NEW - Revolutionary type system benefits** ### Build & Dependencies - Zero TypeScript compilation errors - Added @azure/storage-blob and @azure/identity - 591 tests passing (23 timeout in long-running neural tests) ## What's NOT in This Release This is a work-in-progress commit. Before v4.0.0 release we need: - Storage adapter optimizations (batch operations, compression) - Azure blob tier management (Hot/Cool/Archive) - Cost optimization implementations - Additional performance testing at billion-scale - Migration guides for v3.x users ## Testing - Clean build: ✅ - Type checking: ✅ (zero errors) - Test suite: ✅ (591/614 passing, timeouts in neural tests only) 🔐 Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
8d6dd07e1d
commit
92c96246fb
35 changed files with 4524 additions and 1026 deletions
|
|
@ -362,8 +362,11 @@ export class LSMTree {
|
|||
const storageKey = `${this.config.storagePrefix}-${sstable.metadata.id}`
|
||||
|
||||
await this.storage.saveMetadata(storageKey, {
|
||||
type: 'lsm-sstable',
|
||||
data: Array.from(data) // Convert Uint8Array to number[] for JSON storage
|
||||
noun: 'thing', // Required for NounMetadata
|
||||
data: {
|
||||
type: 'lsm-sstable',
|
||||
data: Array.from(data) // Convert Uint8Array to number[] for JSON storage
|
||||
}
|
||||
})
|
||||
|
||||
// Add to L0 SSTables
|
||||
|
|
@ -424,8 +427,11 @@ export class LSMTree {
|
|||
const storageKey = `${this.config.storagePrefix}-${merged.metadata.id}`
|
||||
|
||||
await this.storage.saveMetadata(storageKey, {
|
||||
type: 'lsm-sstable',
|
||||
data: Array.from(data)
|
||||
noun: 'thing', // Required for NounMetadata
|
||||
data: {
|
||||
type: 'lsm-sstable',
|
||||
data: Array.from(data)
|
||||
}
|
||||
})
|
||||
|
||||
// Delete old SSTables from storage
|
||||
|
|
@ -500,12 +506,13 @@ export class LSMTree {
|
|||
*/
|
||||
private async loadManifest(): Promise<void> {
|
||||
try {
|
||||
const data = await this.storage.getMetadata(`${this.config.storagePrefix}-manifest`)
|
||||
const metadata = await this.storage.getMetadata(`${this.config.storagePrefix}-manifest`)
|
||||
|
||||
if (data) {
|
||||
if (metadata && metadata.data) {
|
||||
const data = metadata.data as any
|
||||
this.manifest.sstables = new Map(Object.entries(data.sstables || {}))
|
||||
this.manifest.lastCompaction = data.lastCompaction || Date.now()
|
||||
this.manifest.totalRelationships = data.totalRelationships || 0
|
||||
this.manifest.lastCompaction = (data.lastCompaction as number) || Date.now()
|
||||
this.manifest.totalRelationships = (data.totalRelationships as number) || 0
|
||||
|
||||
// Load SSTables from storage
|
||||
await this.loadSSTables()
|
||||
|
|
@ -525,17 +532,20 @@ export class LSMTree {
|
|||
const loadPromise = (async () => {
|
||||
try {
|
||||
const storageKey = `${this.config.storagePrefix}-${sstableId}`
|
||||
const data = await this.storage.getMetadata(storageKey)
|
||||
const metadata = await this.storage.getMetadata(storageKey)
|
||||
|
||||
if (data && data.type === 'lsm-sstable') {
|
||||
// Convert number[] back to Uint8Array
|
||||
const uint8Data = new Uint8Array(data.data)
|
||||
const sstable = SSTable.deserialize(uint8Data)
|
||||
if (metadata && metadata.data) {
|
||||
const data = metadata.data as any
|
||||
if (data.type === 'lsm-sstable') {
|
||||
// Convert number[] back to Uint8Array
|
||||
const uint8Data = new Uint8Array(data.data)
|
||||
const sstable = SSTable.deserialize(uint8Data)
|
||||
|
||||
if (!this.sstablesByLevel.has(level)) {
|
||||
this.sstablesByLevel.set(level, [])
|
||||
if (!this.sstablesByLevel.has(level)) {
|
||||
this.sstablesByLevel.set(level, [])
|
||||
}
|
||||
this.sstablesByLevel.get(level)!.push(sstable)
|
||||
}
|
||||
this.sstablesByLevel.get(level)!.push(sstable)
|
||||
}
|
||||
} catch (error) {
|
||||
prodLog.warn(`LSMTree: Failed to load SSTable ${sstableId}`, error)
|
||||
|
|
@ -554,15 +564,16 @@ export class LSMTree {
|
|||
*/
|
||||
private async saveManifest(): Promise<void> {
|
||||
try {
|
||||
const manifestData = {
|
||||
sstables: Object.fromEntries(this.manifest.sstables),
|
||||
lastCompaction: this.manifest.lastCompaction,
|
||||
totalRelationships: this.manifest.totalRelationships
|
||||
}
|
||||
|
||||
await this.storage.saveMetadata(
|
||||
`${this.config.storagePrefix}-manifest`,
|
||||
manifestData
|
||||
{
|
||||
noun: 'thing', // Required for NounMetadata
|
||||
data: {
|
||||
sstables: Object.fromEntries(this.manifest.sstables),
|
||||
lastCompaction: this.manifest.lastCompaction,
|
||||
totalRelationships: this.manifest.totalRelationships
|
||||
}
|
||||
}
|
||||
)
|
||||
} catch (error) {
|
||||
prodLog.error('LSMTree: Failed to save manifest', error)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue