feat(v4.0.0): Complete metadata/vector separation architecture with Azure support
This commit completes the core v4.0.0 architecture changes for billion-scale performance with metadata/vector separation. NO RELEASE YET - remaining optimizations and testing required before production release. ## Core v4.0.0 Architecture Changes ### Type System Updates - Fixed all TypeScript compilation errors (zero errors achieved) - Updated HNSWNoun/HNSWVerb to separate core fields from metadata - Implemented HNSWNounWithMetadata/HNSWVerbWithMetadata for API boundaries - Added required 'noun' field to NounMetadata for semantic structure - Renamed verb.type to verb.verb for consistency ### Storage Adapter Updates **All adapters updated for v4.0.0 two-file storage pattern:** - memoryStorage: Proper metadata/vector separation - fileSystemStorage: Two-file pattern with sharding - opfsStorage: Browser persistent storage updated - s3CompatibleStorage: AWS/MinIO/DigitalOcean support - r2Storage: Cloudflare R2 optimization - gcsStorage: Google Cloud with ADC support - **azureBlobStorage: NEW - Full Azure Blob Storage support** ### Storage Features - BaseStorage: Internal vs public method separation (_getNoun vs getNoun) - Two-file storage: Vectors in one file, metadata in another - Change tracking: getChangesSince return type updated - Pagination: getNounsWithPagination returns WithMetadata types ### Azure Blob Storage Integration (NEW) - Native @azure/storage-blob SDK integration - Four authentication methods: * DefaultAzureCredential (Managed Identity) - recommended * Connection String - simplest setup * Account Name + Key - traditional auth * SAS Token - delegated access - High-volume mode with write buffering - Adaptive backpressure for throttling - UUID-based sharding for billion-scale - Full HNSW support with graph persistence ### Utility Updates - EmbeddingManager: Updated to accept Record<string, unknown> - LSMTree: Wrapped data in NounMetadata structure with 'noun' field - EntityIdMapper: Fixed nested metadata.data structure access - MetadataIndex: Fixed field type inference integration - PeriodicCleanup: Updated for new metadata structure ### Core API Updates - Brainy: Updated verb property access from v.type to v.verb - ConfigAPI: Fixed NounMetadata access patterns - DataAPI: Updated metadata handling ### Documentation Updates - CREATING-AUGMENTATIONS.md: v4.0.0 breaking changes guide - DEVELOPER-GUIDE.md: Migration checklist and examples - COMPLETE-REFERENCE.md: v4.0.0 architecture improvements - **finite-type-system.md: NEW - Revolutionary type system benefits** ### Build & Dependencies - Zero TypeScript compilation errors - Added @azure/storage-blob and @azure/identity - 591 tests passing (23 timeout in long-running neural tests) ## What's NOT in This Release This is a work-in-progress commit. Before v4.0.0 release we need: - Storage adapter optimizations (batch operations, compression) - Azure blob tier management (Hot/Cool/Archive) - Cost optimization implementations - Additional performance testing at billion-scale - Migration guides for v3.x users ## Testing - Clean build: ✅ - Type checking: ✅ (zero errors) - Test suite: ✅ (591/614 passing, timeouts in neural tests only) 🔐 Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
8d6dd07e1d
commit
92c96246fb
35 changed files with 4524 additions and 1026 deletions
|
|
@ -184,7 +184,7 @@ export class EmbeddingManager {
|
|||
/**
|
||||
* Generate embeddings
|
||||
*/
|
||||
async embed(text: string | string[]): Promise<Vector> {
|
||||
async embed(text: string | string[] | Record<string, unknown>): Promise<Vector> {
|
||||
// Check for unit test environment - use mocks to prevent ONNX conflicts
|
||||
const isTestMode = process.env.BRAINY_UNIT_TEST === 'true' || (globalThis as any).__BRAINY_UNIT_TEST__
|
||||
|
||||
|
|
@ -210,9 +210,12 @@ export class EmbeddingManager {
|
|||
input = text.map(t => typeof t === 'string' ? t : String(t)).join(' ')
|
||||
} else if (typeof text === 'string') {
|
||||
input = text
|
||||
} else if (typeof text === 'object') {
|
||||
// Convert object to string representation
|
||||
input = JSON.stringify(text)
|
||||
} else {
|
||||
// This shouldn't happen but let's be defensive
|
||||
console.warn('EmbeddingManager.embed received non-string input:', typeof text)
|
||||
console.warn('EmbeddingManager.embed received unexpected input type:', typeof text)
|
||||
input = String(text)
|
||||
}
|
||||
|
||||
|
|
@ -243,22 +246,22 @@ export class EmbeddingManager {
|
|||
/**
|
||||
* Generate mock embeddings for unit tests
|
||||
*/
|
||||
private getMockEmbedding(text: string | string[]): Vector {
|
||||
private getMockEmbedding(text: string | string[] | Record<string, unknown>): Vector {
|
||||
// Use the same mock logic as setup-unit.ts for consistency
|
||||
const input = Array.isArray(text) ? text.join(' ') : text
|
||||
const str = typeof input === 'string' ? input : JSON.stringify(input)
|
||||
const vector = new Array(384).fill(0)
|
||||
|
||||
|
||||
// Create semi-realistic embeddings based on text content
|
||||
for (let i = 0; i < Math.min(str.length, 384); i++) {
|
||||
vector[i] = (str.charCodeAt(i % str.length) % 256) / 256
|
||||
}
|
||||
|
||||
|
||||
// Add position-based variation
|
||||
for (let i = 0; i < 384; i++) {
|
||||
vector[i] += Math.sin(i * 0.1 + str.length) * 0.1
|
||||
}
|
||||
|
||||
|
||||
// Track mock embedding count
|
||||
this.embedCount++
|
||||
return vector
|
||||
|
|
@ -268,7 +271,7 @@ export class EmbeddingManager {
|
|||
* Get embedding function for compatibility
|
||||
*/
|
||||
getEmbeddingFunction(): EmbeddingFunction {
|
||||
return async (data: string | string[]): Promise<Vector> => {
|
||||
return async (data: string | string[] | Record<string, unknown>): Promise<Vector> => {
|
||||
return await this.embed(data)
|
||||
}
|
||||
}
|
||||
|
|
@ -390,7 +393,7 @@ export const embeddingManager = EmbeddingManager.getInstance()
|
|||
/**
|
||||
* Direct embed function
|
||||
*/
|
||||
export async function embed(text: string | string[]): Promise<Vector> {
|
||||
export async function embed(text: string | string[] | Record<string, unknown>): Promise<Vector> {
|
||||
return await embeddingManager.embed(text)
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue