brainy/src/augmentations
David Snelling 92c96246fb feat(v4.0.0): Complete metadata/vector separation architecture with Azure support
This commit completes the core v4.0.0 architecture changes for billion-scale
performance with metadata/vector separation. NO RELEASE YET - remaining optimizations
and testing required before production release.

## Core v4.0.0 Architecture Changes

### Type System Updates
- Fixed all TypeScript compilation errors (zero errors achieved)
- Updated HNSWNoun/HNSWVerb to separate core fields from metadata
- Implemented HNSWNounWithMetadata/HNSWVerbWithMetadata for API boundaries
- Added required 'noun' field to NounMetadata for semantic structure
- Renamed verb.type to verb.verb for consistency

### Storage Adapter Updates
**All adapters updated for v4.0.0 two-file storage pattern:**
- memoryStorage: Proper metadata/vector separation
- fileSystemStorage: Two-file pattern with sharding
- opfsStorage: Browser persistent storage updated
- s3CompatibleStorage: AWS/MinIO/DigitalOcean support
- r2Storage: Cloudflare R2 optimization
- gcsStorage: Google Cloud with ADC support
- **azureBlobStorage: NEW - Full Azure Blob Storage support**

### Storage Features
- BaseStorage: Internal vs public method separation (_getNoun vs getNoun)
- Two-file storage: Vectors in one file, metadata in another
- Change tracking: getChangesSince return type updated
- Pagination: getNounsWithPagination returns WithMetadata types

### Azure Blob Storage Integration (NEW)
- Native @azure/storage-blob SDK integration
- Four authentication methods:
  * DefaultAzureCredential (Managed Identity) - recommended
  * Connection String - simplest setup
  * Account Name + Key - traditional auth
  * SAS Token - delegated access
- High-volume mode with write buffering
- Adaptive backpressure for throttling
- UUID-based sharding for billion-scale
- Full HNSW support with graph persistence

### Utility Updates
- EmbeddingManager: Updated to accept Record<string, unknown>
- LSMTree: Wrapped data in NounMetadata structure with 'noun' field
- EntityIdMapper: Fixed nested metadata.data structure access
- MetadataIndex: Fixed field type inference integration
- PeriodicCleanup: Updated for new metadata structure

### Core API Updates
- Brainy: Updated verb property access from v.type to v.verb
- ConfigAPI: Fixed NounMetadata access patterns
- DataAPI: Updated metadata handling

### Documentation Updates
- CREATING-AUGMENTATIONS.md: v4.0.0 breaking changes guide
- DEVELOPER-GUIDE.md: Migration checklist and examples
- COMPLETE-REFERENCE.md: v4.0.0 architecture improvements
- **finite-type-system.md: NEW - Revolutionary type system benefits**

### Build & Dependencies
- Zero TypeScript compilation errors
- Added @azure/storage-blob and @azure/identity
- 591 tests passing (23 timeout in long-running neural tests)

## What's NOT in This Release
This is a work-in-progress commit. Before v4.0.0 release we need:
- Storage adapter optimizations (batch operations, compression)
- Azure blob tier management (Hot/Cool/Archive)
- Cost optimization implementations
- Additional performance testing at billion-scale
- Migration guides for v3.x users

## Testing
- Clean build: 
- Type checking:  (zero errors)
- Test suite:  (591/614 passing, timeouts in neural tests only)

🔐 Generated with Claude Code
https://claude.com/claude-code

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-17 12:29:27 -07:00
..
discovery feat: add browser environment compatibility support 2025-09-17 15:48:02 -07:00
display feat: add complete silent mode for TUI applications 2025-09-16 13:18:49 -07:00
intelligentImport feat: add intelligent import for CSV, Excel, and PDF files 2025-10-01 16:51:03 -07:00
typeMatching perf: pre-compute type embeddings at build time (zero runtime cost) 2025-10-09 18:08:57 -07:00
apiServerAugmentation.ts feat: remove legacy ImportManager, standardize getStats() API 2025-10-09 11:40:31 -07:00
auditLogAugmentation.ts feat: add browser environment compatibility support 2025-09-17 15:48:02 -07:00
AugmentationMetadataContract.ts feat: comprehensive metadata namespace architecture and cleanup system 2025-08-27 15:38:48 -07:00
batchProcessingAugmentation.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
brainyAugmentation.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
cacheAugmentation.ts feat: add complete silent mode for TUI applications 2025-09-16 13:18:49 -07:00
conduitAugmentations.ts feat: modernize API architecture and deprecation handling 2025-09-17 11:54:20 -07:00
configResolver.ts feat: add browser environment compatibility support 2025-09-17 14:53:54 -07:00
connectionPoolAugmentation.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
defaultAugmentations.ts feat: add intelligent import for CSV, Excel, and PDF files 2025-10-01 16:51:03 -07:00
discovery.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
entityRegistryAugmentation.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
indexAugmentation.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
intelligentVerbScoringAugmentation.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
manifest.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
metadataEnforcer.ts feat: comprehensive metadata namespace architecture and cleanup system 2025-08-27 15:38:48 -07:00
metricsAugmentation.ts feat: add complete silent mode for TUI applications 2025-09-16 13:18:49 -07:00
monitoringAugmentation.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
neuralImport.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
rateLimitAugmentation.ts feat: add distributed scaling and enterprise features for v3 2025-09-08 14:26:09 -07:00
README.md feat: add intelligent import for CSV, Excel, and PDF files 2025-10-01 16:51:03 -07:00
requestDeduplicatorAugmentation.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
storageAugmentation.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
storageAugmentations.ts feat(v4.0.0): Complete metadata/vector separation architecture with Azure support 2025-10-17 12:29:27 -07:00
synapseAugmentation.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
universalDisplayAugmentation.ts feat: add complete silent mode for TUI applications 2025-09-16 13:18:49 -07:00

Brainy Logo

Brainy Augmentations

This directory contains the augmentation implementations for Brainy. Augmentations are pluggable components that extend Brainy's functionality in various ways.

Available Augmentations

Core Augmentations

IntelligentImportAugmentation

Automatically detects and processes CSV, Excel, and PDF files with intelligent extraction. This augmentation is enabled by default and provides:

  • CSV Support: Auto-detection of encoding, delimiters, and field types
  • Excel Support: Multi-sheet extraction with metadata preservation
  • PDF Support: Text extraction, table detection, and metadata extraction
  • Type Inference: Automatically infers data types (string, number, boolean, date)
  • Neural Integration: Seamlessly integrates with entity extraction and relationship detection
import { Brainy } from '@soulcraft/brainy'

const brain = new Brainy({
  intelligentImport: {
    enableCSV: true,
    enableExcel: true,
    enablePDF: true,
    maxFileSize: 100 * 1024 * 1024  // 100MB
  }
})
await brain.init()

// Import CSV with auto-detection
await brain.import('customers.csv')

// Import Excel with specific sheets
await brain.import('sales-data.xlsx', {
  excelSheets: ['Q1', 'Q2']
})

// Import PDF with table extraction
await brain.import('report.pdf', {
  pdfExtractTables: true
})

See: Import Anything Guide | Example

Conduit Augmentations

Conduit augmentations provide data synchronization between Brainy instances.

WebSocketConduitAugmentation

A conduit augmentation that syncs Brainy instances using WebSockets. This is used for syncing between browsers and servers, or between servers.

import { createConduitAugmentation, augmentationPipeline } from '@soulcraft/brainy'

// Create a WebSocket conduit augmentation
const wsConduit = await createConduitAugmentation('websocket', 'my-websocket-sync')

// Register the augmentation with the pipeline
augmentationPipeline.register(wsConduit)

// Connect to another Brainy instance
const connectionResult = await wsConduit.establishConnection(
  'wss://your-websocket-server.com/brainy-sync',
  { protocols: 'brainy-sync' }
)

WebRTCConduitAugmentation

A conduit augmentation that syncs Brainy instances using WebRTC. This is used for direct peer-to-peer syncing between browsers.

import { createConduitAugmentation, augmentationPipeline } from '@soulcraft/brainy'

// Create a WebRTC conduit augmentation
const webrtcConduit = await createConduitAugmentation('webrtc', 'my-webrtc-sync')

// Register the augmentation with the pipeline
augmentationPipeline.register(webrtcConduit)

// Connect to a peer
const connectionResult = await webrtcConduit.establishConnection(
  'peer-id-to-connect-to',
  {
    signalServerUrl: 'wss://your-signal-server.com',
    localPeerId: 'my-peer-id',
    iceServers: [{ urls: 'stun:stun.l.google.com:19302' }]
  }
)

ServerSearchConduitAugmentation

A specialized conduit augmentation that provides functionality for searching a server-hosted Brainy instance and storing results locally. This allows you to:

  • Search a server-hosted Brainy instance from a browser
  • Store the search results in a local Brainy instance
  • Perform further searches against the local instance without needing to query the server again
  • Add data to both local and server instances
import {
  ServerSearchConduitAugmentation,
  createServerSearchAugmentations,
  augmentationPipeline
} from '@soulcraft/brainy'

// Using the factory function (recommended)
const { conduit, activation, connection } = await createServerSearchAugmentations(
  'wss://your-brainy-server.com/ws',
  { protocols: 'brainy-sync' }
)

// Register the augmentations with the pipeline
augmentationPipeline.register(conduit)
augmentationPipeline.register(activation)

// Search the server and store results locally
const serverSearchResult = await conduit.searchServer(
  connection.connectionId,
  'your search query',
  5 // limit
)

// Search the local instance
const localSearchResult = await conduit.searchLocal('your search query', 5)

// Perform a combined search (local first, then server if needed)
const combinedSearchResult = await conduit.searchCombined(
  connection.connectionId,
  'your search query',
  5
)

// Add data to both local and server
const addResult = await conduit.addToBoth(
  connection.connectionId,
  'Text to add',
  { /* metadata */ }
)

Activation Augmentations

Activation augmentations dictate how Brainy initiates actions, responses, or data manipulations.

ServerSearchActivationAugmentation

An activation augmentation that provides actions for server search functionality. This works in conjunction with the ServerSearchConduitAugmentation to provide a complete solution for browser-server search.

import {
  ServerSearchActivationAugmentation,
  createServerSearchAugmentations,
  augmentationPipeline
} from '@soulcraft/brainy'

// Using the factory function (recommended)
const { conduit, activation, connection } = await createServerSearchAugmentations(
  'wss://your-brainy-server.com/ws',
  { protocols: 'brainy-sync' }
)

// Register the augmentations with the pipeline
augmentationPipeline.register(conduit)
augmentationPipeline.register(activation)

// Use the activation augmentation to search the server
const serverSearchAction = activation.triggerAction('searchServer', {
  connectionId: connection.connectionId,
  query: 'your search query',
  limit: 5
})

if (serverSearchAction.success) {
  // The data property contains a promise that will resolve to the search results
  const serverSearchResult = await serverSearchAction.data
  console.log('Server search results:', serverSearchResult)
}

// Other available actions:
// - 'connectToServer': Connect to a server
// - 'searchLocal': Search the local instance
// - 'searchCombined': Search both local and server
// - 'addToBoth': Add data to both local and server

Using the Augmentation Pipeline

The augmentation pipeline provides a way to execute augmentations based on their type.

import { augmentationPipeline } from '@soulcraft/brainy'

// Execute a conduit augmentation
const conduitResults = await augmentationPipeline.executeConduitPipeline(
  'methodName',
  [arg1, arg2, ...],
  { /* options */ }
)

// Execute an activation augmentation
const activationResults = await augmentationPipeline.executeActivationPipeline(
  'methodName',
  [arg1, arg2, ...],
  { /* options */ }
)

Creating Custom Augmentations

To create a custom augmentation, implement one of the augmentation interfaces:

  • ISenseAugmentation: For processing raw data
  • IConduitAugmentation: For data synchronization
  • ICognitionAugmentation: For reasoning and inference
  • IMemoryAugmentation: For data storage
  • IPerceptionAugmentation: For data interpretation and visualization
  • IDialogAugmentation: For natural language processing
  • IActivationAugmentation: For triggering actions

Example:

import { AugmentationType, IActivationAugmentation } from '@soulcraft/brainy'

class MyCustomActivation implements IActivationAugmentation {
  readonly
  name = 'my-custom-activation'
  readonly
  description = 'My custom activation augmentation'
  enabled = true

  getType(): AugmentationType {
    return AugmentationType.ACTIVATION
  }

  async initialize(): Promise<void> {
    // Initialization code
  }

  async shutDown(): Promise<void> {
    // Cleanup code
  }

  async getStatus(): Promise<'active' | 'inactive' | 'error'> {
    return 'active'
  }

  triggerAction(actionName: string, parameters

?:

  Record<string, unknown>

):

  AugmentationResponse<unknown> {
    // Implementation
  }

  generateOutput(knowledgeId: string, format: string): AugmentationResponse<string | Record<string, unknown

>> {
  // Implementation
}

interactExternal(systemId
:
string, payload
:
Record < string, unknown >
):
AugmentationResponse < unknown > {
  // Implementation
}
}