soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

No commit message can be generated for the provided diff as it appears to be incomplete. Please provide a more descriptive or complete diff to enable me to generate a relevant commit message for you.

Browse source
This commit is contained in:
David Snelling 2025-07-28 16:25:11 -07:00
parent 94c88e128c
commit 524be5010b
5 changed files with 430 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.gitignore vendored
View file

@ -46,6 +46,7 @@ Thumbs.db
# Test files
/test-worker.js
/test-node24-worker.js
/test-results.json
# Generated files
/encoded-image.html

1
.npmignore
View file

@ -38,3 +38,4 @@ scalingStrategy.md
npm-debug.log*
yarn-debug.log*
yarn-error.log*
test-results.json

182
REALTIME_UPDATES.md Normal file
View file

@ -0,0 +1,182 @@
# Real-time Updates in Brainy
This document explains the real-time update features in Brainy, which ensure that the in-memory index and statistics are always up-to-date with the latest data in storage.
## Overview
When running Brainy inside a web service with data being constantly added in a stream (using S3 or any other storage option), the new data needs to be searchable in real-time. The real-time update feature periodically checks for new data in storage and updates the in-memory index and statistics accordingly.
## Configuration
Real-time updates can be configured when creating a BrainyData instance:
```typescript
import { BrainyData } from '@soulcraft/brainy'
const db = new BrainyData({
// ... other configuration options ...
// Real-time update configuration
realtimeUpdates: {
// Whether to enable automatic updates (default: false)
enabled: true,
// The interval in milliseconds at which to check for updates (default: 30000 - 30 seconds)
interval: 10000, // 10 seconds
// Whether to update statistics when checking for updates (default: true)
updateStatistics: true,
// Whether to update the index when checking for updates (default: true)
updateIndex: true
}
})
```
## Runtime Control
Real-time updates can also be controlled at runtime:
### Enable Real-time Updates
```typescript
// Enable with default configuration
db.enableRealtimeUpdates()
// Enable with custom configuration
db.enableRealtimeUpdates({
interval: 5000, // 5 seconds
updateStatistics: true,
updateIndex: true
})
```
### Disable Real-time Updates
```typescript
db.disableRealtimeUpdates()
```
### Get Current Configuration
```typescript
const config = db.getRealtimeUpdateConfig()
console.log(`Real-time updates enabled: ${config.enabled}`)
console.log(`Update interval: ${config.interval}ms`)
```
### Manual Update Check
You can also manually check for updates at any time, regardless of whether automatic updates are enabled:
```typescript
await db.checkForUpdatesNow()
```
## How It Works
When real-time updates are enabled, Brainy will:
1. Periodically check for new data in storage at the specified interval.
2. If new data is found, update the in-memory index with the new data.
3. Update the statistics to reflect the latest data.
This ensures that search operations and statistics always reflect the latest data, even when data is being added by external processes.
### Incremental Updates
The real-time update mechanism is designed to be efficient and only processes new data:
- **Incremental Indexing**: Brainy only adds new items to the index that aren't already there, rather than reloading the entire index. It compares the IDs of items in storage with those already in the index to identify only the new items that need to be added.
- **Efficient Statistics Updates**: Statistics are updated incrementally as well, with changes being batched for performance.
### Handling Large Indices
Brainy is designed to handle indices that are too large to fit entirely in memory:
- **Optimized HNSW Implementation**: Brainy uses the `HNSWIndexOptimized` class which supports large datasets through:
- **Product Quantization**: Compresses vectors to reduce memory usage while maintaining search quality
- **Disk-Based Storage**: Can offload parts of the index to disk when memory is constrained
- **Memory Management**: When the index grows too large for available memory:
1. The most frequently accessed items are kept in memory for fast access
2. Less frequently accessed items may be stored on disk and loaded when needed
3. The system automatically balances memory usage based on access patterns
- **Configurable Trade-offs**: You can configure the balance between memory usage and performance through the HNSW configuration options when creating the database.
## Best Practices
- For high-volume data streams, set a reasonable update interval to balance real-time updates with performance.
- If you only need occasional updates, disable automatic updates and use `checkForUpdatesNow()` when needed.
- For web services with multiple instances, each instance will maintain its own in-memory index and statistics.
## Compatibility
Real-time updates work with all storage options supported by Brainy, including:
- File system storage
- Memory storage
- S3 storage
- Custom storage adapters
## Example: Web Service with S3 Storage
```typescript
import { BrainyData } from '@soulcraft/brainy'
import express from 'express'
const app = express()
// Create a BrainyData instance with S3 storage and real-time updates
const db = new BrainyData({
storage: {
s3Storage: {
bucketName: 'my-brainy-bucket',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
region: 'us-west-2'
}
},
realtimeUpdates: {
enabled: true,
interval: 30000 // 30 seconds
}
})
// Initialize the database
await db.init()
// API endpoint to search
app.get('/search', async (req, res) => {
const { query, limit } = req.query
const results = await db.searchText(query, parseInt(limit) || 10)
res.json(results)
})
// API endpoint to get statistics
app.get('/stats', async (req, res) => {
const stats = await db.getStatistics()
res.json(stats)
})
// API endpoint to manually check for updates
app.post('/update', async (req, res) => {
await db.checkForUpdatesNow()
res.json({ success: true })
})
// Start the server
app.listen(3000, () => {
console.log('Server running on port 3000')
})
// Graceful shutdown
process.on('SIGINT', async () => {
await db.shutDown()
process.exit(0)
})
```
In this example, the BrainyData instance will automatically check for new data in the S3 bucket every 30 seconds, ensuring that search results and statistics are always up-to-date.

245
src/brainyData.ts
View file

@ -134,6 +134,37 @@ export interface BrainyDataConfig {
*/
verbose?: boolean
}
/**
* Real-time update configuration
* Controls how the database handles updates when data is added by external processes
*/
realtimeUpdates?: {
/**
* Whether to enable automatic updates of the index and statistics
* When true, the database will periodically check for new data in storage
* Default: false
*/
enabled?: boolean
/**
* The interval (in milliseconds) at which to check for updates
* Default: 30000 (30 seconds)
*/
interval?: number
/**
* Whether to update statistics when checking for updates
* Default: true
*/
updateStatistics?: boolean
/**
* Whether to update the index when checking for updates
* Default: true
*/
updateIndex?: boolean
}
}
export class BrainyData<T = any> implements BrainyDataInterface<T> {
@ -149,6 +180,17 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
private useOptimizedIndex: boolean = false
private _dimensions: number
private loggingConfig: BrainyDataConfig['logging'] = {verbose: true}
// Real-time update properties
private realtimeUpdateConfig: Required<NonNullable<BrainyDataConfig['realtimeUpdates']>> = {
enabled: false,
interval: 30000, // 30 seconds
updateStatistics: true,
updateIndex: true
}
private updateTimerId: NodeJS.Timeout | null = null
private lastUpdateTime = 0
private lastKnownNounCount = 0
// Remote server properties
private remoteServerConfig: BrainyDataConfig['remoteServer'] | null = null
@ -230,6 +272,14 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
if (config.remoteServer) {
this.remoteServerConfig = config.remoteServer
}
// Initialize real-time update configuration if provided
if (config.realtimeUpdates) {
this.realtimeUpdateConfig = {
...this.realtimeUpdateConfig,
...config.realtimeUpdates
}
}
}
/**
@ -243,6 +293,189 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
)
}
}
/**
* Start real-time updates if enabled in the configuration
* This will periodically check for new data in storage and update the in-memory index and statistics
*/
private startRealtimeUpdates(): void {
// If real-time updates are not enabled, do nothing
if (!this.realtimeUpdateConfig.enabled) {
return
}
// If the update timer is already running, do nothing
if (this.updateTimerId !== null) {
return
}
// Set the initial last known noun count
this.getNounCount().then(count => {
this.lastKnownNounCount = count
}).catch(error => {
console.warn('Failed to get initial noun count for real-time updates:', error)
})
// Start the update timer
this.updateTimerId = setInterval(() => {
this.checkForUpdates().catch(error => {
console.warn('Error during real-time update check:', error)
})
}, this.realtimeUpdateConfig.interval)
if (this.loggingConfig?.verbose) {
console.log(`Real-time updates started with interval: ${this.realtimeUpdateConfig.interval}ms`)
}
}
/**
* Stop real-time updates
*/
private stopRealtimeUpdates(): void {
// If the update timer is not running, do nothing
if (this.updateTimerId === null) {
return
}
// Stop the update timer
clearInterval(this.updateTimerId)
this.updateTimerId = null
if (this.loggingConfig?.verbose) {
console.log('Real-time updates stopped')
}
}
/**
* Manually check for updates in storage and update the in-memory index and statistics
* This can be called by the user to force an update check even if automatic updates are not enabled
*/
public async checkForUpdatesNow(): Promise<void> {
await this.ensureInitialized()
return this.checkForUpdates()
}
/**
* Enable real-time updates with the specified configuration
* @param config Configuration for real-time updates
*/
public enableRealtimeUpdates(config?: Partial<BrainyDataConfig['realtimeUpdates']>): void {
// Update configuration if provided
if (config) {
this.realtimeUpdateConfig = {
...this.realtimeUpdateConfig,
...config
}
}
// Enable updates
this.realtimeUpdateConfig.enabled = true
// Start updates if initialized
if (this.isInitialized) {
this.startRealtimeUpdates()
}
}
/**
* Disable real-time updates
*/
public disableRealtimeUpdates(): void {
// Disable updates
this.realtimeUpdateConfig.enabled = false
// Stop updates if running
this.stopRealtimeUpdates()
}
/**
* Get the current real-time update configuration
* @returns The current real-time update configuration
*/
public getRealtimeUpdateConfig(): Required<NonNullable<BrainyDataConfig['realtimeUpdates']>> {
return { ...this.realtimeUpdateConfig }
}
/**
* Check for updates in storage and update the in-memory index and statistics if needed
* This is called periodically by the update timer when real-time updates are enabled
*/
private async checkForUpdates(): Promise<void> {
// If the database is not initialized, do nothing
if (!this.isInitialized || !this.storage) {
return
}
try {
// Record the current time
const startTime = Date.now()
// Update statistics if enabled
if (this.realtimeUpdateConfig.updateStatistics) {
await this.storage.flushStatisticsToStorage()
// Clear the statistics cache to force a reload from storage
await this.getStatistics({ forceRefresh: true })
}
// Update index if enabled
if (this.realtimeUpdateConfig.updateIndex) {
// Get the current noun count
const currentCount = await this.getNounCount()
// If the noun count has changed, update the index
if (currentCount !== this.lastKnownNounCount) {
// Get all nouns from storage
const nouns = await this.storage.getAllNouns()
// Get all nouns currently in the index
const indexNouns = this.index.getNouns()
const indexNounIds = new Set(indexNouns.keys())
// Find nouns that are in storage but not in the index
const newNouns = nouns.filter(noun => !indexNounIds.has(noun.id))
// Add new nouns to the index
for (const noun of newNouns) {
// Check if the vector dimensions match the expected dimensions
if (noun.vector.length !== this._dimensions) {
console.warn(
`Skipping noun ${noun.id} due to dimension mismatch: expected ${this._dimensions}, got ${noun.vector.length}`
)
continue
}
// Add to index
await this.index.addItem({
id: noun.id,
vector: noun.vector
})
if (this.loggingConfig?.verbose) {
console.log(`Added new noun ${noun.id} to index during real-time update`)
}
}
// Update the last known noun count
this.lastKnownNounCount = currentCount
if (this.loggingConfig?.verbose && newNouns.length > 0) {
console.log(`Real-time update: Added ${newNouns.length} new nouns to index`)
}
}
}
// Update the last update time
this.lastUpdateTime = Date.now()
if (this.loggingConfig?.verbose) {
const duration = this.lastUpdateTime - startTime
console.log(`Real-time update completed in ${duration}ms`)
}
} catch (error) {
console.error('Failed to check for updates:', error)
// Don't rethrow the error to avoid disrupting the update timer
}
}
/**
* Get the current augmentation name if available
@ -408,6 +641,9 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
this.isInitialized = true
this.isInitializing = false
// Start real-time updates if enabled
this.startRealtimeUpdates()
} catch (error) {
console.error('Failed to initialize BrainyData:', error)
this.isInitializing = false
@ -1998,6 +2234,7 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
*/
public async getStatistics(options: {
service?: string | string[] // Filter statistics by service(s)
forceRefresh?: boolean // Force a refresh of statistics from storage
} = {}): Promise<{
nounCount: number
verbCount: number
@ -2025,6 +2262,11 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
await this.ensureInitialized()
try {
// If forceRefresh is true, flush statistics to storage first
if (options.forceRefresh && this.storage) {
await this.storage.flushStatisticsToStorage()
}
// Get statistics from storage
const stats = await this.storage!.getStatistics()
@ -2808,6 +3050,9 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
*/
public async shutDown(): Promise<void> {
try {
// Stop real-time updates if they're running
this.stopRealtimeUpdates()
// Flush statistics to ensure they're saved before shutting down
if (this.storage && this.isInitialized) {
try {

2
test-results.json
View file

File diff suppressed because one or more lines are too long