**chore: remove outdated changelog and summary documents**
- Deleted `CHANGES.md`, `CHANGES_SUMMARY.md`, `CONCURRENCY_ANALYSIS.md`, `CONCURRENCY_IMPLEMENTATION_SUMMARY.md`, and related developer documentation files. - Removed redundant or legacy content no longer aligned with the current codebase and workflows. - Updated repository to reflect streamlined documentation approach, reducing clutter and improving maintainability. **Purpose**: Simplify and declutter repository by removing obsolete documentation files, ensuring it remains focused and relevant.
This commit is contained in:
parent
0932e15274
commit
d015d1a94d
30 changed files with 343 additions and 3 deletions
127
archive/CHANGES.md
Normal file
127
archive/CHANGES.md
Normal file
|
|
@ -0,0 +1,127 @@
|
|||
# Brainy Changelog and Implementation Summaries
|
||||
|
||||
<div align="center">
|
||||
<img src="./brainy.png" alt="Brainy Logo" width="200"/>
|
||||
</div>
|
||||
|
||||
This document provides a comprehensive record of changes and implementation summaries for the Brainy project.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Recent Changes](#recent-changes)
|
||||
- [Statistics Optimizations Implementation](#statistics-optimizations-implementation)
|
||||
|
||||
## Recent Changes
|
||||
|
||||
### 2025-07-28
|
||||
|
||||
#### Bug Fixes
|
||||
|
||||
- Fixed an issue in FileSystemStorage constructor where path operations were performed before the path module was fully loaded. The fix defers path operations until the init() method is called, when the path module is guaranteed to be loaded.
|
||||
|
||||
#### Details
|
||||
|
||||
The issue was in the FileSystemStorage constructor where it was using the path module synchronously:
|
||||
|
||||
```typescript
|
||||
constructor(rootDirectory: string) {
|
||||
super()
|
||||
this.rootDir = rootDirectory
|
||||
this.nounsDir = path.join(this.rootDir, NOUNS_DIR) // Error here - path could be undefined
|
||||
this.verbsDir = path.join(this.rootDir, VERBS_DIR)
|
||||
this.metadataDir = path.join(this.rootDir, METADATA_DIR)
|
||||
this.indexDir = path.join(this.rootDir, INDEX_DIR)
|
||||
}
|
||||
```
|
||||
|
||||
However, the path module was being loaded asynchronously via dynamic imports:
|
||||
|
||||
```typescript
|
||||
try {
|
||||
// Using dynamic imports to avoid issues in browser environments
|
||||
const fsPromise = import('fs')
|
||||
const pathPromise = import('path')
|
||||
|
||||
Promise.all([fsPromise, pathPromise]).then(([fsModule, pathModule]) => {
|
||||
fs = fsModule
|
||||
path = pathModule.default
|
||||
}).catch(error => {
|
||||
console.error('Failed to load Node.js modules:', error)
|
||||
})
|
||||
} catch (error) {
|
||||
console.error(
|
||||
'FileSystemStorage: Failed to load Node.js modules. This adapter is not supported in this environment.',
|
||||
error
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
The fix:
|
||||
1. Modified the constructor to only store the rootDirectory and defer path operations
|
||||
2. Updated the init() method to initialize directory paths when the path module is guaranteed to be loaded
|
||||
|
||||
This ensures that path operations are only performed when the path module is available, preventing the "Cannot read properties of undefined (reading 'join')" error.
|
||||
|
||||
## Statistics Optimizations Implementation
|
||||
|
||||
### Overview
|
||||
|
||||
This section summarizes the changes made to implement statistics optimizations across all storage adapters in the Brainy project. The optimizations were originally implemented for the s3CompatibleStorage adapter and have now been extended to all storage adapters.
|
||||
|
||||
### Changes Made
|
||||
|
||||
#### 1. BaseStorageAdapter Enhancements
|
||||
|
||||
The BaseStorageAdapter class was refactored to include shared optimizations:
|
||||
|
||||
- Added in-memory caching of statistics data
|
||||
- Implemented batched updates with adaptive flush timing
|
||||
- Added error handling and retry mechanisms
|
||||
- Updated core statistics methods to use the new caching and batching approach
|
||||
|
||||
Specific changes:
|
||||
- Added properties for caching and batch update management
|
||||
- Implemented `scheduleBatchUpdate()` and `flushStatistics()` methods
|
||||
- Updated `saveStatistics()`, `getStatistics()`, `incrementStatistic()`, `decrementStatistic()`, and `updateHnswIndexSize()` methods
|
||||
|
||||
#### 2. Storage Adapter Updates
|
||||
|
||||
##### FileSystemStorage
|
||||
|
||||
- Implemented time-based partitioning for statistics files
|
||||
- Added fallback mechanisms to check multiple storage locations
|
||||
- Maintained backward compatibility with legacy statistics files
|
||||
|
||||
##### MemoryStorage
|
||||
|
||||
- Updated to be compatible with the BaseStorageAdapter changes
|
||||
- Leverages the in-memory nature of this adapter for efficient caching
|
||||
|
||||
##### OPFSStorage (Origin Private File System)
|
||||
|
||||
- Implemented time-based partitioning for statistics files
|
||||
- Added fallback mechanisms to check multiple storage locations
|
||||
- Maintained backward compatibility with legacy statistics files
|
||||
|
||||
#### 3. Documentation Updates
|
||||
|
||||
- Updated statistics.md to reflect that optimizations are implemented across all storage adapters
|
||||
- Added a new section describing the implementation across different adapter types
|
||||
|
||||
### Benefits
|
||||
|
||||
These changes provide several benefits:
|
||||
|
||||
1. **Improved Performance**: Reduced storage operations through caching and batching
|
||||
2. **Better Scalability**: Time-based partitioning helps avoid rate limits and reduces contention
|
||||
3. **Historical Data**: Daily statistics files provide a historical record of database usage
|
||||
4. **Consistent Experience**: All storage adapters now provide the same optimizations
|
||||
5. **Backward Compatibility**: Legacy statistics files are still supported
|
||||
|
||||
### Testing
|
||||
|
||||
The changes have been tested to ensure they don't break existing functionality. The specific statistics test requires additional setup (dotenv package and AWS credentials) but general tests are passing.
|
||||
|
||||
### Conclusion
|
||||
|
||||
The statistics optimizations originally implemented for the s3CompatibleStorage adapter have been successfully extended to all storage adapters in the Brainy project. This ensures consistent performance and scalability across different storage backends.
|
||||
63
archive/CHANGES_SUMMARY.md
Normal file
63
archive/CHANGES_SUMMARY.md
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
# Statistics Optimizations Implementation Summary
|
||||
|
||||
## Overview
|
||||
|
||||
This document summarizes the changes made to implement statistics optimizations across all storage adapters in the Brainy project. The optimizations were originally implemented for the s3CompatibleStorage adapter and have now been extended to all storage adapters.
|
||||
|
||||
## Changes Made
|
||||
|
||||
### 1. BaseStorageAdapter Enhancements
|
||||
|
||||
The BaseStorageAdapter class was refactored to include shared optimizations:
|
||||
|
||||
- Added in-memory caching of statistics data
|
||||
- Implemented batched updates with adaptive flush timing
|
||||
- Added error handling and retry mechanisms
|
||||
- Updated core statistics methods to use the new caching and batching approach
|
||||
|
||||
Specific changes:
|
||||
- Added properties for caching and batch update management
|
||||
- Implemented `scheduleBatchUpdate()` and `flushStatistics()` methods
|
||||
- Updated `saveStatistics()`, `getStatistics()`, `incrementStatistic()`, `decrementStatistic()`, and `updateHnswIndexSize()` methods
|
||||
|
||||
### 2. Storage Adapter Updates
|
||||
|
||||
#### FileSystemStorage
|
||||
|
||||
- Implemented time-based partitioning for statistics files
|
||||
- Added fallback mechanisms to check multiple storage locations
|
||||
- Maintained backward compatibility with legacy statistics files
|
||||
|
||||
#### MemoryStorage
|
||||
|
||||
- Updated to be compatible with the BaseStorageAdapter changes
|
||||
- Leverages the in-memory nature of this adapter for efficient caching
|
||||
|
||||
#### OPFSStorage (Origin Private File System)
|
||||
|
||||
- Implemented time-based partitioning for statistics files
|
||||
- Added fallback mechanisms to check multiple storage locations
|
||||
- Maintained backward compatibility with legacy statistics files
|
||||
|
||||
### 3. Documentation Updates
|
||||
|
||||
- Updated statistics.md to reflect that optimizations are implemented across all storage adapters
|
||||
- Added a new section describing the implementation across different adapter types
|
||||
|
||||
## Benefits
|
||||
|
||||
These changes provide several benefits:
|
||||
|
||||
1. **Improved Performance**: Reduced storage operations through caching and batching
|
||||
2. **Better Scalability**: Time-based partitioning helps avoid rate limits and reduces contention
|
||||
3. **Historical Data**: Daily statistics files provide a historical record of database usage
|
||||
4. **Consistent Experience**: All storage adapters now provide the same optimizations
|
||||
5. **Backward Compatibility**: Legacy statistics files are still supported
|
||||
|
||||
## Testing
|
||||
|
||||
The changes have been tested to ensure they don't break existing functionality. The specific statistics test requires additional setup (dotenv package and AWS credentials) but general tests are passing.
|
||||
|
||||
## Conclusion
|
||||
|
||||
The statistics optimizations originally implemented for the s3CompatibleStorage adapter have been successfully extended to all storage adapters in the Brainy project. This ensures consistent performance and scalability across different storage backends.
|
||||
65
archive/demo.md
Normal file
65
archive/demo.md
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
# Running the Brainy Demo
|
||||
|
||||
The Brainy interactive demo showcases the library's features in a web browser. Follow these steps to run it:
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Make sure you have Node.js installed (version 24.4.0 or higher)
|
||||
- Ensure the project is built (run both `npm run build` and `npm run build:browser`)
|
||||
|
||||
## Running the Demo
|
||||
|
||||
### Option 1: Using the npm script (recommended)
|
||||
|
||||
Run the following command from the project root:
|
||||
|
||||
```bash
|
||||
npm run demo
|
||||
```
|
||||
|
||||
This will start an HTTP server and automatically open the demo in your default browser.
|
||||
|
||||
### Option 2: Manual setup
|
||||
|
||||
1. Start an HTTP server in the project root:
|
||||
|
||||
```bash
|
||||
npx http-server
|
||||
```
|
||||
|
||||
2. Open your browser and navigate to:
|
||||
http://localhost:8080/demo/index.html
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
If you see the error "Could not load Brainy library. Please ensure the project is built and served over HTTP", check the
|
||||
following:
|
||||
|
||||
1. Make sure you've built the project with `npm run build && npm run build:browser`
|
||||
2. Ensure you're accessing the demo through HTTP (not by opening the file directly)
|
||||
3. Check your browser's console for additional error messages
|
||||
|
||||
If issues persist, try clearing your browser cache or using a private/incognito window.
|
||||
|
||||
## Build Process
|
||||
|
||||
The Brainy library uses a two-step build process:
|
||||
|
||||
1. `npm run build` - Compiles TypeScript files to JavaScript (used for Node.js environments)
|
||||
2. `npm run build:browser` - Creates a browser-compatible bundle using Rollup
|
||||
|
||||
You can run both steps together with:
|
||||
|
||||
```bash
|
||||
npm run build && npm run build:browser
|
||||
```
|
||||
|
||||
Or simply use the demo script which does this for you:
|
||||
|
||||
```bash
|
||||
npm run demo
|
||||
```
|
||||
|
||||
The browser bundle is created from `src/unified.ts`, which provides environment detection and adapts to browser,
|
||||
Node.js, or serverless environments. This unified approach ensures that the library works correctly across all
|
||||
environments.
|
||||
34
archive/fix-documentation.md
Normal file
34
archive/fix-documentation.md
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
# Fix for "process.memoryUsage is not a function" Error
|
||||
|
||||
## Issue
|
||||
During test runs with Vitest, the following error was occurring:
|
||||
|
||||
```
|
||||
TypeError: process.memoryUsage is not a function
|
||||
❯ VitestTestRunner.onAfterRunSuite node_modules/vitest/dist/runners.js:150:95
|
||||
```
|
||||
|
||||
This error was happening because Vitest was trying to use `process.memoryUsage()` to log heap usage statistics, but this function was not available in the current environment.
|
||||
|
||||
## Solution
|
||||
The issue was fixed by disabling the heap usage logging in the Vitest configuration:
|
||||
|
||||
In `vitest.config.ts`, changed:
|
||||
```typescript
|
||||
// Show test statistics
|
||||
logHeapUsage: true,
|
||||
```
|
||||
|
||||
To:
|
||||
```typescript
|
||||
// Show test statistics
|
||||
logHeapUsage: false,
|
||||
```
|
||||
|
||||
## Explanation
|
||||
The `logHeapUsage` option in Vitest attempts to use Node.js's `process.memoryUsage()` function to track and report memory usage during test runs. However, this function might not be available in all environments, particularly in certain browser-like environments or when using specific Node.js versions or configurations.
|
||||
|
||||
By setting `logHeapUsage: false`, we prevent Vitest from attempting to call this function, which resolves the error while still allowing tests to run successfully.
|
||||
|
||||
## Verification
|
||||
After making this change, the tests run without any unhandled errors, confirming that the issue has been resolved.
|
||||
32
archive/test-issue-summary.md
Normal file
32
archive/test-issue-summary.md
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
# API Integration Test Issue Resolution
|
||||
|
||||
## Issue Summary
|
||||
The API integration test was failing because it was trying to insert data into Brainy and then search for it, but the data wasn't being properly embedded/vectorized or wasn't being found in the search results.
|
||||
|
||||
## Key Changes That Fixed the Issue
|
||||
|
||||
### Test Modifications
|
||||
1. Removed the explicit `dimensions: 512` parameter from BrainyData initialization
|
||||
- This allows it to use the default dimensions that match the embedding model
|
||||
|
||||
2. Changed from using `addItem()` to using `add()` with `forceEmbed: true`
|
||||
- This ensures proper embedding of the text data
|
||||
|
||||
3. Increased the wait time for indexing from 500ms to 2000ms
|
||||
- Gives the HNSW index more time to update before searching
|
||||
|
||||
4. Added more detailed logging to help diagnose issues
|
||||
|
||||
### Embedding Functionality Improvements
|
||||
1. Fixed how the Universal Sentence Encoder is loaded
|
||||
- Now ensures it uses the bundled model from the package
|
||||
|
||||
2. Improved type handling for TextDecoder to avoid potential compatibility issues
|
||||
|
||||
## Why It Works Now
|
||||
The test is now passing because:
|
||||
1. The data is being properly embedded through the `add()` method with forced embedding
|
||||
2. The system has enough time to index the data before searching for it
|
||||
3. The embedding model is being loaded correctly without dimension mismatches
|
||||
|
||||
These changes ensure that when data is inserted into Brainy, it's properly embedded and vectorized, and then can be successfully retrieved through semantic search without needing to run in Express or any other server environment.
|
||||
Loading…
Add table
Add a link
Reference in a new issue