**chore(docs): improve code formatting and readability in README.md**
- Reformatted code examples for consistency in spacing and alignment: - Adjusted indentation
This commit is contained in:
parent
8fb17aeb51
commit
e71f5e0fae
1 changed files with 394 additions and 399 deletions
101
README.md
101
README.md
|
|
@ -27,7 +27,8 @@ it gets - learning from your data to provide increasingly relevant results and c
|
||||||
|
|
||||||
- **Run Everywhere** - Works in browsers, Node.js, serverless functions, and containers
|
- **Run Everywhere** - Works in browsers, Node.js, serverless functions, and containers
|
||||||
- **Vector Search** - Find semantically similar content using embeddings
|
- **Vector Search** - Find semantically similar content using embeddings
|
||||||
- **Advanced JSON Document Search** - Search within specific fields of JSON documents with field prioritization and service-based field standardization
|
- **Advanced JSON Document Search** - Search within specific fields of JSON documents with field prioritization and
|
||||||
|
service-based field standardization
|
||||||
- **Graph Relationships** - Connect data with meaningful relationships
|
- **Graph Relationships** - Connect data with meaningful relationships
|
||||||
- **Streaming Pipeline** - Process data in real-time as it flows through the system
|
- **Streaming Pipeline** - Process data in real-time as it flows through the system
|
||||||
- **Extensible Augmentations** - Customize and extend functionality with pluggable components
|
- **Extensible Augmentations** - Customize and extend functionality with pluggable components
|
||||||
|
|
@ -434,6 +435,7 @@ const verbTypeMap = getVerbTypeMap() // { RelatedTo: 'relatedTo', Contains: 'con
|
||||||
```
|
```
|
||||||
|
|
||||||
These utility functions make it easy to:
|
These utility functions make it easy to:
|
||||||
|
|
||||||
- Get a complete list of available noun and verb types
|
- Get a complete list of available noun and verb types
|
||||||
- Validate user input against valid types
|
- Validate user input against valid types
|
||||||
- Create dynamic UI components that display or select from available types
|
- Create dynamic UI components that display or select from available types
|
||||||
|
|
@ -534,7 +536,9 @@ const restoreResult = await db.restore(backupData, {clearExisting: true})
|
||||||
|
|
||||||
### Database Statistics
|
### Database Statistics
|
||||||
|
|
||||||
Brainy provides a way to get statistics about the current state of the database. For detailed information about the statistics system, including implementation details, scalability improvements, and usage examples, see our [Statistics Guide](STATISTICS.md).
|
Brainy provides a way to get statistics about the current state of the database. For detailed information about the
|
||||||
|
statistics system, including implementation details, scalability improvements, and usage examples, see
|
||||||
|
our [Statistics Guide](STATISTICS.md).
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
import { BrainyData, getStatistics } from '@soulcraft/brainy'
|
import { BrainyData, getStatistics } from '@soulcraft/brainy'
|
||||||
|
|
@ -608,7 +612,8 @@ const authorResults = await db.searchByStandardField("author", "johndoe", 10, {
|
||||||
|
|
||||||
### Field Standardization and Service Tracking
|
### Field Standardization and Service Tracking
|
||||||
|
|
||||||
Brainy automatically tracks field names from JSON documents and associates them with the service that inserted the data. This enables powerful cross-service search capabilities:
|
Brainy automatically tracks field names from JSON documents and associates them with the service that inserted the data.
|
||||||
|
This enables powerful cross-service search capabilities:
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// Get all available field names organized by service
|
// Get all available field names organized by service
|
||||||
|
|
@ -698,8 +703,10 @@ db.setReadOnly(false)
|
||||||
db.setWriteOnly(false)
|
db.setWriteOnly(false)
|
||||||
```
|
```
|
||||||
|
|
||||||
- **Read-Only Mode**: When enabled, prevents all write operations (add, update, delete). Useful for deployment scenarios where you want to prevent modifications to the database.
|
- **Read-Only Mode**: When enabled, prevents all write operations (add, update, delete). Useful for deployment scenarios
|
||||||
- **Write-Only Mode**: When enabled, prevents all search operations. Useful for initial data loading or when you want to optimize for write performance.
|
where you want to prevent modifications to the database.
|
||||||
|
- **Write-Only Mode**: When enabled, prevents all search operations. Useful for initial data loading or when you want to
|
||||||
|
optimize for write performance.
|
||||||
|
|
||||||
### Embedding
|
### Embedding
|
||||||
|
|
||||||
|
|
@ -1410,7 +1417,9 @@ terabyte-scale data that can't fit entirely in memory, we provide several approa
|
||||||
- **Distributed HNSW**: Sharding and partitioning across multiple machines
|
- **Distributed HNSW**: Sharding and partitioning across multiple machines
|
||||||
- **Hybrid Solutions**: Combining quantization techniques with multi-tier architectures
|
- **Hybrid Solutions**: Combining quantization techniques with multi-tier architectures
|
||||||
|
|
||||||
For detailed information on how to scale Brainy for large datasets, vector dimension standardization, threading implementation, storage testing, and other technical topics, see our comprehensive [Technical Guides](TECHNICAL_GUIDES.md).
|
For detailed information on how to scale Brainy for large datasets, vector dimension standardization, threading
|
||||||
|
implementation, storage testing, and other technical topics, see our
|
||||||
|
comprehensive [Technical Guides](TECHNICAL_GUIDES.md).
|
||||||
|
|
||||||
## Recent Changes and Performance Improvements
|
## Recent Changes and Performance Improvements
|
||||||
|
|
||||||
|
|
@ -1418,7 +1427,9 @@ For detailed information on how to scale Brainy for large datasets, vector dimen
|
||||||
|
|
||||||
Brainy has been significantly improved to handle larger datasets more efficiently:
|
Brainy has been significantly improved to handle larger datasets more efficiently:
|
||||||
|
|
||||||
- **Pagination Support**: All data retrieval methods now support pagination to avoid loading entire datasets into memory at once. The deprecated `getAllNouns()` and `getAllVerbs()` methods have been replaced with `getNouns()` and `getVerbs()` methods that support pagination, filtering, and cursor-based navigation.
|
- **Pagination Support**: All data retrieval methods now support pagination to avoid loading entire datasets into memory
|
||||||
|
at once. The deprecated `getAllNouns()` and `getAllVerbs()` methods have been replaced with `getNouns()` and
|
||||||
|
`getVerbs()` methods that support pagination, filtering, and cursor-based navigation.
|
||||||
|
|
||||||
- **Multi-level Caching**: A sophisticated three-level caching strategy has been implemented:
|
- **Multi-level Caching**: A sophisticated three-level caching strategy has been implemented:
|
||||||
- **Level 1**: Hot cache (most accessed nodes) - RAM (automatically detecting and adjusting in each environment)
|
- **Level 1**: Hot cache (most accessed nodes) - RAM (automatically detecting and adjusting in each environment)
|
||||||
|
|
@ -1429,13 +1440,16 @@ Brainy has been significantly improved to handle larger datasets more efficientl
|
||||||
- In Node.js: Uses 10% of free memory (minimum 1000 entries)
|
- In Node.js: Uses 10% of free memory (minimum 1000 entries)
|
||||||
- In browsers: Scales based on device memory (500 entries per GB, minimum 1000)
|
- In browsers: Scales based on device memory (500 entries per GB, minimum 1000)
|
||||||
|
|
||||||
- **Intelligent Cache Eviction**: Implements a Least Recently Used (LRU) policy that evicts the oldest 20% of items when the cache reaches the configured threshold.
|
- **Intelligent Cache Eviction**: Implements a Least Recently Used (LRU) policy that evicts the oldest 20% of items when
|
||||||
|
the cache reaches the configured threshold.
|
||||||
|
|
||||||
- **Prefetching Strategy**: Implements batch prefetching to improve performance while avoiding overwhelming system resources.
|
- **Prefetching Strategy**: Implements batch prefetching to improve performance while avoiding overwhelming system
|
||||||
|
resources.
|
||||||
|
|
||||||
### S3-Compatible Storage Improvements
|
### S3-Compatible Storage Improvements
|
||||||
|
|
||||||
- **Enhanced Cloud Storage**: Improved support for S3-compatible storage services including AWS S3, Cloudflare R2, and others.
|
- **Enhanced Cloud Storage**: Improved support for S3-compatible storage services including AWS S3, Cloudflare R2, and
|
||||||
|
others.
|
||||||
|
|
||||||
- **Optimized Data Access**: Batch operations and error handling for efficient cloud storage access.
|
- **Optimized Data Access**: Batch operations and error handling for efficient cloud storage access.
|
||||||
|
|
||||||
|
|
@ -1445,9 +1459,11 @@ Brainy has been significantly improved to handle larger datasets more efficientl
|
||||||
|
|
||||||
Yes, you can use existing data indexed from an old version. Brainy includes robust data migration capabilities:
|
Yes, you can use existing data indexed from an old version. Brainy includes robust data migration capabilities:
|
||||||
|
|
||||||
- **Vector Regeneration**: If vectors are missing in imported data, they will be automatically created using the embedding function.
|
- **Vector Regeneration**: If vectors are missing in imported data, they will be automatically created using the
|
||||||
|
embedding function.
|
||||||
|
|
||||||
- **HNSW Index Reconstruction**: The system can reconstruct the HNSW index from backup data, ensuring compatibility with previous versions.
|
- **HNSW Index Reconstruction**: The system can reconstruct the HNSW index from backup data, ensuring compatibility with
|
||||||
|
previous versions.
|
||||||
|
|
||||||
- **Sparse Data Import**: Support for importing sparse data (without vectors) through the `importSparseData()` method.
|
- **Sparse Data Import**: Support for importing sparse data (without vectors) through the `importSparseData()` method.
|
||||||
|
|
||||||
|
|
@ -1487,7 +1503,8 @@ Read-only mode prevents all write operations (add, update, delete) and is optimi
|
||||||
|
|
||||||
#### Write-Only Mode
|
#### Write-Only Mode
|
||||||
|
|
||||||
Write-only mode prevents all search operations and is optimized for initial data loading or when you want to optimize for write performance.
|
Write-only mode prevents all search operations and is optimized for initial data loading or when you want to optimize
|
||||||
|
for write performance.
|
||||||
|
|
||||||
- **Memory**:
|
- **Memory**:
|
||||||
- Minimum: 512MB RAM
|
- Minimum: 512MB RAM
|
||||||
|
|
@ -1503,7 +1520,9 @@ Write-only mode prevents all search operations and is optimized for initial data
|
||||||
|
|
||||||
### Performance Tuning Parameters
|
### Performance Tuning Parameters
|
||||||
|
|
||||||
Brainy offers comprehensive configuration options for performance tuning, with enhanced support for large datasets in S3 or other remote storage. **All configuration is optional** - the system automatically detects the optimal settings based on your environment, dataset size, and usage patterns.
|
Brainy offers comprehensive configuration options for performance tuning, with enhanced support for large datasets in S3
|
||||||
|
or other remote storage. **All configuration is optional** - the system automatically detects the optimal settings based
|
||||||
|
on your environment, dataset size, and usage patterns.
|
||||||
|
|
||||||
#### Intelligent Defaults
|
#### Intelligent Defaults
|
||||||
|
|
||||||
|
|
@ -1577,11 +1596,13 @@ const brainy = new BrainyData({
|
||||||
});
|
});
|
||||||
```
|
```
|
||||||
|
|
||||||
These configuration options make Brainy more efficient, scalable, and adaptable to different environments and usage patterns, especially for large datasets in cloud storage.
|
These configuration options make Brainy more efficient, scalable, and adaptable to different environments and usage
|
||||||
|
patterns, especially for large datasets in cloud storage.
|
||||||
|
|
||||||
## Testing
|
## Testing
|
||||||
|
|
||||||
Brainy uses Vitest for testing. For detailed information about testing in Brainy, including test configuration, scripts, reporting tools, and best practices, see our [Testing Guide](TESTING.md).
|
Brainy uses Vitest for testing. For detailed information about testing in Brainy, including test configuration, scripts,
|
||||||
|
reporting tools, and best practices, see our [Testing Guide](TESTING.md).
|
||||||
|
|
||||||
Here are some common test commands:
|
Here are some common test commands:
|
||||||
|
|
||||||
|
|
@ -1605,45 +1626,18 @@ see [DEVELOPERS.md](DEVELOPERS.md).
|
||||||
|
|
||||||
We have a [Code of Conduct](CODE_OF_CONDUCT.md) that all contributors are expected to follow.
|
We have a [Code of Conduct](CODE_OF_CONDUCT.md) that all contributors are expected to follow.
|
||||||
|
|
||||||
## Release Workflow
|
|
||||||
|
|
||||||
Brainy uses a streamlined release workflow that automates version updates, changelog generation, GitHub releases, and NPM deployment.
|
|
||||||
|
|
||||||
### Automated Release Process
|
|
||||||
|
|
||||||
The release workflow combines several steps into a single command:
|
|
||||||
|
|
||||||
1. **Build the project** - Ensures the code compiles correctly
|
|
||||||
2. **Run tests** - Verifies that all tests pass
|
|
||||||
3. **Update version** - Bumps the version number (patch, minor, or major)
|
|
||||||
4. **Generate changelog** - Automatically updates CHANGELOG.md with commit messages since the last release
|
|
||||||
5. **Create GitHub release** - Creates a GitHub release with auto-generated notes
|
|
||||||
6. **Publish to NPM** - Deploys the package to NPM
|
|
||||||
|
|
||||||
### Release Commands
|
|
||||||
|
|
||||||
Use one of the following commands to release a new version:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Release with patch version update (0.0.x)
|
|
||||||
npm run workflow:patch
|
|
||||||
|
|
||||||
# Release with minor version update (0.x.0)
|
|
||||||
npm run workflow:minor
|
|
||||||
|
|
||||||
# Release with major version update (x.0.0)
|
|
||||||
npm run workflow:major
|
|
||||||
|
|
||||||
# Default workflow (same as patch)
|
|
||||||
npm run workflow
|
|
||||||
|
|
||||||
# Dry run (build, test, and simulate version update without making changes)
|
|
||||||
npm run workflow:dry-run
|
|
||||||
```
|
|
||||||
|
|
||||||
### Commit Message Format
|
### Commit Message Format
|
||||||
|
|
||||||
For best results with automatic changelog generation, follow the [Conventional Commits](https://www.conventionalcommits.org/) specification for your commit messages:
|
For best results with automatic changelog generation, follow
|
||||||
|
the [Conventional Commits](https://www.conventionalcommits.org/) specification for your commit messages:
|
||||||
|
|
||||||
|
```
|
||||||
|
AI Template for automated commit messages:
|
||||||
|
|
||||||
|
Use Conventional Commit format
|
||||||
|
Specify the changes in a structured format
|
||||||
|
Add information about the purpose of the commit
|
||||||
|
```
|
||||||
|
|
||||||
```
|
```
|
||||||
<type>(<scope>): <description>
|
<type>(<scope>): <description>
|
||||||
|
|
@ -1654,6 +1648,7 @@ For best results with automatic changelog generation, follow the [Conventional C
|
||||||
```
|
```
|
||||||
|
|
||||||
Where `<type>` is one of:
|
Where `<type>` is one of:
|
||||||
|
|
||||||
- `feat`: A new feature (maps to **Added** section)
|
- `feat`: A new feature (maps to **Added** section)
|
||||||
- `fix`: A bug fix (maps to **Fixed** section)
|
- `fix`: A bug fix (maps to **Fixed** section)
|
||||||
- `chore`: Regular maintenance tasks (maps to **Changed** section)
|
- `chore`: Regular maintenance tasks (maps to **Changed** section)
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue