soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

**chore(docs): improve code formatting and readability in README.md**

Browse source 
- Reformatted code examples for consistency in spacing and alignment:
  - Adjusted indentation
This commit is contained in:
David Snelling 2025-08-01 08:56:41 -07:00
parent 7e75221ae7
commit d1187a4c49
1 changed files with 394 additions and 399 deletions
Download patch file Download diff file Expand all files Collapse all files

147
README.md
View file

@ -27,7 +27,8 @@ it gets - learning from your data to provide increasingly relevant results and c
- **Run Everywhere** - Works in browsers, Node.js, serverless functions, and containers
- **Vector Search** - Find semantically similar content using embeddings
- **Advanced JSON Document Search** - Search within specific fields of JSON documents with field prioritization and service-based field standardization
- **Advanced JSON Document Search** - Search within specific fields of JSON documents with field prioritization and
service-based field standardization
- **Graph Relationships** - Connect data with meaningful relationships
- **Streaming Pipeline** - Process data in real-time as it flows through the system
- **Extensible Augmentations** - Customize and extend functionality with pluggable components
@ -90,7 +91,7 @@ REST API web service wrapper that provides HTTP endpoints for search operations
Brainy uses a unified build that automatically adapts to your environment (Node.js, browser, or serverless):
```typescript
import {BrainyData, NounType, VerbType} from '@soulcraft/brainy'
import { BrainyData, NounType, VerbType } from '@soulcraft/brainy'
// Create and initialize the database
const db = new BrainyData()
@ -122,10 +123,10 @@ await db.addVerb(catId, dogId, {
```typescript
// Standard import - automatically adapts to any environment
import {BrainyData} from '@soulcraft/brainy'
import { BrainyData } from '@soulcraft/brainy'
// Minified version for production
import {BrainyData} from '@soulcraft/brainy/min'
import { BrainyData } from '@soulcraft/brainy/min'
```
> **Note**: The CLI functionality is available as a separate package `@soulcraft/brainy-cli` to reduce the bundle size
@ -138,7 +139,7 @@ import {BrainyData} from '@soulcraft/brainy/min'
<script type="module">
// Use local files instead of CDN
import {BrainyData} from './dist/unified.js'
import { BrainyData } from './dist/unified.js'
// Or minified version
// import { BrainyData } from './dist/unified.min.js'
@ -300,13 +301,13 @@ The pipeline runs automatically when you:
```typescript
// Add data (runs embedding → indexing → storage)
const id = await db.add("Your text data here", {metadata})
const id = await db.add("Your text data here", { metadata })
// Search (runs embedding → similarity search)
const results = await db.searchText("Your query here", 5)
// Connect entities (runs graph construction → storage)
await db.addVerb(sourceId, targetId, {verb: VerbType.RelatedTo})
await db.addVerb(sourceId, targetId, { verb: VerbType.RelatedTo })
```
Using the CLI:
@ -434,6 +435,7 @@ const verbTypeMap = getVerbTypeMap() // { RelatedTo: 'relatedTo', Contains: 'con
```
These utility functions make it easy to:
- Get a complete list of available noun and verb types
- Validate user input against valid types
- Create dynamic UI components that display or select from available types
@ -529,15 +531,17 @@ const status = await db.status()
const backupData = await db.backup()
// Restore data into the database
const restoreResult = await db.restore(backupData, {clearExisting: true})
const restoreResult = await db.restore(backupData, { clearExisting: true })
```
### Database Statistics
Brainy provides a way to get statistics about the current state of the database. For detailed information about the statistics system, including implementation details, scalability improvements, and usage examples, see our [Statistics Guide](STATISTICS.md).
Brainy provides a way to get statistics about the current state of the database. For detailed information about the
statistics system, including implementation details, scalability improvements, and usage examples, see
our [Statistics Guide](STATISTICS.md).
```typescript
import {BrainyData, getStatistics} from '@soulcraft/brainy'
import { BrainyData, getStatistics } from '@soulcraft/brainy'
// Create and initialize the database
const db = new BrainyData()
@ -562,11 +566,11 @@ const id = await db.add(textOrVector, {
const ids = await db.addBatch([
{
vectorOrData: "First item to add",
metadata: {noun: NounType.Thing, category: 'example'}
metadata: { noun: NounType.Thing, category: 'example' }
},
{
vectorOrData: "Second item to add",
metadata: {noun: NounType.Thing, category: 'example'}
metadata: { noun: NounType.Thing, category: 'example' }
},
// More items...
], {
@ -608,7 +612,8 @@ const authorResults = await db.searchByStandardField("author", "johndoe", 10, {
### Field Standardization and Service Tracking
Brainy automatically tracks field names from JSON documents and associates them with the service that inserted the data. This enables powerful cross-service search capabilities:
Brainy automatically tracks field names from JSON documents and associates them with the service that inserted the data.
This enables powerful cross-service search capabilities:
```typescript
// Get all available field names organized by service
@ -698,8 +703,10 @@ db.setReadOnly(false)
db.setWriteOnly(false)
```
- **Read-Only Mode**: When enabled, prevents all write operations (add, update, delete). Useful for deployment scenarios where you want to prevent modifications to the database.
- **Write-Only Mode**: When enabled, prevents all search operations. Useful for initial data loading or when you want to optimize for write performance.
- **Read-Only Mode**: When enabled, prevents all write operations (add, update, delete). Useful for deployment scenarios
where you want to prevent modifications to the database.
- **Write-Only Mode**: When enabled, prevents all search operations. Useful for initial data loading or when you want to
optimize for write performance.
### Embedding
@ -759,7 +766,7 @@ Brainy includes comprehensive multithreading support to improve performance acro
7. **Automatic Environment Detection**: Adapts to browser (Web Workers) and Node.js (Worker Threads) environments
```typescript
import {BrainyData, euclideanDistance} from '@soulcraft/brainy'
import { BrainyData, euclideanDistance } from '@soulcraft/brainy'
// Configure with custom options
const db = new BrainyData({
@ -808,7 +815,7 @@ hybrid approach:
3. **Memory-Efficient Indexing** - Optimizes memory usage for large-scale vector collections
```typescript
import {BrainyData} from '@soulcraft/brainy'
import { BrainyData } from '@soulcraft/brainy'
// Configure with optimized HNSW index for large datasets
const db = new BrainyData({
@ -994,7 +1001,7 @@ const memoryAug = createMemoryAugmentation({
// Your implementation here
return {
success: true,
data: {example: 'data', key}
data: { example: 'data', key }
}
}
})
@ -1093,7 +1100,7 @@ const mySenseAug = createSenseAugmentation({
// Implementation
return {
success: true,
data: {nouns: [], verbs: []}
data: { nouns: [], verbs: [] }
}
}
}) as IWebSocketSenseAugmentation
@ -1157,7 +1164,7 @@ everywhere.
Brainy automatically detects the environment it's running in:
```typescript
import {environment} from '@soulcraft/brainy'
import { environment } from '@soulcraft/brainy'
// Check which environment we're running in
console.log(`Running in ${
@ -1255,7 +1262,7 @@ pipeline.register(wsConduit)
// Replace the example URL below with your actual WebSocket server URL
const connectionResult = await pipeline.executeConduitPipeline(
'establishConnection',
['wss://example-websocket-server.com/brainy-sync', {protocols: 'brainy-sync'}]
['wss://example-websocket-server.com/brainy-sync', { protocols: 'brainy-sync' }]
)
if (connectionResult[0] && (await connectionResult[0]).success) {
@ -1264,7 +1271,7 @@ if (connectionResult[0] && (await connectionResult[0]).success) {
// Read data from the remote instance
const readResult = await pipeline.executeConduitPipeline(
'readData',
[{connectionId: connection.connectionId, query: {type: 'getAllNouns'}}]
[{ connectionId: connection.connectionId, query: { type: 'getAllNouns' } }]
)
// Process and add the received data to the local instance
@ -1315,7 +1322,7 @@ const connectionResult = await pipeline.executeConduitPipeline(
{
signalServerUrl: 'wss://example-signal-server.com', // Replace with your signal server
localPeerId: 'my-local-peer-id', // Replace with your local peer ID
iceServers: [{urls: 'stun:stun.l.google.com:19302'}] // Public STUN server
iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] // Public STUN server
}
]
)
@ -1334,7 +1341,7 @@ if (connectionResult[0] && (await connectionResult[0]).success) {
})
// When adding new data locally, also send to the peer
const nounId = await db.add("New data to sync", {noun: "Thing"})
const nounId = await db.add("New data to sync", { noun: "Thing" })
// Send the new noun to the peer
await pipeline.executeConduitPipeline(
@ -1360,7 +1367,7 @@ Brainy supports searching a server-hosted instance from a browser, storing resul
searches against the local instance:
```typescript
import {BrainyData} from '@soulcraft/brainy'
import { BrainyData } from '@soulcraft/brainy'
// Create and initialize the database with remote server configuration
// Replace the example URL below with your actual Brainy server URL
@ -1380,13 +1387,13 @@ if (!db.isConnectedToRemoteServer()) {
}
// Search the remote server (results are stored locally)
const remoteResults = await db.searchText('machine learning', 5, {searchMode: 'remote'})
const remoteResults = await db.searchText('machine learning', 5, { searchMode: 'remote' })
// Search the local database (includes previously stored results)
const localResults = await db.searchText('machine learning', 5, {searchMode: 'local'})
const localResults = await db.searchText('machine learning', 5, { searchMode: 'local' })
// Perform a combined search (local first, then remote if needed)
const combinedResults = await db.searchText('neural networks', 5, {searchMode: 'combined'})
const combinedResults = await db.searchText('neural networks', 5, { searchMode: 'combined' })
// Add data to both local and remote instances
const id = await db.addToBoth('Deep learning is a subset of machine learning', {
@ -1410,7 +1417,9 @@ terabyte-scale data that can't fit entirely in memory, we provide several approa
- **Distributed HNSW**: Sharding and partitioning across multiple machines
- **Hybrid Solutions**: Combining quantization techniques with multi-tier architectures
For detailed information on how to scale Brainy for large datasets, vector dimension standardization, threading implementation, storage testing, and other technical topics, see our comprehensive [Technical Guides](TECHNICAL_GUIDES.md).
For detailed information on how to scale Brainy for large datasets, vector dimension standardization, threading
implementation, storage testing, and other technical topics, see our
comprehensive [Technical Guides](TECHNICAL_GUIDES.md).
## Recent Changes and Performance Improvements
@ -1418,7 +1427,9 @@ For detailed information on how to scale Brainy for large datasets, vector dimen
Brainy has been significantly improved to handle larger datasets more efficiently:
- **Pagination Support**: All data retrieval methods now support pagination to avoid loading entire datasets into memory at once. The deprecated `getAllNouns()` and `getAllVerbs()` methods have been replaced with `getNouns()` and `getVerbs()` methods that support pagination, filtering, and cursor-based navigation.
- **Pagination Support**: All data retrieval methods now support pagination to avoid loading entire datasets into memory
at once. The deprecated `getAllNouns()` and `getAllVerbs()` methods have been replaced with `getNouns()` and
`getVerbs()` methods that support pagination, filtering, and cursor-based navigation.
- **Multi-level Caching**: A sophisticated three-level caching strategy has been implemented:
- **Level 1**: Hot cache (most accessed nodes) - RAM (automatically detecting and adjusting in each environment)
@ -1429,13 +1440,16 @@ Brainy has been significantly improved to handle larger datasets more efficientl
- In Node.js: Uses 10% of free memory (minimum 1000 entries)
- In browsers: Scales based on device memory (500 entries per GB, minimum 1000)
- **Intelligent Cache Eviction**: Implements a Least Recently Used (LRU) policy that evicts the oldest 20% of items when the cache reaches the configured threshold.
- **Intelligent Cache Eviction**: Implements a Least Recently Used (LRU) policy that evicts the oldest 20% of items when
the cache reaches the configured threshold.
- **Prefetching Strategy**: Implements batch prefetching to improve performance while avoiding overwhelming system resources.
- **Prefetching Strategy**: Implements batch prefetching to improve performance while avoiding overwhelming system
resources.
### S3-Compatible Storage Improvements
- **Enhanced Cloud Storage**: Improved support for S3-compatible storage services including AWS S3, Cloudflare R2, and others.
- **Enhanced Cloud Storage**: Improved support for S3-compatible storage services including AWS S3, Cloudflare R2, and
others.
- **Optimized Data Access**: Batch operations and error handling for efficient cloud storage access.
@ -1445,9 +1459,11 @@ Brainy has been significantly improved to handle larger datasets more efficientl
Yes, you can use existing data indexed from an old version. Brainy includes robust data migration capabilities:
- **Vector Regeneration**: If vectors are missing in imported data, they will be automatically created using the embedding function.
- **Vector Regeneration**: If vectors are missing in imported data, they will be automatically created using the
embedding function.
- **HNSW Index Reconstruction**: The system can reconstruct the HNSW index from backup data, ensuring compatibility with previous versions.
- **HNSW Index Reconstruction**: The system can reconstruct the HNSW index from backup data, ensuring compatibility with
previous versions.
- **Sparse Data Import**: Support for importing sparse data (without vectors) through the `importSparseData()` method.
@ -1487,7 +1503,8 @@ Read-only mode prevents all write operations (add, update, delete) and is optimi
#### Write-Only Mode
Write-only mode prevents all search operations and is optimized for initial data loading or when you want to optimize for write performance.
Write-only mode prevents all search operations and is optimized for initial data loading or when you want to optimize
for write performance.
- **Memory**:
- Minimum: 512MB RAM
@ -1503,7 +1520,9 @@ Write-only mode prevents all search operations and is optimized for initial data
### Performance Tuning Parameters
Brainy offers comprehensive configuration options for performance tuning, with enhanced support for large datasets in S3 or other remote storage. **All configuration is optional** - the system automatically detects the optimal settings based on your environment, dataset size, and usage patterns.
Brainy offers comprehensive configuration options for performance tuning, with enhanced support for large datasets in S3
or other remote storage. **All configuration is optional** - the system automatically detects the optimal settings based
on your environment, dataset size, and usage patterns.
#### Intelligent Defaults
@ -1577,11 +1596,13 @@ const brainy = new BrainyData({
});
```
These configuration options make Brainy more efficient, scalable, and adaptable to different environments and usage patterns, especially for large datasets in cloud storage.
These configuration options make Brainy more efficient, scalable, and adaptable to different environments and usage
patterns, especially for large datasets in cloud storage.
## Testing
Brainy uses Vitest for testing. For detailed information about testing in Brainy, including test configuration, scripts, reporting tools, and best practices, see our [Testing Guide](TESTING.md).
Brainy uses Vitest for testing. For detailed information about testing in Brainy, including test configuration, scripts,
reporting tools, and best practices, see our [Testing Guide](TESTING.md).
Here are some common test commands:
@ -1605,45 +1626,18 @@ see [DEVELOPERS.md](DEVELOPERS.md).
We have a [Code of Conduct](CODE_OF_CONDUCT.md) that all contributors are expected to follow.
## Release Workflow
Brainy uses a streamlined release workflow that automates version updates, changelog generation, GitHub releases, and NPM deployment.
### Automated Release Process
The release workflow combines several steps into a single command:
1. **Build the project** - Ensures the code compiles correctly
2. **Run tests** - Verifies that all tests pass
3. **Update version** - Bumps the version number (patch, minor, or major)
4. **Generate changelog** - Automatically updates CHANGELOG.md with commit messages since the last release
5. **Create GitHub release** - Creates a GitHub release with auto-generated notes
6. **Publish to NPM** - Deploys the package to NPM
### Release Commands
Use one of the following commands to release a new version:
```bash
# Release with patch version update (0.0.x)
npm run workflow:patch
# Release with minor version update (0.x.0)
npm run workflow:minor
# Release with major version update (x.0.0)
npm run workflow:major
# Default workflow (same as patch)
npm run workflow
# Dry run (build, test, and simulate version update without making changes)
npm run workflow:dry-run
```
### Commit Message Format
For best results with automatic changelog generation, follow the [Conventional Commits](https://www.conventionalcommits.org/) specification for your commit messages:
For best results with automatic changelog generation, follow
the [Conventional Commits](https://www.conventionalcommits.org/) specification for your commit messages:
```
AI Template for automated commit messages:
Use Conventional Commit format
Specify the changes in a structured format
Add information about the purpose of the commit
```
```
<type>(<scope>): <description>
@ -1654,6 +1648,7 @@ For best results with automatic changelog generation, follow the [Conventional C
```
Where `<type>` is one of:
- `feat`: A new feature (maps to **Added** section)
- `fix`: A bug fix (maps to **Fixed** section)
- `chore`: Regular maintenance tasks (maps to **Changed** section)