soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat(docs): add comprehensive user guides and installation instructions for Brainy

Browse source
This commit is contained in:
David Snelling 2025-08-03 17:33:52 -07:00
parent c92eeae234
commit 5799a79f0c
15 changed files with 1859 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

114
docs/user-guides/README.md Normal file
View file

@ -0,0 +1,114 @@
# User Guides
Comprehensive guides for using Brainy's features effectively in your applications.
## 📖 Available Guides
### 🔍 [Search and Metadata Guide](SEARCH_AND_METADATA_GUIDE.md)
Master advanced search techniques and metadata management.
- Field-specific search strategies
- JSON document search patterns
- Metadata optimization techniques
- Search performance tips
### ✏️ [Write-Only Mode Implementation](WRITEONLY_MODE_IMPLEMENTATION.md)
Optimize data ingestion with write-only mode.
- High-throughput data loading
- Batch operation strategies
- Performance optimization for writes
- Use cases and implementations
### 💾 [Cache Configuration](../guides/cache-configuration.md)
Configure caching for optimal performance.
- Multi-level caching strategies
- Memory optimization techniques
- Cache hit rate improvement
- Environment-specific settings
### 📄 [JSON Document Search](../guides/json-document-search.md)
Search within JSON document fields.
- Field prioritization strategies
- Service-based field mapping
- Cross-service search patterns
- Advanced querying techniques
### 🎯 [HNSW Field Search](../guides/hnsw-field-search.md)
Field-specific vector search techniques.
- Targeted field searches
- Search scope optimization
- Performance considerations
- Integration patterns
### 🤖 [Model Management](../guides/model-management.md)
Manage AI models and embeddings effectively.
- Embedding model selection
- Custom embedding functions
- Model performance optimization
- Version management strategies
### 🚀 [Production Migration Guide](../guides/production-migration-guide.md)
Best practices for production deployment.
- Environment transition strategies
- Data migration techniques
- Performance monitoring setup
- Scaling considerations
### 🔌 [Service Identification](../guides/service-identification.md)
Track and manage data from multiple services.
- Service-based data organization
- Cross-service search capabilities
- Data source attribution
- Integration patterns
### 📦 [Optional Model Bundling](../guides/optional-model-bundling.md)
Optimize bundle size with selective model loading.
- Bundle size optimization
- Conditional model loading
- Performance trade-offs
- Deployment strategies
## 🎯 How to Use These Guides
### For New Users
1. Start with **[Search and Metadata Guide](SEARCH_AND_METADATA_GUIDE.md)** to understand core search concepts
2. Review **[JSON Document Search](../guides/json-document-search.md)** for working with structured data
3. Check **[Cache Configuration](../guides/cache-configuration.md)** for performance optimization
### For Production Deployments
1. Read **[Production Migration Guide](../guides/production-migration-guide.md)** for deployment best practices
2. Implement **[Write-Only Mode](WRITEONLY_MODE_IMPLEMENTATION.md)** for high-throughput scenarios
3. Configure **[Model Management](../guides/model-management.md)** for optimal AI model usage
### For Advanced Use Cases
1. Master **[HNSW Field Search](../guides/hnsw-field-search.md)** for specialized search patterns
2. Implement **[Service Identification](../guides/service-identification.md)** for multi-service architectures
3. Optimize with **[Optional Model Bundling](../guides/optional-model-bundling.md)** for specific deployment needs
## 🔗 Related Documentation
- **[Getting Started](../getting-started/)** - Basic setup and first steps
- **[Optimization Guides](../optimization-guides/)** - Performance and scaling
- **[API Reference](../api-reference/)** - Complete API documentation
- **[Examples](../examples/)** - Practical code examples
## 💡 Pro Tips
- **Read Guides in Order**: Each guide builds on concepts from previous ones
- **Test Examples**: All guides include working code examples
- **Check Performance**: Monitor metrics after implementing guide recommendations
- **Stay Updated**: Guides are updated with new features and best practices
## 🆘 Need Help?
- **[Troubleshooting](../troubleshooting/)** - Common issues and solutions
- **[GitHub Issues](https://github.com/soulcraft-research/brainy/issues)** - Report bugs or request features
- **[GitHub Discussions](https://github.com/soulcraft-research/brainy/discussions)** - Community support and questions

322
docs/user-guides/SEARCH_AND_METADATA_GUIDE.md Normal file
View file

@ -0,0 +1,322 @@
# Brainy Library: Search and Metadata Retrieval Guide
## Overview
The Brainy library provides a seamless, fast, and easy way to search for similar content and retrieve complete metadata for nouns (entities) and verbs (relationships) in your knowledge graph. This guide explains how the search and metadata retrieval process works under the hood and how to use it effectively.
## How It Works: The Complete Workflow
### 1. Search Process Architecture
The Brainy library uses a sophisticated multi-layered approach for search and metadata retrieval:
```
User Query → Vector Embedding → HNSW Index Search → Metadata Retrieval → Complete Results
```
### 2. Core Components
#### SearchResult Structure
Every search returns results in a consistent format:
```typescript
interface SearchResult<T = any> {
id: string // Unique identifier
score: number // Similarity score (0-1, higher = more similar)
vector: Vector // The vector representation
metadata?: T // Complete metadata object
}
```
#### GraphNoun Metadata
Nouns contain rich metadata including:
```typescript
interface GraphNoun {
id: string // Unique identifier
createdBy: CreatorMetadata // Source/augmentation info
noun: NounType // Type (Person, Location, Thing, etc.)
createdAt: Timestamp // Creation time
updatedAt: Timestamp // Last update time
label?: string // Descriptive label
data?: Record<string, any> // Flexible additional data
embeddedVerbs?: EmbeddedGraphVerb[] // Related relationships
embedding?: number[] // Vector representation
}
```
#### GraphVerb Metadata
Verbs contain relationship metadata:
```typescript
interface GraphVerb {
id: string // Unique identifier
source: string // Source noun ID
target: string // Target noun ID
label?: string // Descriptive label
verb: VerbType // Relationship type
createdAt: Timestamp // Creation time
updatedAt: Timestamp // Last update time
createdBy: CreatorMetadata // Source/augmentation info
data?: Record<string, any> // Flexible additional data
embedding?: number[] // Vector representation
confidence?: number // Confidence score (0-1)
weight?: number // Relationship strength
}
```
## Usage Examples
### 1. Basic Search with Full Metadata
```typescript
import { BrainyData } from 'brainy'
const brainy = new BrainyData({
dimensions: 384,
// ... other config
})
await brainy.init()
// Search for similar content - returns complete metadata automatically
const results = await brainy.search("artificial intelligence", 10)
results.forEach(result => {
console.log(`ID: ${result.id}`)
console.log(`Similarity: ${result.score}`)
console.log(`Metadata:`, result.metadata)
// Metadata contains ALL the noun/verb information:
// - Type classification
// - Creation timestamps
// - Creator information
// - Custom data fields
// - Embedded relationships
})
```
### 2. Search Specific Noun Types
```typescript
// Search only within specific entity types
const personResults = await brainy.search("John Smith", 5, {
nounTypes: ['Person']
})
const locationResults = await brainy.search("New York", 5, {
nounTypes: ['Location', 'Organization']
})
```
### 3. Search for Relationships (Verbs)
```typescript
// Search for relationships directly
const relationshipResults = await brainy.search("works at", 10, {
searchVerbs: true,
verbTypes: ['Employment', 'Association']
})
relationshipResults.forEach(result => {
console.log(`Relationship: ${result.metadata.verb}`)
console.log(`From: ${result.metadata.source}`)
console.log(`To: ${result.metadata.target}`)
console.log(`Confidence: ${result.metadata.confidence}`)
})
```
### 4. Search Connected Entities
```typescript
// Find entities connected through relationships
const connectedResults = await brainy.search("technology company", 10, {
searchConnectedNouns: true,
verbTypes: ['Partnership', 'Investment'],
verbDirection: 'both'
})
```
### 5. Multi-Modal Search
```typescript
// Search with different input types
const textResults = await brainy.search("machine learning")
const vectorResults = await brainy.search([0.1, 0.2, 0.3, 0.4]) // Direct vector
const objectResults = await brainy.search({
title: "AI Research",
description: "Latest developments in artificial intelligence"
})
```
## Performance Optimizations
### 1. Seamless Operation
- **Single API Call**: One `search()` call returns both similarity results AND complete metadata
- **Automatic Embedding**: Text queries are automatically converted to vectors
- **Flexible Input**: Accepts text, objects, or pre-computed vectors
- **Consistent Output**: Always returns the same `SearchResult` format
### 2. Speed Optimizations
#### HNSW Index
- Uses Hierarchical Navigable Small World (HNSW) algorithm for sub-linear search time
- Approximate nearest neighbor search with high accuracy
- Scales efficiently to millions of vectors
#### Lazy Loading
```typescript
const brainy = new BrainyData({
readOnly: true,
lazyLoadInReadOnlyMode: true // Load nodes on-demand during search
})
```
#### Parallel Processing
- Metadata retrieval happens in parallel for all search results
- Multiple noun types searched concurrently
- Batch operations for efficiency
#### Caching
```typescript
const brainy = new BrainyData({
storage: {
cacheConfig: {
hotCacheMaxSize: 10000, // Keep frequently accessed items in memory
warmCacheTTL: 3600000, // 1 hour TTL for warm cache
batchSize: 100, // Batch size for operations
autoTune: true // Automatically optimize cache settings
}
}
})
```
### 3. Easy Usage Features
#### Search Modes
```typescript
// Local search only
const localResults = await brainy.search("query", 10, {
searchMode: 'local'
})
// Remote server search
const remoteResults = await brainy.search("query", 10, {
searchMode: 'remote'
})
// Combined local + remote
const combinedResults = await brainy.search("query", 10, {
searchMode: 'combined'
})
```
#### Field-Specific Search
```typescript
// Search within specific JSON fields
const titleResults = await brainy.search("AI", 10, {
searchField: 'title'
})
// Prioritize certain fields
const prioritizedResults = await brainy.search("research", 10, {
priorityFields: ['title', 'abstract', 'keywords']
})
```
#### Service Filtering
```typescript
// Filter results by the service that created them
const serviceResults = await brainy.search("data", 10, {
service: 'my-data-ingestion-service'
})
```
## Advanced Features
### 1. Augmentation Pipeline
The library supports augmentation pipelines that can enrich search results:
```typescript
const brainy = new BrainyData({
augmentations: [
{
type: AugmentationType.ServerSearch,
config: { /* server config */ }
}
]
})
```
### 2. Custom Embedding Functions
```typescript
const brainy = new BrainyData({
embeddingFunction: async (data) => {
// Your custom embedding logic
return await myCustomEmbedder.embed(data)
}
})
```
### 3. Storage Adapters
Multiple storage backends supported for optimal performance:
```typescript
const brainy = new BrainyData({
storage: {
// Cloud storage for scalability
s3Storage: {
bucketName: 'my-brainy-data',
region: 'us-east-1'
},
// Local caching for speed
cacheConfig: {
hotCacheMaxSize: 50000
}
}
})
```
## Key Benefits
### 🚀 **Seamless**
- Single API call gets both search results and complete metadata
- No need for separate metadata lookup calls
- Consistent interface across all search types
### ⚡ **Quick**
- HNSW index provides sub-linear search time
- Parallel metadata retrieval
- Intelligent caching and lazy loading
- Optimized storage access patterns
### 🎯 **Easy**
- Simple `search()` method handles all complexity
- Flexible input types (text, vectors, objects)
- Rich configuration options with sensible defaults
- Comprehensive TypeScript types for IDE support
## Error Handling
The library provides robust error handling:
```typescript
try {
const results = await brainy.search("query", 10)
} catch (error) {
if (error.message.includes('write-only mode')) {
// Handle write-only mode error
} else if (error.message.includes('not initialized')) {
// Handle initialization error
}
}
```
## Conclusion
The Brainy library's search and metadata retrieval system is designed to be:
1. **Seamless**: One call gets everything you need
2. **Fast**: Optimized algorithms and caching strategies
3. **Easy**: Simple API with powerful features under the hood
Whether you're building a recommendation system, knowledge graph explorer, or semantic search application, Brainy handles the complexity of vector search and metadata management so you can focus on your application logic.

149
docs/user-guides/WRITEONLY_MODE_IMPLEMENTATION.md Normal file
View file

@ -0,0 +1,149 @@
# Write-Only Mode Implementation Summary
## Overview
This implementation addresses the GitHub issue regarding write-only mode behavior in Brainy, specifically:
1. Enabling existence checks in write-only mode via direct storage queries
2. Improving placeholder noun handling to avoid indexing mock data
3. Ensuring auto-configuration works seamlessly
## Changes Made
### 1. Enhanced `get()` Method for Write-Only Mode
**File**: `src/brainyData.ts` (lines 2084-2105)
- Modified to query storage directly when in write-only mode since index is not loaded
- Added fallback logic for normal mode to check storage if item not found in index
- Maintains backward compatibility while enabling existence checks in write-only mode
```typescript
// In write-only mode, query storage directly since index is not loaded
if (this.writeOnly) {
try {
noun = await this.storage!.getNoun(id)
} catch (storageError) {
return null
}
} else {
// Normal mode: Get noun from index first, fallback to storage
noun = this.index.getNouns().get(id)
if (!noun && this.storage) {
try {
noun = await this.storage.getNoun(id)
} catch (storageError) {
return null
}
}
}
```
### 2. Enhanced `add()` Method for Existence Checks
**File**: `src/brainyData.ts` (lines 1211-1247)
- Added comprehensive existence checking for both write-only and normal modes
- Detects and handles placeholder noun replacement when real data is provided
- Skips index operations in write-only mode while maintaining storage operations
```typescript
// Check for existing noun (both write-only and normal modes)
let existingNoun: HNSWNoun | undefined
if (options.id) {
// Check if existing noun is a placeholder and replace with real data
const isPlaceholder = existingMetadata &&
typeof existingMetadata === 'object' &&
(existingMetadata as any).isPlaceholder
if (isPlaceholder) {
console.log(`Replacing placeholder noun ${options.id} with real data`)
}
}
```
### 3. Improved Placeholder Noun Handling
**File**: `src/brainyData.ts` (lines 2614, 2636)
- Added `isPlaceholder: true` flag to placeholder nouns created in `addVerb()` method
- Ensures placeholder nouns are marked as non-searchable mock data
```typescript
const sourceMetadata = options.missingNounMetadata || {
autoCreated: true,
writeOnlyMode: true,
isPlaceholder: true, // Mark as placeholder to exclude from search results
// ... other metadata
}
```
### 4. Search Result Filtering
**File**: `src/brainyData.ts` (lines 1998-2006)
- Added filtering logic to exclude placeholder nouns from search results
- Prevents mock data from appearing in user-facing search results
```typescript
// Filter out placeholder nouns from search results
searchResults = searchResults.filter(result => {
if (result.metadata && typeof result.metadata === 'object') {
const metadata = result.metadata as Record<string, any>
return !metadata.isPlaceholder
}
return true
})
```
### 5. Enhanced Error Messages
**File**: `src/brainyData.ts` (lines 534-540)
- Updated `checkWriteOnly()` method to provide more helpful error messages
- Guides users to use `get()` for existence checks in write-only mode
```typescript
private checkWriteOnly(allowExistenceChecks: boolean = false): void {
if (this.writeOnly && !allowExistenceChecks) {
throw new Error(
'Cannot perform search operation: database is in write-only mode. Use get() for existence checks.'
)
}
}
```
## Key Features Implemented
### ✅ Existence Checks in Write-Only Mode
- `get()` method now works in write-only mode by querying storage directly
- No need for separate writeOnlyMode parameter on addverb - the system auto-detects
### ✅ Placeholder Noun Management
- Placeholder nouns are marked with `isPlaceholder: true` flag
- Automatically filtered out of search results to prevent mock data visibility
- Mechanism to replace placeholders when real data is found
### ✅ Auto-Configuration
- Brainy automatically detects write-only mode and skips index loading
- Seamless fallback to storage queries when index is not available
- No additional configuration required from users
### ✅ Backward Compatibility
- All existing functionality preserved
- Enhanced error messages guide users to proper usage
- Graceful handling of edge cases and race conditions
## Testing Results
The implementation was thoroughly tested with a comprehensive reproduction script that verified:
1. ✅ Search operations properly blocked in write-only mode with helpful error message
2. ✅ Existence checks (get operations) work in write-only mode via storage
3. ✅ Add operations can check for existing data in write-only mode
4. ✅ Placeholder nouns are filtered out of search results
5. ✅ Mechanism implemented to update placeholder nouns when real data is found
6. ✅ Auto-configuration: Brainy detects write-only mode and skips index loading
## Impact
This implementation fully addresses the original GitHub issue requirements:
- Existence checks are no longer ignored in write-only mode
- They are performed directly against underlying storage as requested
- Placeholder nouns are properly handled and don't appear in search results
- Auto-configuration ensures the best user experience with minimal setup
The solution maintains the principle of making Brainy "auto configure and auto-tune itself so the user experience is simple as possible" while providing robust write-only mode functionality for high-performance data insertion scenarios.