feat(distributed): add distributed mode with multi-instance coordination

Implements Phase 1 and Phase 2 of distributed enhancements for horizontal scaling:

Phase 1 - Zero-Config Distributed Mode:
- Add DistributedConfigManager for shared S3 configuration coordination
- Implement explicit role configuration (reader/writer/hybrid) for safety
- Add instance registration with heartbeat and health monitoring
- Create hash-based partitioner for deterministic data distribution

Phase 2 - Intelligent Data Management:
- Add DomainDetector for automatic data categorization (medical, legal, product, etc.)
- Implement domain-aware search filtering for improved relevance
- Create role-based operational modes with specific optimizations
- Add HealthMonitor for comprehensive metrics tracking

Key Features:
- Multi-writer support with consistent hash partitioning
- Reader instances optimize for 80% cache utilization
- Writer instances optimize for batched writes
- Automatic domain detection and tagging
- Real-time health monitoring across all instances
- Cross-platform crypto utilities for browser compatibility

Safety Improvements:
- Require explicit role configuration (no automatic assignment)
- Validate role compatibility on startup
- Track instance health and performance metrics

Testing:
- Add comprehensive test suite for distributed features
- All 25 distributed tests passing
- Fixed domain filtering in search functionality

Documentation:
- Update README with distributed mode highlights
- Add examples showing reader/writer setup
- Document new capabilities and benefits

🤖 Generated with Claude Code
https://claude.ai/code

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
David Snelling 2025-08-04 12:18:58 -07:00
parent 29625cc1af
commit 26e9c26852
12 changed files with 3301 additions and 3 deletions

47
src/utils/crypto.ts Normal file
View file

@ -0,0 +1,47 @@
/**
* Cross-platform crypto utilities
* Provides hashing functions that work in both Node.js and browser environments
*/
/**
* Simple string hash function that works in all environments
* Uses djb2 algorithm - fast and good distribution
* @param str - String to hash
* @returns Positive integer hash
*/
export function hashString(str: string): number {
let hash = 5381
for (let i = 0; i < str.length; i++) {
const char = str.charCodeAt(i)
hash = ((hash << 5) + hash) + char // hash * 33 + char
}
// Ensure positive number
return Math.abs(hash)
}
/**
* Alternative: FNV-1a hash algorithm
* Good distribution and fast
* @param str - String to hash
* @returns Positive integer hash
*/
export function fnv1aHash(str: string): number {
let hash = 2166136261
for (let i = 0; i < str.length; i++) {
hash ^= str.charCodeAt(i)
hash = (hash * 16777619) >>> 0
}
return hash
}
/**
* Generate a deterministic hash for partitioning
* Uses the most appropriate algorithm for the environment
* @param input - Input string to hash
* @returns Positive integer hash suitable for modulo operations
*/
export function getPartitionHash(input: string): number {
// Use djb2 by default as it's fast and has good distribution
// This ensures consistent partitioning across all environments
return hashString(input)
}