feat(distributed): add distributed mode with multi-instance coordination
Implements Phase 1 and Phase 2 of distributed enhancements for horizontal scaling: Phase 1 - Zero-Config Distributed Mode: - Add DistributedConfigManager for shared S3 configuration coordination - Implement explicit role configuration (reader/writer/hybrid) for safety - Add instance registration with heartbeat and health monitoring - Create hash-based partitioner for deterministic data distribution Phase 2 - Intelligent Data Management: - Add DomainDetector for automatic data categorization (medical, legal, product, etc.) - Implement domain-aware search filtering for improved relevance - Create role-based operational modes with specific optimizations - Add HealthMonitor for comprehensive metrics tracking Key Features: - Multi-writer support with consistent hash partitioning - Reader instances optimize for 80% cache utilization - Writer instances optimize for batched writes - Automatic domain detection and tagging - Real-time health monitoring across all instances - Cross-platform crypto utilities for browser compatibility Safety Improvements: - Require explicit role configuration (no automatic assignment) - Validate role compatibility on startup - Track instance health and performance metrics Testing: - Add comprehensive test suite for distributed features - All 25 distributed tests passing - Fixed domain filtering in search functionality Documentation: - Update README with distributed mode highlights - Add examples showing reader/writer setup - Document new capabilities and benefits 🤖 Generated with Claude Code https://claude.ai/code Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
29625cc1af
commit
26e9c26852
12 changed files with 3301 additions and 3 deletions
47
src/utils/crypto.ts
Normal file
47
src/utils/crypto.ts
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
/**
|
||||
* Cross-platform crypto utilities
|
||||
* Provides hashing functions that work in both Node.js and browser environments
|
||||
*/
|
||||
|
||||
/**
|
||||
* Simple string hash function that works in all environments
|
||||
* Uses djb2 algorithm - fast and good distribution
|
||||
* @param str - String to hash
|
||||
* @returns Positive integer hash
|
||||
*/
|
||||
export function hashString(str: string): number {
|
||||
let hash = 5381
|
||||
for (let i = 0; i < str.length; i++) {
|
||||
const char = str.charCodeAt(i)
|
||||
hash = ((hash << 5) + hash) + char // hash * 33 + char
|
||||
}
|
||||
// Ensure positive number
|
||||
return Math.abs(hash)
|
||||
}
|
||||
|
||||
/**
|
||||
* Alternative: FNV-1a hash algorithm
|
||||
* Good distribution and fast
|
||||
* @param str - String to hash
|
||||
* @returns Positive integer hash
|
||||
*/
|
||||
export function fnv1aHash(str: string): number {
|
||||
let hash = 2166136261
|
||||
for (let i = 0; i < str.length; i++) {
|
||||
hash ^= str.charCodeAt(i)
|
||||
hash = (hash * 16777619) >>> 0
|
||||
}
|
||||
return hash
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate a deterministic hash for partitioning
|
||||
* Uses the most appropriate algorithm for the environment
|
||||
* @param input - Input string to hash
|
||||
* @returns Positive integer hash suitable for modulo operations
|
||||
*/
|
||||
export function getPartitionHash(input: string): number {
|
||||
// Use djb2 by default as it's fast and has good distribution
|
||||
// This ensures consistent partitioning across all environments
|
||||
return hashString(input)
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue