David Snelling
26e9c26852
Implements Phase 1 and Phase 2 of distributed enhancements for horizontal scaling: Phase 1 - Zero-Config Distributed Mode: - Add DistributedConfigManager for shared S3 configuration coordination - Implement explicit role configuration (reader/writer/hybrid) for safety - Add instance registration with heartbeat and health monitoring - Create hash-based partitioner for deterministic data distribution Phase 2 - Intelligent Data Management: - Add DomainDetector for automatic data categorization (medical, legal, product, etc.) - Implement domain-aware search filtering for improved relevance - Create role-based operational modes with specific optimizations - Add HealthMonitor for comprehensive metrics tracking Key Features: - Multi-writer support with consistent hash partitioning - Reader instances optimize for 80% cache utilization - Writer instances optimize for batched writes - Automatic domain detection and tagging - Real-time health monitoring across all instances - Cross-platform crypto utilities for browser compatibility Safety Improvements: - Require explicit role configuration (no automatic assignment) - Validate role compatibility on startup - Track instance health and performance metrics Testing: - Add comprehensive test suite for distributed features - All 25 distributed tests passing - Fixed domain filtering in search functionality Documentation: - Update README with distributed mode highlights - Add examples showing reader/writer setup - Document new capabilities and benefits 🤖 Generated with Claude Code https://claude.ai/code Co-Authored-By: Claude <noreply@anthropic.com>
47 lines
No EOL
1.3 KiB
TypeScript
47 lines
No EOL
1.3 KiB
TypeScript
/**
|
|
* Cross-platform crypto utilities
|
|
* Provides hashing functions that work in both Node.js and browser environments
|
|
*/
|
|
|
|
/**
|
|
* Simple string hash function that works in all environments
|
|
* Uses djb2 algorithm - fast and good distribution
|
|
* @param str - String to hash
|
|
* @returns Positive integer hash
|
|
*/
|
|
export function hashString(str: string): number {
|
|
let hash = 5381
|
|
for (let i = 0; i < str.length; i++) {
|
|
const char = str.charCodeAt(i)
|
|
hash = ((hash << 5) + hash) + char // hash * 33 + char
|
|
}
|
|
// Ensure positive number
|
|
return Math.abs(hash)
|
|
}
|
|
|
|
/**
|
|
* Alternative: FNV-1a hash algorithm
|
|
* Good distribution and fast
|
|
* @param str - String to hash
|
|
* @returns Positive integer hash
|
|
*/
|
|
export function fnv1aHash(str: string): number {
|
|
let hash = 2166136261
|
|
for (let i = 0; i < str.length; i++) {
|
|
hash ^= str.charCodeAt(i)
|
|
hash = (hash * 16777619) >>> 0
|
|
}
|
|
return hash
|
|
}
|
|
|
|
/**
|
|
* Generate a deterministic hash for partitioning
|
|
* Uses the most appropriate algorithm for the environment
|
|
* @param input - Input string to hash
|
|
* @returns Positive integer hash suitable for modulo operations
|
|
*/
|
|
export function getPartitionHash(input: string): number {
|
|
// Use djb2 by default as it's fast and has good distribution
|
|
// This ensures consistent partitioning across all environments
|
|
return hashString(input)
|
|
} |