feat: COW always-on architecture + cloud storage clear() fix (v5.11.0)
Major architectural improvements and critical bug fixes: ## COW Always-On Architecture - Removed cowEnabled flag from BaseStorage (COW cannot be disabled) - Eliminated marker file system (checkClearMarker, createClearMarker) - Simplified all code paths to assume COW is always enabled - COW automatically re-initializes after clear() operations ## Critical Bug Fix: Cloud Storage clear() - Fixed GCS clear() using correct paths (branches/ instead of entities/nouns/) - Fixed S3Compatible clear() path structure - Fixed R2 clear() implementation - Fixed Azure, FileSystem, OPFS, Memory clear() COW flag handling - clear() now deletes: branches/, _cow/, _system/ - Result: Cloud buckets can now be fully cleared (previously impossible) ## Container Memory Detection - Auto-detect Docker/K8s/Cloud Run memory limits (cgroup v1/v2) - Smart memory allocation (75% graph data, 25% query operations) - Environment variable support (CLOUD_RUN_MEMORY, MEMORY_LIMIT) - Production-grade containerized deployment support ## CommitLog streamHistory Feature - Added streamable commit history with pagination - Efficient memory usage for large commit histories - Support for branch filtering and time ranges ## Comprehensive Storage Documentation - Complete v5.11.0 file structure reference - Detailed path construction algorithms - 8 common storage scenarios with examples - Type-first storage, sharding, COW architecture explained - Public docs: docs/architecture/data-storage-architecture.md (1063 lines) ## Files Modified (14 files) - All 8 storage adapters (GCS, S3, R2, Azure, FS, OPFS, Memory, Historical) - BaseStorage core architecture - CommitLog with streaming - Brainy memory configuration - Parameter validation with container detection - Storage architecture documentation ## Breaking Changes NONE - COW was already enabled by default. This removes the ability to disable it. ## Migration No action required. Upgrade and clear() will work correctly on cloud storage. ## Impact - Users can now clear cloud storage buckets completely - No more corrupted buckets after clear() operations - Container deployments automatically optimize memory allocation - COW is mandatory and always enabled (safer, simpler) v5.11.0 - Production ready
This commit is contained in:
parent
28160a3052
commit
3e8b9aacc8
17 changed files with 2925 additions and 1207 deletions
|
|
@ -267,6 +267,54 @@ export class CommitLog {
|
|||
return commits
|
||||
}
|
||||
|
||||
/**
|
||||
* Stream commit history (memory-efficient for large histories)
|
||||
*
|
||||
* Yields commits one at a time without accumulating in memory.
|
||||
* Use this for large commit histories (1000s of commits) where
|
||||
* memory efficiency is important.
|
||||
*
|
||||
* @param ref - Starting ref
|
||||
* @param options - Walk options
|
||||
* @yields Commits in reverse chronological order (newest first)
|
||||
*
|
||||
* @example
|
||||
* ```typescript
|
||||
* // Stream all commits without memory accumulation
|
||||
* for await (const commit of commitLog.streamHistory('main', { maxCount: 10000 })) {
|
||||
* console.log(commit.message)
|
||||
* }
|
||||
* ```
|
||||
*/
|
||||
async *streamHistory(
|
||||
ref: string,
|
||||
options?: {
|
||||
maxCount?: number
|
||||
since?: number
|
||||
until?: number
|
||||
}
|
||||
): AsyncIterableIterator<CommitObject> {
|
||||
let count = 0
|
||||
|
||||
for await (const commit of this.walk(ref, {
|
||||
maxDepth: options?.maxCount,
|
||||
until: options?.until
|
||||
})) {
|
||||
// Filter by since timestamp if provided
|
||||
if (options?.since && commit.timestamp < options.since) {
|
||||
continue
|
||||
}
|
||||
|
||||
yield commit
|
||||
count++
|
||||
|
||||
// Stop after maxCount commits
|
||||
if (options?.maxCount && count >= options.maxCount) {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Count commits between two commits
|
||||
*
|
||||
|
|
|
|||
Reference in a new issue