feat: COW always-on architecture + cloud storage clear() fix (v5.11.0)

Major architectural improvements and critical bug fixes:

## COW Always-On Architecture
- Removed cowEnabled flag from BaseStorage (COW cannot be disabled)
- Eliminated marker file system (checkClearMarker, createClearMarker)
- Simplified all code paths to assume COW is always enabled
- COW automatically re-initializes after clear() operations

## Critical Bug Fix: Cloud Storage clear()
- Fixed GCS clear() using correct paths (branches/ instead of entities/nouns/)
- Fixed S3Compatible clear() path structure
- Fixed R2 clear() implementation
- Fixed Azure, FileSystem, OPFS, Memory clear() COW flag handling
- clear() now deletes: branches/, _cow/, _system/
- Result: Cloud buckets can now be fully cleared (previously impossible)

## Container Memory Detection
- Auto-detect Docker/K8s/Cloud Run memory limits (cgroup v1/v2)
- Smart memory allocation (75% graph data, 25% query operations)
- Environment variable support (CLOUD_RUN_MEMORY, MEMORY_LIMIT)
- Production-grade containerized deployment support

## CommitLog streamHistory Feature
- Added streamable commit history with pagination
- Efficient memory usage for large commit histories
- Support for branch filtering and time ranges

## Comprehensive Storage Documentation
- Complete v5.11.0 file structure reference
- Detailed path construction algorithms
- 8 common storage scenarios with examples
- Type-first storage, sharding, COW architecture explained
- Public docs: docs/architecture/data-storage-architecture.md (1063 lines)

## Files Modified (14 files)
- All 8 storage adapters (GCS, S3, R2, Azure, FS, OPFS, Memory, Historical)
- BaseStorage core architecture
- CommitLog with streaming
- Brainy memory configuration
- Parameter validation with container detection
- Storage architecture documentation

## Breaking Changes
NONE - COW was already enabled by default. This removes the ability to disable it.

## Migration
No action required. Upgrade and clear() will work correctly on cloud storage.

## Impact
- Users can now clear cloud storage buckets completely
- No more corrupted buckets after clear() operations
- Container deployments automatically optimize memory allocation
- COW is mandatory and always enabled (safer, simpler)

v5.11.0 - Production ready
This commit is contained in:
David Snelling 2025-11-18 13:44:02 -08:00
parent 28160a3052
commit 3e8b9aacc8
17 changed files with 2925 additions and 1207 deletions

View file

@ -267,6 +267,54 @@ export class CommitLog {
return commits
}
/**
* Stream commit history (memory-efficient for large histories)
*
* Yields commits one at a time without accumulating in memory.
* Use this for large commit histories (1000s of commits) where
* memory efficiency is important.
*
* @param ref - Starting ref
* @param options - Walk options
* @yields Commits in reverse chronological order (newest first)
*
* @example
* ```typescript
* // Stream all commits without memory accumulation
* for await (const commit of commitLog.streamHistory('main', { maxCount: 10000 })) {
* console.log(commit.message)
* }
* ```
*/
async *streamHistory(
ref: string,
options?: {
maxCount?: number
since?: number
until?: number
}
): AsyncIterableIterator<CommitObject> {
let count = 0
for await (const commit of this.walk(ref, {
maxDepth: options?.maxCount,
until: options?.until
})) {
// Filter by since timestamp if provided
if (options?.since && commit.timestamp < options.since) {
continue
}
yield commit
count++
// Stop after maxCount commits
if (options?.maxCount && count >= options.maxCount) {
break
}
}
}
/**
* Count commits between two commits
*