feat: COW always-on architecture + cloud storage clear() fix (v5.11.0)

Major architectural improvements and critical bug fixes:

## COW Always-On Architecture
- Removed cowEnabled flag from BaseStorage (COW cannot be disabled)
- Eliminated marker file system (checkClearMarker, createClearMarker)
- Simplified all code paths to assume COW is always enabled
- COW automatically re-initializes after clear() operations

## Critical Bug Fix: Cloud Storage clear()
- Fixed GCS clear() using correct paths (branches/ instead of entities/nouns/)
- Fixed S3Compatible clear() path structure
- Fixed R2 clear() implementation
- Fixed Azure, FileSystem, OPFS, Memory clear() COW flag handling
- clear() now deletes: branches/, _cow/, _system/
- Result: Cloud buckets can now be fully cleared (previously impossible)

## Container Memory Detection
- Auto-detect Docker/K8s/Cloud Run memory limits (cgroup v1/v2)
- Smart memory allocation (75% graph data, 25% query operations)
- Environment variable support (CLOUD_RUN_MEMORY, MEMORY_LIMIT)
- Production-grade containerized deployment support

## CommitLog streamHistory Feature
- Added streamable commit history with pagination
- Efficient memory usage for large commit histories
- Support for branch filtering and time ranges

## Comprehensive Storage Documentation
- Complete v5.11.0 file structure reference
- Detailed path construction algorithms
- 8 common storage scenarios with examples
- Type-first storage, sharding, COW architecture explained
- Public docs: docs/architecture/data-storage-architecture.md (1063 lines)

## Files Modified (14 files)
- All 8 storage adapters (GCS, S3, R2, Azure, FS, OPFS, Memory, Historical)
- BaseStorage core architecture
- CommitLog with streaming
- Brainy memory configuration
- Parameter validation with container detection
- Storage architecture documentation

## Breaking Changes
NONE - COW was already enabled by default. This removes the ability to disable it.

## Migration
No action required. Upgrade and clear() will work correctly on cloud storage.

## Impact
- Users can now clear cloud storage buckets completely
- No more corrupted buckets after clear() operations
- Container deployments automatically optimize memory allocation
- COW is mandatory and always enabled (safer, simpler)

v5.11.0 - Production ready
This commit is contained in:
David Snelling 2025-11-18 13:44:02 -08:00
parent 28160a3052
commit 3e8b9aacc8
17 changed files with 2925 additions and 1207 deletions

View file

@ -488,19 +488,11 @@ export class OPFSStorage extends BaseStorage {
// Delete the entire _cow/ directory (not just contents)
await this.rootDir!.removeEntry('_cow', { recursive: true })
// CRITICAL: Reset COW state to prevent automatic reinitialization
// When COW data is cleared, we must also clear the COW managers
// Otherwise initializeCOW() will auto-recreate initial commit on next operation
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use
this.refManager = undefined
this.blobStorage = undefined
this.commitLog = undefined
this.cowEnabled = false
// v5.10.4: Create persistent marker file (CRITICAL FIX)
// Bug: cowEnabled = false only affects current instance, not future instances
// Fix: Create marker file that persists across instance restarts
// When new instance calls initializeCOW(), it checks for this marker
await this.createClearMarker()
} catch (error: any) {
// Ignore if _cow directory doesn't exist (not all instances use COW)
if (error.name !== 'NotFoundError') {
@ -528,46 +520,10 @@ export class OPFSStorage extends BaseStorage {
* @returns true if marker file exists, false otherwise
* @protected
*/
protected async checkClearMarker(): Promise<boolean> {
await this.ensureInitialized()
try {
// Get system directory (may not exist yet)
const systemDir = await this.rootDir!.getDirectoryHandle('system', { create: false })
// Try to get the marker file
await systemDir.getFileHandle('cow-disabled', { create: false })
return true // Marker exists
} catch (error: any) {
if (error.name === 'NotFoundError') {
return false // Marker doesn't exist (or system dir doesn't exist)
}
// Other errors (permissions, etc.) - treat as marker not existing
console.warn('OPFSStorage.checkClearMarker: Error checking marker', error)
return false
}
}
/**
* Create marker indicating COW has been explicitly disabled
* v5.10.4: Called by clear() to prevent COW reinitialization on new instances
* @protected
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
protected async createClearMarker(): Promise<void> {
await this.ensureInitialized()
try {
// Get or create system directory
const systemDir = await this.rootDir!.getDirectoryHandle('system', { create: true })
// Create empty marker file
const fileHandle = await systemDir.getFileHandle('cow-disabled', { create: true })
const writable = await fileHandle.createWritable()
await writable.write(new Uint8Array(0)) // Empty file
await writable.close()
} catch (error) {
console.error('OPFSStorage.createClearMarker: Failed to create marker file', error)
// Don't throw - marker creation failure shouldn't break clear()
}
}
// Quota monitoring configuration (v4.0.0)
private quotaWarningThreshold = 0.8 // Warn at 80% usage