feat: COW always-on architecture + cloud storage clear() fix (v5.11.0)
Major architectural improvements and critical bug fixes: ## COW Always-On Architecture - Removed cowEnabled flag from BaseStorage (COW cannot be disabled) - Eliminated marker file system (checkClearMarker, createClearMarker) - Simplified all code paths to assume COW is always enabled - COW automatically re-initializes after clear() operations ## Critical Bug Fix: Cloud Storage clear() - Fixed GCS clear() using correct paths (branches/ instead of entities/nouns/) - Fixed S3Compatible clear() path structure - Fixed R2 clear() implementation - Fixed Azure, FileSystem, OPFS, Memory clear() COW flag handling - clear() now deletes: branches/, _cow/, _system/ - Result: Cloud buckets can now be fully cleared (previously impossible) ## Container Memory Detection - Auto-detect Docker/K8s/Cloud Run memory limits (cgroup v1/v2) - Smart memory allocation (75% graph data, 25% query operations) - Environment variable support (CLOUD_RUN_MEMORY, MEMORY_LIMIT) - Production-grade containerized deployment support ## CommitLog streamHistory Feature - Added streamable commit history with pagination - Efficient memory usage for large commit histories - Support for branch filtering and time ranges ## Comprehensive Storage Documentation - Complete v5.11.0 file structure reference - Detailed path construction algorithms - 8 common storage scenarios with examples - Type-first storage, sharding, COW architecture explained - Public docs: docs/architecture/data-storage-architecture.md (1063 lines) ## Files Modified (14 files) - All 8 storage adapters (GCS, S3, R2, Azure, FS, OPFS, Memory, Historical) - BaseStorage core architecture - CommitLog with streaming - Brainy memory configuration - Parameter validation with container detection - Storage architecture documentation ## Breaking Changes NONE - COW was already enabled by default. This removes the ability to disable it. ## Migration No action required. Upgrade and clear() will work correctly on cloud storage. ## Impact - Users can now clear cloud storage buckets completely - No more corrupted buckets after clear() operations - Container deployments automatically optimize memory allocation - COW is mandatory and always enabled (safer, simpler) v5.11.0 - Production ready
This commit is contained in:
parent
28160a3052
commit
3e8b9aacc8
17 changed files with 2925 additions and 1207 deletions
|
|
@ -995,31 +995,21 @@ export class GcsStorage extends BaseStorage {
|
|||
}
|
||||
}
|
||||
|
||||
// Clear all data directories
|
||||
await deleteObjectsWithPrefix(this.nounPrefix)
|
||||
await deleteObjectsWithPrefix(this.verbPrefix)
|
||||
await deleteObjectsWithPrefix(this.metadataPrefix)
|
||||
await deleteObjectsWithPrefix(this.verbMetadataPrefix)
|
||||
await deleteObjectsWithPrefix(this.systemPrefix)
|
||||
// v5.11.0: Clear ALL data using correct paths
|
||||
// Delete entire branches/ directory (includes ALL entities, ALL types, ALL VFS data, ALL forks)
|
||||
await deleteObjectsWithPrefix('branches/')
|
||||
|
||||
// v5.6.1: Clear COW (copy-on-write) version control data
|
||||
// This includes all git-like versioning data (commits, trees, blobs, refs)
|
||||
// Must be deleted to fully clear all data including version history
|
||||
// Delete COW version control data
|
||||
await deleteObjectsWithPrefix('_cow/')
|
||||
|
||||
// CRITICAL: Reset COW state to prevent automatic reinitialization
|
||||
// When COW data is cleared, we must also clear the COW managers
|
||||
// Otherwise initializeCOW() will auto-recreate initial commit on next operation
|
||||
// Delete system metadata
|
||||
await deleteObjectsWithPrefix('_system/')
|
||||
|
||||
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled)
|
||||
// COW will re-initialize automatically on next use
|
||||
this.refManager = undefined
|
||||
this.blobStorage = undefined
|
||||
this.commitLog = undefined
|
||||
this.cowEnabled = false
|
||||
|
||||
// v5.10.4: Create persistent marker object (CRITICAL FIX)
|
||||
// Bug: cowEnabled = false only affects current instance, not future instances
|
||||
// Fix: Create marker object that persists across instance restarts
|
||||
// When new instance calls initializeCOW(), it checks for this marker
|
||||
await this.createClearMarker()
|
||||
|
||||
// Clear caches
|
||||
this.nounCacheManager.clear()
|
||||
|
|
@ -1081,38 +1071,10 @@ export class GcsStorage extends BaseStorage {
|
|||
* @returns true if marker object exists, false otherwise
|
||||
* @protected
|
||||
*/
|
||||
protected async checkClearMarker(): Promise<boolean> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
try {
|
||||
const markerPath = `${this.systemPrefix}cow-disabled`
|
||||
const file = this.bucket!.file(markerPath)
|
||||
const [exists] = await file.exists()
|
||||
return exists
|
||||
} catch (error) {
|
||||
this.logger.warn('GCSStorage.checkClearMarker: Error checking marker', error)
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Create marker indicating COW has been explicitly disabled
|
||||
* v5.10.4: Called by clear() to prevent COW reinitialization on new instances
|
||||
* @protected
|
||||
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
|
||||
* COW is now always enabled - marker files are no longer used
|
||||
*/
|
||||
protected async createClearMarker(): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
try {
|
||||
const markerPath = `${this.systemPrefix}cow-disabled`
|
||||
const file = this.bucket!.file(markerPath)
|
||||
// Create empty marker object
|
||||
await file.save('', { contentType: 'text/plain' })
|
||||
} catch (error) {
|
||||
this.logger.error('GCSStorage.createClearMarker: Failed to create marker object', error)
|
||||
// Don't throw - marker creation failure shouldn't break clear()
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Save statistics data to storage
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue