soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix: move metadata routing to base class, fix GCS/S3 system key crashes

Browse source 
Critical fix for GCS/S3 native storage adapters crashing on metadata index keys.

Problem:
- GCS and S3 adapters crashed with "Invalid UUID format" errors
- System keys like __metadata_field_index__status are NOT UUIDs
- Adapters incorrectly tried to shard all metadata keys as UUIDs

Solution:
- Move sharding/routing logic from adapters to BaseStorage class
- Add analyzeKey() method to detect system keys vs entity UUIDs
- System keys route to _system/ directory (no sharding)
- Entity UUIDs route to sharded directories (256 shards)
- All adapters now implement 4 primitive operations:
  * writeObjectToPath(path, data)
  * readObjectFromPath(path)
  * deleteObjectFromPath(path)
  * listObjectsUnderPath(prefix)

Benefits:
- Impossible for future adapters to repeat this mistake
- Zero breaking changes, full backward compatibility
- No data migration required
- Cleaner architecture with better separation of concerns

Updated adapters: GcsStorage, S3CompatibleStorage, OPFSStorage,
FileSystemStorage, MemoryStorage

Added: docs/architecture/data-storage-architecture.md
Updated: README.md with architecture docs link
This commit is contained in:
David Snelling 2025-10-09 13:10:06 -07:00
parent 13303c20c2
commit 1966c39f24
9 changed files with 1330 additions and 630 deletions
Download patch file Download diff file Expand all files Collapse all files

42
CHANGELOG.md
View file

@ -2,6 +2,48 @@
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
### [3.30.1] - Critical Storage Architecture Fix (2025-10-09)
#### 🐛 Critical Bug Fixes
**Fixed: GCS/S3 Storage Crash on System Metadata Keys**
- GCS and S3 native adapters were crashing with "Invalid UUID format" errors when saving metadata index keys
- Root cause: Storage adapters incorrectly assumed ALL metadata keys are UUIDs
- System keys like `__metadata_field_index__status` and `statistics_` are NOT UUIDs and should not be sharded
**Architecture Improvement: Base Class Enforcement Pattern**
- Moved sharding/routing logic from individual adapters to BaseStorage class
- All adapters now implement 4 primitive operations instead of metadata-specific methods:
- `writeObjectToPath(path, data)` - Write any object to storage
- `readObjectFromPath(path)` - Read any object from storage
- `deleteObjectFromPath(path)` - Delete object from storage
- `listObjectsUnderPath(prefix)` - List objects under path prefix
- BaseStorage.analyzeKey() now routes ALL metadata operations through primitive layer
- System keys automatically routed to `_system/` directory (no sharding)
- Entity UUIDs automatically sharded to `entities/{type}/metadata/{shard}/` directories
**Benefits:**
- Impossible for future adapters to make the same mistake
- Cleaner separation of concerns (routing vs. storage primitives)
- Zero breaking changes for users
- No data migration required
- Full backward compatibility maintained
**Updated Adapters:**
- GcsStorage: Implements primitive operations using GCS bucket.file() API
- S3CompatibleStorage: Implements primitive operations using AWS SDK
- OPFSStorage: Implements primitive operations using browser FileSystem API
- FileSystemStorage: Implements primitive operations using Node.js fs.promises
- MemoryStorage: Implements primitive operations using Map data structures
**Documentation:**
- Added comprehensive storage architecture documentation: `docs/architecture/data-storage-architecture.md`
- Linked from README for easy discovery
**Impact:** CRITICAL FIX - GCS/S3 native storage now fully functional for metadata indexing
---
### [3.30.0](https://github.com/soulcraftlabs/brainy/compare/v3.29.1...v3.30.0) (2025-10-09)
- feat: remove legacy ImportManager, standardize getStats() API (58daf09)