Critical bug fix that restores data persistence after application restart for all storage adapters (GCS, S3, OPFS, FileSystem).
**Root Cause:**
Storage adapters were loading entity counts from _system/counts.json but not returning totalCount in pagination methods, causing rebuildIndexesIfNeeded() to see undefined and incorrectly assume no data existed.
**Changes:**
- GCS Storage: Return totalCount in getNodesWithPagination() and getVerbsWithPagination()
- S3 Storage: Call initializeCounts() in init() and return totalCount in pagination methods
- OPFS Storage: Call initializeCounts() in init() to load pre-calculated counts
- BaseStorage: Add defensive validation to prevent future adapters from missing totalCount
**Impact:**
- Production deployments now work correctly (Cloud Run, AWS, Kubernetes)
- Data persists correctly across container restarts
- Serverless scale-to-zero deployments now functional
- All storage adapters use O(1) pre-calculated counts (scales to millions of entities)
**Testing:**
- Build passes with no TypeScript errors
- All existing tests pass
- Defensive validation ensures future storage adapters won't have this bug
Fixes critical production issue affecting all persistent storage deployments.
Critical fix for GCS/S3 native storage adapters crashing on metadata index keys.
Problem:
- GCS and S3 adapters crashed with "Invalid UUID format" errors
- System keys like __metadata_field_index__status are NOT UUIDs
- Adapters incorrectly tried to shard all metadata keys as UUIDs
Solution:
- Move sharding/routing logic from adapters to BaseStorage class
- Add analyzeKey() method to detect system keys vs entity UUIDs
- System keys route to _system/ directory (no sharding)
- Entity UUIDs route to sharded directories (256 shards)
- All adapters now implement 4 primitive operations:
* writeObjectToPath(path, data)
* readObjectFromPath(path)
* deleteObjectFromPath(path)
* listObjectsUnderPath(prefix)
Benefits:
- Impossible for future adapters to repeat this mistake
- Zero breaking changes, full backward compatibility
- No data migration required
- Cleaner architecture with better separation of concerns
Updated adapters: GcsStorage, S3CompatibleStorage, OPFSStorage,
FileSystemStorage, MemoryStorage
Added: docs/architecture/data-storage-architecture.md
Updated: README.md with architecture docs link
