fix: move metadata routing to base class, fix GCS/S3 system key crashes

Critical fix for GCS/S3 native storage adapters crashing on metadata index keys.

Problem:
- GCS and S3 adapters crashed with "Invalid UUID format" errors
- System keys like __metadata_field_index__status are NOT UUIDs
- Adapters incorrectly tried to shard all metadata keys as UUIDs

Solution:
- Move sharding/routing logic from adapters to BaseStorage class
- Add analyzeKey() method to detect system keys vs entity UUIDs
- System keys route to _system/ directory (no sharding)
- Entity UUIDs route to sharded directories (256 shards)
- All adapters now implement 4 primitive operations:
  * writeObjectToPath(path, data)
  * readObjectFromPath(path)
  * deleteObjectFromPath(path)
  * listObjectsUnderPath(prefix)

Benefits:
- Impossible for future adapters to repeat this mistake
- Zero breaking changes, full backward compatibility
- No data migration required
- Cleaner architecture with better separation of concerns

Updated adapters: GcsStorage, S3CompatibleStorage, OPFSStorage,
FileSystemStorage, MemoryStorage

Added: docs/architecture/data-storage-architecture.md
Updated: README.md with architecture docs link
This commit is contained in:
David Snelling 2025-10-09 13:10:06 -07:00
parent 13303c20c2
commit 1966c39f24
9 changed files with 1330 additions and 630 deletions

View file

@ -888,6 +888,7 @@ We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
- [Getting Started Guide](docs/guides/getting-started.md)
- [API Reference](docs/api/README.md)
- [Architecture Overview](docs/architecture/overview.md)
- [Data Storage Architecture](docs/architecture/data-storage-architecture.md) - Deep dive into storage, indexing, and sharding
- [Natural Language Guide](docs/guides/natural-language.md)
- [Triple Intelligence](docs/architecture/triple-intelligence.md)
- [Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)