David Snelling
2f3357132d
Fixes critical scalability bottleneck where metadata was stored in non-sharded
directories, causing performance degradation at scale (1M+ entities).
Changes:
- Add UUID-based sharding to metadata operations in S3Compatible, FileSystem, and OpFS storage
- Implement complete UUID-based sharding for OpFS storage (nouns, verbs, metadata)
- Update pagination methods to iterate through all 256 UUID-based shards
- Add integration tests verifying sharding behavior across storage adapters
Impact:
- Metadata now scales to millions of entities without directory bottlenecks
- All storage adapters now use consistent UUID-based sharding (256 buckets: 00-ff)
- Improves GCS/S3/R2/OpFS performance at scale
- Path format: entities/{type}/{subtype}/{shard}/{id}.json
Breaking change: Requires data migration for existing S3/GCS/R2/OpFS deployments.
See .strategy/UNIFIED-UUID-SHARDING.md for migration guidance.
|
||
|---|---|---|
| .. | ||
| api | ||
| augmentations/intelligentImport | ||
| benchmarks | ||
| comprehensive | ||
| configs | ||
| fixtures/import | ||
| helpers | ||
| integration | ||
| performance | ||
| scripts | ||
| unit | ||
| vfs | ||
| brainy-3.test.ts | ||
| cli.test.ts | ||
| critical-error-handling.test.ts | ||
| critical-neural-validation.test.ts | ||
| critical-performance-benchmark.test.ts | ||
| distributed-demo.test.ts | ||
| model-loading.test.ts | ||
| opfs-storage.test.ts | ||
| package-size-breakdown.test.ts | ||
| package-size-limit.test.ts | ||
| setup-integration.ts | ||
| setup-unit.ts | ||
| setup.ts | ||
| streaming-pipeline.test.ts | ||
| type-utils.test.ts | ||