fix: implement unified UUID-based sharding for metadata across all storage adapters

Fixes critical scalability bottleneck where metadata was stored in non-sharded
directories, causing performance degradation at scale (1M+ entities).

Changes:
- Add UUID-based sharding to metadata operations in S3Compatible, FileSystem, and OpFS storage
- Implement complete UUID-based sharding for OpFS storage (nouns, verbs, metadata)
- Update pagination methods to iterate through all 256 UUID-based shards
- Add integration tests verifying sharding behavior across storage adapters

Impact:
- Metadata now scales to millions of entities without directory bottlenecks
- All storage adapters now use consistent UUID-based sharding (256 buckets: 00-ff)
- Improves GCS/S3/R2/OpFS performance at scale
- Path format: entities/{type}/{subtype}/{shard}/{id}.json

Breaking change: Requires data migration for existing S3/GCS/R2/OpFS deployments.
See .strategy/UNIFIED-UUID-SHARDING.md for migration guidance.
This commit is contained in:
David Snelling 2025-10-08 13:26:35 -07:00
parent 6d50fe4054
commit 2f3357132d
6 changed files with 1011 additions and 217 deletions

View file

@ -776,31 +776,52 @@ const exported = await data.export({
---
### `getStats(): StatsResult`
Gets complete statistics about entities and relationships.
Gets complete statistics about entities and relationships. All stats are **O(1) pre-calculated** - updated when entities/relationships are added/removed.
**Returns:**
```typescript
{
entities: {
total: number
byType: Record<string, number>
total: number // Total entity count
byType: Record<string, number> // Entity count by type
}
relationships: {
totalRelationships: number
byType: Record<string, number>
totalRelationships: number // Total relationship/edge count
relationshipsByType: Record<string, number> // Relationship count by type
uniqueSourceNodes: number // Number of unique source entities
uniqueTargetNodes: number // Number of unique target entities
totalNodes: number // Total unique entities in relationships
}
density: number // relationships per entity
density: number // Relationships per entity ratio
}
```
**Example:**
```typescript
const stats = brain.getStats()
console.log(`Total entities: ${stats.entities.total}`)
console.log(`Total relationships: ${stats.relationships.totalRelationships}`)
console.log(`Graph density: ${stats.density.toFixed(2)}`)
// Total counts (O(1) operations)
const totalNouns = stats.entities.total
const totalVerbs = stats.relationships.totalRelationships
const totalRelations = stats.relationships.totalRelationships // alias
// Counts by type (O(1) operations)
const nounTypes = stats.entities.byType
const verbTypes = stats.relationships.relationshipsByType
// Graph metrics
console.log(`Entities: ${totalNouns}`)
console.log(`Relationships: ${totalVerbs}`)
console.log(`Density: ${stats.density.toFixed(2)}`)
console.log(`Types:`, Object.keys(nounTypes))
```
**Performance:**
- ✅ All counts pre-calculated in memory
- ✅ O(1) access time
- ✅ Updated automatically on add/remove
- ✅ No expensive full scans required
**Note:** For more granular counting operations, see the `brain.counts` API below.
---