fix: implement unified UUID-based sharding for metadata across all storage adapters
Fixes critical scalability bottleneck where metadata was stored in non-sharded
directories, causing performance degradation at scale (1M+ entities).
Changes:
- Add UUID-based sharding to metadata operations in S3Compatible, FileSystem, and OpFS storage
- Implement complete UUID-based sharding for OpFS storage (nouns, verbs, metadata)
- Update pagination methods to iterate through all 256 UUID-based shards
- Add integration tests verifying sharding behavior across storage adapters
Impact:
- Metadata now scales to millions of entities without directory bottlenecks
- All storage adapters now use consistent UUID-based sharding (256 buckets: 00-ff)
- Improves GCS/S3/R2/OpFS performance at scale
- Path format: entities/{type}/{subtype}/{shard}/{id}.json
Breaking change: Requires data migration for existing S3/GCS/R2/OpFS deployments.
See .strategy/UNIFIED-UUID-SHARDING.md for migration guidance.
This commit is contained in:
parent
6d50fe4054
commit
2f3357132d
6 changed files with 1011 additions and 217 deletions
|
|
@ -776,31 +776,52 @@ const exported = await data.export({
|
|||
---
|
||||
|
||||
### `getStats(): StatsResult`
|
||||
Gets complete statistics about entities and relationships.
|
||||
Gets complete statistics about entities and relationships. All stats are **O(1) pre-calculated** - updated when entities/relationships are added/removed.
|
||||
|
||||
**Returns:**
|
||||
```typescript
|
||||
{
|
||||
entities: {
|
||||
total: number
|
||||
byType: Record<string, number>
|
||||
total: number // Total entity count
|
||||
byType: Record<string, number> // Entity count by type
|
||||
}
|
||||
relationships: {
|
||||
totalRelationships: number
|
||||
byType: Record<string, number>
|
||||
totalRelationships: number // Total relationship/edge count
|
||||
relationshipsByType: Record<string, number> // Relationship count by type
|
||||
uniqueSourceNodes: number // Number of unique source entities
|
||||
uniqueTargetNodes: number // Number of unique target entities
|
||||
totalNodes: number // Total unique entities in relationships
|
||||
}
|
||||
density: number // relationships per entity
|
||||
density: number // Relationships per entity ratio
|
||||
}
|
||||
```
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
const stats = brain.getStats()
|
||||
console.log(`Total entities: ${stats.entities.total}`)
|
||||
console.log(`Total relationships: ${stats.relationships.totalRelationships}`)
|
||||
console.log(`Graph density: ${stats.density.toFixed(2)}`)
|
||||
|
||||
// Total counts (O(1) operations)
|
||||
const totalNouns = stats.entities.total
|
||||
const totalVerbs = stats.relationships.totalRelationships
|
||||
const totalRelations = stats.relationships.totalRelationships // alias
|
||||
|
||||
// Counts by type (O(1) operations)
|
||||
const nounTypes = stats.entities.byType
|
||||
const verbTypes = stats.relationships.relationshipsByType
|
||||
|
||||
// Graph metrics
|
||||
console.log(`Entities: ${totalNouns}`)
|
||||
console.log(`Relationships: ${totalVerbs}`)
|
||||
console.log(`Density: ${stats.density.toFixed(2)}`)
|
||||
console.log(`Types:`, Object.keys(nounTypes))
|
||||
```
|
||||
|
||||
**Performance:**
|
||||
- ✅ All counts pre-calculated in memory
|
||||
- ✅ O(1) access time
|
||||
- ✅ Updated automatically on add/remove
|
||||
- ✅ No expensive full scans required
|
||||
|
||||
**Note:** For more granular counting operations, see the `brain.counts` API below.
|
||||
|
||||
---
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue