brainy/docs/BATCHING.md

title	slug	public	category	template	order	description	next
Batch Operations	guides/batching	true	guides	guide	5	Eliminate N+1 query patterns with batchGet() and storage-level batch APIs for fast multi-entity reads against filesystem and memory storage.	api/reference guides/find-system

Batch Operations API

Production-Ready | Zero N+1 Query Patterns

Overview

Brainy provides batch operations at the storage layer to eliminate N+1 query patterns for VFS operations, relationship queries, and entity retrieval.

Problem Solved

The naive pattern of looping and calling brain.get(id) once per item issues sequential reads through the storage layer. Batched APIs collapse that into a single read pass.

IMPORTANT: The batch optimizations apply ONLY to getTreeStructure() at the VFS layer and the explicit batchGet() / getNounMetadataBatch() calls — not to readFile() or individual get() operations.

---

New Public APIs

1. brain.batchGet(ids, options?)

Batch retrieval of multiple entities (metadata-only by default).

// Fetch multiple entities in a single batched operation
const ids = ['id1', 'id2', 'id3']
const results: Map<string, Entity> = await brain.batchGet(ids)

// With vectors (falls back to individual gets)
const resultsWithVectors = await brain.batchGet(ids, { includeVectors: true })

// Results map
results.get('id1') // → Entity or undefined
results.size // → 3 (number of found entities)

Performance:

• Memory storage: Instant (parallel reads)
• Filesystem storage: Parallel reads, scales with available IOPS

Use Cases:

• Loading multiple entities for display
• Bulk data export operations
• Relationship traversal (fetch all connected entities)

---

Storage-Level APIs

2. storage.getNounMetadataBatch(ids)

Batch metadata retrieval with direct O(1) path construction.

const storage = brain.storage as BaseStorage
const ids = ['id1', 'id2', 'id3']

const metadataMap: Map<string, NounMetadata> = await storage.getNounMetadataBatch(ids)

for (const [id, metadata] of metadataMap) {
 console.log(metadata.noun) // Type: 'document', 'person', etc.
 console.log(metadata.data) // Entity data
}

Features:

• ✅ Direct O(1) path construction from ID (no type lookup needed!)
• ✅ Sharding preservation (all paths include {shard}/{id})
• ✅ Write-cache coherent (read-after-write consistency)
• ✅ 40x faster than v5.x type-first architecture

Performance:

• ~1ms per 100 entities (consistent, no cache misses!)
• Filesystem: parallel reads bounded by IOPS
• No type search delays — every ID maps directly to storage path

---

3. storage.getVerbsBySourceBatch(sourceIds, verbType?)

Batch relationship queries by source entity IDs.

const storage = brain.storage as BaseStorage

// Get all relationships from multiple sources
const results: Map<string, GraphVerb[]> = await storage.getVerbsBySourceBatch([
 'person1',
 'person2'
])

// Filter by verb type
const createsResults = await storage.getVerbsBySourceBatch(
 ['person1', 'person2'],
 'creates'
)

// Process results
for (const [sourceId, verbs] of results) {
 console.log(`${sourceId} has ${verbs.length} relationships`)
 verbs.forEach(verb => {
 console.log(` → ${verb.verb} → ${verb.targetId}`)
 })
}

Use Cases:

• Social graph traversal (fetch all connections for multiple users)
• Knowledge graph queries (find all relationships of specific type)
• Bulk export of relationship data

Performance:

• Memory storage: <10ms for 1000 relationships
• Filesystem storage: parallel reads through the metadata index

---

VFS Integration

VFS operations automatically use batch APIs for maximum performance.

Directory Traversal

// Tree traversal uses batched reads under the hood
const tree = await brain.vfs.getTreeStructure('/my-dir')
// ✅ PathResolver.getChildren() uses brain.batchGet() internally
// ✅ Parallel traversal of directories at the same tree level
// ✅ 2-3 batched calls instead of 22 sequential calls

Architecture:

VFS.getTreeStructure()
 ↓ PARALLEL (breadth-first traversal)
 → PathResolver.getChildren() [all dirs at level processed in parallel]
 ↓ BATCHED
 → brain.batchGet(childIds) [1 call instead of N]
 ↓ BATCHED
 → storage.getNounMetadataBatch(ids) [1 call instead of N]
 ↓ ADAPTER
 → Filesystem: Promise.all() parallel reads
 → Memory: Promise.all() parallel reads

---

Advanced Features Compatibility

✅ ID-First Storage Architecture

All batch operations use direct ID-first paths - no type lookup needed!

ID-First Path Structure:

entities/nouns/{SHARD}/{ID}/metadata.json
entities/verbs/{SHARD}/{ID}/metadata.json

Direct O(1) Path Construction:

// Every ID maps directly to exactly ONE path - 40x faster!
const id = 'abc-123'
const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
const path = `entities/nouns/${shard}/${id}/metadata.json`

// No type cache needed!
// No type search needed!
// No multi-type fallback needed!
// Just pure O(1) lookup!

Benefits:

• 40x faster path lookups (eliminates 42-type sequential search)
• Simpler code - removed 500+ lines of type cache complexity
• Scalable - works at billion-scale without type tracking overhead

---

✅ Sharding

All batch paths include shard IDs calculated via getShardIdFromUuid(id):

const id = 'a3c4e5f7-...'
const shard = getShardIdFromUuid(id) // → 'a3' (first 2 hex chars)
const path = `entities/nouns/${shard}/${id}/metadata.json`

Distribution: 256 shards (00-ff) for optimal load distribution.

---

✅ Generational MVCC (8.0)

Batch reads always serve the live generation through the fast paths shown above. Point-in-time reads go through the Db API instead: a pinned Db (brain.now(), brain.asOf()) resolves changed ids from immutable generation records and unchanged ids from the same live paths batch reads use — see the consistency model.

const db = brain.now()                       // pinned view
const entity = await db.get(id)              // correct at the pinned generation
const results = await brain.batchGet(ids)    // live state, batched
await db.release()

---

Performance Benchmarks

VFS Operations (12 Files)

Storage	Before optimization	After optimization	Improvement
Memory	150ms	50ms	67% faster
Filesystem	~500ms	~80ms	84% faster

Entity Batch Retrieval (100 Entities)

Storage	Individual Gets	Batch Get	Improvement
Memory	180ms	15ms	92% faster
Filesystem	~1.2s	~120ms	90% faster

Throughput (Entities/Second)

Storage	Individual	Batch	Improvement
Memory	556 ent/s	6667 ent/s	12x
Filesystem	~80 ent/s	~800 ent/s	10x

---

Error Handling

Partial Batch Failures

Batch operations gracefully handle missing or invalid entities:

const validId = 'abc-123-...'
const invalidIds = [
 '11111111-1111-1111-1111-111111111111',
 '22222222-2222-2222-2222-222222222222'
]

const results = await brain.batchGet([validId, ...invalidIds])

results.size // → 1 (only valid entity)
results.has(validId) // → true
results.has(invalidIds[0]) // → false (silently skipped)

Behavior:

• Invalid UUIDs: Silently skipped (not included in results)
• Missing entities: Silently skipped (not included in results)
• Storage errors: Logged, entity excluded from results
• No exceptions thrown for partial failures

Empty Batches

const results = await brain.batchGet([])
results.size // → 0 (empty map)

Duplicate IDs

const results = await brain.batchGet(['id1', 'id1', 'id1'])
results.size // → 1 (deduplicated automatically)

---

Migration Guide

From Individual Gets

Before:

const entities = []
for (const id of ids) {
 const entity = await brain.get(id)
 if (entity) entities.push(entity)
}

After:

const results = await brain.batchGet(ids)
const entities = Array.from(results.values())

Performance Gain: 10-20x faster on filesystem storage.

---

From Individual Relationship Queries

Before:

const allVerbs = []
for (const sourceId of sourceIds) {
 const verbs = await brain.related({ from: sourceId })
 allVerbs.push(...verbs)
}

After:

const storage = brain.storage as BaseStorage
const results = await storage.getVerbsBySourceBatch(sourceIds)

const allVerbs = []
for (const verbs of results.values()) {
 allVerbs.push(...verbs)
}

Performance Gain: 5-10x faster due to batched metadata fetches.

---

Best Practices

1. Use Batching for Multiple Entity Operations

// ✅ GOOD: Batch fetch
const results = await brain.batchGet(ids)

// ❌ BAD: Individual gets in loop
for (const id of ids) {
 await brain.get(id)
}

2. Batch Size Recommendations

Storage	Optimal Batch Size	Max Batch Size
Memory	Unlimited	Unlimited
Filesystem	100-500	1000

Guideline: For batches >1000, split into chunks of 500-1000.

3. Metadata-Only by Default

// Default: Metadata-only (fast)
const results = await brain.batchGet(ids) // No vectors

// Only load vectors if needed
const withVectors = await brain.batchGet(ids, { includeVectors: true })

4. Error Handling

// Batch operations never throw for missing entities
const results = await brain.batchGet(ids)

// Check results
for (const id of ids) {
 if (results.has(id)) {
 // Entity exists
 const entity = results.get(id)
 } else {
 // Entity missing (not an error)
 console.log(`Entity ${id} not found`)
 }
}

---

Testing

Comprehensive test coverage in tests/integration/storage-batch-operations.test.ts:

npx vitest run tests/integration/storage-batch-operations.test.ts

Test Coverage:

• ✅ brain.batchGet() high-level API
• ✅ storage.getNounMetadataBatch() with ID-first paths
• ✅ COW integration (branch isolation, inheritance)
• ✅ storage.getVerbsBySourceBatch() relationship queries
• ✅ VFS integration (PathResolver.getChildren())
• ✅ Performance benchmarks (N+1 elimination)
• ✅ Error handling (partial failures, empty batches, duplicates)
• ✅ ID-first storage verification
• ✅ Sharding preservation

Results: 23 tests passing ✅

---

Implementation Details

Architecture Layers

User Code (brain.batchGet)
 ↓
High-Level API (src/brainy.ts)
 ↓
Storage Layer (src/storage/baseStorage.ts)
 ↓
Adapter Layer (readBatchFromAdapter)
 ↓
Storage Adapter (FileSystemStorage / MemoryStorage)

Parallel Reads

Both shipped adapters fall back to Promise.all over individual reads:

// BaseStorage.readBatchFromAdapter()
return await Promise.all(resolvedPaths.map(path => this.read(path)))

Shipped Adapters:

• MemoryStorage
• FileSystemStorage

---

API Summary

• brain.batchGet(ids, options?) - High-level batch entity retrieval
• storage.getNounMetadataBatch(ids) - Storage-level metadata batch
• storage.getVerbsBySourceBatch(sourceIds, verbType?) - Batch relationship queries

Performance Improvements:

• VFS operations: 90%+ faster than the naive per-entity loop
• Entity retrieval: 10-20x throughput improvement
• Zero N+1 query patterns

Compatibility:

• ✅ ID-first storage
• ✅ Sharding (256 shards)
• ✅ Generational MVCC — batch reads serve the live generation; pinned Db views serve the past
• ✅ All indexes respected (vector, metadata, graph adjacency)

---

Support

• Documentation: /docs/BATCHING.md, /docs/PERFORMANCE.md
• Tests: /tests/integration/storage-batch-operations.test.ts
• Issues: https://github.com/soulcraft/brainy/issues
• Discussions: https://github.com/soulcraft/brainy/discussions

---

Built with ❤️ for enterprise-scale knowledge graphs