brainy/docs/BATCHING.md

title	slug	public	category	template	order	description	next
Batch Operations	guides/batching	true	guides	guide	5	Eliminate N+1 query patterns with batchGet() and storage-level batch APIs for fast multi-entity reads against filesystem and memory storage.	api/reference guides/find-system

Batch Operations API

Production-Ready | Zero N+1 Query Patterns

Overview

Brainy provides batch operations at the storage layer to eliminate N+1 query patterns for VFS operations, relationship queries, and entity retrieval.

Problem Solved

The naive pattern of looping and calling brain.get(id) once per item issues sequential reads through the storage layer. Batched APIs collapse that into a single read pass.

IMPORTANT: The batch optimizations apply ONLY to getTreeStructure() at the VFS layer and the explicit batchGet() / getNounMetadataBatch() calls — not to readFile() or individual get() operations.

---

New Public APIs

1. brain.batchGet(ids, options?)

Batch retrieval of multiple entities (metadata-only by default).

// Fetch multiple entities in a single batched operation
const ids = ['id1', 'id2', 'id3']
const results: Map<string, Entity> = await brain.batchGet(ids)

// With vectors (falls back to individual gets)
const resultsWithVectors = await brain.batchGet(ids, { includeVectors: true })

// Results map
results.get('id1') // → Entity or undefined
results.size // → 3 (number of found entities)

Performance:

• Memory storage: Instant (parallel reads)
• Filesystem storage: Parallel reads, scales with available IOPS

Use Cases:

• Loading multiple entities for display
• Bulk data export operations
• Relationship traversal (fetch all connected entities)

---

Storage-Level APIs

2. storage.getNounMetadataBatch(ids)

Batch metadata retrieval with direct O(1) path construction.

const storage = brain.storage as BaseStorage
const ids = ['id1', 'id2', 'id3']

const metadataMap: Map<string, NounMetadata> = await storage.getNounMetadataBatch(ids)

for (const [id, metadata] of metadataMap) {
 console.log(metadata.noun) // Type: 'document', 'person', etc.
 console.log(metadata.data) // Entity data
}

Features:

• ✅ Direct O(1) path construction from ID (no type lookup needed!)
• ✅ Sharding preservation (all paths include {shard}/{id})
• ✅ Write-cache coherent (read-after-write consistency)
• ✅ O(1) path construction — eliminates the per-entity type search of the old type-first layout

Performance:

• Constant-time path construction per ID — no type-cache misses
• Filesystem: parallel reads bounded by IOPS
• No type search delays — every ID maps directly to storage path

---

3. storage.getVerbsBySourceBatch(sourceIds, verbType?)

Batch relationship queries by source entity IDs.

const storage = brain.storage as BaseStorage

// Get all relationships from multiple sources
const results: Map<string, GraphVerb[]> = await storage.getVerbsBySourceBatch([
 'person1',
 'person2'
])

// Filter by verb type
const createsResults = await storage.getVerbsBySourceBatch(
 ['person1', 'person2'],
 'creates'
)

// Process results
for (const [sourceId, verbs] of results) {
 console.log(`${sourceId} has ${verbs.length} relationships`)
 verbs.forEach(verb => {
 console.log(` → ${verb.verb} → ${verb.targetId}`)
 })
}

Use Cases:

• Social graph traversal (fetch all connections for multiple users)
• Knowledge graph queries (find all relationships of specific type)
• Bulk export of relationship data

Performance:

• Memory storage: single in-memory pass over the metadata index
• Filesystem storage: parallel reads through the metadata index

---

VFS Integration

VFS operations automatically use batch APIs for maximum performance.

Directory Traversal

// Tree traversal uses batched reads under the hood
const tree = await brain.vfs.getTreeStructure('/my-dir')
// ✅ PathResolver.getChildren() uses brain.batchGet() internally
// ✅ Parallel traversal of directories at the same tree level
// ✅ 2-3 batched calls instead of 22 sequential calls

Architecture:

VFS.getTreeStructure()
 ↓ PARALLEL (breadth-first traversal)
 → PathResolver.getChildren() [all dirs at level processed in parallel]
 ↓ BATCHED
 → brain.batchGet(childIds) [1 call instead of N]
 ↓ BATCHED
 → storage.getNounMetadataBatch(ids) [1 call instead of N]
 ↓ ADAPTER
 → Filesystem: Promise.all() parallel reads
 → Memory: Promise.all() parallel reads

---

Advanced Features Compatibility

✅ ID-First Storage Architecture

All batch operations use direct ID-first paths - no type lookup needed!

ID-First Path Structure:

entities/nouns/{SHARD}/{ID}/metadata.json
entities/verbs/{SHARD}/{ID}/metadata.json

Direct O(1) Path Construction:

// Every ID maps directly to exactly ONE path - O(1), no type search
const id = 'abc-123'
const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
const path = `entities/nouns/${shard}/${id}/metadata.json`

// No type cache needed!
// No type search needed!
// No multi-type fallback needed!
// Just pure O(1) lookup!

Benefits:

• O(1) path lookups (eliminates the 42-type sequential search the old type-first layout required)
• Simpler code - removed 500+ lines of type cache complexity
• Scalable - works at large scale without type tracking overhead

---

✅ Sharding

All batch paths include shard IDs calculated via getShardIdFromUuid(id):

const id = 'a3c4e5f7-...'
const shard = getShardIdFromUuid(id) // → 'a3' (first 2 hex chars)
const path = `entities/nouns/${shard}/${id}/metadata.json`

Distribution: 256 shards (00-ff) for optimal load distribution.

---

✅ Generational MVCC (8.0)

Batch reads always serve the live generation through the fast paths shown above. Point-in-time reads go through the Db API instead: a pinned Db (brain.now(), brain.asOf()) resolves changed ids from immutable generation records and unchanged ids from the same live paths batch reads use — see the consistency model.

const db = brain.now()                       // pinned view
const entity = await db.get(id)              // correct at the pinned generation
const results = await brain.batchGet(ids)    // live state, batched
await db.release()

---

Why Batching Is Faster

Batching's advantage is structural, not a fixed multiplier (the actual speedup depends on storage backend, IOPS, and batch size):

• N+1 elimination — N sequential reads collapse into a single parallel pass (Promise.all over the batch).
• O(1) path construction — every ID maps directly to one storage path, with no per-type cache lookup.
• One metadata round-trip — relationship batches fetch all sources' metadata in a single pass instead of one query per source.

The integration test tests/integration/storage-batch-operations.test.ts exercises batch vs. individual reads and asserts that batch retrieval is not slower than the per-entity loop for large batches; it does not pin a specific multiplier, since that is hardware- and IOPS-dependent.

---

Error Handling

Partial Batch Failures

Batch operations gracefully handle missing or invalid entities:

const validId = 'abc-123-...'
const invalidIds = [
 '11111111-1111-1111-1111-111111111111',
 '22222222-2222-2222-2222-222222222222'
]

const results = await brain.batchGet([validId, ...invalidIds])

results.size // → 1 (only valid entity)
results.has(validId) // → true
results.has(invalidIds[0]) // → false (silently skipped)

Behavior:

• Invalid UUIDs: Silently skipped (not included in results)
• Missing entities: Silently skipped (not included in results)
• Storage errors: Logged, entity excluded from results
• No exceptions thrown for partial failures

Empty Batches

const results = await brain.batchGet([])
results.size // → 0 (empty map)

Duplicate IDs

const results = await brain.batchGet(['id1', 'id1', 'id1'])
results.size // → 1 (deduplicated automatically)

---

Migration Guide

From Individual Gets

Before:

const entities = []
for (const id of ids) {
 const entity = await brain.get(id)
 if (entity) entities.push(entity)
}

After:

const results = await brain.batchGet(ids)
const entities = Array.from(results.values())

Performance Gain: Replaces N sequential reads with a single batched pass — no fixed multiplier, it scales with storage IOPS.

---

From Individual Relationship Queries

Before:

const allVerbs = []
for (const sourceId of sourceIds) {
 const verbs = await brain.related({ from: sourceId })
 allVerbs.push(...verbs)
}

After:

const storage = brain.storage as BaseStorage
const results = await storage.getVerbsBySourceBatch(sourceIds)

const allVerbs = []
for (const verbs of results.values()) {
 allVerbs.push(...verbs)
}

Performance Gain: One batched metadata fetch instead of one query per source entity.

---

Best Practices

1. Use Batching for Multiple Entity Operations

// ✅ GOOD: Batch fetch
const results = await brain.batchGet(ids)

// ❌ BAD: Individual gets in loop
for (const id of ids) {
 await brain.get(id)
}

2. Batch Size Recommendations

Storage	Optimal Batch Size	Max Batch Size
Memory	Unlimited	Unlimited
Filesystem	100-500	1000

Guideline: For batches >1000, split into chunks of 500-1000.

3. Metadata-Only by Default

// Default: Metadata-only (fast)
const results = await brain.batchGet(ids) // No vectors

// Only load vectors if needed
const withVectors = await brain.batchGet(ids, { includeVectors: true })

4. Error Handling

// Batch operations never throw for missing entities
const results = await brain.batchGet(ids)

// Check results
for (const id of ids) {
 if (results.has(id)) {
 // Entity exists
 const entity = results.get(id)
 } else {
 // Entity missing (not an error)
 console.log(`Entity ${id} not found`)
 }
}

---

Testing

Comprehensive test coverage in tests/integration/storage-batch-operations.test.ts:

npx vitest run tests/integration/storage-batch-operations.test.ts

Test Coverage:

• ✅ brain.batchGet() high-level API
• ✅ storage.getNounMetadataBatch() with ID-first paths
• ✅ COW integration (branch isolation, inheritance)
• ✅ storage.getVerbsBySourceBatch() relationship queries
• ✅ VFS integration (PathResolver.getChildren())
• ✅ Performance benchmarks (N+1 elimination)
• ✅ Error handling (partial failures, empty batches, duplicates)
• ✅ ID-first storage verification
• ✅ Sharding preservation

Results: 23 tests passing ✅

---

Implementation Details

Architecture Layers

User Code (brain.batchGet)
 ↓
High-Level API (src/brainy.ts)
 ↓
Storage Layer (src/storage/baseStorage.ts)
 ↓
Adapter Layer (readBatchFromAdapter)
 ↓
Storage Adapter (FileSystemStorage / MemoryStorage)

Parallel Reads

Both shipped adapters fall back to Promise.all over individual reads:

// BaseStorage.readBatchFromAdapter()
return await Promise.all(resolvedPaths.map(path => this.read(path)))

Shipped Adapters:

• MemoryStorage
• FileSystemStorage

---

API Summary

• brain.batchGet(ids, options?) - High-level batch entity retrieval
• storage.getNounMetadataBatch(ids) - Storage-level metadata batch
• storage.getVerbsBySourceBatch(sourceIds, verbType?) - Batch relationship queries

Performance Improvements:

• VFS operations: single batched pass instead of N sequential reads
• Entity retrieval: N+1 reads collapsed into one batched pass
• Zero N+1 query patterns

Compatibility:

• ✅ ID-first storage
• ✅ Sharding (256 shards)
• ✅ Generational MVCC — batch reads serve the live generation; pinned Db views serve the past
• ✅ All indexes respected (vector, metadata, graph adjacency)

---

Support

• Documentation: /docs/BATCHING.md, /docs/PERFORMANCE.md
• Tests: /tests/integration/storage-batch-operations.test.ts
• Issues: https://github.com/soulcraft/brainy/issues
• Discussions: https://github.com/soulcraft/brainy/discussions

---

Built with ❤️ for enterprise-scale knowledge graphs