fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location

Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.

Three concurrent fixes:

A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
  (reservedQueryMemory / containerMemory / freeMemory) all divided by
  100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
  with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
  entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
  20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
  unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
  unchanged in behavior.

B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
  call site (dedup keyed on caller stack frame + limit value), query
  proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
  safety-cap limits keeps working; the warning teaches the recipe so consumers
  can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
  format. Real OOM territory; the cap stops being a recommendation and becomes
  a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
  against a 9 K-cap box) without disabling OOM protection. Real OOM territory
  on a JS in-memory brain is hundreds of thousands of results, not 10x the
  safety cap.

C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
  name the three escape valves (maxQueryLimit / reservedQueryMemory /
  pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
  src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
  the limit enforcement (7.30.2) share one implementation without circular
  imports.

DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
  exists, the four memory sources the auto-config considers, the three escape
  valves with when-to-use-which guidance, and an explicit "pagination is the
  future-proof pattern" callout (8.0 may tighten the cap further; pagination
  keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
  to the new guide.
- RELEASES.md v7.30.2 entry.

TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
  soft-tier warns once per call site (dedup verified by exercising same vs.
  different source lines via wrapper closures); soft-tier message format
  (names all three escape valves + docs link); soft-tier message includes
  caller location; hard-tier throws; hard-tier message format same as
  soft-tier; consumer maxQueryLimit override raises the cap and shifts both
  tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
  cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
  assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
  the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
  throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
  enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.

CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
  runs in ValidationConfig.constructor(), two-tier enforcement runs in
  validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
  per-call limits to keep snapshot semantics cheap; pagination remains the
  pattern that's guaranteed to keep working.

REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.

Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
This commit is contained in:
David Snelling 2026-06-08 12:34:05 -07:00
parent 34e8271c53
commit 9e307e457f
35 changed files with 819 additions and 154 deletions

View file

@ -251,6 +251,8 @@ const results = await brain.find({
- `hybridAlpha?`: `number` - Balance between text (0.0) and semantic (1.0) search. Auto-detected by query length if not specified.
- `excludeVFS?`: `boolean` - Exclude VFS entities from results (default: false)
> **`limit` tip:** Brainy caps `limit` against an auto-configured maximum (based on container/free memory, ~25 KB per result). Above the cap you get a one-time warning per call site; above 2× the cap it throws. To raise the cap, pass `new Brainy({ maxQueryLimit: N })` or `{ reservedQueryMemory: bytes }`. For queries that need ALL matches, paginate with `{ limit, offset }` — that's the only pattern guaranteed to keep working across Brainy versions. See [Query Limits & Pagination](../guides/find-limits.md).
**Returns:** `Promise<Result[]>` - Matching entities with scores
---

View file

@ -150,12 +150,6 @@ Add Cortex and you also unlock memory-mapped storage — aggregate state lives d
- **Safe experiments** — Let teams branch the knowledge base, experiment independently, and merge when ready — just like branching code.
- **Unified business platforms** — Combine booking, CRM, inventory, and analytics in one queryable knowledge graph with no sync pipeline.
### Built with Brainy
### What Brainy is good at
Real products built on Brainy — live in production at Soulcraft:
- **Workshop** — AI-powered IDE and creation studio where imagination meets creation.
- **Venue** — Unified physical + digital business operations, from food truck to franchise.
- **Memory** — Infinite context for AI agents that never forgets.
- **Collective** — Multi-agent coordination with shared knowledge and automatic task routing.
- **Heart** — Emotional intelligence and empathy layer for AI communication.
Brainy is the engine underneath production systems that need to combine semantic search, structured filtering, and graph traversal in a single query — agent memory, knowledge-base platforms, business operations consoles, multi-agent coordination, and more. The combination of vector + graph + metadata search in one indexed call is what differentiates it from running three engines side by side.

149
docs/guides/find-limits.md Normal file
View file

@ -0,0 +1,149 @@
---
title: Query Limits & Pagination
slug: guides/find-limits
public: true
category: guides
template: guide
order: 8
description: How Brainy caps `find({ limit })` to prevent OOM, the three escape valves when the cap is too tight, and why pagination is the future-proof pattern.
next:
- guides/aggregation
- api/reference
---
# Query Limits & Pagination
Brainy's `find()` returns entities into a JavaScript array. The size of that array is bounded by an auto-configured cap so a single query can never run the host out of memory. This guide explains the cap, the three ways to raise it when your use case justifies it, and the one pattern that scales no matter what cap is in effect: pagination.
## Why the cap exists
Every entity Brainy returns carries:
- A 384-dim float32 embedding vector (1.5 KB)
- Standard fields: `id`, `type`, `subtype`, timestamps, confidence, weight (~200 bytes)
- User metadata (variable — typical 5-10 KB, can spike to 20+ KB)
Conservative budget: **25 KB per result**. A `find({ limit: 100_000 })` against a brain with rich metadata can claim ~2.5 GB before Brainy's iteration starts. JavaScript's GC + V8's heap targets can't absorb that swing without paging or OOM in production.
The cap is a safety net. It's not the only reason your query might be slow — graph traversal and HNSW search have their own perf characteristics — but it's the one that turns a slow query into a sudden runtime error.
## The auto-configured cap (7.30.2+)
Brainy picks `maxLimit` from the first of these that's available:
| Priority | Source | Formula |
|---|---|---|
| 1 | Constructor option `maxQueryLimit` | Hard cap at supplied value, max 100 000 |
| 2 | Constructor option `reservedQueryMemory` | `floor(reservedQueryMemory / 25 KB)` capped at 100 000 |
| 3 | Detected container memory limit (Cloud Run, Kubernetes, cgroups v1/v2) | `floor(containerLimit × 0.25 / 25 KB)` capped at 100 000 |
| 4 | Free system memory | `floor(availableMemory / 25 KB)` capped at 100 000 |
Worked example: a 4 GB Cloud Run container picks priority 3 → `floor(4 GB × 0.25 / 25 KB) = floor(40 960) = 40 000` results. A 900 MB free-memory box on priority 4 gets `floor(900 MB / 25 KB) = ~36 000`.
> **Calibration note.** Pre-7.30.2 used 100 KB per result instead of 25 KB, which produced caps that were 4× too tight for typical workloads (an 8 KB / result reality). 7.30.2 recalibrated to match observed entity sizes; existing `limit: 10_000` safety patterns now pass silently on any reasonably-sized box.
## What happens when you exceed the cap
`find({ limit })` enforces in **two tiers**:
### Soft tier: `maxLimit < limit ≤ 2 × maxLimit`
You get a one-time warning per call site:
```
[Brainy] find({ limit: 50000 }) exceeds the auto-configured query limit of
40000 (basis: detected container memory limit). Choose one:
• Increase the cap: new Brainy({ maxQueryLimit: 50000 })
• Reserve more memory: new Brainy({ reservedQueryMemory: 1310720000 })
• Paginate: split the query with { limit, offset } pages
at YourService.loadDashboard (/app/src/dashboard.ts:142:18)
Docs: https://soulcraft.com/docs/guides/find-limits
```
**The query proceeds.** Brainy returns the result set you asked for; the warning is a teaching signal, not a block. Existing code that relied on the cap silently allowing safety-cap limits (`limit: 10_000` against a 9 K-cap box) keeps working — the warning shows you the recipe so you can fix it intentionally.
### Hard tier: `limit > 2 × maxLimit`
Same message, but thrown as an error. This is real OOM territory; the cap stops being a recommendation and becomes a guardrail.
## The three escape valves
### 1. Raise the cap at construction — `maxQueryLimit`
When the auto-config is wrong for your workload (e.g. you know your entities are smaller than 25 KB average and you need bigger result sets), set an explicit cap:
```typescript
const brain = new Brainy({
storage: { type: 'filesystem', options: { path: './data' } },
maxQueryLimit: 50_000 // raises the cap; still hard-clamped at 100 000
})
```
This is the right answer when:
- Your entity metadata is genuinely small (e.g. 1-2 KB) and 25 KB per result is over-conservative
- You're running on a box with lots of headroom and 25% of memory underestimates what you can spare for queries
- You need a known-good limit that doesn't change when the box's free-memory wiggles at startup
### 2. Reserve more memory for queries — `reservedQueryMemory`
When you want the cap to be memory-derived but more generous than the default 25% slice:
```typescript
const brain = new Brainy({
reservedQueryMemory: 1024 * 1024 * 1024 // 1 GB → ~40 000 result cap
})
```
This is the right answer when:
- Your host's memory budget for queries is known and stable, regardless of free-memory at startup
- You want the formula to scale with the documented per-result size (25 KB) instead of a hard number
### 3. Paginate — the future-proof pattern
If your query genuinely needs to walk all matches in a category, don't fight the cap — walk in pages:
```typescript
async function findAll<T>(params: FindParams<T>, pageSize = 1000): Promise<Result<T>[]> {
const all: Result<T>[] = []
let offset = 0
while (true) {
const page = await brain.find({ ...params, limit: pageSize, offset })
all.push(...page)
if (page.length < pageSize) break
offset += page.length
}
return all
}
// Use it just like find():
const allEvents = await findAll({ type: NounType.Event, where: { status: 'open' } })
```
For very large brains, prefer the streaming API which avoids holding the full result set in memory at all:
```typescript
for await (const entity of brain.streaming.entities({ type: NounType.Event })) {
// process one entity at a time
}
```
## When to use which
| Situation | Recommended valve |
|---|---|
| The cap is unreasonably low for your known entity size | `maxQueryLimit` |
| You want a memory-derived cap but more generous than 25% | `reservedQueryMemory` |
| Your query needs ALL matches in a category | Pagination or `brain.streaming.entities()` |
| You hit the cap once during a one-off migration | `maxQueryLimit` or `migrateField` (which already paginates internally) |
| You're hitting the cap on a recurring user-facing query | Pagination — the cap will get tighter in 8.0, not looser |
## A note on Brainy 8.0
8.0's Datomic-style `Db` API may make per-call limits stricter to keep snapshot semantics cheap. **Pagination is the only pattern that's guaranteed to keep working unchanged.** Code that paginates today doesn't need to revisit when 8.0 ships.
## Reference
- `BrainyConfig.maxQueryLimit?: number` — explicit cap override (max 100 000)
- `BrainyConfig.reservedQueryMemory?: number` — memory budget for queries (bytes)
- `find({ limit, offset })` — paginated find
- `brain.streaming.entities(filter)` — streaming alternative for very large traversals

View file

@ -464,7 +464,7 @@ await brain.add({ type: NounType.Person, subtype: 'employee', data: '...' })
When a platform layer like the Soulcraft SDK registers `requireSubtype()` rules on behalf of every consumer's brain, every downstream product that calls `brain.add()` / `brain.relate()` against those types must pass a matching `subtype`. Skipping the field — or passing one outside the registered vocabulary — throws at the boundary.
This pattern is powerful but surfaces a class of latent bug: any `brain.add()` call site that was written before strict-mode adoption starts rejecting writes. The Venue team hit this in production on 2026-06-08 when their `/book` flow 500'd on every request because `BookingDraftService.getOrCreateByToken` called `brain.add({ type: NounType.Event, ... })` without subtype.
This pattern is powerful but surfaces a class of latent bug: any `brain.add()` call site that was written before strict-mode adoption starts rejecting writes. A representative production incident: a booking flow started returning 500 on every request because a service method called `brain.add({ type: NounType.Event, ... })` without subtype, and an SDK layer had just registered `requireSubtype()` for `NounType.Event` on every brain instance.
The fix is a four-step migration recipe — and Brainy 7.30.1+ ships diagnostic tools to make it deterministic.
@ -488,7 +488,7 @@ The fix is a four-step migration recipe — and Brainy 7.30.1+ ships diagnostic
2. **Bulk-migrate any existing convention** with `brain.migrateField()` if a legacy field can be lifted:
```typescript
// Venue chose subtype = same string as metadata.entityType:
// Common pattern: subtype mirrors a discriminator field already in metadata
await brain.migrateField({
from: 'metadata.entityType',
to: 'subtype',
@ -496,7 +496,7 @@ The fix is a four-step migration recipe — and Brainy 7.30.1+ ships diagnostic
})
```
3. **Hand-fix the remaining call sites.** The exact list is in `report.entitiesWithoutSubtype`. For each call site, add `subtype: '<value>'` to the `brain.add()` / `brain.relate()` params. Choose a stable convention (Venue chose `subtype = metadata.entityType`; any rule that's deterministic from the data works).
3. **Hand-fix the remaining call sites.** The exact list is in `report.entitiesWithoutSubtype`. For each call site, add `subtype: '<value>'` to the `brain.add()` / `brain.relate()` params. Choose a stable convention (e.g. mirror `metadata.entityType` if you have one; any rule that's deterministic from the data works).
4. **Verify with `brain.audit()` again.** Re-run; total should be `0`. If you turn on brain-wide strict mode at this point, all future writes are protected.