brainy/CLAUDE.md
David Snelling 9e307e457f fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.

Three concurrent fixes:

A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
  (reservedQueryMemory / containerMemory / freeMemory) all divided by
  100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
  with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
  entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
  20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
  unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
  unchanged in behavior.

B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
  call site (dedup keyed on caller stack frame + limit value), query
  proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
  safety-cap limits keeps working; the warning teaches the recipe so consumers
  can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
  format. Real OOM territory; the cap stops being a recommendation and becomes
  a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
  against a 9 K-cap box) without disabling OOM protection. Real OOM territory
  on a JS in-memory brain is hundreds of thousands of results, not 10x the
  safety cap.

C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
  name the three escape valves (maxQueryLimit / reservedQueryMemory /
  pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
  src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
  the limit enforcement (7.30.2) share one implementation without circular
  imports.

DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
  exists, the four memory sources the auto-config considers, the three escape
  valves with when-to-use-which guidance, and an explicit "pagination is the
  future-proof pattern" callout (8.0 may tighten the cap further; pagination
  keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
  to the new guide.
- RELEASES.md v7.30.2 entry.

TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
  soft-tier warns once per call site (dedup verified by exercising same vs.
  different source lines via wrapper closures); soft-tier message format
  (names all three escape valves + docs link); soft-tier message includes
  caller location; hard-tier throws; hard-tier message format same as
  soft-tier; consumer maxQueryLimit override raises the cap and shifts both
  tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
  cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
  assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
  the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
  throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
  enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.

CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
  runs in ValidationConfig.constructor(), two-tier enforcement runs in
  validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
  per-call limits to keep snapshot semantics cheap; pagination remains the
  pattern that's guaranteed to keep working.

REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.

Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-08 12:49:43 -07:00

8.4 KiB

Brainy - Claude Code Project Guide

This file provides guidance for Claude Code (and human contributors) when working on the Brainy codebase.

Cross-Project Coordination

Handoff file: /home/dpsifr/.strategy/PLATFORM-HANDOFF.md

At session START: Read the handoff. Find rows where Owner = Brainy. Act on those first.

At session END: Mark completed actions , delete rows you finished, delete threads with zero remaining actions. File must not grow. If you shipped anything consumers need to know about, update RELEASES.md before closing.

Brainy's current open actions: None. MIT open-source — no platform-specific actions.

Current version: @soulcraft/brainy@7.19.10


Project Overview

Brainy is a Universal Knowledge Protocol -- a Triple Intelligence database that combines vector similarity search, graph traversal, and metadata filtering into a single TypeScript library. Published as @soulcraft/brainy on npm under the MIT license.

Getting Started

npm install           # Install dependencies
npm run build         # Build the project
npm test              # Run test suite (Vitest)

Architecture

Full architecture reference: .claude/skills/architecture.md

Core Systems

  • Storage (src/storage/): Pluggable storage backends via StorageAdapter interface (src/coreTypes.ts)
  • Vector Search (src/hnsw/): HNSW approximate nearest neighbor search
  • Graph Engine (src/graph/): Relationship traversal with adjacency index and pathfinding
  • Metadata Index (src/utils/metadataIndex.ts): O(1) exact match, O(log n) range queries
  • Triple Intelligence (src/triple/): Unified query combining all three intelligence types
  • Aggregation Engine (src/aggregation/): Write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows
  • Virtual Filesystem (src/vfs/): Full VFS with semantic search

Type System

  • NounType (42 types): Entity classification -- Person, Concept, Collection, Document, Task, etc.
  • VerbType (127 types): Relationship types -- Contains, RelatedTo, PartOf, Creates, DependsOn, etc.
  • Defined in src/types/graphTypes.ts

Code Standards

TypeScript

  • Strict mode enabled
  • Target: ES2020, NodeNext module resolution
  • All new code must be TypeScript
  • Follow existing patterns -- read related code before writing

Quality

  • All code must compile without errors
  • All code must have working tests that exercise real behavior
  • No stub returns (return {} as any)
  • No incomplete implementations with TODO comments
  • If something can't be fully implemented, throw an explicit error rather than faking it

Verification Before Code Changes

  1. Check that interfaces and methods actually exist before using them
  2. Check that type properties are in the type definitions
  3. Run npm test -- tests must pass
  4. Run npm run build -- build must succeed

Testing

  • Framework: Vitest
  • Tests in tests/ (unit, integration, benchmarks, comprehensive)
  • Use in-memory storage for speed where possible
  • Tests must exercise real behavior, not mock it
  • Benchmarks are in tests/benchmarks/ (not tests/performance/)

Commit Conventions

Use Conventional Commits:

feat: add new feature        (minor version bump)
fix: resolve bug             (patch version bump)
docs: update documentation   (patch version bump)
perf: improve performance    (patch version bump)
refactor: restructure code   (patch version bump)
test: add/update tests       (patch version bump)

Important: Never use BREAKING CHANGE in commit messages. Major version bumps are manual decisions only (npm run release:major).

Docs Pipeline — soulcraft.com/docs

Docs in docs/**/*.md are published with the npm package (included in files) and synced to soulcraft.com/docs on every portal deploy. Frontmatter controls what appears publicly.

Docs check triggers

Run the docs check whenever the user says ANY of:

  • "commit, publish, release" / "release" / "publish"
  • "update the docs" / "make sure docs are accurate" / "check the docs"
  • "review docs" / "clean up docs"

Pre-release docs check (MANDATORY before every release)

When the user says "commit, publish, release" or any variation, before committing:

  1. Scan all files changed in this session (and any recently added docs/*.md files)
  2. For each changed/new doc, decide: is this useful to external users?
    • Yes → ensure it has complete frontmatter (add or update it)
    • No (internal, migration, dev-only) → ensure it has no frontmatter or public: false
  3. For docs that already have frontmatter, verify:
    • description still matches the actual content
    • next links still exist and are still the right follow-up pages
    • title matches the doc's h1
  4. Include frontmatter changes in the commit

Frontmatter format

---
title: Human-readable title
slug: category/page-name          # URL: soulcraft.com/docs/category/page-name
public: true                      # false or absent = not published
category: getting-started | concepts | guides | api
template: guide | concept | api   # controls layout on soulcraft.com
order: 1                          # sidebar position within category (lower = first)
description: One sentence. What this doc covers and why it matters.
next:                             # "Next steps" links shown at bottom of page
  - category/other-slug
---

Category guide

category use for
getting-started installation, quick start, first steps
concepts how the system works, mental models
guides how to do specific things, recipes
api method reference, signatures, parameters

What stays internal (no frontmatter / public: false)

  • Release guides, developer learning paths
  • Migration guides for old versions (v3→v4, v5.11)
  • Architecture analysis docs (clustering algorithms, etc.)
  • Anything in docs/internal/
  • Deployment/ops/cost docs (cloud-run, kubernetes, cost-optimization)

Release Process

Fully automated via scripts/release.sh:

npm run release:dry          # Preview (no changes)
npm run release:patch        # Bug fixes
npm run release:minor        # New features
npm run release:major        # Breaking changes (rare, manual decision)

The script: verifies clean git state, builds, tests, bumps version, updates CHANGELOG.md, commits, tags, pushes, publishes to npm, and creates a GitHub release.

After a successful release, remind the user:

"Published. Deploy portal to pick up the new docs → go to the portal project and deploy."

Do NOT deploy portal from here. Portal is always deployed separately from within the portal project.

Closed-Source Product Names — HARD RULE

Brainy is the only Soulcraft open-source project. Nothing in this repo — code, JSDoc, tests, docs, RELEASES.md, CHANGELOG.md, commit messages — may reference closed-source Soulcraft products by name (Workshop, Venue, Memory, Muse, Hall, Forge, Academy, Pulse, Heart, Collective, SDK) or by their specific class/method names (BookingDraftService, getDemandHeatmap, systemKind, etc.).

When recording a consumer-reported bug, regression scenario, or release note:

  • Refer to "a consumer", "a downstream application", "a production deployment", or "an internal report" — never name the product.
  • All doc examples must use generic domain values ('employee', 'customer', 'invoice', 'milestone', OrderService, /orders/...), not product-specific schemas.
  • Internal session artifacts (.strategy/, ~/.claude/plans/, handoff files outside the repo) MAY name products — those are not public.

If you catch yourself typing a product name into a tracked file, stop and rephrase.

Performance Claims

When documenting performance characteristics:

  • MEASURED: Cite the test file and line number
  • PROJECTED: Clearly label as extrapolated from tested scale
  • Never claim a performance figure without context or evidence

Debugging

When a bug persists through 2+ fix attempts, switch to systematic debugging:

  1. Add comprehensive logging at every step
  2. Test with production-like data
  3. Trace the complete execution path
  4. Check both library code and consumer code
  5. Verify with actual test execution before declaring fixed

Key Paths

  • Main class: src/brainy.ts
  • Public API: src/index.ts (38+ exports)
  • Storage interface: src/coreTypes.ts
  • Type definitions: src/types/
  • Strategy/planning docs: .strategy/ (gitignored, not public)