brainy/docs/eli5.md

156 lines
8.6 KiB
Markdown
Raw Normal View History

---
title: What is Brainy?
slug: getting-started/what-is-brainy
public: true
category: getting-started
template: guide
order: 0
description: Plain-language guide covering what Brainy does, how it compares to other tools, and what you can build with it. No jargon, no code — just clear analogies.
next:
- getting-started/installation
- getting-started/quick-start
---
# Brainy and Cor — Explained Simply
*A plain-language guide for anyone who wants to understand what this thing actually does.*
---
## What is Brainy?
Imagine you have the world's smartest librarian.
You walk up and say *"I'm looking for something about climate change — but only books published after 2020, and only ones written by authors I've already read."* A normal library would make you dig through a card catalogue, then cross-reference a list of authors, then scan the shelves yourself. That takes a while.
Your smart librarian does all three at the same time — in less than the time it takes to blink.
That's Brainy. It's a knowledge database that can search by **meaning**, follow **connections**, and filter by **labels** — all at once, in a single question.
---
## The Three Superpowers
### 1. Meaning Search (the "fuzzy" superpower)
When you search for "automobile," Brainy also finds results about "car," "vehicle," and "sedan" — because it understands what words *mean*, not just how they're spelled. It reads your data the way a person would, not the way a search box does.
Think of it like the librarian who finds books on "heartbreak" when you ask for something about "loneliness."
### 2. Relationship Walking (the "follow the thread" superpower)
Every piece of information can be connected to other pieces. A Person *works at* a Company. A Project *depends on* a Tool. A Recipe *contains* Ingredients.
Brainy can follow these connections across many hops in one step. Ask for "everything connected to this author, two steps out" and Brainy returns the author's books, the books' publishers, the publishers' other authors — without you needing to chain four separate lookups yourself.
Think of it like the librarian who not only hands you the book you asked for, but also knows which shelf it came from, who donated it, and what other books arrived in the same donation.
### 3. Label Filtering (the "narrow it down" superpower)
Sometimes meaning and connections aren't enough — you need precision. "Only recipes with fewer than 500 calories." "Only events from last week." "Only documents tagged as urgent."
Brainy can narrow any result set down by exact labels or ranges in the same breath as the other two searches. No extra steps.
---
## What Else Can It Do?
- **Virtual file cabinet.** Brainy includes a full filesystem you can use to store, organize, and semantically search files — PDFs, documents, anything — the same way you search everything else.
- **Live dashboards.** You can define running totals that Brainy keeps updated automatically — things like "total sales this month by region" or "average response time per service." Every time new data comes in, the numbers stay current with no manual recalculation.
- **Time travel.** Every committed change becomes part of the database's history. You can pin the current state as a frozen view, see the whole knowledge base exactly as it was last week, try out changes in a scratch copy that never touches the real data, and take instant backups.
- **Universal vocabulary.** Brainy ships with a shared language of 42 kinds of things (Person, Document, Task, Concept, Event…) and 127 kinds of connections (Contains, DependsOn, Creates, RelatedTo…). This means data from different sources speaks the same language without you having to translate.
---
## What is Cor?
Cor is a turbocharger for Brainy.
Same car. Same controls. Same fuel. You just swap in a faster engine under the hood, and everything that used to take a moment now happens instantly.
Technically, Cor is an optional plugin written in Rust — a lower-level language that runs much closer to the raw metal of your processor. It plugs into Brainy and takes over the most compute-intensive work: the distance calculations that power meaning search, the number-crunching behind live aggregates, and the set operations that drive label filtering.
You install it with one line, register it with one call, and Brainy automatically uses it everywhere it can help.
---
## How Much Faster?
Plain language:
- **Searches** go from "the blink of an eye" to "faster than a blink." The overall speedup is **5.2× on average** across all operations.
- **Live aggregates** are rebuilt using all CPU cores in parallel, so re-indexing large datasets takes a fraction of the time.
- **Analytics** that aren't even possible in pure JavaScript — real-time anomaly detection, streaming percentile estimates, approximate unique counts — become available because Cor brings the native capabilities required to run them efficiently.
If Brainy is what makes knowledge fast, Cor is what makes Brainy feel instant.
---
## What Does Brainy Replace?
Most applications that need to store and search knowledge end up stitching together several specialized tools. Brainy replaces all of them with one — a single free, open-source library in place of multiple paid services.
### Before and After
**Before Brainy** — a pile of services:
- Pinecone (vectors) + Neo4j (graph) + MongoDB (docs)
- Algolia (search) + Redis (cache) + PostgreSQL + pgvector
- Plus glue code, sync jobs, ETL pipelines, and 3am incidents
**After Brainy** — one thing:
Search, graph, filter, files, time travel, and imports — unified in a single library.
### What Each Tool Is Missing
| Tool | Search | Graph | Filter | VFS | Time travel | Import |
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| **Brainy** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| *— Vector databases —* | | | | | | |
| Pinecone | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| Weaviate | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| Qdrant | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| Chroma | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| *— Graph databases —* | | | | | | |
| Neo4j | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| *— Document stores —* | | | | | | |
| MongoDB | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| Firestore | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| DynamoDB | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| *— Relational + vector —* | | | | | | |
| PostgreSQL + pgvector | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| MySQL | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| SQLite | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| *— Search engines —* | | | | | | |
| Elasticsearch | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| Algolia | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| *— Cache —* | | | | | | |
| Redis | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
Brainy is the only row with every box checked. And it runs all of them in a single query — no stitching services together.
### One library, any scale
Brainy scales from a quick experiment to serious production datasets without changing a line of code. Small datasets live entirely in memory. Larger ones spill to disk, where Brainy shards and compresses everything automatically. Need a backup or a copy? Snapshots are instant — the same API the whole way.
Add Cor and you also unlock memory-mapped storage — aggregate state lives directly in the operating system's memory with zero serialization overhead, as fast as the hardware allows.
---
## What Can You Build?
### Common applications
- **AI agents with persistent memory** — Give any AI an always-on, self-organizing knowledge graph that persists between sessions and across agents.
- **Searchable knowledge bases** — Build institutional memory that links documents automatically and surfaces answers across the full web of related information.
- **Semantic document search** — Index PDFs, code, or media and find them by meaning, not just keywords.
- **Relationship-aware recommendations** — Power product catalogs or content platforms where every recommendation understands what connects to what.
- **Safe experiments** — Test risky changes against a scratch copy of the knowledge base, audit exactly what changed and when, and roll back to any snapshot instantly.
- **Unified business platforms** — Combine booking, CRM, inventory, and analytics in one queryable knowledge graph with no sync pipeline.
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })` to prevent OOM. The cap was sound in intent but ~4x too conservative in calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB (384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB free-memory box the cap derived to 9000 — breaking common safety-cap patterns like `find({ type, where, limit: 10_000 })` that typically return 10-500 entities. Surfaced as a runtime regression with cascading 500s degrading production dashboards. Three concurrent fixes: A. RECALIBRATE THE FORMULA - src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities (reservedQueryMemory / containerMemory / freeMemory) all divided by 100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed entity size. - Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 → 20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides unchanged in behavior. B. TWO-TIER ENFORCEMENT (warn-then-throw) - Below cap (limit <= maxLimit): silent pass, unchanged. - Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per call site (dedup keyed on caller stack frame + limit value), query proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical safety-cap limits keeps working; the warning teaches the recipe so consumers can fix it intentionally. - Hard tier (limit > 2 * maxLimit): throw with the same teaching message format. Real OOM territory; the cap stops being a recommendation and becomes a guardrail. - The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000 against a 9 K-cap box) without disabling OOM protection. Real OOM territory on a JS in-memory brain is hundreds of thousands of results, not 10x the safety cap. C. IMPROVED ERROR / WARNING MESSAGE - Same shape as the 7.30.1 enforcement-error messages: state the problem, name the three escape valves (maxQueryLimit / reservedQueryMemory / pagination), include caller location, link to docs. - Extracted findCallerLocation() helper from brainy.ts to a new src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and the limit enforcement (7.30.2) share one implementation without circular imports. DOCS - New docs/guides/find-limits.md (public: true) — full reference: why the cap exists, the four memory sources the auto-config considers, the three escape valves with when-to-use-which guidance, and an explicit "pagination is the future-proof pattern" callout (8.0 may tighten the cap further; pagination keeps working unchanged). - docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer to the new guide. - RELEASES.md v7.30.2 entry. TESTS - New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass; soft-tier warns once per call site (dedup verified by exercising same vs. different source lines via wrapper closures); soft-tier message format (names all three escape valves + docs link); soft-tier message includes caller location; hard-tier throws; hard-tier message format same as soft-tier; consumer maxQueryLimit override raises the cap and shifts both tiers accordingly; pre-7.30.2 regression scenario explicitly covered. - tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result assumption). - tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover the three-tier semantics (below-cap pass / soft-tier silent / hard-tier throw). - Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and- enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468. CORTEX COMPATIBILITY - Zero Cortex changes required. Every change is JS-side: formula recalibration runs in ValidationConfig.constructor(), two-tier enforcement runs in validateFindParams(), both fire before any storage / index / Cortex call. - The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten per-call limits to keep snapshot semantics cheap; pagination remains the pattern that's guaranteed to keep working. REPO-WIDE CLEANUP Brainy is the only Soulcraft project that is open source. This commit also scrubs closed-source product names and product-specific class/field references from every tracked file in the repo (src/, docs/, tests/, RELEASES.md, CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release notes now refer to "a consumer", "a downstream application", "a production deployment", or "an internal report" — never to the named product. Two product-named test files renamed to neutral diagnostic names. CLAUDE.md gains a project-level guard rule documenting the policy and an example list of the identifiers that may not appear in tracked code. Verification - npx tsc --noEmit: clean - npm test: 1468 / 1468 unit - All four integration subtype + verb + strict + find-limits suites: 78/78 - npm run build: clean - Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
### What Brainy is good at
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })` to prevent OOM. The cap was sound in intent but ~4x too conservative in calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB (384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB free-memory box the cap derived to 9000 — breaking common safety-cap patterns like `find({ type, where, limit: 10_000 })` that typically return 10-500 entities. Surfaced as a runtime regression with cascading 500s degrading production dashboards. Three concurrent fixes: A. RECALIBRATE THE FORMULA - src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities (reservedQueryMemory / containerMemory / freeMemory) all divided by 100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed entity size. - Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 → 20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides unchanged in behavior. B. TWO-TIER ENFORCEMENT (warn-then-throw) - Below cap (limit <= maxLimit): silent pass, unchanged. - Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per call site (dedup keyed on caller stack frame + limit value), query proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical safety-cap limits keeps working; the warning teaches the recipe so consumers can fix it intentionally. - Hard tier (limit > 2 * maxLimit): throw with the same teaching message format. Real OOM territory; the cap stops being a recommendation and becomes a guardrail. - The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000 against a 9 K-cap box) without disabling OOM protection. Real OOM territory on a JS in-memory brain is hundreds of thousands of results, not 10x the safety cap. C. IMPROVED ERROR / WARNING MESSAGE - Same shape as the 7.30.1 enforcement-error messages: state the problem, name the three escape valves (maxQueryLimit / reservedQueryMemory / pagination), include caller location, link to docs. - Extracted findCallerLocation() helper from brainy.ts to a new src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and the limit enforcement (7.30.2) share one implementation without circular imports. DOCS - New docs/guides/find-limits.md (public: true) — full reference: why the cap exists, the four memory sources the auto-config considers, the three escape valves with when-to-use-which guidance, and an explicit "pagination is the future-proof pattern" callout (8.0 may tighten the cap further; pagination keeps working unchanged). - docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer to the new guide. - RELEASES.md v7.30.2 entry. TESTS - New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass; soft-tier warns once per call site (dedup verified by exercising same vs. different source lines via wrapper closures); soft-tier message format (names all three escape valves + docs link); soft-tier message includes caller location; hard-tier throws; hard-tier message format same as soft-tier; consumer maxQueryLimit override raises the cap and shifts both tiers accordingly; pre-7.30.2 regression scenario explicitly covered. - tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result assumption). - tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover the three-tier semantics (below-cap pass / soft-tier silent / hard-tier throw). - Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and- enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468. CORTEX COMPATIBILITY - Zero Cortex changes required. Every change is JS-side: formula recalibration runs in ValidationConfig.constructor(), two-tier enforcement runs in validateFindParams(), both fire before any storage / index / Cortex call. - The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten per-call limits to keep snapshot semantics cheap; pagination remains the pattern that's guaranteed to keep working. REPO-WIDE CLEANUP Brainy is the only Soulcraft project that is open source. This commit also scrubs closed-source product names and product-specific class/field references from every tracked file in the repo (src/, docs/, tests/, RELEASES.md, CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release notes now refer to "a consumer", "a downstream application", "a production deployment", or "an internal report" — never to the named product. Two product-named test files renamed to neutral diagnostic names. CLAUDE.md gains a project-level guard rule documenting the policy and an example list of the identifiers that may not appear in tracked code. Verification - npx tsc --noEmit: clean - npm test: 1468 / 1468 unit - All four integration subtype + verb + strict + find-limits suites: 78/78 - npm run build: clean - Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
Brainy is the engine underneath production systems that need to combine semantic search, structured filtering, and graph traversal in a single query — agent memory, knowledge-base platforms, business operations consoles, multi-agent coordination, and more. The combination of vector + graph + metadata search in one indexed call is what differentiates it from running three engines side by side.