11 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 4e058720b4 |
fix(index): a field holds every value kind it was written with, not the first one
Some checks failed
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
Delta Gate / Delta gate — candidate vs control (push) Failing after 7s
The metadata index fixed a field's value type from the first value it saw.
Every later value of another kind was coerced to that type, and when coercion
failed — `Number('electronics')` is NaN — the value was dropped from the index
with no error at all. The row stayed readable by id and by vector search and
vanished only from equality filters on that one field, which is what made it so
quiet: writing `category: 'electronics'` rows and then `category: 5` rows left
`where { category: 5 }` returning nothing, while the same rows in a
numbers-only corpus answered correctly.
The column store now keeps one posting column per (field, kind), where a kind
is a JavaScript typeof class. The first kind a field sees keeps the historical
`_column_index/<field>/` layout, so a single-kind field is byte-identical to
what earlier versions wrote and an index written before this opens unchanged;
each later kind takes its own column at `_column_index/<field>/k/<kind>/`.
Equality reads the column matching the query value's own kind, so `{c: 5}` and
`{c: '5'}` match different rows and neither is coerced into the other. Ranges
route by the kind of their bounds, and an unbounded range — the "has any value"
probe behind `exists` — reads every kind. A mixed field orders by kind first,
then by value, because a number and a string have no order between them. A
value that cannot be encoded for the column its own kind selected now raises
instead of being skipped: that path is unreachable by construction, and if it
is ever reached it is the silent drop this change exists to end.
Two neighbours fell out of the same routing. A boolean query value is now
encoded to the 1/0 the column stores, so boolean equality matches at all. And
an integer column widens to f64 the first time a non-integer arrives, so 4.5 is
stored as itself rather than rounded to 5 and answering the wrong query.
Field type inference reports every kind a field holds beside its dominant
reading, rather than leaving callers to treat one type as the whole answer.
Pins: mixed-kind equality in both write orders, `5` vs `'5'`, booleans mixed in,
a numeric range over a mixed field's numbers, close/reopen keeping every typed
posting, and an index in the pre-existing on-disk shape still reading.
`tests/critical-neural-validation.test.ts` — which writes `category` as strings
in one test and as numbers in another against one shared brain — passes whole
for the first time.
(cherry picked from commit
|
|||
| 24bf6cdbc5 |
feat(namespace): NO SPECIAL NAMES + storage fidelity — the ruled completion of the field-addressing law
The write side of the law, ruled 2026-08-03: data is either in main space where developers can use anything, or it is in system.*. - The reserved-name write door DIES: add/update/relate/updateRelation metadata bags accept EVERY name (confidence, type, id, data, level, content, ...) as ordinary user fields — indexed, filterable, sortable, aggregatable, identical to any other field. The remap/enforce/warn machinery, the reservedFieldPolicy config (now a typed init refusal), and the compile-time metadata key bans are all removed. The one write refusal left: keys spelled 'system.*' (namespace forgery), now enforced on all four write doors. - STORED RECORDS GO NESTED (v2): engine fields top-level, the user bag nested verbatim under 'metadata', sealed by a format stamp — by-name storage discrimination is unsound once colliders are admitted. Legacy flat records stay readable forever through the shape-aware splitters (sound for them: the old door refused colliders). Time travel rides the same split (generation store snapshots whole records). - Name-based index exclusions DIE: user frame indexes every name; the excludeFields/indexedFields knobs and their silent-[] holes are gone; bulk-payload protection is value-shape only, uniform across names. - Consumer-sweep findings fixed in the same wave: per-type counts read the frozen 'system.type' column (addToIndex sort, affinity tracking, cold-count rehydration, VFS type bitmaps — legacy 'noun' fallback for pre-rebuild reads); resolveHiddenIds addresses 'system.visibility' (bare 'visibility' was a silent no-op under the law — VFS/system entities leaked into default reads). - Fidelity fallout fixed in the owning layers: readEntityFieldAddress reads the bag first (colliders were absent-shadowed by its own guard) and never serves system addresses from the bag; blob history refs read the bag shape-aware; migration transforms now receive ONE normalized view (engine fields + nested bag) regardless of stored era, and stray flat-habit keys refuse with the fix in the message. - THE REOPEN-COLLIDER CONFORMANCE CASE (required before any RC counts as gates-green): all ten collider names + plumbing names written as user fields, verified verbatim + queryable across live reads, flush+reopen, a forced epoch rebuild, and asOf time travel; relation mirror; forgery refusals; legacy flat-record compat. 8/8 green. Gates: unit 1901/1901 (exit 0) · integration 758 (exit 0) · conformance 27/27 (exit 0) · consumer test sweep migrated (10 files). |
|||
| 945d92d29e |
fix: one field-resolution law across aggregation hooks, source.where, removeMany, and find() spellings
Four fixes from a consumer conformance report, one root disease — two field-resolution regimes where there must be one: - The delete/update aggregation hooks fed the engine a partial entity view (type/service/data/metadata only), so a reserved-field groupBy (subtype, visibility, ...) resolved to a nonexistent group on the way down: counts drifted upward forever after deletes, and updates moving an entity between reserved-field groups double-counted. The hooks now pass the full-fidelity view via entityForAggFromRawRecord (every reserved field top-level, mirroring the add path); the update sites pass the full get() view instead of a hand-rolled subset. - Aggregation source.where resolved fields only against the custom metadata bag, so where on a reserved field silently matched nothing. The matcher now resolves each filtered field through resolveEntityField — the same single source of truth groupBy uses. - removeMany() with no usable selector (bare array passed positionally, empty params, ids: []) resolved successfully having deleted nothing. All three now throw; the two legacy tests that pinned the silent no-op as 'graceful' now pin the refusal. - find() where keys accept both spellings: a metadata.-prefixed key falls back to its flattened spelling when the prefixed one is not indexed (metadata is flattened at index time). A literal nested custom key named metadata still wins when indexed as spelled. Five regression pins in aggregate-reserved-fields.test.ts (4 of 5 vary red on the unfixed code). |
|||
| 3f9f140c8c |
test(8.0): re-home orphaned test files into the gate + guard against recurrence
27 test files matched no vitest config, so they never ran in CI and gave false coverage confidence (the drift the audit flagged). - Re-homed 10 functional suites into the gate (renamed to *.unit.test.ts / *.integration.test.ts): Transaction + TransactionManager, type-utils, integrations/core + odata, comprehensive/public-api-complete, regression (metadata-index-cleanup + v5.7.0-deadlock), vfs/tree-operations + vfs-bulkwrite-race. ~192 previously-dark tests now run and pass. - Deleted 8 bit-rotted/redundant files that fail against 8.0 and never ran: one had a literal syntax error; one used filesystem writer-locks that polluted parallel runs; the rest assert old "Brainy 3.0/v3.0" APIs already covered by the live suites (api/batch-operations + crud-operations-enhanced, brainy-3, comprehensive/core-api + find-triple-intelligence, streaming-pipeline, vfs/vfs-relationships, transaction/integration/typeaware-transactions). - Added a coverage guard (tests/unit/test-suite-coverage-guard.test.ts) that FAILS CI if any *.test.ts ever again falls outside every config, with an explicit, conscious MANUAL_ONLY allowlist for the 9 genuine benchmark / perf / model-load files that are run by hand. Gate green: unit 1712, integration 607. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
|||
| 1f7e365a4e | chore(8.0): final pre-RC1 sweep — API consistency, named errors, orphans, zero-cast codebase | |||
| 780fb6444b |
feat(8.0)!: flip requireSubtype default to true (BRAINY-8.0-SUBTYPE-CONTRACT § C-1)
Brainy 8.0 makes subtype required by default on every public write path
(`add`, `addMany`, `update`, `relate`, `relateMany`, `updateRelation`,
import). Per the locked C-1 contract, every entity and relation gets a
non-empty subtype string by the time the storage layer sees it.
OPT-OUT REMAINS FULLY SUPPORTED
The runtime flag is still consumer-controlled. Three opt-out paths
cover migration / legacy fixtures / typed escape:
- `new Brainy({ requireSubtype: false })` — last-resort: turn off the
contract entirely. Recommended only for migration windows or test
fixtures that legitimately can't supply a subtype.
- `new Brainy({ requireSubtype: { except: [NounType.Thing, ...] } })` —
per-type allowlist: strict everywhere except the listed types.
- `brain.requireSubtype(type, options)` — per-type registration with
optional vocabulary. Composes with the brain-wide flag.
Default is now `true`. Opt-out is explicit and documented; nothing
silently degrades.
TEST SWEEP
Bulk-applied `requireSubtype: false` to every `new Brainy({...})` call
site across 120 test files. Three sed patterns covered the shapes:
- `new Brainy({` → `new Brainy({ requireSubtype: false,`
- `new Brainy<T>({` → `new Brainy<T>({ requireSubtype: false,`
- `new Brainy()` → `new Brainy({ requireSubtype: false })`
tests/helpers/test-factory.ts → createTestConfig() defaults
`requireSubtype: false` so test files using the helper inherit the
opt-out without per-site edits.
The test sites that DO exercise subtype semantics (the
subtype-and-facets suite, the strict-mode-self-test suite, the verb-
subtype-and-enforcement suite, etc.) already pass real subtypes — they
were the 7.30.x acceptance tests for this contract. Those tests
continue to pass unchanged.
CHANGES
src/brainy.ts
- normalizeConfig() — `requireSubtype` default `false` → `true`.
Comment refreshed to document the three opt-out paths.
tests/* (120 files)
- Bulk-edited brain construction sites. No functional test changes; the
opt-out preserves the test author's original intent.
tests/helpers/test-factory.ts
- createTestConfig() base config gains `requireSubtype: false`.
NO-OP for consumers who were already passing subtype on every write.
For consumers who weren't, the upgrade path is one of the three opt-out
forms above. Migration recipe documented in 8.0 release notes (next
commit).
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (same pre-existing race-condition outstanding;
no other regressions from the flip)
|
|||
| 9e307e457f |
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
|
|||
| d4cb26c604 |
feat: mmap-vector backend wiring — HNSWIndex consumes vectorStore:mmap (2.4.0 #2)
Cortex already registers the vectorStore:mmap provider (its Rust NativeMmapVectorStore), but brainy has never consumed it — preloadVectors and getVectorSafe still go straight to storage.getNounVector for every id, even when an mmap layer is available. This wires the consumer end. Architecture: - NEW MmapVectorBackend (src/hnsw/mmapVectorBackend.ts) — bridges brainy's UUID-keyed vector reads to a int-slot mmap file via the vectorStore:mmap provider. Slots are addressed by the stable int id from the post-2.4.0 #1 EntityIdMapper (the foundation this depends on). Auto-grows the file (doubling) when a write lands beyond capacity, so HNSWIndex never has to think about sizing. The class never touches per-entity storage — it owns only the mmap layer. - HNSWIndex changes — adds a vectorBackend field + a setVectorBackend setter. The vector read paths (preloadVectors, getVectorSafe) try the mmap layer first; on a storage fallback hit, they LAZILY write back into the mmap slot. An upgraded install converges to the zero-copy fast path under live traffic — no big-bang migration step. The legacy per-entity path is preserved and still used when no backend is set. - brainy.ts wiring — a new private wireMmapVectorBackend() runs once during init, after plugin activation + metadataIndex setup. It activates the backend only when (a) the vectorStore:mmap provider is registered, (b) the storage adapter resolves a real local path via getBinaryBlobPath(), and (c) the metadata index exposes its idMapper. Cloud adapters return null on (b) and the backend is silently skipped; HNSWIndex's behaviour is then identical to pre-2.4.0. - Provider interfaces in plugin.ts — VectorStoreMmapProvider and VectorStoreMmapInstance document the contract cortex's class fulfils (the class IS the provider — static factory methods). Brainy depends on the interfaces, not on cortex; the structural match is verified when cortex 2.4.0 picks up this brainy release. Tests (1428 total, +11 vs pre-2.4.0): - tests/unit/hnsw/mmap-vector-backend.test.ts — 6 unit tests with an in-memory mock provider. Covers round-trip, batch reads with interleaved misses, slot stability (no re-slotting on overwrite), file growth without data loss, idempotent open, and null returns for unwritten slots. The real perf integration with cortex's NativeMmapVectorStore is exercised when cortex 2.4.0 wires this in. - tests/unit/utils/entity-id-mapper-stability.test.ts — moved here from tests/regression/ (which is NOT in the unit-config include glob, so the five #23 tests were not actually being run by npm test). The unit config matches tests/unit/**/*.test.ts. The 2.4.0 #2 follow-up will be the chunked-segment layout for remote storage adapters (S3 / R2 / GCS) where a single growing file doesn't fit immutable objects. For 2.4.0 release: local-FS only. |
|||
| b2408cb5a6 |
feat: stable EntityIdMapper — rebuild() no longer renumbers UUID→int
Previously metadataIndex.rebuild() called idMapper.clear() which reset nextId to 1 and renumbered every UUID by re-insertion order. Any consumer that had persisted int-keyed data against the old map was silently invalidated — and 2.4.0's vector mmap store (#20), graph link compression (#21), and column-store JS↔native interchange all need persisted int indices that survive a rebuild. Remove the unconditional clear() in rebuild(). The rebuild already re-iterates every entity via idMapper.getOrAssign(uuid), which returns the existing int unchanged for known UUIDs. Stale UUID→int entries for entities no longer in storage persist as harmless memory overhead; a dedicated prune step can be added if it ever matters. clearAllIndexData() — the explicit nuclear recovery path — keeps its existing idMapper.clear() call (renumbering is intentional there), and now logs a prodLog.warn making it explicit that any persisted int-keyed data is invalidated and must be rebuilt from canonical sources. Strengthened the EntityIdMapper class JSDoc to document the stability guarantee as a contract — append-only getOrAssign, monotonic nextId, remove() leaves permanent holes, rebuild() never renumbers, only clear() does. Added tests/regression/entity-id-mapper-stability.test.ts pinning down the five-point contract: (1) single-rebuild stability; (2) many-rebuild stability; (3) post-rebuild adds get fresh monotonic ints; (4) removes leave permanent holes — new entities never recycle; (5) clearAllIndexData() explicitly renumbers (the documented destructive path). Foundation for 2.4.0 #2-#4. Full test suite (62 files, 1417 tests) green. |
|||
| 1a628daa83 |
fix: metadata index not cleaned up after delete/deleteMany
Three bugs caused deleted entities to persist in the metadata index: 1. idMapper never cleaned up — EntityIdMapper accumulated UUID→int mappings permanently. idMapper.getAllIntIds() is used as the universe for ne and exists:false operators, so deleted entities returned in those queries indefinitely. Fix: removeFromIndex() now calls idMapper.remove(id) and idMapper.flush() after all bitmap operations complete (must be last because removeFromChunk() reads idMapper.getInt(id) internally). 2. Optional fields indexed as __NULL__ but never unindexed — entityForIndexing in add() included confidence, weight, and createdBy as explicit keys even when undefined. Object.entries() preserves undefined-valued keys so extractIndexableFields() indexed them as '__NULL__' bitmap entries. storageMetadata omitted those keys via conditional spreading, so removeFromIndex() passed a structure without those keys and never cleaned them up. Fix: entityForIndexing now uses conditional spreading for confidence, weight, and createdBy matching storageMetadata exactly. 3. result.successful updated before transaction commits — deleteMany() pushed ids to result.successful inside the transaction builder, before transaction.execute() ran. A rollback would leave result.successful containing ids that were never actually deleted. Fix: queued ids are held in a local chunkQueued array and moved to result.successful only after executeTransaction() resolves without throwing. Adds regression test suite (14 tests) covering delete() and deleteMany() for type-index cleanup, ne operator, exists:false operator, optional-field indexing, and partial deletion correctness. Reported by wickworks team. |
|||
| eb9af45bab |
fix: resolve v5.7.0 deadlock by restoring storage layer separation (v5.7.1)
CRITICAL BUG FIX - v5.7.0 caused complete production failure PROBLEM: v5.7.0 introduced circular dependency deadlock during GraphAdjacencyIndex initialization: GraphAdjacencyIndex.rebuild() → storage.getVerbs() → getVerbsBySource_internal() → getGraphIndex() [NEW in v5.7.0] → [waiting for rebuild to complete] → DEADLOCK SYMPTOMS (Production Impact): - ALL imports hung at "Reading Data Structure" for 760+ seconds - brain.add() operations took 12+ seconds per entity (50x slower) - Zero entities imported successfully - 100% of Workshop users unable to import files - No errors thrown - infinite wait - Forced immediate rollback to v5.6.3 ROOT CAUSE: v5.7.0 modified storage internal methods (getVerbsBySource_internal, getVerbsByTarget_internal) to use GraphAdjacencyIndex optimization, creating tight coupling where storage depends on index AND index depends on storage. This violated separation of concerns and created deadlock. SOLUTION (Architectural Fix): Reverted storage internals to v5.6.3 implementation (lines 2320-2444): - Storage layer simple, no index dependencies ✅ - GraphAdjacencyIndex can safely call storage.getVerbs() to rebuild ✅ - No circular dependency possible ✅ - Proper layered architecture restored ✅ LAYERS (Correct Architecture): Layer 3 (Brainy/Queries): CAN use GraphAdjacencyIndex Layer 2 (GraphAdjacencyIndex): Uses storage.getVerbs() to rebuild Layer 1 (Storage Internals): NO GraphAdjacencyIndex calls IMPACT: - Slightly slower GraphAdjacencyIndex.rebuild() (one-time init cost) - High-level queries still use optimized index - Import performance unaffected (writes don't trigger init) - NO breaking changes to public API TESTING: - Added 4 regression tests (tests/regression/v5.7.0-deadlock.test.ts) - All 1146 existing tests pass ✅ - Import + relationships complete in <1 second (not 760+) - No 12+ second delays per entity ✅ FILES CHANGED: - src/storage/baseStorage.ts (reverted lines 2320-2444 to v5.6.3) - tests/regression/v5.7.0-deadlock.test.ts (new regression tests) - CHANGELOG.md (comprehensive v5.7.1 entry with upgrade instructions) VERIFICATION: Workshop team should upgrade immediately: npm install @soulcraft/brainy@5.7.1 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> |
