feat: add confidence/weight to Entity and flatten Result fields for convenient access
Add confidence and weight properties to Entity interface and flatten Result fields to top level for improved developer experience and API consistency.
Breaking Changes: None (all changes are backward compatible)
Phase 2 - Entity Confidence & Weight:
- Add confidence (type classification certainty) and weight (entity importance) to Entity interface
- Add confidence/weight parameters to AddParams and UpdateParams
- Update convertNounToEntity() to extract confidence/weight from storage
- Update add() and update() methods to preserve confidence/weight in metadata
- Enable developers to specify and access entity confidence/weight scores
Phase 3 - Result Field Flattening:
- Flatten commonly-used entity fields (type, metadata, data, confidence, weight) to Result top level
- Add createResult() helper for consistent Result construction
- Update all find() code paths to use createResult()
- Enable direct access: result.metadata instead of result.entity.metadata
- Preserve full entity in result.entity for backward compatibility
VFS Fix (from previous work):
- Fix VFSStructureGenerator to use brain.vfs() cached instance instead of creating separate instance
- Improve VFS error messages with step-by-step guidance
- Update examples to show correct vfs.init() usage
- Add comprehensive VFS import verification tests
Documentation Updates:
- Update API_REFERENCE.md with confidence/weight examples and flattened Result documentation
- Enhance JSDoc for add(), get(), find(), similar() with v4.3.0 examples
- Document Result structure changes and backward compatibility
- Add migration examples showing both old and new access patterns
Tests:
- Add 16 comprehensive tests for Entity confidence/weight exposure
- Add tests for Result field flattening
- Add tests for backward compatibility
- All tests passing (16/16)
API Consistency:
- Entity: direct access to confidence/weight
- Result: flattened fields + nested entity (both work)
- Relation: already had confidence/weight (consistent)
- VFS: inherits from Entity (automatic)
Files Changed:
- src/types/brainy.types.ts - Updated Entity, AddParams, UpdateParams, Result interfaces
- src/brainy.ts - Updated implementation and JSDoc for all affected methods
- tests/integration/entity-confidence-weight.test.ts - 16 comprehensive tests
- docs/API_REFERENCE.md - Updated with v4.3.0 examples
- src/importers/VFSStructureGenerator.ts - VFS fix
- src/vfs/VirtualFileSystem.ts - Improved error messages
- examples/unified-import-example.ts - Added vfs.init() example
- tests/integration/vfs-*-verification.test.ts - VFS verification tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-23 12:19:50 -07:00
|
|
|
/**
|
|
|
|
|
* VFS Debug Test - Minimal reproduction to find the issue
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
import { describe, it, expect, beforeEach } from 'vitest'
|
|
|
|
|
import { Brainy } from '../../src/brainy.js'
|
|
|
|
|
import * as XLSX from 'xlsx'
|
|
|
|
|
|
|
|
|
|
describe('VFS Debug', () => {
|
|
|
|
|
it('minimal VFS writeFile test', async () => {
|
|
|
|
|
const brain = new Brainy({ storage: { type: 'memory' } })
|
|
|
|
|
await brain.init()
|
|
|
|
|
|
|
|
|
|
console.log('✅ Brain initialized')
|
|
|
|
|
|
|
|
|
|
// Get VFS and initialize
|
2025-11-02 11:38:12 -08:00
|
|
|
const vfs = brain.vfs
|
feat: add confidence/weight to Entity and flatten Result fields for convenient access
Add confidence and weight properties to Entity interface and flatten Result fields to top level for improved developer experience and API consistency.
Breaking Changes: None (all changes are backward compatible)
Phase 2 - Entity Confidence & Weight:
- Add confidence (type classification certainty) and weight (entity importance) to Entity interface
- Add confidence/weight parameters to AddParams and UpdateParams
- Update convertNounToEntity() to extract confidence/weight from storage
- Update add() and update() methods to preserve confidence/weight in metadata
- Enable developers to specify and access entity confidence/weight scores
Phase 3 - Result Field Flattening:
- Flatten commonly-used entity fields (type, metadata, data, confidence, weight) to Result top level
- Add createResult() helper for consistent Result construction
- Update all find() code paths to use createResult()
- Enable direct access: result.metadata instead of result.entity.metadata
- Preserve full entity in result.entity for backward compatibility
VFS Fix (from previous work):
- Fix VFSStructureGenerator to use brain.vfs() cached instance instead of creating separate instance
- Improve VFS error messages with step-by-step guidance
- Update examples to show correct vfs.init() usage
- Add comprehensive VFS import verification tests
Documentation Updates:
- Update API_REFERENCE.md with confidence/weight examples and flattened Result documentation
- Enhance JSDoc for add(), get(), find(), similar() with v4.3.0 examples
- Document Result structure changes and backward compatibility
- Add migration examples showing both old and new access patterns
Tests:
- Add 16 comprehensive tests for Entity confidence/weight exposure
- Add tests for Result field flattening
- Add tests for backward compatibility
- All tests passing (16/16)
API Consistency:
- Entity: direct access to confidence/weight
- Result: flattened fields + nested entity (both work)
- Relation: already had confidence/weight (consistent)
- VFS: inherits from Entity (automatic)
Files Changed:
- src/types/brainy.types.ts - Updated Entity, AddParams, UpdateParams, Result interfaces
- src/brainy.ts - Updated implementation and JSDoc for all affected methods
- tests/integration/entity-confidence-weight.test.ts - 16 comprehensive tests
- docs/API_REFERENCE.md - Updated with v4.3.0 examples
- src/importers/VFSStructureGenerator.ts - VFS fix
- src/vfs/VirtualFileSystem.ts - Improved error messages
- examples/unified-import-example.ts - Added vfs.init() example
- tests/integration/vfs-*-verification.test.ts - VFS verification tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-23 12:19:50 -07:00
|
|
|
await vfs.init()
|
|
|
|
|
|
|
|
|
|
console.log('✅ VFS initialized')
|
|
|
|
|
|
|
|
|
|
// Write a single file
|
|
|
|
|
await vfs.writeFile('/test.txt', 'Hello World')
|
|
|
|
|
|
|
|
|
|
console.log('✅ File written')
|
|
|
|
|
|
|
|
|
|
// Check if entity exists using find()
|
|
|
|
|
const allEntities = await brain.find({ limit: 100 })
|
|
|
|
|
console.log(`📊 Total entities (via find): ${allEntities.length}`)
|
|
|
|
|
|
|
|
|
|
console.log('Entities from find() - CHECKING STRUCTURE:')
|
|
|
|
|
allEntities.forEach((e, i) => {
|
|
|
|
|
console.log(` ${i+1}. Result object keys: ${Object.keys(e).join(', ')}`)
|
|
|
|
|
console.log(` e.id: ${e.id}`)
|
|
|
|
|
console.log(` e.score: ${(e as any).score}`)
|
|
|
|
|
console.log(` e.entity: ${(e as any).entity ? 'EXISTS' : 'MISSING'}`)
|
|
|
|
|
if ((e as any).entity) {
|
|
|
|
|
console.log(` e.entity.type: ${(e as any).entity.type}`)
|
|
|
|
|
console.log(` e.entity.metadata: ${JSON.stringify((e as any).entity.metadata)}`)
|
|
|
|
|
}
|
|
|
|
|
console.log(` e.type (direct): ${e.type}`)
|
|
|
|
|
console.log(` e.metadata (direct): ${JSON.stringify(e.metadata)}`)
|
|
|
|
|
})
|
|
|
|
|
|
|
|
|
|
const vfsEntities = allEntities.filter(e => e.metadata?.vfsType)
|
|
|
|
|
console.log(`📊 VFS entities (via find): ${vfsEntities.length}`)
|
|
|
|
|
|
|
|
|
|
// Now try getting entities directly
|
|
|
|
|
console.log('\nChecking entities directly with brain.get():')
|
|
|
|
|
for (const entity of allEntities) {
|
|
|
|
|
const direct = await brain.get(entity.id)
|
|
|
|
|
if (direct) {
|
|
|
|
|
console.log(` ${direct.id}:`)
|
|
|
|
|
console.log(` metadata: ${JSON.stringify(direct.metadata)}`)
|
|
|
|
|
if (direct.metadata?.vfsType) {
|
|
|
|
|
console.log(` ✅ HAS vfsType: ${direct.metadata.vfsType}`)
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Try reading the file
|
|
|
|
|
const content = await vfs.readFile('/test.txt')
|
|
|
|
|
console.log(`\n📄 File content: "${content.toString()}"`)
|
|
|
|
|
|
|
|
|
|
// Try VFS directory listing
|
|
|
|
|
console.log('\n📂 VFS Directory Listing:')
|
|
|
|
|
const rootContents = await vfs.readdir('/')
|
|
|
|
|
console.log(` Root contents: ${rootContents.join(', ')}`)
|
|
|
|
|
|
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
|
|
|
// Try getDirectChildren (the consumer's method)
|
feat: add confidence/weight to Entity and flatten Result fields for convenient access
Add confidence and weight properties to Entity interface and flatten Result fields to top level for improved developer experience and API consistency.
Breaking Changes: None (all changes are backward compatible)
Phase 2 - Entity Confidence & Weight:
- Add confidence (type classification certainty) and weight (entity importance) to Entity interface
- Add confidence/weight parameters to AddParams and UpdateParams
- Update convertNounToEntity() to extract confidence/weight from storage
- Update add() and update() methods to preserve confidence/weight in metadata
- Enable developers to specify and access entity confidence/weight scores
Phase 3 - Result Field Flattening:
- Flatten commonly-used entity fields (type, metadata, data, confidence, weight) to Result top level
- Add createResult() helper for consistent Result construction
- Update all find() code paths to use createResult()
- Enable direct access: result.metadata instead of result.entity.metadata
- Preserve full entity in result.entity for backward compatibility
VFS Fix (from previous work):
- Fix VFSStructureGenerator to use brain.vfs() cached instance instead of creating separate instance
- Improve VFS error messages with step-by-step guidance
- Update examples to show correct vfs.init() usage
- Add comprehensive VFS import verification tests
Documentation Updates:
- Update API_REFERENCE.md with confidence/weight examples and flattened Result documentation
- Enhance JSDoc for add(), get(), find(), similar() with v4.3.0 examples
- Document Result structure changes and backward compatibility
- Add migration examples showing both old and new access patterns
Tests:
- Add 16 comprehensive tests for Entity confidence/weight exposure
- Add tests for Result field flattening
- Add tests for backward compatibility
- All tests passing (16/16)
API Consistency:
- Entity: direct access to confidence/weight
- Result: flattened fields + nested entity (both work)
- Relation: already had confidence/weight (consistent)
- VFS: inherits from Entity (automatic)
Files Changed:
- src/types/brainy.types.ts - Updated Entity, AddParams, UpdateParams, Result interfaces
- src/brainy.ts - Updated implementation and JSDoc for all affected methods
- tests/integration/entity-confidence-weight.test.ts - 16 comprehensive tests
- docs/API_REFERENCE.md - Updated with v4.3.0 examples
- src/importers/VFSStructureGenerator.ts - VFS fix
- src/vfs/VirtualFileSystem.ts - Improved error messages
- examples/unified-import-example.ts - Added vfs.init() example
- tests/integration/vfs-*-verification.test.ts - VFS verification tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-23 12:19:50 -07:00
|
|
|
const children = await vfs.getDirectChildren('/')
|
|
|
|
|
console.log(` Direct children: ${children.length}`)
|
|
|
|
|
children.forEach(child => {
|
|
|
|
|
console.log(` - ${child.metadata.name} (${child.metadata.vfsType})`)
|
|
|
|
|
})
|
|
|
|
|
|
|
|
|
|
// THE REAL TEST: Can we query VFS?
|
|
|
|
|
expect(children.length).toBeGreaterThan(0)
|
|
|
|
|
expect(rootContents.length).toBeGreaterThan(0)
|
|
|
|
|
})
|
|
|
|
|
})
|