Step 2 of the brainy 8.0 rename scaffolding. Adds the new algorithm-neutral
config surface alongside the legacy config.hnsw path (which becomes a compat
shim removed in the final cleanup commit).
CHANGES
src/utils/recallPreset.ts (NEW)
- DEFAULT_RECALL = 'balanced'
- HNSW_PRESETS table — fast/balanced/accurate → (M, efConstruction, efSearch).
'balanced' equals brainy 7.x DEFAULT_CONFIG byte-for-byte so a 7.x → 8.0
upgrade with no explicit recall value is a no-op.
- resolveRecallHnswKnobs(preset) — preset → knobs.
- resolveJsHnswConfig(vectorConfig) — preset first, then `advanced.hnsw` overrides
win. Explicit knobs always beat the preset.
src/types/brainy.types.ts
- New optional `vector` field on BrainyConfig:
vector?: {
recall?: 'fast' | 'balanced' | 'accurate'
quantization?: { enabled?, bits?: 8 | 4, rerankMultiplier? }
vectorStorage?: 'memory' | 'lazy'
advanced?: {
hnsw?: { M?, efConstruction?, efSearch?, ml? }
diskann?: { pqM?, pqKsub?, maxDegree?, searchListSize?, alpha?, ... }
}
}
- `hnsw` field marked @deprecated with note that it stays as a compat shim
during the 8.0 rename and is removed in the final cleanup commit.
src/utils/unifiedCache.ts
- Cache-category union widened from
'hnsw' | 'metadata' | 'embedding' | 'other'
to
'hnsw' | 'vectors' | 'metadata' | 'embedding' | 'other'
- 5 union sites + 4 typed-counter maps (typeAccessCounts, checkFairness
typeSizes/typeCounts/accessRatios/sizeRatios, getStats typeSizes/typeCounts,
evictType loop) all gained the 'vectors' key with initial value 0.
- The hard-coded fairness-check loop now iterates both keys.
- Final 8.0 cleanup will narrow back to 'vectors' | 'metadata' | 'embedding' | 'other'.
src/brainy.ts (normalizeConfig)
- Adds the new `vector` field to the Required<BrainyConfig> return shape so
TS compiles. Reads config?.vector ?? undefined as any (same pattern as
the other optional config fields).
NO-OP scope
No behavioural change. The new config.vector path is silently ignored at
runtime in this commit — wired up in step 4 (storage + index resolution).
Pure surface plumbing. Tests + build green.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
Native accelerators (cortex) register providers for metadataIndex, graphIndex,
hnsw, entityIdMapper, cache, columnStore, and aggregation. Until now the exact
method/property surface brainy calls on each was implicit — a provider could
drop a member brainy depends on and only fail at runtime when that path ran.
This defines and exports the provider contracts from the stable
@soulcraft/brainy/plugin entrypoint:
- MetadataIndexProvider, GraphIndexProvider, HnswProvider,
EntityIdMapperProvider, CacheProvider — each typed as exactly the surface
brainy calls (optional/feature-detected members like HNSW setPersistMode and
enableCOW are intentionally excluded).
- Re-exports ColumnStoreProvider and AggregationProvider (and the aggregate
types) from the same entrypoint so a plugin author can import the whole
provider surface from one place.
Brainy's own baseline classes now `implements` these contracts
(MetadataIndexManager, GraphAdjacencyIndex, HNSWIndex, EntityIdMapper,
UnifiedCache), so the interfaces can never silently diverge from what brainy
ships — and any provider that declares `implements` gets a compile error the
moment brainy starts requiring a new member.
The embeddings/embedBatch providers stay typed by the existing EmbeddingFunction
(no duplicate interface added). Type-only changes; no runtime behavior change.
UnifiedCache.setMaxSize() allows runtime resizing of the cache maximum.
When the new limit is smaller than current usage, lowest-value items are
evicted until the cache fits. Used by Cortex's ResourceManager to
rebalance cache vs instance memory as brainy instances are created and
evicted in multi-tenant deployments.
Also adds getMaxSize() for observability.
- Wire PluginRegistry into Brainy init() with provider resolution for distance,
metadataIndex, graphIndex, embeddings, roaring, msgpack, and storage adapters
- Add setupStorage() factory that resolves storage:* providers from plugins before
falling back to built-in createStorage()
- Export internals API (setGlobalCache, UnifiedCache, EntityIdMapper, etc.) for
cortex plugin consumption
- Add plugin.test.ts verifying registration, activation, and provider resolution
- Deprecate browser support (OPFS, Web Workers, WASM embeddings) with warnings
in preparation for v8.0 server-only release
- FileSystemStorage: fix setupStorage resolution for mmap-filesystem provider
The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:
1. validateConsistency() to falsely detect corruption on every startup,
triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
and report inflated totalEntries/totalIds stats
Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
Bug 1: verbCountsByType optimization skipping requested verb types
- After restart, stale statistics could cause VerbType.Contains to be skipped
- readdir() would return empty/incomplete results
- Fixed by never skipping verb types explicitly requested in filter
- Added fast path for sourceId + verbType combo (common VFS pattern)
- Save statistics on first entity of each type (not just every 100th)
Bug 2: UnifiedCache not invalidated on path deletion
- rmdir() cleared local caches but NOT the global UnifiedCache
- When folder recreated, resolve() returned stale entity ID
- Caused "Source entity not found" errors
- Fixed by adding deleteByPrefix() to UnifiedCache
- Fixed invalidatePath() to also clear UnifiedCache entries
Files modified:
- src/storage/baseStorage.ts (verbCountsByType fix + fast path)
- src/utils/unifiedCache.ts (deleteByPrefix method)
- src/vfs/PathResolver.ts (cache invalidation fix)
Tests added:
- tests/unit/storage/vfs-mkdir-bug.test.ts (7 tests)
Reported by: Soulcraft Workshop team
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
v3.40.1 fixed the eviction formula but fairness parameters were too gentle,
allowing metadata to regenerate faster than eviction could keep up. This
caused cache thrashing: evict 20% → regenerate → evict 20% → repeat.
Changes to prevent thrashing:
1. Fairness interval: 60s → 30s (faster response to imbalances)
2. Size threshold: 90% → 70% (earlier intervention)
3. Access threshold: <10% → <15% (catch more imbalances)
4. Eviction amount: 20% → 50% (more aggressive cleanup)
5. Proactive checking: Added immediate fairness check during set() operations
to prevent imbalance formation rather than just reacting to it
This should eliminate the "still creating relationships" slowdown reported
by Soulcraft Studio while maintaining the OOM crash prevention from v3.40.1.
Fixed inverted eviction scoring formula in UnifiedCache that was causing
metadata (cheap to rebuild) to be retained while HNSW vectors (expensive,
frequently accessed) were evicted. This was causing OOM crashes during
large Excel imports with relationship extraction.
Changes:
- evictLowestValue(): Changed accessScore / rebuildCost to accessScore * rebuildCost
- evictForSize(): Changed accessScore / rebuildCost to accessScore * rebuildCost
- evictType(): Changed accessScore / rebuildCost to accessScore * rebuildCost
With the corrected formula, items with higher access counts AND higher
rebuild costs get higher scores and are protected from eviction.
Test coverage: Added comprehensive eviction scoring tests
Fixes: Type metadata hogging 99.7% of cache with only 3.7% access rate
