- Wire PluginRegistry into Brainy init() with provider resolution for distance,
metadataIndex, graphIndex, embeddings, roaring, msgpack, and storage adapters
- Add setupStorage() factory that resolves storage:* providers from plugins before
falling back to built-in createStorage()
- Export internals API (setGlobalCache, UnifiedCache, EntityIdMapper, etc.) for
cortex plugin consumption
- Add plugin.test.ts verifying registration, activation, and provider resolution
- Deprecate browser support (OPFS, Web Workers, WASM embeddings) with warnings
in preparation for v8.0 server-only release
- FileSystemStorage: fix setupStorage resolution for mmap-filesystem provider
- Remove detectAndRepairCorruption from init() hot path — was loading all
metadata chunks sequentially on startup. Now available via checkHealth()
and repairIndex() methods.
- Short-circuit warmCache on empty workspace — skip 4-6 wasted storage reads.
- Parallelize MetadataIndex.init() and getGraphIndex() via Promise.all().
- Defer metadata writes during rebuild to batch boundaries (every 5000
entities) instead of flushing per-entity.
- Skip pre-reads for new entities in transactions — saves 2 storage
round-trips per add() on cloud storage.
- Switch vitest pool from threads to forks for process isolation
- Disable v8 coverage by default (causes vitest worker RPC timeout)
- Increase hookTimeout 30s to 60s for MetadataIndexManager init under load
- Reduce O(1) space test entity count, replace brittle timing assertions
- Add docs/guides/storage-adapters.md with verified batch config values
brain.add() was generating 26-40 immediate cloud writes per call, causing
HTTP 429 rate limit errors and high latency on GCS/S3/R2/Azure. Three-layer
fix: (1) deferred metadata writes with dirty-marking, (2) MetadataWriteBuffer
for write coalescing, (3) retry/backoff on all cloud storage adapters.
All metadata index files were stored under a single _system/ prefix, causing
per-prefix rate limiting on GCS/S3/R2/Azure during cold starts and bulk imports.
Distributes high-volume system keys across 256 sub-prefixes using FNV-1a hash.
Backward-compatible with legacy path fallback.
When rmdir({ recursive: true }) deleted a directory tree, child file paths
remained in contentCache and statCache, causing readFile() to return stale
data for deleted files. Adds recursive flag to invalidateCaches() that
evicts all descendant keys by prefix.
Uses embedBatch() to pre-compute all vectors in a single WASM forward
pass instead of N individual embed() calls. Items that already have
vectors are skipped.
Before: 100 entities = 100 separate WASM calls
After: 100 entities = 1 batched WASM call (micro-batched internally)
highlight() used Promise.race with a 10s timeout, but the losing
semantic phase promise continued running 25 WASM micro-batches,
saturating the event loop and degrading all subsequent operations
(find() going from ~200ms to ~10,000ms).
Add AbortController to highlight() so the semantic phase stops
immediately on timeout or error. Pass abort signal through
embedBatch() → EmbeddingManager → micro-batch loop.
Also add defensive hardening:
- CandleEmbeddingEngine: try/catch around WASM calls resets engine
state on failure so next call triggers re-initialization
- WASMEmbeddingEngine: initialize() now checks underlying Candle
engine state, not just its own flag, completing the recovery chain
embedBatch() with large inputs (e.g. 500 chunks from highlight()) runs
a single synchronous WASM forward pass that blocks the event loop for
200-500ms. Split large batches into micro-batches of 20 with setTimeout(0)
yields between each, keeping max blocking per batch to ~10-30ms.
Also change Cargo.toml opt-level from "z" (size) to 3 (speed) for
~15-20% faster WASM inference. Requires WASM rebuild to take effect.
The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:
1. validateConsistency() to falsely detect corruption on every startup,
triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
and report inflated totalEntries/totalIds stats
Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
Replace document-centric categories (prose/heading/code/label) with a
universal set that works across documents, code, and UI:
- title: headings, identifiers, labels, JSON keys
- annotation: comments, docstrings, captions, alt text
- content: paragraphs, list items, flowing text
- value: string literals, numbers, form values
- code: unparsed code blocks
- structural: keywords, operators, punctuation
Built-in extractors now produce title/content/code. All 6 categories
are available for custom parsers (e.g. tree-sitter).
Also adds inline code detection in Markdown: backtick spans within
prose lines are split into separate code/content segments.
Fix highlight() hanging on structured text input by addressing 3 root causes:
1. embedBatch() now uses native WASM batch API (single forward pass instead
of N individual embed() calls via Promise.all)
2. highlight() auto-detects content type (plain text, rich-text JSON, HTML,
Markdown) and extracts meaningful text segments. Supports TipTap, Slate.js,
Lexical, Draft.js, and Quill Delta formats. New contentType hint and
contentExtractor callback for custom parsers.
3. Semantic matching phase has 10s timeout - falls back to text-only matches
instead of hanging indefinitely.
Also fixes extractTextContent() array check: uses type-based detection
(typeof data[0] === 'number') instead of length-based (data.length > 10)
so arrays of objects are properly indexed for text search.
New types: ContentType, ContentCategory, ExtractedSegment
New fields: HighlightParams.contentType, HighlightParams.contentExtractor,
Highlight.contentCategory
- Add textMatches, textScore, semanticScore, matchSource to search results
- Add highlight() method for zero-config text + semantic highlighting
- Increase word indexing limit to 5000 (handles articles/chapters)
- Optimize findMatchingWords() with O(1) fast path for semantic-only results
- Add production safety limits (500 chunks for highlight)
- Add comprehensive tests for new features (35 tests)
- Update docs with match visibility and highlight() API
CRITICAL: Fixed metadata index corruption on update() operations where
removalMetadata only contained custom metadata + type, while entityForIndexing
contained ALL indexed fields. This caused 7 fields to accumulate on every
update, eventually making queries return 0 results.
- Fix removalMetadata to include all indexed fields (src/brainy.ts)
- Add validateIndexConsistency() and getIndexStats() public APIs
- Add auto-corruption detection and repair on startup
- Add getOrAssignSync() for EntityIdMapper persistence
- Add comprehensive regression tests
- PathResolver.getChildren() now deduplicates by entity ID (v7.4.1)
This handles duplicate relationship records that can occur when multiple
Brainy instances create relationships concurrently for the same storage path.
- brain.clear() now invalidates GraphAdjacencyIndex (v7.4.1)
Prevents stale in-memory index data after clearing storage, which could
cause relate()'s duplicate check to fail.
Fixes: Workshop bug where readdir('/') returned same directory 13+ times
- Add native config option: `new Brainy({ integrations: true })`
- OData integration for Excel Power Query and Power BI
- Google Sheets integration with Apps Script
- Server-Sent Events (SSE) for real-time streaming
- Webhooks for push notifications
- Zero-config with sensible defaults
- Full tree-shaking when disabled
Bug: After brain.clear(), VFS operations failed with
"Source entity 00000000-0000-0000-0000-000000000000 not found"
Root causes fixed:
- VFS instance remained in memory pointing to deleted root entity
- FileSystemStorage.clear() set blobStorage=undefined but didn't reinit
- Write-through cache returned stale entity data after clear()
Changes:
- Re-initialize COW (BlobStorage) after storage.clear() in brainy.ts
- Reset and reinitialize VFS following checkout() pattern
- Add clearWriteCache() to BaseStorage, call in Memory/FileSystem adapters
- Add 7 integration tests for VFS clear functionality
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
