Native accelerators (cortex) register providers for metadataIndex, graphIndex,
hnsw, entityIdMapper, cache, columnStore, and aggregation. Until now the exact
method/property surface brainy calls on each was implicit — a provider could
drop a member brainy depends on and only fail at runtime when that path ran.
This defines and exports the provider contracts from the stable
@soulcraft/brainy/plugin entrypoint:
- MetadataIndexProvider, GraphIndexProvider, HnswProvider,
EntityIdMapperProvider, CacheProvider — each typed as exactly the surface
brainy calls (optional/feature-detected members like HNSW setPersistMode and
enableCOW are intentionally excluded).
- Re-exports ColumnStoreProvider and AggregationProvider (and the aggregate
types) from the same entrypoint so a plugin author can import the whole
provider surface from one place.
Brainy's own baseline classes now `implements` these contracts
(MetadataIndexManager, GraphAdjacencyIndex, HNSWIndex, EntityIdMapper,
UnifiedCache), so the interfaces can never silently diverge from what brainy
ships — and any provider that declares `implements` gets a compile error the
moment brainy starts requiring a new member.
The embeddings/embedBatch providers stay typed by the existing EmbeddingFunction
(no duplicate interface added). Type-only changes; no runtime behavior change.
Root cause: metadata index failed to reconstruct after deploy restart
because the field registry file (__metadata_field_registry__) was lost
during an interrupted flush. init() silently assumed the workspace was
empty even though 5000+ entities existed on disk.
Fix 1 (root cause): init() now probes storage for entities when field
registry is missing. If entities exist, triggers rebuild instead of
silently skipping. Never trusts a missing registry as "empty."
Fix 2 (safety net): after rebuildIndexesIfNeeded(), verifies metadata
index entry count matches storage entity count. Forces second rebuild
if mismatch detected.
Fix 3 (prevention): flush() now always saves field registry and
EntityIdMapper, even when no dirty fields exist. These tiny files are
the critical link that init() needs to discover persisted indices.
Fix 5 (safe rebuild): rebuild() no longer deletes the field registry
file before rewriting. If rebuild fails partway, the registry survives
for the next init() to discover and re-trigger rebuild.
Fix 6 (collision guard): EntityIdMapper init() warns when mapper file
is missing but entities exist on disk, preventing silent ID collisions
from nextId starting at 1.
Shutdown/close/flush now properly flushes all 4 components in parallel:
metadataIndex, graphIndex, HNSW dirty nodes, and storage counts. Previously
only counts were flushed, causing native provider data loss on restart.
Also:
- Wire roaring, msgpack, entityIdMapper provider consumption from plugins
- Fix allOf filter O(n²) intersection → O(n) Set-based
- Fix ne/exists negation filter to use Set-based exclusion
- Add setMsgpackImplementation() swap in SSTable for native msgpack
- Add setRoaringImplementation() swap for native CRoaring bitmaps
- Add getAllIntIds() to EntityIdMapper for bitmap operations
- Remove TypeAwareHNSWIndex from default index creation path
- Export memory detection utilities from internals
- Clean up 26 permanently-skipped dead tests
The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:
1. validateConsistency() to falsely detect corruption on every startup,
triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
and report inflated totalEntries/totalIds stats
Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
CRITICAL: Fixed metadata index corruption on update() operations where
removalMetadata only contained custom metadata + type, while entityForIndexing
contained ALL indexed fields. This caused 7 fields to accumulate on every
update, eventually making queries return 0 results.
- Fix removalMetadata to include all indexed fields (src/brainy.ts)
- Add validateIndexConsistency() and getIndexStats() public APIs
- Add auto-corruption detection and repair on startup
- Add getOrAssignSync() for EntityIdMapper persistence
- Add comprehensive regression tests
- Fix update() method saving data as '_data' instead of 'data'
- Fix update() passing wrong entity structure to metadata index
- Add guard against undefined IDs in analyzeKey() for clustering
- Fix EntityIdMapper to read from top-level metadata in v4.8.0
- Fix PatternSignal tests to use NounType.Measurement
- Update test expectations for v4.8.0 entity structure
Fixes augmentations-simplified.test.ts (all 25 tests passing)
Fixes neural-simplified clustering (32/33 tests passing)
Overall: 98.3% test pass rate (988/997 tests)
- Change EntityIdMapper to use StorageAdapter interface instead of BaseStorage
- Use RoaringBitmap32.orMany() for multiple bitmap OR operations
- Use manual reduction for AND operations (no andMany available)
- Fixes TypeScript compilation errors
- Add .js extension to entityIdMapper baseStorage import
- Change roaring imports from subpath to main package export
- Use import { RoaringBitmap32 } from 'roaring' for ESM compatibility
