- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
relate() was setting verb.source = fromEntity.type (a NounType like
"concept") instead of the entity UUID. Cortex's graph index indexes by
verb.source, so lookups by UUID found nothing — causing in-session
reads to return 0 results.
Also fixes:
- PathResolver calling private getIdsFromChunks() → public getIds()
- Plugin auto-detection removed; cortex loads only via explicit config
- GraphVerb types accept number timestamps and sourceId/targetId aliases
- Dead autoDetect() method removed from PluginRegistry
- In-session regression tests added for getRelations after relate()
The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:
1. validateConsistency() to falsely detect corruption on every startup,
triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
and report inflated totalEntries/totalIds stats
Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
CRITICAL FIX: VFS bug that persisted through v4.5.1-v4.7.4 is NOW FIXED.
Root Cause:
- Storage adapters were not properly extracting standard fields from metadata
- This caused getVerbsBySource_internal() to return 0 relationships despite relationships existing
- VFS PathResolver couldn't navigate directory structure
Solution - Metadata Architecture Refactoring:
1. Move standard fields to top-level of HNSWNounWithMetadata and HNSWVerbWithMetadata
- type, createdAt, updatedAt, confidence, weight, service, data, createdBy
2. Update all 9 storage adapters to extract standard fields from metadata on load
3. Maintain backward compatibility at storage layer (metadata files unchanged)
Changes:
- src/coreTypes.ts: Update HNSWNounWithMetadata and HNSWVerbWithMetadata interfaces
- Add top-level standard fields
- Change data type from unknown to Record<string, any>
- Add confidence field to GraphVerb
- src/storage/baseStorage.ts: Add type cast pattern for standard field extraction
- src/storage/adapters/*.ts: Fix all 9 adapters (memoryStorage, fileSystemStorage, gcsStorage,
s3CompatibleStorage, r2Storage, opfsStorage, azureBlobStorage, typeAwareStorageAdapter)
- Extract standard fields from metadata on load
- Place at top-level of returned entities
- src/api/DataAPI.ts: Read fields from top-level instead of metadata
- src/graph/graphAdjacencyIndex.ts: Convert HNSWVerbWithMetadata to GraphVerb format
- src/utils/metadataIndex.ts: Fix typo (metadata → entityOrMetadata)
- src/types/brainy.types.ts: Add createdBy field to AddParams
- src/types/graphTypes.ts: Add service field to GraphVerb
Test Results:
✅ VFS bug FIXED - vfs.readdir('/') now returns files (was returning empty array)
✅ getVerbsBySource_internal() now returns relationships correctly
✅ Build succeeds with ZERO compilation errors
✅ 95.7% of tests pass (954/997)
Breaking Changes:
- None - backward compatibility maintained at storage layer
Version: 4.8.0
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Progressive intervals adjust dynamically based on current entity count
(not total), making them work for both known and unknown totals.
**Key Features:**
- 0-999 entities: Flush every 100 (frequent early updates for UX)
- 1K-9.9K: Flush every 1000 (balanced performance)
- 10K+: Flush every 5000 (minimal overhead ~0.3%)
**Benefits:**
- Works with known totals (file imports)
- Works with unknown totals (streaming APIs, database cursors)
- Adapts automatically as import grows
- Zero configuration required
**Implementation:**
- Replaced adaptive intervals (requires total count) with progressive
- Added interval transition logging for observability
- Enhanced documentation to highlight engineering sophistication
- Final flush with statistics reporting
**Documentation:**
- Added "Engineering Insight" section showcasing advanced approach
- Updated all interval references from "adaptive" to "progressive"
- Added comprehensive examples in streaming-imports.md
Generated with Claude Code (https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
