feat: stable EntityIdMapper — rebuild() no longer renumbers (2.4.0 #1, the foundation) #1
Merged
dpsifr
merged 4 commits from 2026-05-28 19:37:48 +02:00
feat/2.4.0-storage into main
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 42cd241305 |
feat: graph link compression — delta-varint connections (2.4.0 #3)
Cortex registers a graph:compression provider exposing `encode`/`decode`
for HNSW connection lists (delta-varint, ~4× smaller edges than the
JSON-UUID-array shape brainy has persisted since the beginning). This
wires the consumer side without changing the saveHNSWData/getHNSWData
adapter signatures.
Architecture:
- NEW ConnectionsCodec (src/hnsw/connectionsCodec.ts) — translates between
UUID-keyed Map<level, Set<UUID>> and a single compact per-node buffer
via the provider + stable EntityIdMapper. Custom wire format:
[count: u8] [[level: u8] [len: u32 LE] [bytes...]]*. One blob per node
regardless of level count, so the storage I/O is identical to the
legacy path (one saveHNSWData call) plus a single saveBinaryBlob.
- HNSWIndex changes — a `connectionsCodec: ConnectionsCodec | null` field
with `setConnectionsCodec()` setter. THREE save sites (deferred flush,
immediate-mode entity persist, immediate-mode neighbor updates) now go
through a single `persistNodeConnections(nodeId, noun)` helper: when the
codec is wired AND the storage adapter exposes saveBinaryBlob, it
encodes the connections, stores them at `_hnsw_conn/<nodeId>`, and
records `connections: {}` in saveHNSWData as the marker. Otherwise the
legacy JSON-array path is taken. TWO load sites use a matching
`restoreNodeConnections` helper that tries loadBinaryBlob first and
falls back to the legacy hnswData.connections field on miss / decode
error. Format convergence is lazy: pre-2.4.0 nodes still load via the
legacy path, then write the compressed form on next dirty save.
- brainy.ts wireConnectionsCodec — runs alongside wireMmapVectorBackend
during init. Activates when (a) the graph:compression provider is
registered and (b) the metadata index exposes its idMapper. Unlike the
mmap-vector backend, this layer engages on EVERY brainy 7.25.0
adapter — the blob primitive itself is universal; only the codec
presence gates activation.
- Provider interface GraphCompressionProvider in plugin.ts — encode +
decode static signatures. Brainy depends on the interface; cortex's
registered { encode: encodeConnections, decode: decodeConnections }
satisfies it structurally.
Tests (1437 total, +4 vs prior tip):
- tests/unit/hnsw/connections-codec.test.ts — 4 unit tests with a JSON
mock provider: single-level round-trip, empty-Map round-trip (one-byte
buffer), multi-level fan-out round-trip, and silent-drop of UUIDs the
decoding idMapper no longer knows. The real cross-language byte
format is exercised when cortex 2.4.0 wires its delta-varint
encode/decode in.
This completes the brainy half of the 2.4.0 storage foundation: stable
ids (#23), mmap vectors (#24), graph link compression (#25), and
column-store interchange (#26). Coordinated release as brainy 7.26.0 +
cortex 2.4.0 follows.
|
|||
| ec157afc15 |
feat: column-store JS↔native interchange — raw-blob unify (2.4.0 #4)
Brainy's JS ColumnStore.flushBuffer now writes segment payloads + the
per-field DELETED bitmap via the binary-blob primitive (saveBinaryBlob) at
the cortex-shared key convention `_column_index/<field>/L<level>-NNNNNN`
(suffix-free; adapter appends its own). Byte-for-byte identical to what
cortex's NativeColumnStore writes — JS and native engines now read each
other's segments with no envelope re-encoding.
Closes the gap that the cortex 2.3.1 read-side fallback opened: cortex was
reading both formats but the JS engine was still WRITING the legacy
{_binary, base64} envelope, so a fresh JS write always required the cortex
fallback to kick in (and migrate lazily on first read). With this commit
JS writes natively in the unified format, and cortex's fallback only
fires on indexes persisted by older brainy releases.
Backward compat:
- ColumnStore.loadSegmentCursor tries the raw blob first, then falls back
to the legacy `{_binary, base64}` envelope at the `.cidx` object-path.
Indexes written by pre-2.4.0 brainy keep loading correctly.
- ColumnStore.init's DELETED-bitmap load has the same dual-format read.
- Adapters without the binary-blob primitive (custom adapters that didn't
follow the 7.25.0 surface) fall through to the legacy envelope writer
too, so writes still succeed there.
Tests (1433 total, +5 vs prior tip):
- tests/unit/indexes/columnStore/column-store-interchange.test.ts —
pins down the contract: (1) flush writes the raw blob, NO legacy
envelope; (2) DELETED bitmap likewise; (3) a legacy-format on-disk
segment loads correctly via the fallback; (4) legacy DELETED bitmap
ditto; (5) round-trip in the new format.
- All 101 existing ColumnStore tests pass — the new write path is
exercised by the existing lifecycle tests (MemoryStorage has the blob
primitive, so the new branch fires).
|
|||
| c18ccbb497 |
feat: mmap-vector backend wiring — HNSWIndex consumes vectorStore:mmap (2.4.0 #2)
Cortex already registers the vectorStore:mmap provider (its Rust NativeMmapVectorStore), but brainy has never consumed it — preloadVectors and getVectorSafe still go straight to storage.getNounVector for every id, even when an mmap layer is available. This wires the consumer end. Architecture: - NEW MmapVectorBackend (src/hnsw/mmapVectorBackend.ts) — bridges brainy's UUID-keyed vector reads to a int-slot mmap file via the vectorStore:mmap provider. Slots are addressed by the stable int id from the post-2.4.0 #1 EntityIdMapper (the foundation this depends on). Auto-grows the file (doubling) when a write lands beyond capacity, so HNSWIndex never has to think about sizing. The class never touches per-entity storage — it owns only the mmap layer. - HNSWIndex changes — adds a vectorBackend field + a setVectorBackend setter. The vector read paths (preloadVectors, getVectorSafe) try the mmap layer first; on a storage fallback hit, they LAZILY write back into the mmap slot. An upgraded install converges to the zero-copy fast path under live traffic — no big-bang migration step. The legacy per-entity path is preserved and still used when no backend is set. - brainy.ts wiring — a new private wireMmapVectorBackend() runs once during init, after plugin activation + metadataIndex setup. It activates the backend only when (a) the vectorStore:mmap provider is registered, (b) the storage adapter resolves a real local path via getBinaryBlobPath(), and (c) the metadata index exposes its idMapper. Cloud adapters return null on (b) and the backend is silently skipped; HNSWIndex's behaviour is then identical to pre-2.4.0. - Provider interfaces in plugin.ts — VectorStoreMmapProvider and VectorStoreMmapInstance document the contract cortex's class fulfils (the class IS the provider — static factory methods). Brainy depends on the interfaces, not on cortex; the structural match is verified when cortex 2.4.0 picks up this brainy release. Tests (1428 total, +11 vs pre-2.4.0): - tests/unit/hnsw/mmap-vector-backend.test.ts — 6 unit tests with an in-memory mock provider. Covers round-trip, batch reads with interleaved misses, slot stability (no re-slotting on overwrite), file growth without data loss, idempotent open, and null returns for unwritten slots. The real perf integration with cortex's NativeMmapVectorStore is exercised when cortex 2.4.0 wires this in. - tests/unit/utils/entity-id-mapper-stability.test.ts — moved here from tests/regression/ (which is NOT in the unit-config include glob, so the five #23 tests were not actually being run by npm test). The unit config matches tests/unit/**/*.test.ts. The 2.4.0 #2 follow-up will be the chunked-segment layout for remote storage adapters (S3 / R2 / GCS) where a single growing file doesn't fit immutable objects. For 2.4.0 release: local-FS only. |
|||
| 01cf4b1b7e |
feat: stable EntityIdMapper — rebuild() no longer renumbers UUID→int
Previously metadataIndex.rebuild() called idMapper.clear() which reset nextId to 1 and renumbered every UUID by re-insertion order. Any consumer that had persisted int-keyed data against the old map was silently invalidated — and 2.4.0's vector mmap store (#20), graph link compression (#21), and column-store JS↔native interchange all need persisted int indices that survive a rebuild. Remove the unconditional clear() in rebuild(). The rebuild already re-iterates every entity via idMapper.getOrAssign(uuid), which returns the existing int unchanged for known UUIDs. Stale UUID→int entries for entities no longer in storage persist as harmless memory overhead; a dedicated prune step can be added if it ever matters. clearAllIndexData() — the explicit nuclear recovery path — keeps its existing idMapper.clear() call (renumbering is intentional there), and now logs a prodLog.warn making it explicit that any persisted int-keyed data is invalidated and must be rebuilt from canonical sources. Strengthened the EntityIdMapper class JSDoc to document the stability guarantee as a contract — append-only getOrAssign, monotonic nextId, remove() leaves permanent holes, rebuild() never renumbers, only clear() does. Added tests/regression/entity-id-mapper-stability.test.ts pinning down the five-point contract: (1) single-rebuild stability; (2) many-rebuild stability; (3) post-rebuild adds get fresh monotonic ints; (4) removes leave permanent holes — new entities never recycle; (5) clearAllIndexData() explicitly renumbers (the documented destructive path). Foundation for 2.4.0 #2-#4. Full test suite (62 files, 1417 tests) green. |
