feat: graph link compression — delta-varint connections (2.4.0 #3)
Cortex registers a graph:compression provider exposing `encode`/`decode`
for HNSW connection lists (delta-varint, ~4× smaller edges than the
JSON-UUID-array shape brainy has persisted since the beginning). This
wires the consumer side without changing the saveHNSWData/getHNSWData
adapter signatures.
Architecture:
- NEW ConnectionsCodec (src/hnsw/connectionsCodec.ts) — translates between
UUID-keyed Map<level, Set<UUID>> and a single compact per-node buffer
via the provider + stable EntityIdMapper. Custom wire format:
[count: u8] [[level: u8] [len: u32 LE] [bytes...]]*. One blob per node
regardless of level count, so the storage I/O is identical to the
legacy path (one saveHNSWData call) plus a single saveBinaryBlob.
- HNSWIndex changes — a `connectionsCodec: ConnectionsCodec | null` field
with `setConnectionsCodec()` setter. THREE save sites (deferred flush,
immediate-mode entity persist, immediate-mode neighbor updates) now go
through a single `persistNodeConnections(nodeId, noun)` helper: when the
codec is wired AND the storage adapter exposes saveBinaryBlob, it
encodes the connections, stores them at `_hnsw_conn/<nodeId>`, and
records `connections: {}` in saveHNSWData as the marker. Otherwise the
legacy JSON-array path is taken. TWO load sites use a matching
`restoreNodeConnections` helper that tries loadBinaryBlob first and
falls back to the legacy hnswData.connections field on miss / decode
error. Format convergence is lazy: pre-2.4.0 nodes still load via the
legacy path, then write the compressed form on next dirty save.
- brainy.ts wireConnectionsCodec — runs alongside wireMmapVectorBackend
during init. Activates when (a) the graph:compression provider is
registered and (b) the metadata index exposes its idMapper. Unlike the
mmap-vector backend, this layer engages on EVERY brainy 7.25.0
adapter — the blob primitive itself is universal; only the codec
presence gates activation.
- Provider interface GraphCompressionProvider in plugin.ts — encode +
decode static signatures. Brainy depends on the interface; cortex's
registered { encode: encodeConnections, decode: decodeConnections }
satisfies it structurally.
Tests (1437 total, +4 vs prior tip):
- tests/unit/hnsw/connections-codec.test.ts — 4 unit tests with a JSON
mock provider: single-level round-trip, empty-Map round-trip (one-byte
buffer), multi-level fan-out round-trip, and silent-drop of UUIDs the
decoding idMapper no longer knows. The real cross-language byte
format is exercised when cortex 2.4.0 wires its delta-varint
encode/decode in.
This completes the brainy half of the 2.4.0 storage foundation: stable
ids (#23), mmap vectors (#24), graph link compression (#25), and
column-store interchange (#26). Coordinated release as brainy 7.26.0 +
cortex 2.4.0 follows.
This commit is contained in:
parent
71bc30b829
commit
617c156feb
5 changed files with 473 additions and 40 deletions
|
|
@ -253,6 +253,32 @@ export interface CacheProvider {
|
|||
// uses at the `getProvider('embeddings')` call site. No separate interface is
|
||||
// added here to avoid a duplicate, unwired contract.
|
||||
|
||||
/**
|
||||
* The `'graph:compression'` provider — pure-function encode/decode for HNSW
|
||||
* connection lists as compact delta-varint byte sequences (cortex's
|
||||
* `encodeConnections` / `decodeConnections`).
|
||||
*
|
||||
* Brainy's `HNSWIndex` consumes this via a `ConnectionsCodec` that translates
|
||||
* UUIDs to stable int slots via the `EntityIdMapper`, encodes, and persists
|
||||
* the compressed bytes through the binary-blob primitive. On load, the blob
|
||||
* is fetched + decoded back into UUID sets — `setConnectionsCodec()` on
|
||||
* `HNSWIndex` is the injection point. Read path is dual-format: when no blob
|
||||
* exists for a node, the connections fall back to the legacy JSON-array path
|
||||
* embedded in `saveHNSWData`, so pre-2.4.0 indexes keep loading unchanged
|
||||
* and convergence to the compressed form happens lazily on next save.
|
||||
*
|
||||
* Activated only when the storage adapter exposes the binary-blob primitive
|
||||
* AND the metadata index resolves a stable idMapper. Cloud adapters that
|
||||
* lack a real local-path resolution still benefit, since the blob primitive
|
||||
* itself works across every adapter as of brainy 7.25.0.
|
||||
*/
|
||||
export interface GraphCompressionProvider {
|
||||
/** Encode a list of u32 ints to compact delta-varint bytes. Sorts internally. */
|
||||
encode(ids: number[]): Buffer
|
||||
/** Decode delta-varint bytes back to a u32 list. */
|
||||
decode(data: Buffer): number[]
|
||||
}
|
||||
|
||||
/**
|
||||
* The `'vectorStore:mmap'` provider — a static factory class for an mmap-backed
|
||||
* vector file. Brainy's HNSWIndex uses the static `.create()` / `.open()`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue