feat: graph link compression — delta-varint connections (2.4.0 #3)

Cortex registers a graph:compression provider exposing `encode`/`decode`
for HNSW connection lists (delta-varint, ~4× smaller edges than the
JSON-UUID-array shape brainy has persisted since the beginning). This
wires the consumer side without changing the saveHNSWData/getHNSWData
adapter signatures.

Architecture:

- NEW ConnectionsCodec (src/hnsw/connectionsCodec.ts) — translates between
  UUID-keyed Map<level, Set<UUID>> and a single compact per-node buffer
  via the provider + stable EntityIdMapper. Custom wire format:
  [count: u8] [[level: u8] [len: u32 LE] [bytes...]]*. One blob per node
  regardless of level count, so the storage I/O is identical to the
  legacy path (one saveHNSWData call) plus a single saveBinaryBlob.

- HNSWIndex changes — a `connectionsCodec: ConnectionsCodec | null` field
  with `setConnectionsCodec()` setter. THREE save sites (deferred flush,
  immediate-mode entity persist, immediate-mode neighbor updates) now go
  through a single `persistNodeConnections(nodeId, noun)` helper: when the
  codec is wired AND the storage adapter exposes saveBinaryBlob, it
  encodes the connections, stores them at `_hnsw_conn/<nodeId>`, and
  records `connections: {}` in saveHNSWData as the marker. Otherwise the
  legacy JSON-array path is taken. TWO load sites use a matching
  `restoreNodeConnections` helper that tries loadBinaryBlob first and
  falls back to the legacy hnswData.connections field on miss / decode
  error. Format convergence is lazy: pre-2.4.0 nodes still load via the
  legacy path, then write the compressed form on next dirty save.

- brainy.ts wireConnectionsCodec — runs alongside wireMmapVectorBackend
  during init. Activates when (a) the graph:compression provider is
  registered and (b) the metadata index exposes its idMapper. Unlike the
  mmap-vector backend, this layer engages on EVERY brainy 7.25.0
  adapter — the blob primitive itself is universal; only the codec
  presence gates activation.

- Provider interface GraphCompressionProvider in plugin.ts — encode +
  decode static signatures. Brainy depends on the interface; cortex's
  registered { encode: encodeConnections, decode: decodeConnections }
  satisfies it structurally.

Tests (1437 total, +4 vs prior tip):

- tests/unit/hnsw/connections-codec.test.ts — 4 unit tests with a JSON
  mock provider: single-level round-trip, empty-Map round-trip (one-byte
  buffer), multi-level fan-out round-trip, and silent-drop of UUIDs the
  decoding idMapper no longer knows. The real cross-language byte
  format is exercised when cortex 2.4.0 wires its delta-varint
  encode/decode in.

This completes the brainy half of the 2.4.0 storage foundation: stable
ids (#23), mmap vectors (#24), graph link compression (#25), and
column-store interchange (#26). Coordinated release as brainy 7.26.0 +
cortex 2.4.0 follows.
This commit is contained in:
David Snelling 2026-05-28 10:37:01 -07:00
parent 71bc30b829
commit 617c156feb
5 changed files with 473 additions and 40 deletions

View file

@ -253,6 +253,32 @@ export interface CacheProvider {
// uses at the `getProvider('embeddings')` call site. No separate interface is
// added here to avoid a duplicate, unwired contract.
/**
* The `'graph:compression'` provider pure-function encode/decode for HNSW
* connection lists as compact delta-varint byte sequences (cortex's
* `encodeConnections` / `decodeConnections`).
*
* Brainy's `HNSWIndex` consumes this via a `ConnectionsCodec` that translates
* UUIDs to stable int slots via the `EntityIdMapper`, encodes, and persists
* the compressed bytes through the binary-blob primitive. On load, the blob
* is fetched + decoded back into UUID sets `setConnectionsCodec()` on
* `HNSWIndex` is the injection point. Read path is dual-format: when no blob
* exists for a node, the connections fall back to the legacy JSON-array path
* embedded in `saveHNSWData`, so pre-2.4.0 indexes keep loading unchanged
* and convergence to the compressed form happens lazily on next save.
*
* Activated only when the storage adapter exposes the binary-blob primitive
* AND the metadata index resolves a stable idMapper. Cloud adapters that
* lack a real local-path resolution still benefit, since the blob primitive
* itself works across every adapter as of brainy 7.25.0.
*/
export interface GraphCompressionProvider {
/** Encode a list of u32 ints to compact delta-varint bytes. Sorts internally. */
encode(ids: number[]): Buffer
/** Decode delta-varint bytes back to a u32 list. */
decode(data: Buffer): number[]
}
/**
* The `'vectorStore:mmap'` provider a static factory class for an mmap-backed
* vector file. Brainy's HNSWIndex uses the static `.create()` / `.open()`