feat: graph link compression — delta-varint connections (2.4.0 #3)

Cortex registers a graph:compression provider exposing `encode`/`decode`
for HNSW connection lists (delta-varint, ~4× smaller edges than the
JSON-UUID-array shape brainy has persisted since the beginning). This
wires the consumer side without changing the saveHNSWData/getHNSWData
adapter signatures.

Architecture:

- NEW ConnectionsCodec (src/hnsw/connectionsCodec.ts) — translates between
  UUID-keyed Map<level, Set<UUID>> and a single compact per-node buffer
  via the provider + stable EntityIdMapper. Custom wire format:
  [count: u8] [[level: u8] [len: u32 LE] [bytes...]]*. One blob per node
  regardless of level count, so the storage I/O is identical to the
  legacy path (one saveHNSWData call) plus a single saveBinaryBlob.

- HNSWIndex changes — a `connectionsCodec: ConnectionsCodec | null` field
  with `setConnectionsCodec()` setter. THREE save sites (deferred flush,
  immediate-mode entity persist, immediate-mode neighbor updates) now go
  through a single `persistNodeConnections(nodeId, noun)` helper: when the
  codec is wired AND the storage adapter exposes saveBinaryBlob, it
  encodes the connections, stores them at `_hnsw_conn/<nodeId>`, and
  records `connections: {}` in saveHNSWData as the marker. Otherwise the
  legacy JSON-array path is taken. TWO load sites use a matching
  `restoreNodeConnections` helper that tries loadBinaryBlob first and
  falls back to the legacy hnswData.connections field on miss / decode
  error. Format convergence is lazy: pre-2.4.0 nodes still load via the
  legacy path, then write the compressed form on next dirty save.

- brainy.ts wireConnectionsCodec — runs alongside wireMmapVectorBackend
  during init. Activates when (a) the graph:compression provider is
  registered and (b) the metadata index exposes its idMapper. Unlike the
  mmap-vector backend, this layer engages on EVERY brainy 7.25.0
  adapter — the blob primitive itself is universal; only the codec
  presence gates activation.

- Provider interface GraphCompressionProvider in plugin.ts — encode +
  decode static signatures. Brainy depends on the interface; cortex's
  registered { encode: encodeConnections, decode: decodeConnections }
  satisfies it structurally.

Tests (1437 total, +4 vs prior tip):

- tests/unit/hnsw/connections-codec.test.ts — 4 unit tests with a JSON
  mock provider: single-level round-trip, empty-Map round-trip (one-byte
  buffer), multi-level fan-out round-trip, and silent-drop of UUIDs the
  decoding idMapper no longer knows. The real cross-language byte
  format is exercised when cortex 2.4.0 wires its delta-varint
  encode/decode in.

This completes the brainy half of the 2.4.0 storage foundation: stable
ids (#23), mmap vectors (#24), graph link compression (#25), and
column-store interchange (#26). Coordinated release as brainy 7.26.0 +
cortex 2.4.0 follows.
This commit is contained in:
David Snelling 2026-05-28 10:37:01 -07:00
parent 71bc30b829
commit 617c156feb
5 changed files with 473 additions and 40 deletions

View file

@ -37,8 +37,14 @@ import { setGlobalCache } from './utils/unifiedCache.js'
import type { UnifiedCache } from './utils/unifiedCache.js'
import { rankIndicesByScore, reorderByIndices } from './utils/resultRanking.js'
import { PluginRegistry } from './plugin.js'
import type { BrainyPlugin, BrainyPluginContext, VectorStoreMmapProvider } from './plugin.js'
import type {
BrainyPlugin,
BrainyPluginContext,
GraphCompressionProvider,
VectorStoreMmapProvider
} from './plugin.js'
import { MmapVectorBackend } from './hnsw/mmapVectorBackend.js'
import { ConnectionsCodec } from './hnsw/connectionsCodec.js'
import { TransactionManager } from './transaction/TransactionManager.js'
import {
ValidationConfig,
@ -569,6 +575,14 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// works, just without the zero-copy fast path.
await this.wireMmapVectorBackend()
// Wire the connections codec (2.4.0 #3). When the graph:compression
// provider is registered AND the metadata index exposes a stable
// idMapper, inject a codec that encodes HNSW connections as
// delta-varint blobs at save time and decodes on load. The blob
// primitive itself works on every brainy 7.25.0 adapter, so unlike the
// mmap-vector backend this layer engages even on cloud adapters.
this.wireConnectionsCodec()
// Rebuild indexes if needed for existing data
await this.rebuildIndexesIfNeeded()
@ -7792,6 +7806,35 @@ export class Brainy<T = any> implements BrainyInterface<T> {
}
}
/**
* @description Wire the HNSW connections codec (2.4.0 #3). Activates when
* BOTH (a) the `graph:compression` provider is registered (cortex registers
* `{ encode: encodeConnections, decode: decodeConnections }`), and (b) the
* metadata index exposes a stable idMapper. Failures are non-fatal: HNSW
* keeps working via the legacy JSON-array path.
*
* Unlike the mmap-vector backend, this layer does NOT require a real local
* path the binary-blob primitive saves the compressed bytes through the
* storage adapter directly, so the codec is engaged on cloud adapters too.
* Format convergence is lazy: pre-2.4.0 nodes load via the legacy path,
* then the next dirty save writes the compressed form (and the legacy
* field of saveHNSWData becomes empty), so all reads converge over time
* without an explicit migration step.
*/
private wireConnectionsCodec(): void {
const provider = this.pluginRegistry.getProvider<GraphCompressionProvider>('graph:compression')
if (!provider) return
const idMapper = this.metadataIndex.getIdMapper?.()
if (!idMapper) return
const codec = new ConnectionsCodec(provider, idMapper)
this.index.setConnectionsCodec(codec)
if (!this.config.silent) {
console.log('[brainy] graph link compression wired (delta-varint connections via graph:compression provider)')
}
}
/**
* Rebuild indexes from persisted data if needed (LAZY LOADING)
*