feat: graph link compression — delta-varint connections (2.4.0 #3)
Cortex registers a graph:compression provider exposing `encode`/`decode`
for HNSW connection lists (delta-varint, ~4× smaller edges than the
JSON-UUID-array shape brainy has persisted since the beginning). This
wires the consumer side without changing the saveHNSWData/getHNSWData
adapter signatures.
Architecture:
- NEW ConnectionsCodec (src/hnsw/connectionsCodec.ts) — translates between
UUID-keyed Map<level, Set<UUID>> and a single compact per-node buffer
via the provider + stable EntityIdMapper. Custom wire format:
[count: u8] [[level: u8] [len: u32 LE] [bytes...]]*. One blob per node
regardless of level count, so the storage I/O is identical to the
legacy path (one saveHNSWData call) plus a single saveBinaryBlob.
- HNSWIndex changes — a `connectionsCodec: ConnectionsCodec | null` field
with `setConnectionsCodec()` setter. THREE save sites (deferred flush,
immediate-mode entity persist, immediate-mode neighbor updates) now go
through a single `persistNodeConnections(nodeId, noun)` helper: when the
codec is wired AND the storage adapter exposes saveBinaryBlob, it
encodes the connections, stores them at `_hnsw_conn/<nodeId>`, and
records `connections: {}` in saveHNSWData as the marker. Otherwise the
legacy JSON-array path is taken. TWO load sites use a matching
`restoreNodeConnections` helper that tries loadBinaryBlob first and
falls back to the legacy hnswData.connections field on miss / decode
error. Format convergence is lazy: pre-2.4.0 nodes still load via the
legacy path, then write the compressed form on next dirty save.
- brainy.ts wireConnectionsCodec — runs alongside wireMmapVectorBackend
during init. Activates when (a) the graph:compression provider is
registered and (b) the metadata index exposes its idMapper. Unlike the
mmap-vector backend, this layer engages on EVERY brainy 7.25.0
adapter — the blob primitive itself is universal; only the codec
presence gates activation.
- Provider interface GraphCompressionProvider in plugin.ts — encode +
decode static signatures. Brainy depends on the interface; cortex's
registered { encode: encodeConnections, decode: decodeConnections }
satisfies it structurally.
Tests (1437 total, +4 vs prior tip):
- tests/unit/hnsw/connections-codec.test.ts — 4 unit tests with a JSON
mock provider: single-level round-trip, empty-Map round-trip (one-byte
buffer), multi-level fan-out round-trip, and silent-drop of UUIDs the
decoding idMapper no longer knows. The real cross-language byte
format is exercised when cortex 2.4.0 wires its delta-varint
encode/decode in.
This completes the brainy half of the 2.4.0 storage foundation: stable
ids (#23), mmap vectors (#24), graph link compression (#25), and
column-store interchange (#26). Coordinated release as brainy 7.26.0 +
cortex 2.4.0 follows.
This commit is contained in:
parent
71bc30b829
commit
617c156feb
5 changed files with 473 additions and 40 deletions
|
|
@ -37,8 +37,14 @@ import { setGlobalCache } from './utils/unifiedCache.js'
|
|||
import type { UnifiedCache } from './utils/unifiedCache.js'
|
||||
import { rankIndicesByScore, reorderByIndices } from './utils/resultRanking.js'
|
||||
import { PluginRegistry } from './plugin.js'
|
||||
import type { BrainyPlugin, BrainyPluginContext, VectorStoreMmapProvider } from './plugin.js'
|
||||
import type {
|
||||
BrainyPlugin,
|
||||
BrainyPluginContext,
|
||||
GraphCompressionProvider,
|
||||
VectorStoreMmapProvider
|
||||
} from './plugin.js'
|
||||
import { MmapVectorBackend } from './hnsw/mmapVectorBackend.js'
|
||||
import { ConnectionsCodec } from './hnsw/connectionsCodec.js'
|
||||
import { TransactionManager } from './transaction/TransactionManager.js'
|
||||
import {
|
||||
ValidationConfig,
|
||||
|
|
@ -569,6 +575,14 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// works, just without the zero-copy fast path.
|
||||
await this.wireMmapVectorBackend()
|
||||
|
||||
// Wire the connections codec (2.4.0 #3). When the graph:compression
|
||||
// provider is registered AND the metadata index exposes a stable
|
||||
// idMapper, inject a codec that encodes HNSW connections as
|
||||
// delta-varint blobs at save time and decodes on load. The blob
|
||||
// primitive itself works on every brainy 7.25.0 adapter, so unlike the
|
||||
// mmap-vector backend this layer engages even on cloud adapters.
|
||||
this.wireConnectionsCodec()
|
||||
|
||||
// Rebuild indexes if needed for existing data
|
||||
await this.rebuildIndexesIfNeeded()
|
||||
|
||||
|
|
@ -7792,6 +7806,35 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Wire the HNSW connections codec (2.4.0 #3). Activates when
|
||||
* BOTH (a) the `graph:compression` provider is registered (cortex registers
|
||||
* `{ encode: encodeConnections, decode: decodeConnections }`), and (b) the
|
||||
* metadata index exposes a stable idMapper. Failures are non-fatal: HNSW
|
||||
* keeps working via the legacy JSON-array path.
|
||||
*
|
||||
* Unlike the mmap-vector backend, this layer does NOT require a real local
|
||||
* path — the binary-blob primitive saves the compressed bytes through the
|
||||
* storage adapter directly, so the codec is engaged on cloud adapters too.
|
||||
* Format convergence is lazy: pre-2.4.0 nodes load via the legacy path,
|
||||
* then the next dirty save writes the compressed form (and the legacy
|
||||
* field of saveHNSWData becomes empty), so all reads converge over time
|
||||
* without an explicit migration step.
|
||||
*/
|
||||
private wireConnectionsCodec(): void {
|
||||
const provider = this.pluginRegistry.getProvider<GraphCompressionProvider>('graph:compression')
|
||||
if (!provider) return
|
||||
|
||||
const idMapper = this.metadataIndex.getIdMapper?.()
|
||||
if (!idMapper) return
|
||||
|
||||
const codec = new ConnectionsCodec(provider, idMapper)
|
||||
this.index.setConnectionsCodec(codec)
|
||||
if (!this.config.silent) {
|
||||
console.log('[brainy] graph link compression wired (delta-varint connections via graph:compression provider)')
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Rebuild indexes from persisted data if needed (LAZY LOADING)
|
||||
*
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue