David Snelling
c605b34f98
The open-core, Cortex-free half of the library A/B (handoff AJ/AK), authored once in brainy so the proprietary A/B comparison can import it for both legs: - tests/benchmarks/lib/corpus.js — deterministic clustered-mixture corpus generator (recompute-on-demand, O(clusters·dim) memory) for latency/ingest/memory at scale. - tests/benchmarks/lib/metrics.js — percentiles, brute-force recall@k, RSS snapshot. - tests/benchmarks/brainy-scale.js — brainy-alone scaling leg (ingest, find p50/p99 for vector/metadata/graph/triple, RSS). Recall is intentionally NOT measured on synthetic data — see below. - tests/unit/boundary-no-cortex.test.ts — CI guard: fails if @soulcraft/cortex ever appears in a src/ or tests/ import or in any package.json dependency field. - tests/integration/vector-recall.test.ts — semantic-search correctness on REAL embeddings (19-20/20 exact-text top-1). Methodology note: synthetic vectors (random/one-hot/clustered/latent) are near-orthogonal under cosine, so HNSW (any graph ANN, incl. DiskANN) cannot navigate them and recall collapses regardless of engine — a property of the data, not the index. Brainy vector search is verified correct on real embeddings. The A/B recall@10 column is therefore measured on SIFT/BIGANN, identically for both legs.
52 lines
2.7 KiB
TypeScript
52 lines
2.7 KiB
TypeScript
/**
|
|
* @module tests/integration/vector-recall
|
|
* @description Correctness guard for open-core semantic search on REAL
|
|
* embeddings (the production path): distinct documents embedded by Brainy's
|
|
* built-in model, queried by their own text, must return themselves as the top
|
|
* hit. This confirms the JS HNSW + cosine path retrieves correct neighbours on
|
|
* real embedding geometry.
|
|
*
|
|
* Why real embeddings and not a synthetic corpus: HNSW navigation depends on the
|
|
* smooth, locally-structured manifold that real embeddings occupy. Synthetic
|
|
* vectors (uniform-random, one-hot, or tight clusters) are near-orthogonal under
|
|
* cosine — concentration of measure leaves no gradient for greedy descent, so
|
|
* recall collapses regardless of the engine. That is a property of the data, not
|
|
* the index (verified: an exact query for the graph entry point returns it at
|
|
* cosine 1.0). The A/B's headline recall@10 column is therefore measured on
|
|
* SIFT/BIGANN (real data, canonical ground truth), identically for both legs.
|
|
*/
|
|
import { describe, it, expect, afterEach } from 'vitest'
|
|
import { Brainy } from '../../src/brainy.js'
|
|
import { NounType } from '../../src/types/graphTypes.js'
|
|
|
|
const TOPICS = [
|
|
'quantum physics particles', 'french cuisine recipes', 'basketball playoff games',
|
|
'jazz music history', 'volcanic rock formation', 'machine learning models',
|
|
'medieval castle architecture', 'tropical fish species', 'solar panel efficiency',
|
|
'ancient roman empire', 'coffee bean roasting', 'mountain climbing gear',
|
|
'classical piano sonatas', 'desert wildlife survival', 'space telescope images',
|
|
'wine fermentation process', 'bicycle frame materials', 'honeybee colony behavior',
|
|
'glacier ice formation', 'origami paper folding'
|
|
]
|
|
|
|
describe('open-core semantic search correctness (real embeddings)', () => {
|
|
const brains: Brainy[] = []
|
|
afterEach(async () => { for (const b of brains.splice(0)) await b.close() })
|
|
|
|
it('an exact-text query returns its own document as the top hit', async () => {
|
|
const brain = new Brainy({ storage: { type: 'memory' }, requireSubtype: false, silent: true })
|
|
await brain.init()
|
|
brains.push(brain)
|
|
for (let i = 0; i < TOPICS.length; i++) {
|
|
await brain.add({ data: TOPICS[i], type: NounType.Document, metadata: { idx: i } })
|
|
}
|
|
let hits = 0
|
|
for (let i = 0; i < TOPICS.length; i++) {
|
|
const got = await brain.find({ query: TOPICS[i], limit: 1 })
|
|
if ((got[0]?.metadata as { idx: number })?.idx === i) hits++
|
|
}
|
|
// Near-perfect; a couple of genuine semantic near-neighbours (e.g. glacier
|
|
// vs volcanic "rock/ice formation") may swap — allow a small margin.
|
|
expect(hits).toBeGreaterThanOrEqual(TOPICS.length - 3)
|
|
}, 240000)
|
|
})
|