open-brainy

Archived

This repository has been archived on 2026-09-03. You can view files and clone it, but you cannot make any changes to it's state, such as pushing and creating new issues, pull requests or comments.

History

David Snelling c605b34f98 test(8.0): A/B benchmark harness (open leg) — generic corpus+metrics lib, brainy-alone scaling bench, boundary guard, real-embedding recall guard The open-core, Cortex-free half of the library A/B (handoff AJ/AK), authored once in brainy so the proprietary A/B comparison can import it for both legs: - tests/benchmarks/lib/corpus.js — deterministic clustered-mixture corpus generator (recompute-on-demand, O(clusters·dim) memory) for latency/ingest/memory at scale. - tests/benchmarks/lib/metrics.js — percentiles, brute-force recall@k, RSS snapshot. - tests/benchmarks/brainy-scale.js — brainy-alone scaling leg (ingest, find p50/p99 for vector/metadata/graph/triple, RSS). Recall is intentionally NOT measured on synthetic data — see below. - tests/unit/boundary-no-cortex.test.ts — CI guard: fails if @soulcraft/cortex ever appears in a src/ or tests/ import or in any package.json dependency field. - tests/integration/vector-recall.test.ts — semantic-search correctness on REAL embeddings (19-20/20 exact-text top-1). Methodology note: synthetic vectors (random/one-hot/clustered/latent) are near-orthogonal under cosine, so HNSW (any graph ANN, incl. DiskANN) cannot navigate them and recall collapses regardless of engine — a property of the data, not the index. Brainy vector search is verified correct on real embeddings. The A/B recall@10 column is therefore measured on SIFT/BIGANN, identically for both legs.	2026-06-15 15:51:17 -07:00
..
corpus.js	test(8.0): A/B benchmark harness (open leg) — generic corpus+metrics lib, brainy-alone scaling bench, boundary guard, real-embedding recall guard	2026-06-15 15:51:17 -07:00
metrics.js	test(8.0): A/B benchmark harness (open leg) — generic corpus+metrics lib, brainy-alone scaling bench, boundary guard, real-embedding recall guard	2026-06-15 15:51:17 -07:00

David Snelling c605b34f98 test(8.0): A/B benchmark harness (open leg) — generic corpus+metrics lib, brainy-alone scaling bench, boundary guard, real-embedding recall guard

The open-core, Cortex-free half of the library A/B (handoff AJ/AK), authored once in
brainy so the proprietary A/B comparison can import it for both legs:

- tests/benchmarks/lib/corpus.js — deterministic clustered-mixture corpus generator
  (recompute-on-demand, O(clusters·dim) memory) for latency/ingest/memory at scale.
- tests/benchmarks/lib/metrics.js — percentiles, brute-force recall@k, RSS snapshot.
- tests/benchmarks/brainy-scale.js — brainy-alone scaling leg (ingest, find p50/p99 for
  vector/metadata/graph/triple, RSS). Recall is intentionally NOT measured on synthetic
  data — see below.
- tests/unit/boundary-no-cortex.test.ts — CI guard: fails if @soulcraft/cortex ever
  appears in a src/ or tests/ import or in any package.json dependency field.
- tests/integration/vector-recall.test.ts — semantic-search correctness on REAL
  embeddings (19-20/20 exact-text top-1).

Methodology note: synthetic vectors (random/one-hot/clustered/latent) are near-orthogonal
under cosine, so HNSW (any graph ANN, incl. DiskANN) cannot navigate them and recall
collapses regardless of engine — a property of the data, not the index. Brainy vector
search is verified correct on real embeddings. The A/B recall@10 column is therefore
measured on SIFT/BIGANN, identically for both legs.

2026-06-15 15:51:17 -07:00

corpus.js

test(8.0): A/B benchmark harness (open leg) — generic corpus+metrics lib, brainy-alone scaling bench, boundary guard, real-embedding recall guard

2026-06-15 15:51:17 -07:00

metrics.js

test(8.0): A/B benchmark harness (open leg) — generic corpus+metrics lib, brainy-alone scaling bench, boundary guard, real-embedding recall guard

2026-06-15 15:51:17 -07:00