soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat(8.0): vector allowedIds predicate-pushdown into find() (#46)

Browse source 
Filtered semantic search now keeps its recall. A find({ query, where }) that
pairs vector search with a metadata filter restricts the HNSW beam walk to the
matching candidates INSIDE the traversal (walk-all, collect-allowed) rather than
filtering the top-k afterward — so a query whose nearest vectors are all filtered
out still returns the best matches that DO pass the filter, instead of empty.

- JS HNSW search() honors the 8.0 `allowedIds: OpaqueIdSet | ReadonlySet<string>`
  contract param (ANDs with candidateIds; ignores the opaque Buffer form it can't
  decode — only a native provider consumes that).
- MetadataIndexProvider gains an OPTIONAL `getIdSetForFilter(filter): OpaqueIdSet`
  producer — the native (cor) metadata index returns its roaring filter result as
  a serialized buffer; the JS index does not implement it.
- find() forwards the matched universe to the vector walk: the opaque buffer
  (zero id materialization) when the native producer is present, alongside the
  materialized visibility-precise candidateIds for the JS index. Visibility stays
  correct via the existing post-search hard filter.

Tested: JS recall/restriction/AND/opaque-ignore on the index directly, plus
find() forwarding the opaque universe through to the beam walk via a stubbed
native producer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
David Snelling 2026-06-23 13:18:34 -07:00
parent 632d90aac5
commit dd325f2f94
5 changed files with 279 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

7
RELEASES.md
View file

@ -66,6 +66,13 @@ no separate migration doc. **`8.0.0-rc.2` is live** — install with
> engine when present, and fall back to pure-TS kernels (PageRank, connected components / Tarjan
> SCC, BFS / Dijkstra) otherwise. Both paths return identical shapes and respect the default
> visibility filter (opt in with `includeInternal` / `includeSystem`).
> - **Filtered vector search keeps its recall (`allowedIds` pushdown).** A `find({ query, where })`
> that combines semantic search with a metadata filter now restricts the vector walk to the
> matching candidates *inside* the search (walk-all, collect-allowed) instead of filtering the
> top-k afterward — so a query whose nearest vectors are all filtered out still returns the best
> matches that DO pass the filter, rather than coming back empty. No API change. With the native
> `@soulcraft/cor` 3.0 stack the matched universe is forwarded as an opaque roaring buffer with
> zero id materialization in TypeScript; the pure-JS path restricts the beam walk with a string set.
### Headline: Database as a Value