feat(8.0): vector allowedIds predicate-pushdown into find() (#46)
Filtered semantic search now keeps its recall. A find({ query, where }) that
pairs vector search with a metadata filter restricts the HNSW beam walk to the
matching candidates INSIDE the traversal (walk-all, collect-allowed) rather than
filtering the top-k afterward — so a query whose nearest vectors are all filtered
out still returns the best matches that DO pass the filter, instead of empty.
- JS HNSW search() honors the 8.0 `allowedIds: OpaqueIdSet | ReadonlySet<string>`
contract param (ANDs with candidateIds; ignores the opaque Buffer form it can't
decode — only a native provider consumes that).
- MetadataIndexProvider gains an OPTIONAL `getIdSetForFilter(filter): OpaqueIdSet`
producer — the native (cor) metadata index returns its roaring filter result as
a serialized buffer; the JS index does not implement it.
- find() forwards the matched universe to the vector walk: the opaque buffer
(zero id materialization) when the native producer is present, alongside the
materialized visibility-precise candidateIds for the JS index. Visibility stays
correct via the existing post-search hard filter.
Tested: JS recall/restriction/AND/opaque-ignore on the index directly, plus
find() forwarding the opaque universe through to the beam walk via a stubbed
native producer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
632d90aac5
commit
dd325f2f94
5 changed files with 279 additions and 15 deletions
|
|
@ -130,6 +130,22 @@ export interface MetadataIndexProvider {
|
|||
|
||||
getIds(field: string, value: any): Promise<string[]>
|
||||
getIdsForFilter(filter: any): Promise<string[]>
|
||||
/**
|
||||
* @description OPTIONAL: resolve a `where` filter to its matching id universe as
|
||||
* an {@link OpaqueIdSet} (a serialized roaring `Buffer`) WITHOUT materializing
|
||||
* the ids in TypeScript — the producer half of the predicate-pushdown
|
||||
* (`CTX-PUSHDOWN-ALLOWEDIDS`) and graph query→expand fusion. Brainy forwards the
|
||||
* returned set opaquely to `VectorIndexProvider.search({ allowedIds })` and
|
||||
* `GraphAccelerationProvider.traverse(seeds)`; it NEVER inspects it. A native
|
||||
* (cor) metadata index returns its roaring filter result directly (zero
|
||||
* crossing); the JS index does not implement this — Brainy falls back to
|
||||
* {@link MetadataIndexProvider.getIdsForFilter} + a `ReadonlySet<string>` when
|
||||
* absent. Present ⟺ the native stack is the active provider, so the matching
|
||||
* native vector/graph engines can decode the same envelope.
|
||||
* @param filter - The same filter shape accepted by `getIdsForFilter`.
|
||||
* @returns The matching id universe as an opaque set.
|
||||
*/
|
||||
getIdSetForFilter?(filter: any): Promise<OpaqueIdSet>
|
||||
getIdsForTextQuery(query: string): Promise<Array<{ id: string; matchCount: number }>>
|
||||
getSortedIdsForFilter(filter: any, orderBy: string, order?: 'asc' | 'desc'): Promise<string[]>
|
||||
getFilterValues(field: string): Promise<string[]>
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue