fix(find): the hybrid legs rank inside the filter, and only the page is read
A hybrid find fuses a text leg and a semantic leg. The semantic leg already
walked only the metadata filter's universe. The text leg did not: it ranked
the WHOLE store, took the top `limit * 4`, read every one of those rows from
canonical, and only then intersected with the filter. On a large store with a
selective filter that is hundreds of rows read to return a handful — and a row
matching both the query and the filter, but sitting outside the store-wide
text prefix, was silently dropped. The same defect `find({ connected })`
carried before the graph-first law, one leg over.
Both legs now rank ids inside the universe and neither reads canonical. The
text leg goes through a new optional `getIdsForTextQueryWithin` door on
MetadataIndexProvider — the text twin of `filterIdsWithin`, so a native index
can intersect its postings before any string crosses the boundary; the
reference index implements it from its own posting-list merge, so the two
doors can never disagree, and a provider without it is served by the
whole-store answer intersected here. The fusion ranks shells, the page is cut
from them, and canonical is read once for exactly that page — with the row
rebuilt in full, so a hydrated row is indistinguishable from an eagerly-built
one (same flattened fields, same entity, same match visibility, same key
order). The eager forms of both legs stay for the search modes whose leg
output IS the answer.
Measured on the production recall shape (query + type list + `missing`
negation + excludeVFS, limit 60) the old order read 241 rows in two batches to
return one; the new order reads the page.
Pinned in tests/integration/find-hybrid-filter-before-hydrate.test.ts. The
oracle there is the pre-change pipeline itself, replayed on the same brain
through the same doors: where the filter does not truncate the text leg the
answer is identical — rows, order, scores, match visibility and row shape —
across hybrid + where, + type list + excludeVFS + a `missing` negation, +
connected, with and without offset. Where it does truncate, the correction is
held by name: the old order's text leg contributed nothing at all, the new one
returns the matching rows and paging reaches every one of them. The cost pins
read the engine's own counters: one batchGet of `limit` ids, the whole-store
text door never called, and what the text leg marshals bounded by the universe.
This commit is contained in:
parent
67ae0046de
commit
b1c7054467
4 changed files with 903 additions and 118 deletions
353
src/brainy.ts
353
src/brainy.ts
|
|
@ -7651,6 +7651,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
const searchMode = params.searchMode || 'auto'
|
||||
const limit = params.limit || 10
|
||||
|
||||
// HYDRATE LAST (the hybrid path): its legs and its fusion rank IDS, and
|
||||
// canonical is read at the two page exits below — never for a row the
|
||||
// metadata filter is about to discard. This closure re-applies a hybrid
|
||||
// row's match visibility once its entity is in hand; it is set only by
|
||||
// the hybrid branch, so every other path hydrates unchanged.
|
||||
let finishHybridRow: ((row: Result<T>, pending: Result<T>) => void) | undefined
|
||||
|
||||
// Handle text-only query (user explicitly wants text search)
|
||||
if (searchMode === 'text' && params.query && params.query.trim() !== '') {
|
||||
results = await this.executeTextSearch(params.query, limit * 2)
|
||||
|
|
@ -7661,20 +7668,32 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
// Handle explicit hybrid or auto mode with query
|
||||
else if ((searchMode === 'auto' || searchMode === 'hybrid') && params.query && params.query.trim() !== '' && !params.vector) {
|
||||
// Zero-config hybrid: combine text + semantic search with RRF fusion
|
||||
const [textResults, semanticResults] = await Promise.all([
|
||||
this.executeTextSearch(params.query, limit * 2),
|
||||
this.executeVectorSearch(params, preResolvedMetadataIds ?? undefined, preResolvedAllowedIds)
|
||||
// Zero-config hybrid: combine text + semantic search with RRF fusion.
|
||||
// BOTH legs are held to the metadata filter's universe: the vector leg
|
||||
// walks it as its candidate set, and the text leg ranks inside it
|
||||
// instead of ranking the whole store and discarding what the filter
|
||||
// would drop. Neither leg reads canonical — the page does, once.
|
||||
const [textScored, semanticScored] = await Promise.all([
|
||||
this.executeTextSearchScored(params.query, limit * 2, preResolvedMetadataIds ?? undefined),
|
||||
this.executeVectorSearchScored(params, preResolvedMetadataIds ?? undefined, preResolvedAllowedIds)
|
||||
])
|
||||
|
||||
// Use user-specified alpha or auto-detect based on query length
|
||||
const alpha = params.hybridAlpha ?? this.autoAlpha(params.query)
|
||||
|
||||
// Tokenize query for match visibility
|
||||
// Tokenize query for match visibility. The word list needs the entity,
|
||||
// so it is computed on the page, at hydration.
|
||||
const queryWords = this.metadataIndex.tokenize(params.query)
|
||||
const textResultIds = new Set(textScored.map((r) => r.id))
|
||||
finishHybridRow = (row, pending) => {
|
||||
row.textMatches = this.findMatchingWords(row.entity, queryWords, textResultIds)
|
||||
row.textScore = pending.textScore
|
||||
row.semanticScore = pending.semanticScore
|
||||
row.matchSource = pending.matchSource
|
||||
}
|
||||
|
||||
// RRF fusion combines both result sets with match visibility
|
||||
results = await this.rrfFusion(textResults, semanticResults, alpha, queryWords)
|
||||
// RRF fusion combines both ranked id sets with match visibility
|
||||
results = this.rrfFusion(textScored, semanticScored, alpha)
|
||||
}
|
||||
// Handle direct vector search (no query text) - no hybrid needed
|
||||
else if (params.vector && !params.query) {
|
||||
|
|
@ -7736,19 +7755,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
const order = rankIndicesByScore(results.map(r => r.score), k, true)
|
||||
results = reorderByIndices(results, order).slice(offset, k)
|
||||
|
||||
// Batch-load entities only for the paginated results (10x faster on GCS)
|
||||
const idsToLoad = results.filter(r => !r.entity).map(r => r.id)
|
||||
if (idsToLoad.length > 0) {
|
||||
const entitiesMap = await this.batchGet(idsToLoad)
|
||||
for (const result of results) {
|
||||
if (!result.entity) {
|
||||
const entity = entitiesMap.get(result.id)
|
||||
if (entity) {
|
||||
result.entity = entity
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// Batch-load entities only for the paginated results (10x faster on GCS).
|
||||
// This is the hydrate-last seam for the deferring paths: a row that
|
||||
// arrives as a ranked shell is rebuilt in full here — flattened
|
||||
// fields, entity and match visibility — never `entity` alone.
|
||||
results = await this.hydrateResultPage(results, finishHybridRow)
|
||||
|
||||
// Early return if no other processing needed
|
||||
if (!params.connected && !params.fusion) {
|
||||
|
|
@ -7854,8 +7865,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
|
||||
const finalOffset = params.offset || 0
|
||||
|
||||
// Efficient pagination - only slice what we need (limit already defined above)
|
||||
return results.slice(finalOffset, finalOffset + limit)
|
||||
// Efficient pagination - only slice what we need (limit already defined
|
||||
// above), THEN read canonical for the page. Rows that arrived hydrated
|
||||
// pass straight through; a deferred path reads exactly these rows.
|
||||
return await this.hydrateResultPage(
|
||||
results.slice(finalOffset, finalOffset + limit),
|
||||
finishHybridRow
|
||||
)
|
||||
})()
|
||||
|
||||
// Index-integrity guard — applied ONCE here so every find() path (metadata,
|
||||
|
|
@ -12949,6 +12965,31 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The text-leg twin of {@link filterIdsWithinBelted}: rank `query` INSIDE the
|
||||
* candidate universe, through the provider's own posting-list merge so the
|
||||
* answer can never drift from `getIdsForTextQuery`'s. A provider without the
|
||||
* door is served by its whole-store answer intersected here — the same rows
|
||||
* in the same order, but it pays the whole-store marshal.
|
||||
*
|
||||
* @param query - The text query.
|
||||
* @param ids - The candidate universe (the metadata filter's ids).
|
||||
* @returns `{ id, matchCount }` rows inside `ids`, ranked by match count.
|
||||
*/
|
||||
private async textIdsWithinBelted(
|
||||
query: string,
|
||||
ids: readonly string[]
|
||||
): Promise<Array<{ id: string; matchCount: number }>> {
|
||||
this.ensureIndexesLoaded(['metadata'])
|
||||
const mip = this.metadataIndex as unknown as MetadataIndexProvider
|
||||
if (typeof mip.getIdsForTextQueryWithin === 'function') {
|
||||
return await mip.getIdsForTextQueryWithin(query, ids)
|
||||
}
|
||||
const within = new Set(ids)
|
||||
const all = await this.metadataIndex.getIdsForTextQuery(query)
|
||||
return all.filter((m) => within.has(m.id))
|
||||
}
|
||||
|
||||
async getIndexStatus(): Promise<{
|
||||
initialized: boolean
|
||||
/** `true` once open()'s index-build-if-needed step has run. Named for API
|
||||
|
|
@ -15841,6 +15882,44 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
candidateIds?: string[],
|
||||
allowedIds?: OpaqueIdSet
|
||||
): Promise<Result<T>[]> {
|
||||
const scored = await this.executeVectorSearchScored(params, candidateIds, allowedIds)
|
||||
|
||||
// Batch-load entities for 10-50x faster cloud storage performance
|
||||
// GCS: 10 results = 1×50ms vs 10×50ms = 500ms (10x faster)
|
||||
const entitiesMap = await this.batchGet(scored.map((s) => s.id))
|
||||
|
||||
const results: Result<T>[] = []
|
||||
for (const { id, score } of scored) {
|
||||
const entity = entitiesMap.get(id)
|
||||
if (entity) {
|
||||
results.push(this.createResult(id, score, entity))
|
||||
}
|
||||
}
|
||||
|
||||
return results
|
||||
}
|
||||
|
||||
/**
|
||||
* The semantic leg WITHOUT hydration — ranked ids and their scores.
|
||||
*
|
||||
* The beam walk is already restricted to the candidate universe (that is what
|
||||
* `candidateIds` / `allowedIds` are for), so the leg's cost is the walk. Its
|
||||
* ROWS, though, are candidates for a fusion that will keep one page of them —
|
||||
* so the hybrid path takes them unhydrated and reads exactly the page it
|
||||
* returns. {@link executeVectorSearch} is the eager form, for the search modes
|
||||
* whose leg output IS the answer.
|
||||
*
|
||||
* @param params - Find parameters (supplies the query/vector and the limit).
|
||||
* @param candidateIds - Optional pre-resolved metadata universe (see
|
||||
* {@link executeVectorSearch}).
|
||||
* @param allowedIds - Optional opaque predicate-pushdown universe.
|
||||
* @returns Ranked `{ id, score }` rows — no entity reads.
|
||||
*/
|
||||
private async executeVectorSearchScored(
|
||||
params: FindParams<T>,
|
||||
candidateIds?: string[],
|
||||
allowedIds?: OpaqueIdSet
|
||||
): Promise<Array<{ id: string; score: number }>> {
|
||||
// Vector cold-read guard: before trusting a semantic/vector result, verify the
|
||||
// vector index actually SERVES a known persisted vector (one-shot per brain).
|
||||
// A pure semantic find({ query }) has no filter, so verifyMetadataLive never
|
||||
|
|
@ -15866,21 +15945,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// HNSW search with optional metadata-first candidate filtering
|
||||
const searchResults: [string, number][] = await this.index.search(vector, limit * 2, undefined, searchOptions)
|
||||
|
||||
// Batch-load entities for 10-50x faster cloud storage performance
|
||||
// GCS: 10 results = 1×50ms vs 10×50ms = 500ms (10x faster)
|
||||
const ids = searchResults.map(([id]) => id)
|
||||
const entitiesMap = await this.batchGet(ids)
|
||||
|
||||
const results: Result<T>[] = []
|
||||
for (const [id, distance] of searchResults) {
|
||||
const entity = entitiesMap.get(id)
|
||||
if (entity) {
|
||||
const score = Math.max(0, Math.min(1, 1 / (1 + distance)))
|
||||
results.push(this.createResult(id, score, entity))
|
||||
}
|
||||
}
|
||||
|
||||
return results
|
||||
return searchResults.map(([id, distance]) => ({
|
||||
id,
|
||||
score: Math.max(0, Math.min(1, 1 / (1 + distance)))
|
||||
}))
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -16180,30 +16248,64 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* @returns Array of Results with scores based on match count
|
||||
*/
|
||||
private async executeTextSearch(query: string, limit: number): Promise<Result<T>[]> {
|
||||
const textMatches = await this.metadataIndex.getIdsForTextQuery(query)
|
||||
if (textMatches.length === 0) return []
|
||||
const scored = await this.executeTextSearchScored(query, limit)
|
||||
if (scored.length === 0) return []
|
||||
|
||||
// Take top matches and load entities
|
||||
const topMatches = textMatches.slice(0, limit * 2) // Get more for filtering
|
||||
const ids = topMatches.map(m => m.id)
|
||||
const entitiesMap = await this.batchGet(ids)
|
||||
// Batch-load entities for the whole leg — this is the eager form, kept for
|
||||
// the text-only search mode whose results ARE the answer.
|
||||
const entitiesMap = await this.batchGet(scored.map((s) => s.id))
|
||||
|
||||
// Create results with scores based on match count
|
||||
const maxMatches = topMatches[0]?.matchCount || 1
|
||||
const results: Result<T>[] = []
|
||||
|
||||
for (const match of topMatches) {
|
||||
const entity = entitiesMap.get(match.id)
|
||||
for (const { id, score } of scored) {
|
||||
const entity = entitiesMap.get(id)
|
||||
if (entity) {
|
||||
// Normalize score to 0-1 range based on match count
|
||||
const score = match.matchCount / maxMatches
|
||||
results.push(this.createResult(match.id, score, entity))
|
||||
results.push(this.createResult(id, score, entity))
|
||||
}
|
||||
}
|
||||
|
||||
return results
|
||||
}
|
||||
|
||||
/**
|
||||
* The text leg WITHOUT hydration — ranked ids and their scores.
|
||||
*
|
||||
* FILTER BEFORE HYDRATE: when the caller already knows the candidate
|
||||
* universe (the metadata filter's ids in a hybrid `find({ query, where })`),
|
||||
* it is passed here and the word index ranks INSIDE that universe. The
|
||||
* earlier order ranked the whole store, took the top `limit * 2`, hydrated
|
||||
* every one of them, and only then intersected with the filter — so a
|
||||
* filtered hybrid find on a large store hydrated hundreds of rows to return
|
||||
* a handful, and a matching row outside the store-wide text prefix was
|
||||
* silently dropped (the same defect `find({ connected })` had before the
|
||||
* graph-first law).
|
||||
*
|
||||
* The score is the match count normalized against the top row's, so a
|
||||
* restricted call normalizes against the top row IN THE UNIVERSE — the same
|
||||
* rule applied to the set actually being ranked.
|
||||
*
|
||||
* @param query - Text query to search for.
|
||||
* @param limit - Result budget; the leg keeps `limit * 2` for the fusion.
|
||||
* @param candidateIds - Optional candidate universe to rank inside.
|
||||
* @returns Ranked `{ id, score }` rows — no entity reads.
|
||||
*/
|
||||
private async executeTextSearchScored(
|
||||
query: string,
|
||||
limit: number,
|
||||
candidateIds?: readonly string[]
|
||||
): Promise<Array<{ id: string; score: number }>> {
|
||||
const textMatches = candidateIds
|
||||
? await this.textIdsWithinBelted(query, candidateIds)
|
||||
: await this.metadataIndex.getIdsForTextQuery(query)
|
||||
if (textMatches.length === 0) return []
|
||||
|
||||
// Take top matches (more than the page, for the fusion to rank)
|
||||
const topMatches = textMatches.slice(0, limit * 2)
|
||||
|
||||
// Normalize score to 0-1 range based on match count
|
||||
const maxMatches = topMatches[0]?.matchCount || 1
|
||||
return topMatches.map((m) => ({ id: m.id, score: m.matchCount / maxMatches }))
|
||||
}
|
||||
|
||||
/**
|
||||
* Auto-detect optimal alpha for hybrid search
|
||||
*
|
||||
|
|
@ -16230,55 +16332,56 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
*
|
||||
* Formula: score(d) = sum(1 / (k + rank(d))) for each list
|
||||
*
|
||||
* Now includes match visibility (textMatches, textScore, semanticScore, matchSource)
|
||||
* Now includes match visibility (textScore, semanticScore, matchSource; the
|
||||
* `textMatches` word list needs the entity and is filled at hydration).
|
||||
*
|
||||
* @param textResults - Results from text search
|
||||
* @param semanticResults - Results from semantic search
|
||||
* HYDRATE LAST: both legs arrive as ranked ids + scores, and the fusion ranks
|
||||
* ids — no entity is read here. The rows it returns are ranked SHELLS; the
|
||||
* page is cut from them and only that page is read from canonical (see
|
||||
* {@link hydrateResultPage}). The earlier order hydrated both legs in full —
|
||||
* hundreds of rows — to return one page of them.
|
||||
*
|
||||
* @param textResults - Ranked ids + scores from text search
|
||||
* @param semanticResults - Ranked ids + scores from semantic search
|
||||
* @param alpha - Weight for semantic (0=text only, 1=semantic only)
|
||||
* @param queryWords - Original query words for match tracking
|
||||
* @param k - RRF constant (default: 60, standard in literature)
|
||||
* @returns Fused results sorted by combined score with match visibility
|
||||
* @returns Fused result shells sorted by combined score with match visibility
|
||||
*/
|
||||
private async rrfFusion(
|
||||
textResults: Result<T>[],
|
||||
semanticResults: Result<T>[],
|
||||
private rrfFusion(
|
||||
textResults: ReadonlyArray<{ id: string; score: number }>,
|
||||
semanticResults: ReadonlyArray<{ id: string; score: number }>,
|
||||
alpha: number,
|
||||
queryWords: string[],
|
||||
k: number = 60
|
||||
): Promise<Result<T>[]> {
|
||||
): Result<T>[] {
|
||||
// Track scores and match details per entity
|
||||
interface MatchData {
|
||||
rrf: number
|
||||
textScore?: number
|
||||
semanticScore?: number
|
||||
textMatches: string[]
|
||||
hasText: boolean
|
||||
hasSemantic: boolean
|
||||
}
|
||||
const matchData = new Map<string, MatchData>()
|
||||
const entityMap = new Map<string, Entity<T>>()
|
||||
|
||||
// Text contribution (1 - alpha weight)
|
||||
const textWeight = 1 - alpha
|
||||
textResults.forEach((r, rank) => {
|
||||
const rrfScore = textWeight * (1 / (k + rank + 1))
|
||||
const existing = matchData.get(r.id) || { rrf: 0, textMatches: [], hasText: false, hasSemantic: false }
|
||||
const existing = matchData.get(r.id) || { rrf: 0, hasText: false, hasSemantic: false }
|
||||
existing.rrf += rrfScore
|
||||
existing.textScore = r.score // Original text search score (0-1)
|
||||
existing.hasText = true
|
||||
matchData.set(r.id, existing)
|
||||
if (r.entity) entityMap.set(r.id, r.entity)
|
||||
})
|
||||
|
||||
// Semantic contribution (alpha weight)
|
||||
semanticResults.forEach((r, rank) => {
|
||||
const rrfScore = alpha * (1 / (k + rank + 1))
|
||||
const existing = matchData.get(r.id) || { rrf: 0, textMatches: [], hasText: false, hasSemantic: false }
|
||||
const existing = matchData.get(r.id) || { rrf: 0, hasText: false, hasSemantic: false }
|
||||
existing.rrf += rrfScore
|
||||
existing.semanticScore = r.score // Original semantic search score (0-1)
|
||||
existing.hasSemantic = true
|
||||
matchData.set(r.id, existing)
|
||||
if (r.entity) entityMap.set(r.id, r.entity)
|
||||
})
|
||||
|
||||
// Sort by fused score
|
||||
|
|
@ -16286,51 +16389,93 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
.sort((a, b) => b[1].rrf - a[1].rrf)
|
||||
.map(([id, data]) => ({ id, data }))
|
||||
|
||||
// Build results - need to load any missing entities
|
||||
const missingIds = sortedIds.filter(s => !entityMap.has(s.id)).map(s => s.id)
|
||||
if (missingIds.length > 0) {
|
||||
const loaded = await this.batchGet(missingIds)
|
||||
for (const [id, entity] of loaded) {
|
||||
entityMap.set(id, entity)
|
||||
}
|
||||
}
|
||||
|
||||
// Performance: Build set of text result IDs for O(1) lookup
|
||||
// This avoids re-extracting text for entities that weren't in text results
|
||||
const textResultIds = new Set(textResults.map(r => r.id))
|
||||
|
||||
// Create final results with match visibility
|
||||
// Create ranked shells with match visibility
|
||||
const results: Result<T>[] = []
|
||||
for (const { id, data } of sortedIds) {
|
||||
const entity = entityMap.get(id)
|
||||
if (entity) {
|
||||
// Find which query words matched - uses fast path if entity wasn't in text results
|
||||
const textMatches = this.findMatchingWords(entity, queryWords, textResultIds)
|
||||
|
||||
// Determine match source
|
||||
let matchSource: 'text' | 'semantic' | 'both'
|
||||
if (data.hasText && data.hasSemantic) {
|
||||
matchSource = 'both'
|
||||
} else if (data.hasText) {
|
||||
matchSource = 'text'
|
||||
} else {
|
||||
matchSource = 'semantic'
|
||||
}
|
||||
|
||||
// Create result with match visibility
|
||||
const result = this.createResult(id, data.rrf, entity)
|
||||
result.textMatches = textMatches
|
||||
result.textScore = data.textScore
|
||||
result.semanticScore = data.semanticScore
|
||||
result.matchSource = matchSource
|
||||
|
||||
results.push(result)
|
||||
// Determine match source
|
||||
let matchSource: 'text' | 'semantic' | 'both'
|
||||
if (data.hasText && data.hasSemantic) {
|
||||
matchSource = 'both'
|
||||
} else if (data.hasText) {
|
||||
matchSource = 'text'
|
||||
} else {
|
||||
matchSource = 'semantic'
|
||||
}
|
||||
|
||||
const result = this.pendingResult(id, data.rrf)
|
||||
result.textScore = data.textScore
|
||||
result.semanticScore = data.semanticScore
|
||||
result.matchSource = matchSource
|
||||
|
||||
results.push(result)
|
||||
}
|
||||
|
||||
return results
|
||||
}
|
||||
|
||||
/**
|
||||
* A ranked candidate whose entity has NOT been read yet.
|
||||
*
|
||||
* The shell carries everything the ranking tail needs — the id, the score,
|
||||
* and the match-visibility fields — and nothing that requires canonical. It
|
||||
* is typed `Result` so it flows through the shared dedupe / visibility /
|
||||
* filter / rank / page tail unchanged; {@link hydrateResultPage} turns the
|
||||
* survivors into real results before any caller sees them, and find()'s
|
||||
* index-integrity guard drops any row that never gained an entity.
|
||||
*
|
||||
* @param id - The candidate's canonical id.
|
||||
* @param score - Its rank score.
|
||||
*/
|
||||
private pendingResult(id: string, score: number): Result<T> {
|
||||
return { id, score } as Result<T>
|
||||
}
|
||||
|
||||
/**
|
||||
* Read canonical for exactly the rows that need it — the hydrate-last seam.
|
||||
*
|
||||
* Rows that already carry an entity (the eager legs: metadata, text-only,
|
||||
* semantic-only, proximity, graph) pass through untouched, so this is a no-op
|
||||
* for every path that has not deferred. Rows that are shells are read in ONE
|
||||
* batch and rebuilt through {@link createResult}, so a hydrated row is
|
||||
* indistinguishable from an eagerly-built one — same flattened fields, same
|
||||
* `entity`, same key order — with `finish` re-applying the fields only the
|
||||
* deferring path knows about (a hybrid row's match visibility).
|
||||
*
|
||||
* A shell whose id has no canonical row is dropped, exactly as the eager legs
|
||||
* dropped it; find()'s index-integrity guard makes the same judgement on the
|
||||
* page it returns.
|
||||
*
|
||||
* @param rows - The page's rows, ranked and paged already.
|
||||
* @param finish - Applied to each rebuilt row, with its shell, after the
|
||||
* flattened fields are set.
|
||||
* @returns The page with every surviving row hydrated.
|
||||
*/
|
||||
private async hydrateResultPage(
|
||||
rows: Result<T>[],
|
||||
finish?: (row: Result<T>, pending: Result<T>) => void
|
||||
): Promise<Result<T>[]> {
|
||||
const pendingIds: string[] = []
|
||||
for (const row of rows) {
|
||||
if (!row.entity) pendingIds.push(row.id)
|
||||
}
|
||||
if (pendingIds.length === 0) return rows
|
||||
|
||||
const entitiesMap = await this.batchGet(pendingIds)
|
||||
const hydrated: Result<T>[] = []
|
||||
for (const row of rows) {
|
||||
if (row.entity) {
|
||||
hydrated.push(row)
|
||||
continue
|
||||
}
|
||||
const entity = entitiesMap.get(row.id)
|
||||
if (!entity) continue
|
||||
const filled = this.createResult(row.id, row.score, entity, row.explanation)
|
||||
finish?.(filled, row)
|
||||
hydrated.push(filled)
|
||||
}
|
||||
return hydrated
|
||||
}
|
||||
|
||||
/**
|
||||
* Find which query words match in an entity's text content
|
||||
*
|
||||
|
|
|
|||
|
|
@ -473,6 +473,28 @@ export interface MetadataIndexProvider {
|
|||
graphIndex: unknown
|
||||
): Promise<{ ids: string[]; emptyAt: 'graph' | 'filter' | 'visibility' | 'none' } | null>
|
||||
getIdsForTextQuery(query: string): Promise<Array<{ id: string; matchCount: number }>>
|
||||
/**
|
||||
* @description OPTIONAL: score `query` over `ids` ONLY — the text-leg twin of
|
||||
* {@link filterIdsWithin}, and the door a hybrid `find({ query, where })`
|
||||
* walks. The metadata filter's universe is the candidate set there, so the
|
||||
* text leg must cost O(|ids|) membership checks and marshal at most `|ids|`
|
||||
* rows, never the whole posting list of every query word. A native index
|
||||
* intersects its own postings with the candidate set (membership by entity
|
||||
* int) before any string crosses the boundary; the reference index answers
|
||||
* from its own `getIdsForTextQuery`, so the two doors can never disagree.
|
||||
* Absent → Brainy intersects `getIdsForTextQuery`'s answer with `ids` itself
|
||||
* (correct, and still hydrate-last, but it marshals the whole answer).
|
||||
*
|
||||
* The answer keeps `getIdsForTextQuery`'s contract: `{ id, matchCount }`
|
||||
* sorted by `matchCount` descending, ties in the order the whole-store answer
|
||||
* would have produced. Only rows in `ids` may appear.
|
||||
* @param query - The same text query accepted by `getIdsForTextQuery`.
|
||||
* @param ids - The candidate ids (canonical). The answer is a subset.
|
||||
*/
|
||||
getIdsForTextQueryWithin?(
|
||||
query: string,
|
||||
ids: readonly string[]
|
||||
): Promise<Array<{ id: string; matchCount: number }>>
|
||||
getSortedIdsForFilter(filter: any, orderBy: string, order?: 'asc' | 'desc', topK?: number): Promise<string[]>
|
||||
getFilterValues(field: string): Promise<string[]>
|
||||
getFilterFields(): Promise<string[]>
|
||||
|
|
|
|||
|
|
@ -1509,11 +1509,56 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
|||
* @returns Array of { id, matchCount } sorted by matchCount descending
|
||||
*/
|
||||
async getIdsForTextQuery(query: string): Promise<Array<{ id: string; matchCount: number }>> {
|
||||
return this.scoreTextQuery(query)
|
||||
}
|
||||
|
||||
/**
|
||||
* Score a text query over `ids` ONLY — the reference implementation of the
|
||||
* optional `getIdsForTextQueryWithin` door (see
|
||||
* {@link import('../plugin.js').MetadataIndexProvider}). The hybrid
|
||||
* `find({ query, where })` path passes the metadata filter's universe here so
|
||||
* the text leg ranks INSIDE that universe instead of ranking the whole store
|
||||
* and discarding the rows the filter would have dropped.
|
||||
*
|
||||
* It answers from the same posting-list merge as {@link getIdsForTextQuery},
|
||||
* with the candidate membership applied as each word's postings are counted,
|
||||
* so the two doors can never disagree: the answer is exactly the whole-store
|
||||
* answer restricted to `ids`, in the same order.
|
||||
*
|
||||
* @param query - Text query to search for.
|
||||
* @param ids - Candidate entity ids; only these may appear in the answer.
|
||||
* @returns Array of { id, matchCount } sorted by matchCount descending.
|
||||
*/
|
||||
async getIdsForTextQueryWithin(
|
||||
query: string,
|
||||
ids: readonly string[]
|
||||
): Promise<Array<{ id: string; matchCount: number }>> {
|
||||
if (ids.length === 0) return []
|
||||
return this.scoreTextQuery(query, new Set(ids))
|
||||
}
|
||||
|
||||
/**
|
||||
* The one posting-list merge behind both text doors.
|
||||
*
|
||||
* Each query word contributes AT MOST one match per entity (a posting list
|
||||
* can name an id more than once), and entities are ranked by how many of the
|
||||
* query's words they matched. `within`, when given, restricts the count to
|
||||
* those candidates — applied during the merge, so a restricted call never
|
||||
* materializes a whole-store match map.
|
||||
*
|
||||
* @param query - Text query to search for.
|
||||
* @param within - Optional candidate universe; absent = the whole store.
|
||||
* @returns Array of { id, matchCount } sorted by matchCount descending.
|
||||
*/
|
||||
private async scoreTextQuery(
|
||||
query: string,
|
||||
within?: ReadonlySet<string>
|
||||
): Promise<Array<{ id: string; matchCount: number }>> {
|
||||
const queryWords = this.tokenize(query)
|
||||
if (queryWords.length === 0) return []
|
||||
|
||||
// Get IDs for each word hash
|
||||
const wordIdSets: Map<string, number>[] = []
|
||||
// Count matches per entity, one word's postings at a time.
|
||||
const matchCounts = new Map<string, number>()
|
||||
for (const word of queryWords) {
|
||||
const wordHash = this.hashWord(word)
|
||||
let ids: string[]
|
||||
|
|
@ -1529,19 +1574,12 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
|||
throw err
|
||||
}
|
||||
}
|
||||
const idSet = new Map<string, number>()
|
||||
// One count per (word, entity) — dedupe this word's postings first.
|
||||
const counted = new Set<string>()
|
||||
for (const id of ids) {
|
||||
idSet.set(id, 1)
|
||||
}
|
||||
wordIdSets.push(idSet)
|
||||
}
|
||||
|
||||
if (wordIdSets.length === 0) return []
|
||||
|
||||
// Count matches per entity
|
||||
const matchCounts = new Map<string, number>()
|
||||
for (const idSet of wordIdSets) {
|
||||
for (const [id] of idSet) {
|
||||
if (counted.has(id)) continue
|
||||
counted.add(id)
|
||||
if (within && !within.has(id)) continue
|
||||
matchCounts.set(id, (matchCounts.get(id) || 0) + 1)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Reference in a new issue