fix: validate where-operators and align the in-memory matcher to the documented set

`find({ where })` had two gaps that could silently return nothing. The in-memory
matcher (`matchesQuery`, used for egress re-validation and historical reads) was
missing several documented operators — `in`, `greaterThanOrEqual`,
`lessThanOrEqual` — so it disagreed with the index path, which does handle them.
And an unknown operator key (a typo, or `notIn` written where `not: { in }` was
meant) fell through to a nested-object-field interpretation and matched nothing.

- Align `matchesQuery` to the full documented operator set (add the missing
  aliases; the default branch now throws instead of treating an unknown key as a
  nested field — dotted paths remain the supported nested form).
- Validate the `where` filter up front (`validateWhereFilter`), throwing a typed
  `BrainyError('INVALID_QUERY')` that names the unknown operator, recursing into
  `allOf` / `anyOf` / `not`.
- Stop excluding a user field literally named `level` from the metadata index
  (it collided with an internal never-index key), so numeric/exact filters on it
  resolve instead of returning 0.

Adds unit coverage for the operator set and the throw-on-unknown behavior.
This commit is contained in:
David Snelling 2026-07-07 12:23:36 -07:00
parent 68da66024c
commit 6821e1980b
4 changed files with 176 additions and 13 deletions

View file

@ -1166,8 +1166,17 @@ export class MetadataIndexManager implements MetadataIndexProvider {
private extractIndexableFields(data: any): Array<{ field: string, value: any }> {
const fields: Array<{ field: string, value: any }> = []
// Fields that should NEVER be indexed (vectors, embeddings, large arrays, HNSW internals)
const NEVER_INDEX = new Set(['vector', 'embedding', 'embeddings', 'connections', 'level', 'id'])
// Fields that should NEVER be indexed: bulk structural payloads that would
// blow up the index (the 384-dim vector, embeddings, the adjacency list).
// These are also caught by the array-size guard below, but naming them is
// belt-and-suspenders. NOTE: `level` was previously here (an HNSW node's
// layer) but it never actually reaches this path — every caller passes a
// metadata bag or Entity record, neither of which carries the node's
// `level` — so its only effect was to silently drop a legitimate USER
// metadata field named `level` (log level, skill level, access level…),
// making `where: { level: … }` return nothing. Removed. (`id` stays: it is
// the reserved entity-identity field, resolved specially by find().)
const NEVER_INDEX = new Set(['vector', 'embedding', 'embeddings', 'connections', 'id'])
const extract = (obj: any, prefix = ''): void => {
for (const [key, value] of Object.entries(obj)) {
@ -1278,7 +1287,10 @@ export class MetadataIndexManager implements MetadataIndexProvider {
return data.map(d => this.extractTextContent(d)).filter(Boolean).join(' ')
}
if (typeof data === 'object') {
const skipKeys = new Set(['vector', 'embedding', 'embeddings', 'connections', 'level', 'id'])
// Mirror of NEVER_INDEX for the text-extraction path: bulk structural
// payloads only. `level` removed for the same reason (it silently dropped
// a real user field from hybrid text search too).
const skipKeys = new Set(['vector', 'embedding', 'embeddings', 'connections', 'id'])
const texts: string[] = []
for (const [key, value] of Object.entries(data)) {
// Skip internal fields and numeric keys (array indices)