fix(8.0): multi-valued array fields index every element (contains no longer misses)

addToIndex builds a fieldsMap from the extracted {field,value} entries, but
only __words__ accumulated — every other repeated field did fieldsMap[field]=value,
so a multi-valued field (tags:['a','b','c'] → three 'tags' entries from
extractIndexableFields) collapsed to last-value-wins. columnStore.addEntity
expands an array to one indexed entry per element, but it only ever received the
final scalar, so `contains` matched only the last element and missed the rest.

Accumulate any repeated non-__words__ field into an array before addEntity
(promote scalar→array on the second occurrence). __words__ keeps its always-array
special-case so its multiValue manifest flag stays set even for single-word docs.

find-unified-integration.test.ts → 0 failures (was 4):
- 'array contains' (A.4): now queries first/middle/last element — all match
  (previously only the last-indexed element did).
- 'complete find workflow': $contains→contains (8.0 operator) AND fixed the test
  premise — find() hard-ANDs vector∩graph∩metadata, and connected:{from:X} returns
  X's NEIGHBOURS not X, so the graph anchor must be 'earth' (which the ML concept
  relates to), not the concept itself. Multi-signal scores are reciprocal-rank
  fusion (~1/61), not [0,1] cosine, so assert a positive fusion score, not >0.5.
- 'nonsense vector query': cosine search always returns nearest neighbours, so
  assert the structural array/bound contract, not length===0 (a Tier-2 concern).
- 'hard-ANDs signals': a nonexistent connected.from is an empty graph signal that
  zeros the intersection — assert length===0 (documented AND semantics), was >0.

1469 unit + full find-unified integration green.
This commit is contained in:
David Snelling 2026-06-19 11:37:59 -07:00
parent 009e50681e
commit eccf42000b
2 changed files with 71 additions and 25 deletions

View file

@ -1434,13 +1434,31 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// Write to column store — the single write path for all indexed data.
// Converts extracted fields into a map and feeds the column store.
//
// extractIndexableFields emits ONE {field,value} entry per array element for
// a multi-valued field (e.g. tags:['a','b','c'] → three 'tags' entries). We
// must accumulate every repeated field into an array, not just __words__:
// columnStore.addEntity expands an array value to one indexed entry per
// element, so a scalar overwrite (last-value-wins) would index only the final
// element and `contains` would miss the rest.
if (this.columnStore) {
const entityIntId = this.idMapper.getOrAssign(id)
const fieldsMap: Record<string, unknown> = {}
for (const { field, value } of fields) {
if (field === '__words__') {
// Always an array (keeps the field's multiValue manifest flag set even
// for a single-word document).
if (!fieldsMap.__words__) fieldsMap.__words__ = []
;(fieldsMap.__words__ as number[]).push(value)
;(fieldsMap.__words__ as unknown[]).push(value)
} else if (field in fieldsMap) {
// Repeated field → multi-valued. Promote the scalar to an array on the
// second occurrence, then accumulate.
const existing = fieldsMap[field]
if (Array.isArray(existing)) {
;(existing as unknown[]).push(value)
} else {
fieldsMap[field] = [existing, value]
}
} else {
fieldsMap[field] = value
}