fix(metadata): the indexable-array bound is a named law with a refusal, not a silent skip

An array-valued metadata field indexes one posting per element, so the index has
always carried a ceiling. It was 10, and it was applied by a bare `continue`
deep inside field extraction:

    if (Array.isArray(value) && value.length > 10) continue

A row whose `tags` array held ELEVEN entries therefore had that field skipped
entirely — no posting, no error, no warning. The row then failed to match every
filtered search on `tags`, including a query for a tag it demonstrably held, and
the caller had no way to tell that from "no row matches". Eleven tags is not an
exotic shape; the eleventh tag made the row invisible. Measured on the pin here:
the where-clause returns [] on the base for all eleven values.

The ceiling is not the defect. The silence was.

THE LAW. MAX_INDEXED_ARRAY_LENGTH = 64, hardcoded (the zero-config law: no
knob), sitting far above every legitimate multi-value field — tags, authors,
categories, labels, participants — and far below any real embedding width, so
the two populations do not overlap and nobody has to tune it. Arrays of scalars
index in full up to the bound. Above it the WRITE IS REFUSED by name:
MetadataArrayTooLargeError carries the field (its full dotted address), the
length and the bound, and names the three cures. It fires at all four write
doors — add, update, relate, updateRelation — beside the existing forged-system-
key rejection, and walks nested bags because a nested field indexes under its
dotted address exactly like a top-level one.

THE ONE PLACE THE BOUND STILL SKIPS is a row already on disk, written by an
older engine under the old rule and read back by a rebuild, a catch-up fold or a
remove. extractIndexableFields serves all three, so refusing there would make an
existing store un-rebuildable — the row is admitted and the skipped field is
NARRATED with the field, the length and the bound. Never silent, either way.

tests/integration/metadata-vector-exclusion.test.ts carried the old law as a
green assertion ("should skip indexing large arrays (>10 elements)"). It is
rewritten to the new one, plus a case proving a 64-element array indexes in full
and its eleventh element is searchable. The original bug that suite exists for —
per-dimension numeric field explosion — is still asserted on both paths.
This commit is contained in:
David Snelling 2026-09-02 13:43:14 -07:00
parent a7eb7f5222
commit 0d5ab6077d
6 changed files with 436 additions and 25 deletions

View file

@ -26,6 +26,7 @@ import { describe, it, expect, beforeEach, afterEach } from 'vitest'
import { Brainy } from '../../src/brainy.js'
import { NounType } from '../../src/types/graphTypes.js'
import { existsSync, rmSync } from 'fs'
import { MetadataArrayTooLargeError, MAX_INDEXED_ARRAY_LENGTH } from '../../src/errors/brainyError.js'
describe('Metadata Vector Exclusion Fix', () => {
let brainy: Brainy
@ -155,29 +156,56 @@ describe('Metadata Vector Exclusion Fix', () => {
expect(results[0].entity.metadata?.name).toBe('Bob')
})
it('should skip indexing large arrays (>10 elements)', async () => {
// Add entity with a large array (not a vector, just bulk data).
it('should REFUSE an array over the indexing bound, by name', async () => {
// A large array (not a vector, just bulk data). This used to be SKIPPED in
// silence at a bound of 10 — the field simply vanished from the index and
// the row dropped out of every `where` on it, indistinguishably from "no
// row matches". The bound is now MAX_INDEXED_ARRAY_LENGTH and it REFUSES.
const largeArray = Array.from({ length: 100 }, (_, i) => `item${i}`)
await brainy.add({
type: NounType.Document,
data: 'Doc with large array',
metadata: {
name: 'Doc with large array',
items: largeArray
}
})
const err = await brainy
.add({
type: NounType.Document,
data: 'Doc with large array',
metadata: {
name: 'Doc with large array',
items: largeArray
}
})
.catch((e: any) => e)
// Large arrays (> 10 elements) are deliberately skipped to avoid indexing
// bulk/vector-like payloads: 'items' must NOT appear, and the 100 elements
// must NOT have produced 100 indexed fields.
expect(err).toBeInstanceOf(MetadataArrayTooLargeError)
expect(err.field).toBe('items')
expect(err.length).toBe(100)
expect(err.limit).toBe(MAX_INDEXED_ARRAY_LENGTH)
// Nothing was indexed from the refused write — no 'items' field, and above
// all no per-element numeric fields (the original explosion class).
const fields = await brainy.getAvailableFields()
expect(fields).not.toContain('items')
const numericFields = fields.filter(f => /(^|\.)\d+$/.test(f))
expect(numericFields).toEqual([])
})
// The scalar 'name' field IS indexed.
expect(fields).toContain('name')
it('should index an array UP TO the bound — the old limit of 10 was the bug', async () => {
await brainy.add({
type: NounType.Document,
data: 'Doc with a long-but-legitimate tag list',
metadata: {
name: 'Doc with many tags',
items: Array.from({ length: MAX_INDEXED_ARRAY_LENGTH }, (_, i) => `item${i}`)
}
})
const fields = await brainy.getAvailableFields()
// The field IS indexed now, and still without per-element numeric fields.
expect(fields).toContain('items')
expect(fields.filter(f => /(^|\.)\d+$/.test(f))).toEqual([])
// And the eleventh element — the one the old bound silently dropped the
// whole field for — really is searchable.
const hits = await brainy.find({ where: { items: 'item10' } })
expect(hits.length).toBeGreaterThan(0)
})
it('should preserve HNSW vector search functionality', async () => {