David Snelling
59003fd8bc
Some checks are pending
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
An array-valued metadata field indexes one posting per element, so the index has
always carried a ceiling. It was 10, and it was applied by a bare `continue`
deep inside field extraction:
if (Array.isArray(value) && value.length > 10) continue
A row whose `tags` array held ELEVEN entries therefore had that field skipped
entirely — no posting, no error, no warning. The row then failed to match every
filtered search on `tags`, including a query for a tag it demonstrably held, and
the caller had no way to tell that from "no row matches". Eleven tags is not an
exotic shape; the eleventh tag made the row invisible. Measured on the pin here:
the where-clause returns [] on the base for all eleven values.
The ceiling is not the defect. The silence was.
THE LAW. MAX_INDEXED_ARRAY_LENGTH = 64, hardcoded (the zero-config law: no
knob), sitting far above every legitimate multi-value field — tags, authors,
categories, labels, participants — and far below any real embedding width, so
the two populations do not overlap and nobody has to tune it. Arrays of scalars
index in full up to the bound. Above it the WRITE IS REFUSED by name:
MetadataArrayTooLargeError carries the field (its full dotted address), the
length and the bound, and names the three cures. It fires at all four write
doors — add, update, relate, updateRelation — beside the existing forged-system-
key rejection, and walks nested bags because a nested field indexes under its
dotted address exactly like a top-level one.
THE ONE PLACE THE BOUND STILL SKIPS is a row already on disk, written by an
older engine under the old rule and read back by a rebuild, a catch-up fold or a
remove. extractIndexableFields serves all three, so refusing there would make an
existing store un-rebuildable — the row is admitted and the skipped field is
NARRATED with the field, the length and the bound. Never silent, either way.
tests/integration/metadata-vector-exclusion.test.ts carried the old law as a
green assertion ("should skip indexing large arrays (>10 elements)"). It is
rewritten to the new one, plus a case proving a 64-element array indexes in full
and its eleventh element is searchable. The original bug that suite exists for —
per-dimension numeric field explosion — is still asserted on both paths.
|
||
|---|---|---|
| .. | ||
| roaring | ||
| adaptiveBackpressure.ts | ||
| brainyTypes.ts | ||
| callerLocation.ts | ||
| collation.ts | ||
| contentExtractor.ts | ||
| crc32c.ts | ||
| distance.ts | ||
| embedding.ts | ||
| entityIdMapper.ts | ||
| environment.ts | ||
| errorClassification.ts | ||
| fieldNameTracking.ts | ||
| fieldTypeInference.ts | ||
| idNormalization.ts | ||
| import-progress-tracker.ts | ||
| index.ts | ||
| indexReadiness.ts | ||
| jsonProcessing.ts | ||
| jsonSafeIndexMetadata.ts | ||
| logger.ts | ||
| memoryDetection.ts | ||
| metadataFilter.ts | ||
| metadataIndex.ts | ||
| metadataIndexCache.ts | ||
| metadataIndexChunking.ts | ||
| metadataWriteBuffer.ts | ||
| mutex.ts | ||
| operationUtils.ts | ||
| osLimits.ts | ||
| paramValidation.ts | ||
| performanceMonitor.ts | ||
| projectionWatermark.ts | ||
| rebuildCounts.ts | ||
| recallPreset.ts | ||
| resultRanking.ts | ||
| statisticsCollector.ts | ||
| typeUtils.ts | ||
| typeValidation.ts | ||
| unifiedCache.ts | ||
| version.ts | ||