David Snelling
4e058720b4
Some checks failed
The metadata index fixed a field's value type from the first value it saw.
Every later value of another kind was coerced to that type, and when coercion
failed — `Number('electronics')` is NaN — the value was dropped from the index
with no error at all. The row stayed readable by id and by vector search and
vanished only from equality filters on that one field, which is what made it so
quiet: writing `category: 'electronics'` rows and then `category: 5` rows left
`where { category: 5 }` returning nothing, while the same rows in a
numbers-only corpus answered correctly.
The column store now keeps one posting column per (field, kind), where a kind
is a JavaScript typeof class. The first kind a field sees keeps the historical
`_column_index/<field>/` layout, so a single-kind field is byte-identical to
what earlier versions wrote and an index written before this opens unchanged;
each later kind takes its own column at `_column_index/<field>/k/<kind>/`.
Equality reads the column matching the query value's own kind, so `{c: 5}` and
`{c: '5'}` match different rows and neither is coerced into the other. Ranges
route by the kind of their bounds, and an unbounded range — the "has any value"
probe behind `exists` — reads every kind. A mixed field orders by kind first,
then by value, because a number and a string have no order between them. A
value that cannot be encoded for the column its own kind selected now raises
instead of being skipped: that path is unreachable by construction, and if it
is ever reached it is the silent drop this change exists to end.
Two neighbours fell out of the same routing. A boolean query value is now
encoded to the 1/0 the column stores, so boolean equality matches at all. And
an integer column widens to f64 the first time a non-integer arrives, so 4.5 is
stored as itself rather than rounded to 5 and answering the wrong query.
Field type inference reports every kind a field holds beside its dominant
reading, rather than leaving callers to treat one type as the whole answer.
Pins: mixed-kind equality in both write orders, `5` vs `'5'`, booleans mixed in,
a numeric range over a mixed field's numbers, close/reopen keeping every typed
posting, and an index in the pre-existing on-disk shape still reading.
`tests/critical-neural-validation.test.ts` — which writes `category` as strings
in one test and as numbers in another against one shared brain — passes whole
for the first time.
(cherry picked from commit
|
||
|---|---|---|
| .. | ||
| api | ||
| benchmarks | ||
| comprehensive | ||
| configs | ||
| conformance | ||
| fixtures | ||
| helpers | ||
| integration | ||
| integrations | ||
| lifecycle | ||
| performance | ||
| regression | ||
| scripts | ||
| transaction | ||
| unit | ||
| vfs | ||
| critical-neural-validation.test.ts | ||
| critical-performance-benchmark.test.ts | ||
| model-loading.test.ts | ||
| package-size-breakdown.test.ts | ||
| package-size-limit.test.ts | ||
| setup-integration.ts | ||
| setup-semantic.ts | ||
| setup-unit.ts | ||
| setup.ts | ||
| type-utils.unit.test.ts | ||