fix(index): a field holds every value kind it was written with, not the first one
Some checks failed
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
Delta Gate / Delta gate — candidate vs control (push) Failing after 7s

The metadata index fixed a field's value type from the first value it saw.
Every later value of another kind was coerced to that type, and when coercion
failed — `Number('electronics')` is NaN — the value was dropped from the index
with no error at all. The row stayed readable by id and by vector search and
vanished only from equality filters on that one field, which is what made it so
quiet: writing `category: 'electronics'` rows and then `category: 5` rows left
`where { category: 5 }` returning nothing, while the same rows in a
numbers-only corpus answered correctly.

The column store now keeps one posting column per (field, kind), where a kind
is a JavaScript typeof class. The first kind a field sees keeps the historical
`_column_index/<field>/` layout, so a single-kind field is byte-identical to
what earlier versions wrote and an index written before this opens unchanged;
each later kind takes its own column at `_column_index/<field>/k/<kind>/`.

Equality reads the column matching the query value's own kind, so `{c: 5}` and
`{c: '5'}` match different rows and neither is coerced into the other. Ranges
route by the kind of their bounds, and an unbounded range — the "has any value"
probe behind `exists` — reads every kind. A mixed field orders by kind first,
then by value, because a number and a string have no order between them. A
value that cannot be encoded for the column its own kind selected now raises
instead of being skipped: that path is unreachable by construction, and if it
is ever reached it is the silent drop this change exists to end.

Two neighbours fell out of the same routing. A boolean query value is now
encoded to the 1/0 the column stores, so boolean equality matches at all. And
an integer column widens to f64 the first time a non-integer arrives, so 4.5 is
stored as itself rather than rounded to 5 and answering the wrong query.

Field type inference reports every kind a field holds beside its dominant
reading, rather than leaving callers to treat one type as the whole answer.

Pins: mixed-kind equality in both write orders, `5` vs `'5'`, booleans mixed in,
a numeric range over a mixed field's numbers, close/reopen keeping every typed
posting, and an index in the pre-existing on-disk shape still reading.
`tests/critical-neural-validation.test.ts` — which writes `category` as strings
in one test and as numbers in another against one shared brain — passes whole
for the first time.

(cherry picked from commit a128f0eda5)
This commit is contained in:
David Snelling 2026-09-03 08:52:58 -07:00
parent 4c81d7d4c3
commit 4e058720b4
7 changed files with 1025 additions and 133 deletions

View file

@ -55,8 +55,30 @@ export enum FieldType {
*/
export interface FieldTypeInfo {
field: string
/**
* The DOMINANT reading of the field one type, the most specific one every
* sampled value satisfies.
*
* A field is not obliged to hold one kind, so this is not the whole answer
* for a field that holds several. Read {@link kinds} beside it: a field
* carrying `'electronics'` and `5` infers as STRING here and reports
* `['number', 'string']` there, and the metadata index keeps a separate
* posting column for each of them.
*/
inferredType: FieldType
confidence: number // 0-1 confidence score
/**
* Every value KIND observed in the sample, in the order
* number string boolean. More than one entry means a genuinely
* mixed field, and every one of those kinds is independently filterable.
*
* Kinds are JavaScript `typeof` classes, one level coarser than
* {@link FieldType}: a UUID and a category name are both `'string'`, and an
* integer and a timestamp are both `'number'`.
*
* Optional only for cached analyses written before this was reported.
*/
kinds?: Array<'number' | 'string' | 'boolean'>
sampleSize: number // Number of values analyzed
lastUpdated: number // Timestamp of last analysis
detectionMethod: 'value' // Always 'value' (no fallbacks!)
@ -133,14 +155,71 @@ export class FieldTypeInference {
}
/**
* Analyze values to determine field type
* Analyze values to determine field type, and report every KIND the field
* actually holds alongside it.
*
* The classification below picks ONE type, because every one of its
* heuristics asks `samples.every(...)`: a field carrying `'electronics'` and
* `5` satisfies none of them and lands on STRING. That single answer is true
* as far as it goes string is the dominant reading but on its own it
* says nothing about the numbers also in the field, and a caller that treats
* it as the field's only type reproduces the first-writer freeze the index
* itself no longer has. {@link FieldTypeInfo.kinds} carries the rest.
*/
private async analyzeValues(field: string, values: any[]): Promise<FieldTypeInfo> {
const info = await this.classifyValues(field, values)
info.kinds = FieldTypeInference.observedKinds(values)
if (info.kinds.length > 1 && info.metadata) {
info.metadata.format = `${info.metadata.format} (field also holds: ${info.kinds
.filter((k) => k !== FieldTypeInference.kindOfType(info.inferredType))
.join(', ')})`
}
return info
}
/**
* The distinct value kinds present in a sample, in a stable order.
*
* Kinds are JavaScript `typeof` classes the same classes the metadata
* index keeps separate posting columns for not the finer
* {@link FieldType} readings, which are interpretations layered on top of
* them (a UUID and a category name are both the `string` kind).
*/
private static observedKinds(values: any[]): Array<'number' | 'string' | 'boolean'> {
const order: Array<'number' | 'string' | 'boolean'> = ['number', 'string', 'boolean']
const seen = new Set<'number' | 'string' | 'boolean'>()
for (const v of values) {
if (v === null || v === undefined) continue
const t = typeof v
seen.add(t === 'number' ? 'number' : t === 'boolean' ? 'boolean' : 'string')
}
return order.filter((k) => seen.has(k))
}
/** The value kind a {@link FieldType} reading is an interpretation of. */
private static kindOfType(type: FieldType): 'number' | 'string' | 'boolean' {
switch (type) {
case FieldType.BOOLEAN:
return 'boolean'
case FieldType.INTEGER:
case FieldType.FLOAT:
case FieldType.TIMESTAMP_MS:
case FieldType.TIMESTAMP_S:
return 'number'
default:
return 'string'
}
}
/**
* Classify values into a single field type.
*
* Uses DuckDB-inspired type detection order:
* BOOLEAN INTEGER FLOAT DATE TIMESTAMP UUID STRING
*
* No fallbacks - pure value-based detection
*/
private async analyzeValues(field: string, values: any[]): Promise<FieldTypeInfo> {
private async classifyValues(field: string, values: any[]): Promise<FieldTypeInfo> {
// Filter null/undefined values
const validValues = values.filter(v => v !== null && v !== undefined)