This repository has been archived on 2026-09-03. You can view files and clone it, but you cannot make any changes to it's state, such as pushing and creating new issues, pull requests or comments.
open-brainy/tests/regression/metadata-field-typing.unit.test.ts

123 lines
4.8 KiB
TypeScript
Raw Permalink Normal View History

fix(index): a field holds every value kind it was written with, not the first one The metadata index fixed a field's value type from the first value it saw. Every later value of another kind was coerced to that type, and when coercion failed — `Number('electronics')` is NaN — the value was dropped from the index with no error at all. The row stayed readable by id and by vector search and vanished only from equality filters on that one field, which is what made it so quiet: writing `category: 'electronics'` rows and then `category: 5` rows left `where { category: 5 }` returning nothing, while the same rows in a numbers-only corpus answered correctly. The column store now keeps one posting column per (field, kind), where a kind is a JavaScript typeof class. The first kind a field sees keeps the historical `_column_index/<field>/` layout, so a single-kind field is byte-identical to what earlier versions wrote and an index written before this opens unchanged; each later kind takes its own column at `_column_index/<field>/k/<kind>/`. Equality reads the column matching the query value's own kind, so `{c: 5}` and `{c: '5'}` match different rows and neither is coerced into the other. Ranges route by the kind of their bounds, and an unbounded range — the "has any value" probe behind `exists` — reads every kind. A mixed field orders by kind first, then by value, because a number and a string have no order between them. A value that cannot be encoded for the column its own kind selected now raises instead of being skipped: that path is unreachable by construction, and if it is ever reached it is the silent drop this change exists to end. Two neighbours fell out of the same routing. A boolean query value is now encoded to the 1/0 the column stores, so boolean equality matches at all. And an integer column widens to f64 the first time a non-integer arrives, so 4.5 is stored as itself rather than rounded to 5 and answering the wrong query. Field type inference reports every kind a field holds beside its dominant reading, rather than leaving callers to treat one type as the whole answer. Pins: mixed-kind equality in both write orders, `5` vs `'5'`, booleans mixed in, a numeric range over a mixed field's numbers, close/reopen keeping every typed posting, and an index in the pre-existing on-disk shape still reading. `tests/critical-neural-validation.test.ts` — which writes `category` as strings in one test and as numbers in another against one shared brain — passes whole for the first time. (cherry picked from commit a128f0eda5b450ebf9caeae8e78ecaceff04feeb)
2026-09-03 08:52:58 -07:00
/**
* @module metadata-field-typing.unit.test
* @description Regression: a metadata field that holds more than one value
* KIND stays fully filterable on every kind it holds.
*
* The defect this pins, reproduced on the released engine: the metadata index
* fixed a field's value type from the FIRST value it saw, and every later value
* of a different type was coerced to that type or, when coercion failed,
* dropped from the index in silence. Writing `category: 'electronics'` rows and
* then `category: 5` rows left `find({ where: { category: 5 } })` returning
* nothing — while the same rows in a numbers-only corpus answered correctly.
* The rows themselves were never lost: they stayed readable by id and by vector
* search, and only ever went missing from equality filters on that one field,
* which is what made it so quiet.
*
* Order is the whole point of these cases. Neither writer owns the field, so
* strings-then-numbers and numbers-then-strings must give the same answers.
*/
import { describe, it, expect } from 'vitest'
import { Brainy } from '../../src/brainy.js'
import { NounType } from '../../src/types/graphTypes.js'
/** A brain over memory storage, with a corpus written in the given order. */
async function brainWith(
rows: Array<{ label: string; category: unknown }>
): Promise<Brainy> {
const brainy = new Brainy({ requireSubtype: false, storage: { type: 'memory' } })
await brainy.init()
for (const row of rows) {
await brainy.add({
data: `item ${row.label}`,
type: NounType.Thing,
metadata: { label: row.label, category: row.category }
})
}
return brainy
}
const labelsOf = (results: Array<{ metadata?: Record<string, unknown> }>): string[] =>
results.map((r) => String(r.metadata?.label)).sort()
describe('regression: a mixed-kind metadata field filters on every kind', { timeout: 180_000 }, () => {
it('finds number rows written after string rows', async () => {
const brainy = await brainWith([
{ label: 'e1', category: 'electronics' },
{ label: 'f1', category: 'furniture' },
{ label: 'n1', category: 5 },
{ label: 'n2', category: 5 },
{ label: 'n3', category: 7 }
])
try {
expect(labelsOf(await brainy.find({ where: { category: 5 }, limit: 100 }))).toEqual(['n1', 'n2'])
expect(labelsOf(await brainy.find({ where: { category: 7 }, limit: 100 }))).toEqual(['n3'])
expect(labelsOf(await brainy.find({ where: { category: 'electronics' }, limit: 100 }))).toEqual(['e1'])
expect(labelsOf(await brainy.find({ where: { category: 'furniture' }, limit: 100 }))).toEqual(['f1'])
} finally {
await brainy.close()
}
})
it('finds string rows written after number rows', async () => {
const brainy = await brainWith([
{ label: 'n1', category: 5 },
{ label: 'n2', category: 5 },
{ label: 'e1', category: 'electronics' },
{ label: 'e2', category: 'electronics' }
])
try {
expect(labelsOf(await brainy.find({ where: { category: 'electronics' }, limit: 100 }))).toEqual(['e1', 'e2'])
expect(labelsOf(await brainy.find({ where: { category: 5 }, limit: 100 }))).toEqual(['n1', 'n2'])
} finally {
await brainy.close()
}
})
it('keeps `5` and `\'5\'` apart — a kind is part of the value, not a formatting detail', async () => {
const brainy = await brainWith([
{ label: 'num', category: 5 },
{ label: 'str', category: '5' }
])
try {
expect(labelsOf(await brainy.find({ where: { category: 5 }, limit: 100 }))).toEqual(['num'])
expect(labelsOf(await brainy.find({ where: { category: '5' }, limit: 100 }))).toEqual(['str'])
} finally {
await brainy.close()
}
})
it('serves booleans mixed into a field that already holds strings', async () => {
const brainy = await brainWith([
{ label: 's1', category: 'yes' },
{ label: 'b1', category: true },
{ label: 'b2', category: false }
])
try {
expect(labelsOf(await brainy.find({ where: { category: true }, limit: 100 }))).toEqual(['b1'])
expect(labelsOf(await brainy.find({ where: { category: false }, limit: 100 }))).toEqual(['b2'])
expect(labelsOf(await brainy.find({ where: { category: 'yes' }, limit: 100 }))).toEqual(['s1'])
} finally {
await brainy.close()
}
})
it('ranges over the numeric part of a mixed field', async () => {
const brainy = await brainWith([
{ label: 'unpriced', category: 'on request' },
{ label: 'cheap', category: 100 },
{ label: 'mid', category: 500 },
{ label: 'dear', category: 900 }
])
try {
const found = await brainy.find({
where: { category: { greaterThan: 200 } },
limit: 100
})
expect(labelsOf(found)).toEqual(['dear', 'mid'])
} finally {
await brainy.close()
}
})
})