feat(query): the sparse-store cut — where on a never-carried field serves operator truth, never a refusal
Some checks failed
CI / Node 22 (push) Successful in 12m15s
CI / Node 24 (push) Has been cancelled
CI / Bun (latest) (push) Has been cancelled

A first adopter's namespace migration went 341 red on one class: the
never-carried-field refusal firing on CORRECT filters against fresh and
sparse stores — a freshly provisioned tenant refused its own first
filtered read, with the did-you-mean built for typos firing hardest on
day-one stores where nothing is wrong.

The ruled cut: a WHERE filter naming a field no row carries is SERVED
OPERATOR-TRUTHFULLY — eq/in/range/contains answer [] (nothing carries
it, nothing matches); ne and exists:false answer ALL rows (the equally
true complement — a blanket empty here would be silently wrong, which is
why the simpler cut was rejected); exists:true answers []. Served from
the field registry, with the did-you-mean demoted to a once-per-field
WARN. orderBy and genuinely ambiguous addresses KEEP their hard typed
refusals: no truthful order exists over an uncarried field, and
ambiguity is a contract error while absence is data.

Mechanics: the negative operator absorbs the FIELD_NOT_INDEXED throw as
its empty exclude set (the clause-level catch correctly zeroes positive
operators only); the egress matcher already agreed. Plus the
provider-seam belt: a field refusal thrown by a replacement metadata
manager is normalized to THIS package's UnresolvableFieldError at every
filter call site — one class identity for consumers, instanceof works
(a first adopter's cross-package finding).

Conformance: tests/conformance/sparse-store-cut.test.ts — the shared
operator rows both engines run (positive-empty, negative-all,
fresh-tenant day-one, orderBy refusal kept, compound composition).
Gates: unit 2065/2065 · integration 832 · conformance 36/36.
This commit is contained in:
David Snelling 2026-08-12 15:57:19 -07:00
parent df96fccfd1
commit 7b67db4d0c
5 changed files with 188 additions and 8 deletions

View file

@ -1908,6 +1908,35 @@ export class MetadataIndexManager implements MetadataIndexProvider {
* index (early-stop at `offset+limit`); the JS index returns ALL matches and lets
* the caller window them, so `_opts` is intentionally ignored here.
*/
/** Once-per-field throttle for the sparse-store did-you-mean WARN. */
private readonly warnedNeverCarried = new Set<string>()
/**
* THE SPARSE-STORE CUT (ruled 2026-08-12): a WHERE filter naming a field
* no row carries is SERVED OPERATOR-TRUTHFULLY (eq/range/contains [];
* ne/exists:false all rows; exists:true []) the JS evaluator below
* already computes exactly these truths via complements with the
* did-you-mean demoted to this throttled WARN. A fresh store's first
* filtered read is a correct empty answer, never a refusal. orderBy and
* ambiguous addresses KEEP their hard refusals (no truthful order
* exists; ambiguity is a contract error absence is data).
*/
/** Is this field known to the index at all (any row ever carried it)? */
private fieldRegistryHas(field: string): boolean {
return this.fieldStats.has(field)
}
private warnNeverCarriedOnce(field: string): void {
if (this.warnedNeverCarried.has(field)) return
this.warnedNeverCarried.add(field)
prodLog.warn(
`[MetadataIndex] filter names field '${field}' which no row carries — ` +
`serving the operator-truthful answer (empty for positive matches; ` +
`the complement for ne/exists:false). If this is a typo, check the ` +
`field name; refusals remain on orderBy.`
)
}
async getIdsForFilter(filter: any, _opts?: { limit?: number; offset?: number }): Promise<string[]> {
if (!filter || Object.keys(filter).length === 0) {
return []
@ -1984,6 +2013,17 @@ export class MetadataIndexManager implements MetadataIndexProvider {
const address = parseFieldAddress(rawField, 'entity')
const field = address.scope === 'system' ? `system.${address.field}` : address.field
// Sparse-store cut: a user field no row carries serves operator-truth
// below (the evaluators' complements are already correct) — announce
// it once so a typo is findable without breaking a fresh store.
if (
address.scope !== 'system' &&
!(this.columnStore && this.columnStore.hasField(field)) &&
!this.fieldRegistryHas(field)
) {
this.warnNeverCarriedOnce(field)
}
let fieldResults: string[] = []
try {
@ -2022,7 +2062,18 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// complement as a bitmap difference over the int-id universe rather
// than materializing the whole corpus as UUID strings to filter it.
const excludeInts: number[] = []
for (const uuid of await this.getIds(field, operand)) {
// Sparse-store truth: a never-carried field has NOTHING to
// exclude — the complement of nothing is EVERYTHING. getIds
// throws FIELD_NOT_INDEXED there; the clause-level catch
// would wrongly zero this NEGATIVE operator, so absorb it
// here as the empty exclude set (the ruled operator-truth).
let neMatches: string[] = []
try {
neMatches = await this.getIds(field, operand)
} catch {
neMatches = []
}
for (const uuid of neMatches) {
const intId = this.idMapper.getInt(uuid)
if (intId !== undefined) excludeInts.push(intId)
}