This repository has been archived on 2026-09-03. You can view files and clone it, but you cannot make any changes to it's state, such as pushing and creating new issues, pull requests or comments.
open-brainy/src/db/whereMatcher.ts

293 lines
11 KiB
TypeScript
Raw Normal View History

feat(8.0): generational MVCC storage + Datomic-style Db API (now/transact/asOf/with/persist) One mechanism replaces the COW and versioning subsystems: immutable generation-stamped records behind a Datomic-style database value. Record layer (src/db/generationStore.ts): - Monotonic generation counter in _system/generation.json; bumped once per transact() commit and once per single-operation write (storage hook), so brain.generation() is always a meaningful watermark. - Commit protocol: stage before-images + tx.json delta -> fsync -> execute batch via TransactionManager -> atomic tmp+rename of _system/manifest.json (the rename IS the commit point) -> append _system/tx-log.jsonl. - Crash recovery on open rolls uncommitted generations back byte-identically and forces an index rebuild; refcounted pins gate compactHistory(), which records a horizon (asOf below it throws GenerationCompactedError). Db API: - Db: get/find/search/related pinned at a generation, with() speculative overlays, since() diffs, persist() hard-link-farm snapshots, timestamp, generation, release() + FinalizationRegistry backstop. - Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch (GenerationConflictError CAS), asOf(generation|Date|path), restore(path, {confirm}), compactHistory(), generation(), static open() + load(). - get()/metadata find()/related() are fully correct at any reachable pinned generation; index-accelerated queries at historical generations throw NotYetSupportedAtHistoricalGenerationError - never silently-wrong results. - VersionedIndexProvider (generation/isGenerationVisible/pin/release) in plugin.ts: feature-detected, balanced pin/release in lockstep with Db lifecycle, post-commit applier + replay-gap model documented. Storage primitives (BaseStorage + filesystem/memory adapters): raw-object read/write/list/remove, fsync barrier, noun/verb raw before-image capture, tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and append-in-place exceptions), restoreFromDirectory + derived-state reload. Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation across 200 mutations, batch atomicity under injected execution failure, ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety under pins, with() overlay isolation, generation monotonicity across reopen, crash consistency through the real recovery path, and versioned provider pin/release balance. Design record in docs/ADR-001-generational-mvcc.md.
2026-06-10 14:14:07 -07:00
/**
* @module db/whereMatcher
* @description In-memory evaluation of `find()` metadata filters against a
* single resolved entity the overlay half of historical and speculative
* reads on a `Db` value.
*
* A `Db` pinned at a past generation answers metadata-level `find()` by
* combining the live index's results (for entities untouched since the pin)
* with per-entity evaluation of the same filter against generation records
* (for entities that DID change). The same evaluator backs `db.with()`
* speculative overlays. The semantics here deliberately mirror the index
* evaluator in `src/utils/metadataIndex.ts` (`getIdsForFilter`) operator
* aliases, array-membership equality, `exists`/`missing`, and the
* `allOf`/`anyOf`/`not` logical forms so an entity matches in-memory if and
* only if it would have matched through the index.
*
* **Honesty contract:** an operator this module does not recognize throws
* {@link UnsupportedWhereOperatorError} instead of guessing a historical
* read must never silently return wrong results.
*/
import type { Entity, FindParams } from '../types/brainy.types.js'
/**
* @description Thrown when a `where` clause uses an operator this in-memory
* evaluator does not implement. Callers (historical `find()` on a `Db`)
* convert this into the documented historical-query error rather than
* returning silently-wrong results.
*/
export class UnsupportedWhereOperatorError extends Error {
/** The unrecognized operator name. */
public readonly operator: string
/**
* @param operator - The operator name that could not be evaluated.
*/
constructor(operator: string) {
super(
`The where-operator '${operator}' is not supported for historical/speculative ` +
`in-memory evaluation. Supported: eq/equals/is, ne/notEquals/isNot, in/oneOf, ` +
`gt/greaterThan, gte/greaterThanOrEqual/greaterEqual, lt/lessThan, ` +
`lte/lessThanOrEqual/lessEqual, between, contains, exists, missing, ` +
`plus allOf/anyOf/not.`
)
this.name = 'UnsupportedWhereOperatorError'
this.operator = operator
}
}
/**
* @description Resolve a filter field name to its value on an entity,
* mirroring how `extractIndexableFields` lays entities out for the metadata
* index: standard fields live at the top level (with the `noun` `type`
* alias), everything else is a custom field inside the `metadata` bag.
* Dotted paths (`metadata.priority`, `address.city`) traverse nested objects.
*
* @param entity - The resolved entity to read from.
* @param field - The filter field name.
* @returns The field's value, or `undefined` when absent.
*/
export function resolveEntityField(entity: Entity, field: string): unknown {
switch (field) {
case 'noun':
case 'type':
return entity.type
case 'subtype':
return entity.subtype
case 'id':
return entity.id
case 'createdAt':
return entity.createdAt
case 'updatedAt':
return entity.updatedAt
case 'service':
return entity.service
case 'createdBy':
return entity.createdBy
case 'confidence':
return entity.confidence
case 'weight':
return entity.weight
case '_rev':
return entity._rev
case 'data':
return entity.data
}
if (field.includes('.')) {
// Dotted path: resolve against the whole entity first (`metadata.x`),
// then against the metadata bag (`address.city` on nested metadata).
const fromEntity = resolvePath(entity as unknown as Record<string, unknown>, field)
if (fromEntity !== undefined) return fromEntity
return resolvePath((entity.metadata ?? {}) as Record<string, unknown>, field)
}
return ((entity.metadata ?? {}) as Record<string, unknown>)[field]
}
/** Walk a dotted path through nested plain objects. */
function resolvePath(obj: Record<string, unknown>, path: string): unknown {
let current: unknown = obj
for (const segment of path.split('.')) {
if (current === null || typeof current !== 'object') return undefined
current = (current as Record<string, unknown>)[segment]
}
return current
}
/**
* @description Index-equality semantics: scalar strict equality, with
* array-valued fields matching by membership (the index stores one posting
* per array element, so `eq` on an array field means "contains").
*/
function eqMatches(value: unknown, operand: unknown): boolean {
if (Array.isArray(value)) {
return value.some((element) => element === operand)
}
return value === operand
}
/** Ordered comparison for range operators (numbers, or both-strings lexicographic). */
function compare(value: unknown, operand: unknown): number | null {
if (typeof value === 'number' && typeof operand === 'number') {
return value - operand
}
if (typeof value === 'string' && typeof operand === 'string') {
return value < operand ? -1 : value > operand ? 1 : 0
}
return null
}
/**
* @description Evaluate one field condition (shorthand value or operator
* object) against a resolved field value, mirroring the operator table in
* `metadataIndex.getIdsForFilter`.
*
* @param value - The entity's field value (possibly `undefined`).
* @param condition - The filter condition for this field.
* @returns Whether the value satisfies the condition.
* @throws UnsupportedWhereOperatorError for unrecognized operators.
*/
function fieldConditionMatches(value: unknown, condition: unknown): boolean {
if (condition === null || typeof condition !== 'object' || Array.isArray(condition)) {
// Shorthand for 'eq'.
return eqMatches(value, condition)
}
for (const [op, operand] of Object.entries(condition as Record<string, unknown>)) {
let matches: boolean
switch (op) {
case 'is':
case 'equals':
case 'eq':
matches = eqMatches(value, operand)
break
case 'isNot':
case 'notEquals':
case 'ne':
matches = !eqMatches(value, operand)
break
case 'oneOf':
case 'in':
matches = Array.isArray(operand) && operand.some((candidate) => eqMatches(value, candidate))
break
case 'greaterThan':
case 'gt': {
const cmp = compare(value, operand)
matches = cmp !== null && cmp > 0
break
}
case 'greaterEqual':
case 'greaterThanOrEqual':
case 'gte': {
const cmp = compare(value, operand)
matches = cmp !== null && cmp >= 0
break
}
case 'lessThan':
case 'lt': {
const cmp = compare(value, operand)
matches = cmp !== null && cmp < 0
break
}
case 'lessEqual':
case 'lessThanOrEqual':
case 'lte': {
const cmp = compare(value, operand)
matches = cmp !== null && cmp <= 0
break
}
case 'between': {
if (!Array.isArray(operand) || operand.length !== 2) {
matches = false
break
}
const lower = compare(value, operand[0])
const upper = compare(value, operand[1])
matches = lower !== null && upper !== null && lower >= 0 && upper <= 0
break
}
case 'contains':
// Index semantics: array fields post one entry per element, so
// 'contains' is the same lookup as 'eq' (membership on arrays).
matches = eqMatches(value, operand)
break
case 'exists':
matches = operand ? value !== undefined : value === undefined
break
case 'missing':
matches = operand ? value === undefined : value !== undefined
break
default:
throw new UnsupportedWhereOperatorError(op)
}
if (!matches) return false // Multiple operators on one field AND together.
}
return true
}
/**
* @description Evaluate a full `where` filter (field conditions plus
* `allOf`/`anyOf`/`not` logical composition) against one entity.
*
* @param entity - The resolved entity (historical record or speculative overlay).
* @param where - The `where` clause from `FindParams`.
* @returns Whether the entity satisfies every clause.
* @throws UnsupportedWhereOperatorError for unrecognized operators.
*/
export function whereMatches(entity: Entity, where: Record<string, unknown>): boolean {
for (const [field, condition] of Object.entries(where)) {
if (field === 'allOf') {
if (!Array.isArray(condition)) return false
if (!condition.every((sub) => whereMatches(entity, sub as Record<string, unknown>))) return false
continue
}
if (field === 'anyOf') {
if (!Array.isArray(condition)) return false
if (!condition.some((sub) => whereMatches(entity, sub as Record<string, unknown>))) return false
continue
}
if (field === 'not') {
if (whereMatches(entity, condition as Record<string, unknown>)) return false
continue
}
if (!fieldConditionMatches(resolveEntityField(entity, field), condition)) return false
}
return true
}
/**
* @description Evaluate the metadata-level portions of a `find()` query
* (`type`, `subtype`, `where`, `service`, `excludeVFS`) against one resolved
* entity. Used by historical and speculative `find()` to decide whether a
* changed/overlaid entity belongs in the result set. The caller guarantees
* the query carries no index-only dimensions (semantic `query`/`vector`,
* `connected` traversal, ) those are rejected with the documented
* historical-query error before evaluation starts.
*
* @param entity - The resolved entity.
* @param params - The metadata-level find parameters.
* @returns Whether the entity matches every requested filter.
* @throws UnsupportedWhereOperatorError for unrecognized `where` operators.
*/
export function entityMatchesFind(entity: Entity, params: FindParams): boolean {
if (params.type !== undefined) {
const types = Array.isArray(params.type) ? params.type : [params.type]
if (!types.includes(entity.type)) return false
}
if (params.subtype !== undefined) {
const subtypes = Array.isArray(params.subtype) ? params.subtype : [params.subtype]
if (entity.subtype === undefined || !subtypes.includes(entity.subtype)) return false
}
if (params.service !== undefined && entity.service !== params.service) {
return false
}
if (params.excludeVFS === true) {
// Mirror of find()'s exclusion filter (`vfsType: { exists: false }`,
// `isVFSEntity: { ne: true }`) and Brainy's VFS-marker helper.
const metadata = (entity.metadata ?? {}) as Record<string, unknown>
if (metadata.vfsType !== undefined) return false
if (metadata.isVFSEntity === true || metadata.isVFS === true) return false
}
if (params.where !== undefined) {
if (!whereMatches(entity, params.where as Record<string, unknown>)) return false
}
return true
}