This repository has been archived on 2026-09-03. You can view files and clone it, but you cannot make any changes to it's state, such as pushing and creating new issues, pull requests or comments.
open-brainy/src/db/whereMatcher.ts

292 lines
11 KiB
TypeScript
Raw Normal View History

feat(8.0): generational MVCC storage + Datomic-style Db API (now/transact/asOf/with/persist) One mechanism replaces the COW and versioning subsystems: immutable generation-stamped records behind a Datomic-style database value. Record layer (src/db/generationStore.ts): - Monotonic generation counter in _system/generation.json; bumped once per transact() commit and once per single-operation write (storage hook), so brain.generation() is always a meaningful watermark. - Commit protocol: stage before-images + tx.json delta -> fsync -> execute batch via TransactionManager -> atomic tmp+rename of _system/manifest.json (the rename IS the commit point) -> append _system/tx-log.jsonl. - Crash recovery on open rolls uncommitted generations back byte-identically and forces an index rebuild; refcounted pins gate compactHistory(), which records a horizon (asOf below it throws GenerationCompactedError). Db API: - Db: get/find/search/related pinned at a generation, with() speculative overlays, since() diffs, persist() hard-link-farm snapshots, timestamp, generation, release() + FinalizationRegistry backstop. - Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch (GenerationConflictError CAS), asOf(generation|Date|path), restore(path, {confirm}), compactHistory(), generation(), static open() + load(). - get()/metadata find()/related() are fully correct at any reachable pinned generation; index-accelerated queries at historical generations throw NotYetSupportedAtHistoricalGenerationError - never silently-wrong results. - VersionedIndexProvider (generation/isGenerationVisible/pin/release) in plugin.ts: feature-detected, balanced pin/release in lockstep with Db lifecycle, post-commit applier + replay-gap model documented. Storage primitives (BaseStorage + filesystem/memory adapters): raw-object read/write/list/remove, fsync barrier, noun/verb raw before-image capture, tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and append-in-place exceptions), restoreFromDirectory + derived-state reload. Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation across 200 mutations, batch atomicity under injected execution failure, ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety under pins, with() overlay isolation, generation monotonicity across reopen, crash consistency through the real recovery path, and versioned provider pin/release balance. Design record in docs/ADR-001-generational-mvcc.md.
2026-06-10 14:14:07 -07:00
/**
* @module db/whereMatcher
* @description In-memory evaluation of `find()` metadata filters against a
* single resolved entity the overlay half of historical and speculative
* reads on a `Db` value.
*
* A `Db` pinned at a past generation answers metadata-level `find()` by
* combining the live index's results (for entities untouched since the pin)
* with per-entity evaluation of the same filter against generation records
* (for entities that DID change). The same evaluator backs `db.with()`
* speculative overlays. The semantics here deliberately mirror the index
* evaluator in `src/utils/metadataIndex.ts` (`getIdsForFilter`) operator
* aliases, array-membership equality, `exists`/`missing`, and the
* `allOf`/`anyOf`/`not` logical forms so an entity matches in-memory if and
* only if it would have matched through the index.
*
* **Honesty contract:** an operator this module does not recognize throws
* {@link UnsupportedWhereOperatorError} instead of guessing a historical
* read must never silently return wrong results.
*/
import type { Entity, FindParams } from '../types/brainy.types.js'
/**
* @description Thrown when a `where` clause uses an operator this in-memory
feat(8.0): full query surface at historical generations via ephemeral index materialization Historical Db values (now()/asOf() pins that history has moved past) now serve the COMPLETE query surface - vector/hybrid search, graph traversal, cursor pagination, and aggregation - by materializing ephemeral in-memory indexes over the exact at-generation record set. The historical-query throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted. Materializer (Brainy.materializeAtGeneration): - Copies the at-G record set (live bytes for ids untouched since the pin, immutable before-images otherwise) into a fresh MemoryStorage; a final reconciliation pass under the commit mutex makes the copy exact even when transactions commit mid-build. - Opens a read-only Brainy over the copy: init rebuilds the metadata and graph-adjacency indexes from the records; the vector index is built by inserting every at-G vector (the at-G HNSW graph never existed on disk, so there is nothing to restore). Host embedder and aggregate definitions are shared - no second model load, aggregates backfill at-G values. - Cost is the documented contract: O(n at G) time and memory, ONCE per Db (handle cached; freed by release(), with a FinalizationRegistry backstop that also closes leaked readers). A native VersionedIndexProvider serves the same reads from retained segments with no rebuild. Db routing (src/db/db.ts): metadata-level find()/related() keep the free record path; index-only dimensions (query/vector/near/connected/cursor/ aggregate/includeRelations/non-metadata modes) route to the cached materialization; unsupported where-operators on the record path re-route there too instead of erroring. Speculative with() overlays keep the one honest boundary - SpeculativeOverlayError (overlay entities carry no embeddings, so index reads over them would be silently incomplete); metadata find()/get()/filter related() work on overlays. UpdateParams.vector contract now honored: an explicit pre-computed vector applies directly (with dimension validation) in update() and transact update ops, re-indexing HNSW - previously it was silently ignored unless data also changed. GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees filtered through the live-verb tombstone set (entity->entity edge trees deleted - they carried no verb ids, so removeVerb could never tombstone them and traversal served stale neighbors forever). Neighbor reads batch- load live verbs via the unified cache; addVerb seeds the cache. Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector search finds old vector placement including since-deleted entities; historical graph traversal walks the old wiring after a rewire; historical aggregation computes at-G group values; asOf() pins get the same surface; the materialization builds once per Db and release() closes the ephemeral reader (it refuses reads afterwards); overlays throw the documented error. ADR-001 updated to the no-throws historical model.
2026-06-11 08:12:11 -07:00
* evaluator does not implement. Callers (historical/speculative `find()` on
* a `Db`) never let it surface as silently-wrong results: at a historical
* generation the query is rerouted through the at-generation index
* materialization (which evaluates the full live operator surface); on a
* speculative overlay it becomes a `SpeculativeOverlayError`.
feat(8.0): generational MVCC storage + Datomic-style Db API (now/transact/asOf/with/persist) One mechanism replaces the COW and versioning subsystems: immutable generation-stamped records behind a Datomic-style database value. Record layer (src/db/generationStore.ts): - Monotonic generation counter in _system/generation.json; bumped once per transact() commit and once per single-operation write (storage hook), so brain.generation() is always a meaningful watermark. - Commit protocol: stage before-images + tx.json delta -> fsync -> execute batch via TransactionManager -> atomic tmp+rename of _system/manifest.json (the rename IS the commit point) -> append _system/tx-log.jsonl. - Crash recovery on open rolls uncommitted generations back byte-identically and forces an index rebuild; refcounted pins gate compactHistory(), which records a horizon (asOf below it throws GenerationCompactedError). Db API: - Db: get/find/search/related pinned at a generation, with() speculative overlays, since() diffs, persist() hard-link-farm snapshots, timestamp, generation, release() + FinalizationRegistry backstop. - Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch (GenerationConflictError CAS), asOf(generation|Date|path), restore(path, {confirm}), compactHistory(), generation(), static open() + load(). - get()/metadata find()/related() are fully correct at any reachable pinned generation; index-accelerated queries at historical generations throw NotYetSupportedAtHistoricalGenerationError - never silently-wrong results. - VersionedIndexProvider (generation/isGenerationVisible/pin/release) in plugin.ts: feature-detected, balanced pin/release in lockstep with Db lifecycle, post-commit applier + replay-gap model documented. Storage primitives (BaseStorage + filesystem/memory adapters): raw-object read/write/list/remove, fsync barrier, noun/verb raw before-image capture, tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and append-in-place exceptions), restoreFromDirectory + derived-state reload. Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation across 200 mutations, batch atomicity under injected execution failure, ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety under pins, with() overlay isolation, generation monotonicity across reopen, crash consistency through the real recovery path, and versioned provider pin/release balance. Design record in docs/ADR-001-generational-mvcc.md.
2026-06-10 14:14:07 -07:00
*/
export class UnsupportedWhereOperatorError extends Error {
/** The unrecognized operator name. */
public readonly operator: string
/**
* @param operator - The operator name that could not be evaluated.
*/
constructor(operator: string) {
super(
`The where-operator '${operator}' is not supported for historical/speculative ` +
`in-memory evaluation. Supported: eq/equals, ne/notEquals, in/oneOf, ` +
`gt/greaterThan, gte/greaterThanOrEqual, lt/lessThan, ` +
`lte/lessThanOrEqual, between, contains, exists, missing, ` +
feat(8.0): generational MVCC storage + Datomic-style Db API (now/transact/asOf/with/persist) One mechanism replaces the COW and versioning subsystems: immutable generation-stamped records behind a Datomic-style database value. Record layer (src/db/generationStore.ts): - Monotonic generation counter in _system/generation.json; bumped once per transact() commit and once per single-operation write (storage hook), so brain.generation() is always a meaningful watermark. - Commit protocol: stage before-images + tx.json delta -> fsync -> execute batch via TransactionManager -> atomic tmp+rename of _system/manifest.json (the rename IS the commit point) -> append _system/tx-log.jsonl. - Crash recovery on open rolls uncommitted generations back byte-identically and forces an index rebuild; refcounted pins gate compactHistory(), which records a horizon (asOf below it throws GenerationCompactedError). Db API: - Db: get/find/search/related pinned at a generation, with() speculative overlays, since() diffs, persist() hard-link-farm snapshots, timestamp, generation, release() + FinalizationRegistry backstop. - Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch (GenerationConflictError CAS), asOf(generation|Date|path), restore(path, {confirm}), compactHistory(), generation(), static open() + load(). - get()/metadata find()/related() are fully correct at any reachable pinned generation; index-accelerated queries at historical generations throw NotYetSupportedAtHistoricalGenerationError - never silently-wrong results. - VersionedIndexProvider (generation/isGenerationVisible/pin/release) in plugin.ts: feature-detected, balanced pin/release in lockstep with Db lifecycle, post-commit applier + replay-gap model documented. Storage primitives (BaseStorage + filesystem/memory adapters): raw-object read/write/list/remove, fsync barrier, noun/verb raw before-image capture, tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and append-in-place exceptions), restoreFromDirectory + derived-state reload. Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation across 200 mutations, batch atomicity under injected execution failure, ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety under pins, with() overlay isolation, generation monotonicity across reopen, crash consistency through the real recovery path, and versioned provider pin/release balance. Design record in docs/ADR-001-generational-mvcc.md.
2026-06-10 14:14:07 -07:00
`plus allOf/anyOf/not.`
)
this.name = 'UnsupportedWhereOperatorError'
this.operator = operator
}
}
/**
* @description Resolve a filter field name to its value on an entity,
* mirroring how `extractIndexableFields` lays entities out for the metadata
* index: standard fields live at the top level (with the `noun` `type`
* alias), everything else is a custom field inside the `metadata` bag.
* Dotted paths (`metadata.priority`, `address.city`) traverse nested objects.
*
* @param entity - The resolved entity to read from.
* @param field - The filter field name.
* @returns The field's value, or `undefined` when absent.
*/
export function resolveEntityField(entity: Entity, field: string): unknown {
switch (field) {
case 'noun':
case 'type':
return entity.type
case 'subtype':
return entity.subtype
case 'id':
return entity.id
case 'createdAt':
return entity.createdAt
case 'updatedAt':
return entity.updatedAt
case 'service':
return entity.service
case 'createdBy':
return entity.createdBy
case 'confidence':
return entity.confidence
case 'weight':
return entity.weight
case '_rev':
return entity._rev
case 'data':
return entity.data
}
if (field.includes('.')) {
// Dotted path: resolve against the whole entity first (`metadata.x`),
// then against the metadata bag (`address.city` on nested metadata).
const fromEntity = resolvePath(entity as unknown as Record<string, unknown>, field)
if (fromEntity !== undefined) return fromEntity
return resolvePath((entity.metadata ?? {}) as Record<string, unknown>, field)
}
return ((entity.metadata ?? {}) as Record<string, unknown>)[field]
}
/** Walk a dotted path through nested plain objects. */
function resolvePath(obj: Record<string, unknown>, path: string): unknown {
let current: unknown = obj
for (const segment of path.split('.')) {
if (current === null || typeof current !== 'object') return undefined
current = (current as Record<string, unknown>)[segment]
}
return current
}
/**
* @description Index-equality semantics: scalar strict equality, with
* array-valued fields matching by membership (the index stores one posting
* per array element, so `eq` on an array field means "contains").
*/
function eqMatches(value: unknown, operand: unknown): boolean {
if (Array.isArray(value)) {
return value.some((element) => element === operand)
}
return value === operand
}
/** Ordered comparison for range operators (numbers, or both-strings lexicographic). */
function compare(value: unknown, operand: unknown): number | null {
if (typeof value === 'number' && typeof operand === 'number') {
return value - operand
}
if (typeof value === 'string' && typeof operand === 'string') {
return value < operand ? -1 : value > operand ? 1 : 0
}
return null
}
/**
* @description Evaluate one field condition (shorthand value or operator
* object) against a resolved field value, mirroring the operator table in
* `metadataIndex.getIdsForFilter`.
*
* @param value - The entity's field value (possibly `undefined`).
* @param condition - The filter condition for this field.
* @returns Whether the value satisfies the condition.
* @throws UnsupportedWhereOperatorError for unrecognized operators.
*/
function fieldConditionMatches(value: unknown, condition: unknown): boolean {
if (condition === null || typeof condition !== 'object' || Array.isArray(condition)) {
// Shorthand for 'eq'.
return eqMatches(value, condition)
}
for (const [op, operand] of Object.entries(condition as Record<string, unknown>)) {
let matches: boolean
switch (op) {
case 'equals':
case 'eq':
matches = eqMatches(value, operand)
break
case 'notEquals':
case 'ne':
matches = !eqMatches(value, operand)
break
case 'oneOf':
case 'in':
matches = Array.isArray(operand) && operand.some((candidate) => eqMatches(value, candidate))
break
case 'greaterThan':
case 'gt': {
const cmp = compare(value, operand)
matches = cmp !== null && cmp > 0
break
}
case 'greaterThanOrEqual':
case 'gte': {
const cmp = compare(value, operand)
matches = cmp !== null && cmp >= 0
break
}
case 'lessThan':
case 'lt': {
const cmp = compare(value, operand)
matches = cmp !== null && cmp < 0
break
}
case 'lessThanOrEqual':
case 'lte': {
const cmp = compare(value, operand)
matches = cmp !== null && cmp <= 0
break
}
case 'between': {
if (!Array.isArray(operand) || operand.length !== 2) {
matches = false
break
}
const lower = compare(value, operand[0])
const upper = compare(value, operand[1])
matches = lower !== null && upper !== null && lower >= 0 && upper <= 0
break
}
case 'contains':
// Index semantics: array fields post one entry per element, so
// 'contains' is the same lookup as 'eq' (membership on arrays).
matches = eqMatches(value, operand)
break
case 'exists':
matches = operand ? value !== undefined : value === undefined
break
case 'missing':
matches = operand ? value === undefined : value !== undefined
break
default:
throw new UnsupportedWhereOperatorError(op)
}
if (!matches) return false // Multiple operators on one field AND together.
}
return true
}
/**
* @description Evaluate a full `where` filter (field conditions plus
* `allOf`/`anyOf`/`not` logical composition) against one entity.
*
* @param entity - The resolved entity (historical record or speculative overlay).
* @param where - The `where` clause from `FindParams`.
* @returns Whether the entity satisfies every clause.
* @throws UnsupportedWhereOperatorError for unrecognized operators.
*/
export function whereMatches(entity: Entity, where: Record<string, unknown>): boolean {
for (const [field, condition] of Object.entries(where)) {
if (field === 'allOf') {
if (!Array.isArray(condition)) return false
if (!condition.every((sub) => whereMatches(entity, sub as Record<string, unknown>))) return false
continue
}
if (field === 'anyOf') {
if (!Array.isArray(condition)) return false
if (!condition.some((sub) => whereMatches(entity, sub as Record<string, unknown>))) return false
continue
}
if (field === 'not') {
if (whereMatches(entity, condition as Record<string, unknown>)) return false
continue
}
if (!fieldConditionMatches(resolveEntityField(entity, field), condition)) return false
}
return true
}
/**
* @description Evaluate the metadata-level portions of a `find()` query
* (`type`, `subtype`, `where`, `service`, `excludeVFS`) against one resolved
* entity. Used by historical and speculative `find()` to decide whether a
* changed/overlaid entity belongs in the result set. The caller guarantees
* the query carries no index-only dimensions (semantic `query`/`vector`,
feat(8.0): full query surface at historical generations via ephemeral index materialization Historical Db values (now()/asOf() pins that history has moved past) now serve the COMPLETE query surface - vector/hybrid search, graph traversal, cursor pagination, and aggregation - by materializing ephemeral in-memory indexes over the exact at-generation record set. The historical-query throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted. Materializer (Brainy.materializeAtGeneration): - Copies the at-G record set (live bytes for ids untouched since the pin, immutable before-images otherwise) into a fresh MemoryStorage; a final reconciliation pass under the commit mutex makes the copy exact even when transactions commit mid-build. - Opens a read-only Brainy over the copy: init rebuilds the metadata and graph-adjacency indexes from the records; the vector index is built by inserting every at-G vector (the at-G HNSW graph never existed on disk, so there is nothing to restore). Host embedder and aggregate definitions are shared - no second model load, aggregates backfill at-G values. - Cost is the documented contract: O(n at G) time and memory, ONCE per Db (handle cached; freed by release(), with a FinalizationRegistry backstop that also closes leaked readers). A native VersionedIndexProvider serves the same reads from retained segments with no rebuild. Db routing (src/db/db.ts): metadata-level find()/related() keep the free record path; index-only dimensions (query/vector/near/connected/cursor/ aggregate/includeRelations/non-metadata modes) route to the cached materialization; unsupported where-operators on the record path re-route there too instead of erroring. Speculative with() overlays keep the one honest boundary - SpeculativeOverlayError (overlay entities carry no embeddings, so index reads over them would be silently incomplete); metadata find()/get()/filter related() work on overlays. UpdateParams.vector contract now honored: an explicit pre-computed vector applies directly (with dimension validation) in update() and transact update ops, re-indexing HNSW - previously it was silently ignored unless data also changed. GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees filtered through the live-verb tombstone set (entity->entity edge trees deleted - they carried no verb ids, so removeVerb could never tombstone them and traversal served stale neighbors forever). Neighbor reads batch- load live verbs via the unified cache; addVerb seeds the cache. Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector search finds old vector placement including since-deleted entities; historical graph traversal walks the old wiring after a rewire; historical aggregation computes at-G group values; asOf() pins get the same surface; the materialization builds once per Db and release() closes the ephemeral reader (it refuses reads afterwards); overlays throw the documented error. ADR-001 updated to the no-throws historical model.
2026-06-11 08:12:11 -07:00
* `connected` traversal, ) those are routed to the at-generation index
* materialization (historical) or rejected with `SpeculativeOverlayError`
* (overlays) before evaluation starts.
feat(8.0): generational MVCC storage + Datomic-style Db API (now/transact/asOf/with/persist) One mechanism replaces the COW and versioning subsystems: immutable generation-stamped records behind a Datomic-style database value. Record layer (src/db/generationStore.ts): - Monotonic generation counter in _system/generation.json; bumped once per transact() commit and once per single-operation write (storage hook), so brain.generation() is always a meaningful watermark. - Commit protocol: stage before-images + tx.json delta -> fsync -> execute batch via TransactionManager -> atomic tmp+rename of _system/manifest.json (the rename IS the commit point) -> append _system/tx-log.jsonl. - Crash recovery on open rolls uncommitted generations back byte-identically and forces an index rebuild; refcounted pins gate compactHistory(), which records a horizon (asOf below it throws GenerationCompactedError). Db API: - Db: get/find/search/related pinned at a generation, with() speculative overlays, since() diffs, persist() hard-link-farm snapshots, timestamp, generation, release() + FinalizationRegistry backstop. - Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch (GenerationConflictError CAS), asOf(generation|Date|path), restore(path, {confirm}), compactHistory(), generation(), static open() + load(). - get()/metadata find()/related() are fully correct at any reachable pinned generation; index-accelerated queries at historical generations throw NotYetSupportedAtHistoricalGenerationError - never silently-wrong results. - VersionedIndexProvider (generation/isGenerationVisible/pin/release) in plugin.ts: feature-detected, balanced pin/release in lockstep with Db lifecycle, post-commit applier + replay-gap model documented. Storage primitives (BaseStorage + filesystem/memory adapters): raw-object read/write/list/remove, fsync barrier, noun/verb raw before-image capture, tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and append-in-place exceptions), restoreFromDirectory + derived-state reload. Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation across 200 mutations, batch atomicity under injected execution failure, ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety under pins, with() overlay isolation, generation monotonicity across reopen, crash consistency through the real recovery path, and versioned provider pin/release balance. Design record in docs/ADR-001-generational-mvcc.md.
2026-06-10 14:14:07 -07:00
*
* @param entity - The resolved entity.
* @param params - The metadata-level find parameters.
* @returns Whether the entity matches every requested filter.
* @throws UnsupportedWhereOperatorError for unrecognized `where` operators.
*/
export function entityMatchesFind(entity: Entity, params: FindParams): boolean {
if (params.type !== undefined) {
const types = Array.isArray(params.type) ? params.type : [params.type]
if (!types.includes(entity.type)) return false
}
if (params.subtype !== undefined) {
const subtypes = Array.isArray(params.subtype) ? params.subtype : [params.subtype]
if (entity.subtype === undefined || !subtypes.includes(entity.subtype)) return false
}
if (params.service !== undefined && entity.service !== params.service) {
return false
}
if (params.excludeVFS === true) {
// Mirror of find()'s exclusion filter (`vfsType: { exists: false }`,
// `isVFSEntity: { ne: true }`) and Brainy's VFS-marker helper.
const metadata = (entity.metadata ?? {}) as Record<string, unknown>
if (metadata.vfsType !== undefined) return false
if (metadata.isVFSEntity === true || metadata.isVFS === true) return false
}
if (params.where !== undefined) {
if (!whereMatches(entity, params.where as Record<string, unknown>)) return false
}
return true
}