feat(namespace): merge the field-addressing law train — no special names, system.* scalars, nested-bag storage, epoch-3 index keys
This commit is contained in:
commit
19b477aec0
40 changed files with 2988 additions and 2044 deletions
232
docs/concepts/field-addressing.md
Normal file
232
docs/concepts/field-addressing.md
Normal file
|
|
@ -0,0 +1,232 @@
|
||||||
|
---
|
||||||
|
title: Field addressing: your fields and system fields
|
||||||
|
slug: concepts/field-addressing
|
||||||
|
public: true
|
||||||
|
category: concepts
|
||||||
|
template: concept
|
||||||
|
order: 7
|
||||||
|
description: The one rule for every query-surface field name — a bare name always means your metadata, system.<field> reaches the ten engine scalars explicitly, and anything else refuses by name.
|
||||||
|
next:
|
||||||
|
- concepts/consistency-model
|
||||||
|
---
|
||||||
|
|
||||||
|
# Field addressing: your fields and system fields
|
||||||
|
|
||||||
|
Every query surface in Brainy — `find()`'s `where`, `orderBy`, aggregation
|
||||||
|
`groupBy`, and aggregation `source.where` — resolves field names by one rule,
|
||||||
|
with no exceptions:
|
||||||
|
|
||||||
|
> **A bare field name always means your metadata. `system.<field>` reaches an
|
||||||
|
> engine scalar, and only when you spell it explicitly.**
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
await brain.find({ orderBy: 'level' }) // reads entity.metadata.level — YOUR field
|
||||||
|
await brain.find({ orderBy: 'system.createdAt' }) // reads the engine's createdAt scalar
|
||||||
|
await brain.find({ orderBy: 'metadata.level' }) // identical to bare 'level' — explicit scope
|
||||||
|
```
|
||||||
|
|
||||||
|
There is no priority list, no "try the system field, fall back to metadata"
|
||||||
|
behavior, and no name that resolves differently depending on what else
|
||||||
|
happens to exist on your entities. A field called `level`, `score`,
|
||||||
|
`createdAt`, or `type` in your own `metadata` is read as *your* field, every
|
||||||
|
time, by its bare name.
|
||||||
|
|
||||||
|
## Why this rule exists
|
||||||
|
|
||||||
|
An internal report from a production deployment found that a user metadata
|
||||||
|
field literally named `level` was being silently shadowed by the engine's
|
||||||
|
own internal index layer field of the same name — every sort by `level`
|
||||||
|
returned insertion order, with no error raised. This rule makes that class of
|
||||||
|
bug structurally impossible: bare names belong to you, unconditionally, and
|
||||||
|
anything that isn't yours has to be spelled out.
|
||||||
|
|
||||||
|
## The system scalars
|
||||||
|
|
||||||
|
`system.<field>` addresses exactly ten scalars on an entity — no more, no
|
||||||
|
fewer:
|
||||||
|
|
||||||
|
| System field | What it is |
|
||||||
|
|---|---|
|
||||||
|
| `system.id` | The entity's id |
|
||||||
|
| `system.type` | The entity's `NounType` |
|
||||||
|
| `system.subtype` | The per-app sub-classification passed to `add()` |
|
||||||
|
| `system.createdAt` | When the entity was created |
|
||||||
|
| `system.updatedAt` | When the entity was last written |
|
||||||
|
| `system.confidence` | The `confidence` param (0–1) |
|
||||||
|
| `system.weight` | The `weight` param |
|
||||||
|
| `system.visibility` | `'public'` / `'internal'` (see the visibility tiers in [Consistency Model](./consistency-model.md)) |
|
||||||
|
| `system.service` | The multi-tenancy `service` tag |
|
||||||
|
| `system.createdBy` | Who/what created the entity |
|
||||||
|
|
||||||
|
Relationships mirror the same eight shared scalars (`subtype`, `createdAt`,
|
||||||
|
`updatedAt`, `confidence`, `weight`, `visibility`, `service`, `createdBy`)
|
||||||
|
plus three of their own:
|
||||||
|
|
||||||
|
| System field (relationship) | What it is |
|
||||||
|
|---|---|
|
||||||
|
| `system.verb` | The relationship's `VerbType` |
|
||||||
|
| `system.sourceId` | The id of the entity the relationship starts from |
|
||||||
|
| `system.targetId` | The id of the entity the relationship points to |
|
||||||
|
|
||||||
|
Anything not on these two lists is not a system scalar — `system.<name>` for
|
||||||
|
any other name refuses (see "Refusal semantics" below), even if that name
|
||||||
|
sounds like it should be engine-owned.
|
||||||
|
|
||||||
|
## Invisible plumbing — never addressable, in either spelling
|
||||||
|
|
||||||
|
Five names are pure engine internals. They are not reachable as a bare name,
|
||||||
|
and not reachable as `system.<name>` either — they simply have no place on
|
||||||
|
the query surface:
|
||||||
|
|
||||||
|
- **`vector`** — the stored embedding. It participates in similarity search
|
||||||
|
(`query`, `near`, vector `find()`), never in `where`/`orderBy`/`groupBy`.
|
||||||
|
- **`connections`** — graph adjacency. Reached through `connected` and
|
||||||
|
`brain.related()`, not through field addressing.
|
||||||
|
- **`level`** — the internal index layer number used by the nearest-neighbor
|
||||||
|
graph. It is pure index plumbing with no query-surface meaning at all —
|
||||||
|
which is exactly why a user field of the same name must never be shadowed
|
||||||
|
by it. `level` as a bare name is always yours; there is no engine-owned
|
||||||
|
spelling of it to compete with.
|
||||||
|
- **`data`** — your entity's content payload, not a scalar. It can be a
|
||||||
|
string, a number, or an arbitrary object, so sorting or filtering it as a
|
||||||
|
single comparable value would lie about its actual shape. Content is
|
||||||
|
reached through the content/text-search APIs (`query`, `searchMode:
|
||||||
|
'text'`), not through `where`/`orderBy`.
|
||||||
|
- **`_rev`** — the per-entity revision counter used for optimistic
|
||||||
|
concurrency (`ifRev`). It is a CAS token, not a queryable dimension.
|
||||||
|
|
||||||
|
`system.level`, `system.vector`, and `system.data` all refuse for the same
|
||||||
|
reason: they are not in the ten-scalar system map, full stop.
|
||||||
|
|
||||||
|
## `metadata.<field>` — the explicit spelling of "mine"
|
||||||
|
|
||||||
|
Prefix any field with `metadata.` to say the same thing a bare name already
|
||||||
|
says, spelled out. The two are interchangeable everywhere a field name is
|
||||||
|
accepted, including `orderBy`:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
await brain.find({ where: { 'customer.tier': 'gold' } })
|
||||||
|
await brain.find({ where: { 'metadata.customer.tier': 'gold' } }) // identical
|
||||||
|
await brain.find({ orderBy: 'metadata.score', order: 'desc' }) // identical to orderBy: 'score'
|
||||||
|
```
|
||||||
|
|
||||||
|
Reach for the explicit spelling when it reads more clearly next to a
|
||||||
|
`system.` field in the same query — for example, sorting by your own `score`
|
||||||
|
while filtering on `system.confidence`.
|
||||||
|
|
||||||
|
## No special names — the write side
|
||||||
|
|
||||||
|
The same law governs writes:
|
||||||
|
|
||||||
|
> **Data is either in main space, where developers can use anything, or it
|
||||||
|
> is in `system.*`.**
|
||||||
|
|
||||||
|
There are **no reserved metadata names**. A field called `confidence`,
|
||||||
|
`type`, `id`, `data`, `content`, or anything else inside your `metadata` bag
|
||||||
|
is an ordinary user field: it is stored verbatim, indexed, filterable,
|
||||||
|
sortable, aggregatable, and it survives restarts, index rebuilds, and
|
||||||
|
time-travel (`asOf`) reads exactly as written — even when an engine scalar
|
||||||
|
shares its spelling. The engine's values are written only through their
|
||||||
|
dedicated params (`confidence`, `weight`, `subtype`, `visibility`, …) and
|
||||||
|
read at `system.<field>`; your bag can never touch them and they can never
|
||||||
|
shadow your bag.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const id = await brain.add({
|
||||||
|
data: 'Ada Lovelace',
|
||||||
|
type: NounType.Person,
|
||||||
|
confidence: 0.9, // the ENGINE scalar
|
||||||
|
metadata: { confidence: 'self-rated' } // YOUR field, same spelling — both live
|
||||||
|
})
|
||||||
|
|
||||||
|
await brain.find({ where: { confidence: 'self-rated' } }) // finds it (yours)
|
||||||
|
await brain.find({ where: { 'system.confidence': 0.9 } }) // finds it (engine's)
|
||||||
|
```
|
||||||
|
|
||||||
|
The one spelling a write refuses is a metadata key that literally starts
|
||||||
|
with `system.` — the explicit address namespace cannot be forged as a user
|
||||||
|
field name. That refusal is typed and names the fix.
|
||||||
|
|
||||||
|
Value **shape** rules still apply uniformly to every name (they are not name
|
||||||
|
carve-outs): arrays longer than 10 elements are not turned into posting-list
|
||||||
|
scalars, and very long values are indexed by hash.
|
||||||
|
|
||||||
|
## Refusal semantics
|
||||||
|
|
||||||
|
A name that resolves to neither your metadata nor a system scalar is a typed
|
||||||
|
refusal, not a silent empty result and not a guess. Refusals name **both**
|
||||||
|
candidates, so the fix is always in the error text:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
await brain.find({ orderBy: 'createdAt' })
|
||||||
|
// UnresolvableFieldError: no metadata field 'createdAt' — did you mean
|
||||||
|
// system.createdAt or metadata.createdAt?
|
||||||
|
```
|
||||||
|
|
||||||
|
`UnresolvableFieldError` is exported from the package root:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { UnresolvableFieldError } from '@soulcraft/brainy'
|
||||||
|
|
||||||
|
try {
|
||||||
|
await brain.find({ orderBy: 'createdAt' })
|
||||||
|
} catch (err) {
|
||||||
|
if (err instanceof UnresolvableFieldError) {
|
||||||
|
// err.message names both candidates — usually enough to fix the call site.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
A handful of `find()` options are not implemented yet: `cursor`,
|
||||||
|
`includeRelations`, and `writeOnly`. Rather than accepting them and quietly
|
||||||
|
ignoring the option, `find()` refuses with `UnsupportedFindOptionError` —
|
||||||
|
also exported from the package root — so a call site can never believe an
|
||||||
|
unimplemented option took effect when it didn't.
|
||||||
|
|
||||||
|
## The ordering contract
|
||||||
|
|
||||||
|
`orderBy` behaves identically regardless of which engine (the pure-TypeScript
|
||||||
|
path or a native accelerator) is serving the query:
|
||||||
|
|
||||||
|
- An entity missing the `orderBy` field, or holding `null` on it, sorts
|
||||||
|
**LAST — in both `asc` and `desc`**. It is never treated as "smaller than
|
||||||
|
everything" in one direction and "larger than everything" in the other; it
|
||||||
|
is simply last, either way.
|
||||||
|
- Rows are **never dropped** from an ordered read because they lack the
|
||||||
|
field — a missing value changes position, never presence.
|
||||||
|
- Ties on the `orderBy` field break by **id ascending**, regardless of the
|
||||||
|
primary sort direction.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// employees: [{ score: 9 }, { score: 5 }, { /* no score field */ }]
|
||||||
|
await brain.find({ orderBy: 'score', order: 'desc' }) // [9, 5, missing] — missing is last
|
||||||
|
await brain.find({ orderBy: 'score', order: 'asc' }) // [5, 9, missing] — missing is STILL last
|
||||||
|
```
|
||||||
|
|
||||||
|
## Migrating existing call sites
|
||||||
|
|
||||||
|
If you have call sites written before this rule shipped that rely on a bare
|
||||||
|
system name — `orderBy: 'createdAt'`, `where: { confidence: { greaterThan:
|
||||||
|
0.8 } }`, and similar — they now refuse instead of silently resolving to the
|
||||||
|
engine field. The fix is always in the error: swap the bare name for
|
||||||
|
`system.<field>` (or `metadata.<field>` if you actually meant your own field
|
||||||
|
of that name, and it happens to share a name with a system scalar):
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Before: bare 'createdAt' silently meant the engine's timestamp.
|
||||||
|
await brain.find({ orderBy: 'createdAt' })
|
||||||
|
|
||||||
|
// After: say which one you meant.
|
||||||
|
await brain.find({ orderBy: 'system.createdAt' }) // the engine timestamp
|
||||||
|
await brain.find({ orderBy: 'metadata.createdAt' }) // your own field named createdAt, if you have one
|
||||||
|
```
|
||||||
|
|
||||||
|
There is no silent migration path by design — every ambiguous call site
|
||||||
|
surfaces as a refusal naming its own fix, once, the first time it runs
|
||||||
|
against the new rule.
|
||||||
|
|
||||||
|
## Where to go next
|
||||||
|
|
||||||
|
- [Consistency Model](./consistency-model.md) — visibility tiers, revision
|
||||||
|
counters, and the rest of the read/write contract this page's
|
||||||
|
read-time addressing rule.
|
||||||
|
|
@ -14,7 +14,22 @@
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { StorageAdapter, HNSWNounWithMetadata } from '../coreTypes.js'
|
import type { StorageAdapter, HNSWNounWithMetadata } from '../coreTypes.js'
|
||||||
import { resolveEntityField } from '../coreTypes.js'
|
import { parseFieldAddress, readEntityFieldAddress } from '../db/fieldAddressing.js'
|
||||||
|
import type { HNSWNounWithMetadata as AddressedEntity } from '../coreTypes.js'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Read a user-supplied field name under the one addressing law (sealed
|
||||||
|
* 2026-08-03): bare / `metadata.` = the user's metadata field, `system.<x>` =
|
||||||
|
* the ruled engine scalar, malformed = typed refusal. The aggregation engine
|
||||||
|
* NEVER resolves names any other way — the pre-law resolver made bare
|
||||||
|
* `subtype`/`confidence` read engine scalars, silently shadowing user fields.
|
||||||
|
*/
|
||||||
|
function readAddressed(e: unknown, name: string): unknown {
|
||||||
|
return readEntityFieldAddress(
|
||||||
|
e as AddressedEntity,
|
||||||
|
parseFieldAddress(name, 'entity')
|
||||||
|
)
|
||||||
|
}
|
||||||
import type {
|
import type {
|
||||||
AggregateDefinition,
|
AggregateDefinition,
|
||||||
AggregateGroupState,
|
AggregateGroupState,
|
||||||
|
|
@ -95,11 +110,15 @@ function matchesSource(entity: Record<string, unknown>, source: AggregateDefinit
|
||||||
// live in the custom bag, so those filters could never match anything.
|
// live in the custom bag, so those filters could never match anything.
|
||||||
if (source.where && Object.keys(source.where).length > 0) {
|
if (source.where && Object.keys(source.where).length > 0) {
|
||||||
const e = entity as unknown as HNSWNounWithMetadata
|
const e = entity as unknown as HNSWNounWithMetadata
|
||||||
const resolved: Record<string, unknown> = {}
|
for (const [key, condition] of Object.entries(source.where)) {
|
||||||
for (const key of Object.keys(source.where)) {
|
// Evaluate ONE field at a time under a neutral key: the address may be
|
||||||
resolved[key] = resolveEntityField(e, key)
|
// dotted ('system.subtype'), and the filter evaluator would otherwise
|
||||||
|
// walk dots as a nested path instead of treating the key as an address.
|
||||||
|
const value = readAddressed(e, key)
|
||||||
|
if (!matchesMetadataFilter({ v: value }, { v: condition } as Record<string, unknown>)) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if (!matchesMetadataFilter(resolved, source.where)) return false
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return true
|
return true
|
||||||
|
|
@ -129,11 +148,11 @@ function computeGroupKeys(
|
||||||
|
|
||||||
for (const dim of groupBy) {
|
for (const dim of groupBy) {
|
||||||
if (typeof dim === 'string') {
|
if (typeof dim === 'string') {
|
||||||
const val = resolveEntityField(e, dim)
|
const val = readAddressed(e, dim)
|
||||||
const v = val !== undefined && val !== null ? String(val) : '__null__'
|
const v = val !== undefined && val !== null ? String(val) : '__null__'
|
||||||
for (const k of keys) k[dim] = v
|
for (const k of keys) k[dim] = v
|
||||||
} else if ('unnest' in dim) {
|
} else if ('unnest' in dim) {
|
||||||
const val = resolveEntityField(e, dim.field)
|
const val = readAddressed(e, dim.field)
|
||||||
const raw = Array.isArray(val) ? val : val !== undefined && val !== null ? [val] : []
|
const raw = Array.isArray(val) ? val : val !== undefined && val !== null ? [val] : []
|
||||||
// Distinct elements: an entity with duplicate tags counts once per distinct tag.
|
// Distinct elements: an entity with duplicate tags counts once per distinct tag.
|
||||||
const elems = Array.from(new Set(raw.map(x => String(x))))
|
const elems = Array.from(new Set(raw.map(x => String(x))))
|
||||||
|
|
@ -145,7 +164,7 @@ function computeGroupKeys(
|
||||||
keys = next
|
keys = next
|
||||||
} else {
|
} else {
|
||||||
// Time-windowed field
|
// Time-windowed field
|
||||||
const val = resolveEntityField(e, dim.field)
|
const val = readAddressed(e, dim.field)
|
||||||
const v = typeof val === 'number' ? bucketTimestamp(val, dim.window) : '__null__'
|
const v = typeof val === 'number' ? bucketTimestamp(val, dim.window) : '__null__'
|
||||||
for (const k of keys) k[dim.field] = v
|
for (const k of keys) k[dim.field] = v
|
||||||
}
|
}
|
||||||
|
|
@ -174,7 +193,7 @@ function computeGroupKey(
|
||||||
* in metadata are both handled in one place.
|
* in metadata are both handled in one place.
|
||||||
*/
|
*/
|
||||||
function getNumericField(entity: Record<string, unknown>, field: string): number | undefined {
|
function getNumericField(entity: Record<string, unknown>, field: string): number | undefined {
|
||||||
const val = resolveEntityField(entity as unknown as HNSWNounWithMetadata, field)
|
const val = readAddressed(entity as unknown as HNSWNounWithMetadata, field)
|
||||||
if (typeof val === 'number' && !isNaN(val)) return val
|
if (typeof val === 'number' && !isNaN(val)) return val
|
||||||
if (typeof val === 'string') {
|
if (typeof val === 'string') {
|
||||||
const num = parseFloat(val)
|
const num = parseFloat(val)
|
||||||
|
|
@ -990,7 +1009,7 @@ export class AggregationIndex {
|
||||||
// distinctCount tracks distinct values of ANY type (strings, numbers, booleans),
|
// distinctCount tracks distinct values of ANY type (strings, numbers, booleans),
|
||||||
// keyed by their string form — NOT numeric-coerced, since its primary use is
|
// keyed by their string form — NOT numeric-coerced, since its primary use is
|
||||||
// categorical (distinct categories / users / tags), not numeric columns.
|
// categorical (distinct categories / users / tags), not numeric columns.
|
||||||
const raw = resolveEntityField(entity as unknown as HNSWNounWithMetadata, metricDef.field!)
|
const raw = readAddressed(entity as unknown as HNSWNounWithMetadata, metricDef.field!)
|
||||||
if (raw !== undefined && raw !== null) {
|
if (raw !== undefined && raw !== null) {
|
||||||
if (!state.valueCounts) state.valueCounts = {}
|
if (!state.valueCounts) state.valueCounts = {}
|
||||||
const key = String(raw)
|
const key = String(raw)
|
||||||
|
|
@ -1034,7 +1053,7 @@ export class AggregationIndex {
|
||||||
state.count = Math.max(0, state.count - 1)
|
state.count = Math.max(0, state.count - 1)
|
||||||
state.sum = Math.max(0, state.sum - 1)
|
state.sum = Math.max(0, state.sum - 1)
|
||||||
} else if (metricDef.op === 'distinctCount') {
|
} else if (metricDef.op === 'distinctCount') {
|
||||||
const raw = resolveEntityField(entity as unknown as HNSWNounWithMetadata, metricDef.field!)
|
const raw = readAddressed(entity as unknown as HNSWNounWithMetadata, metricDef.field!)
|
||||||
if (raw !== undefined && raw !== null && state.valueCounts) {
|
if (raw !== undefined && raw !== null && state.valueCounts) {
|
||||||
const key = String(raw)
|
const key = String(raw)
|
||||||
const c = state.valueCounts[key]
|
const c = state.valueCounts[key]
|
||||||
|
|
|
||||||
772
src/brainy.ts
772
src/brainy.ts
|
|
@ -148,7 +148,9 @@ import {
|
||||||
import { NounType, VerbType, TypeUtils } from './types/graphTypes.js'
|
import { NounType, VerbType, TypeUtils } from './types/graphTypes.js'
|
||||||
import {
|
import {
|
||||||
splitNounMetadataRecord,
|
splitNounMetadataRecord,
|
||||||
splitVerbMetadataRecord
|
splitVerbMetadataRecord,
|
||||||
|
buildNounMetadataRecord,
|
||||||
|
buildVerbMetadataRecord
|
||||||
} from './types/reservedFields.js'
|
} from './types/reservedFields.js'
|
||||||
import { BrainyInterface } from './types/brainyInterface.js'
|
import { BrainyInterface } from './types/brainyInterface.js'
|
||||||
import type { IntegrationHub, IntegrationHubConfig } from './integrations/core/IntegrationHub.js'
|
import type { IntegrationHub, IntegrationHubConfig } from './integrations/core/IntegrationHub.js'
|
||||||
|
|
@ -746,6 +748,21 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
private lazyRebuildPromise: Promise<void> | null = null
|
private lazyRebuildPromise: Promise<void> | null = null
|
||||||
|
|
||||||
constructor(config?: BrainyConfig) {
|
constructor(config?: BrainyConfig) {
|
||||||
|
// The reserved-field write policy died with the field-addressing law:
|
||||||
|
// every metadata name is the user's now (engine scalars write via their
|
||||||
|
// dedicated params and read at `system.*`), so there is nothing left for
|
||||||
|
// the policy to govern. A config still passing it refuses loudly rather
|
||||||
|
// than being silently ignored.
|
||||||
|
if (config && 'reservedFieldPolicy' in (config as Record<string, unknown>)) {
|
||||||
|
throw new Error(
|
||||||
|
`reservedFieldPolicy was removed by the field-addressing law: metadata field ` +
|
||||||
|
`names are never reserved anymore — every name in the metadata bag is the ` +
|
||||||
|
`user's and works like any other field. Set engine scalars via their ` +
|
||||||
|
`dedicated params (confidence, weight, subtype, …) and query them as ` +
|
||||||
|
`system.<field>. Remove the reservedFieldPolicy option.`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
// Normalize configuration with defaults
|
// Normalize configuration with defaults
|
||||||
this.config = this.normalizeConfig(config)
|
this.config = this.normalizeConfig(config)
|
||||||
|
|
||||||
|
|
@ -2018,12 +2035,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Zero-config validation (static import for performance)
|
// Zero-config validation (static import for performance)
|
||||||
validateAddParams(params)
|
validateAddParams(params)
|
||||||
|
|
||||||
// Reserved fields arriving via the metadata bag (untyped callers — the
|
|
||||||
// compile-time guard stops TypeScript callers) are normalized to their
|
|
||||||
// canonical top-level location BEFORE any enforcement runs, so a
|
|
||||||
// remapped subtype participates in subtype-pairing enforcement and the
|
|
||||||
// indexed metadata bag carries only custom fields.
|
|
||||||
params = this.remapReservedAddMetadata(params)
|
|
||||||
|
|
||||||
// Tracked-field vocabulary enforcement (Layer 2). Walks both bags so a
|
// Tracked-field vocabulary enforcement (Layer 2). Walks both bags so a
|
||||||
// tracked field declared at top level (e.g. 'subtype') and one declared in
|
// tracked field declared at top level (e.g. 'subtype') and one declared in
|
||||||
|
|
@ -2095,28 +2106,33 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Prepare metadata for storage
|
// Prepare metadata for storage: a v2 nested-bag record — engine fields
|
||||||
// data is stored opaquely in the 'data' field - NOT spread into top-level metadata.
|
// top-level, the user's bag nested VERBATIM (any name, including engine
|
||||||
// Only metadata fields are queryable via find({ where }).
|
// spellings like `confidence` or `type`, is the user's and survives
|
||||||
const storageMetadata = {
|
// faithfully; the field-addressing law).
|
||||||
...params.metadata,
|
const storageMetadata = buildNounMetadataRecord(
|
||||||
// Preserve the caller's original (non-UUID) id when normalized, so reads
|
{
|
||||||
// can surface it. A real UUID passes through with no _originalId.
|
data: params.data,
|
||||||
...(originalId !== undefined && { [ORIGINAL_ID_KEY]: originalId }),
|
noun: params.type,
|
||||||
data: params.data,
|
...(params.subtype !== undefined && { subtype: params.subtype }),
|
||||||
noun: params.type,
|
// visibility: stored only when not 'public' (absent === public, keeps records lean)
|
||||||
...(params.subtype !== undefined && { subtype: params.subtype }),
|
...(params.visibility !== undefined &&
|
||||||
// visibility: stored only when not 'public' (absent === public, keeps records lean)
|
params.visibility !== 'public' && { visibility: params.visibility }),
|
||||||
...(params.visibility !== undefined &&
|
service: params.service,
|
||||||
params.visibility !== 'public' && { visibility: params.visibility }),
|
createdAt: Date.now(),
|
||||||
service: params.service,
|
updatedAt: Date.now(),
|
||||||
createdAt: Date.now(),
|
_rev: 1,
|
||||||
updatedAt: Date.now(),
|
...(params.confidence !== undefined && { confidence: params.confidence }),
|
||||||
_rev: 1,
|
...(params.weight !== undefined && { weight: params.weight }),
|
||||||
...(params.confidence !== undefined && { confidence: params.confidence }),
|
...(params.createdBy && { createdBy: params.createdBy })
|
||||||
...(params.weight !== undefined && { weight: params.weight }),
|
},
|
||||||
...(params.createdBy && { createdBy: params.createdBy })
|
{
|
||||||
}
|
...params.metadata,
|
||||||
|
// Preserve the caller's original (non-UUID) id when normalized, so reads
|
||||||
|
// can surface it. A real UUID passes through with no _originalId.
|
||||||
|
...(originalId !== undefined && { [ORIGINAL_ID_KEY]: originalId })
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
// Build entity structure for indexing (NEW - with top-level fields)
|
// Build entity structure for indexing (NEW - with top-level fields)
|
||||||
// Optional fields must use conditional spreading to match storageMetadata exactly.
|
// Optional fields must use conditional spreading to match storageMetadata exactly.
|
||||||
|
|
@ -2627,320 +2643,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
return entity
|
return entity
|
||||||
}
|
}
|
||||||
|
|
||||||
/** One-shot registry for reserved-field warnings (per process, per method+field). */
|
|
||||||
private static warnedReservedFields = new Set<string>()
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Resolve the human-readable "correct write path" guidance for a
|
|
||||||
* reserved field on a given write method. Single source of truth shared by the
|
|
||||||
* `'throw'` (Error message) and `'warn'` (one-shot warning) paths so the two
|
|
||||||
* never drift. The trio `confidence` / `weight` / `subtype` and the
|
|
||||||
* add()/relate()-time fields `service` / `createdBy` / `visibility` map to a
|
|
||||||
* dedicated param; everything else is system-managed.
|
|
||||||
* @param method - The public write method the bag arrived through.
|
|
||||||
* @param field - The reserved field name found in the metadata bag.
|
|
||||||
* @returns Guidance naming the correct way to set the field.
|
|
||||||
*/
|
|
||||||
private reservedWritePath(
|
|
||||||
method: 'add' | 'update' | 'relate' | 'updateRelation',
|
|
||||||
field: string
|
|
||||||
): string {
|
|
||||||
const typeParam = "the top-level 'type' param"
|
|
||||||
switch (field) {
|
|
||||||
case 'noun':
|
|
||||||
case 'verb':
|
|
||||||
return typeParam
|
|
||||||
case 'data':
|
|
||||||
return "the top-level 'data' param"
|
|
||||||
case 'confidence':
|
|
||||||
return "the 'confidence' param"
|
|
||||||
case 'weight':
|
|
||||||
return "the 'weight' param"
|
|
||||||
case 'subtype':
|
|
||||||
return "the 'subtype' param"
|
|
||||||
case 'visibility':
|
|
||||||
return "the 'visibility' param ('public' | 'internal')"
|
|
||||||
case 'service':
|
|
||||||
return method === 'add'
|
|
||||||
? "the 'service' param of add()"
|
|
||||||
: method === 'relate'
|
|
||||||
? "the 'service' param of relate()"
|
|
||||||
: 'nothing — service is fixed at create time'
|
|
||||||
case 'createdBy':
|
|
||||||
return method === 'add'
|
|
||||||
? "the 'createdBy' param of add()"
|
|
||||||
: 'nothing — createdBy is system-managed'
|
|
||||||
case 'createdAt':
|
|
||||||
return 'nothing — creation time is set automatically'
|
|
||||||
case 'updatedAt':
|
|
||||||
return 'nothing — set automatically on every write'
|
|
||||||
case '_rev':
|
|
||||||
return method === 'update'
|
|
||||||
? "the 'ifRev' param for optimistic concurrency"
|
|
||||||
: 'nothing — revisions are system-managed'
|
|
||||||
default:
|
|
||||||
return 'a dedicated top-level param'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Enforce {@link BrainyConfig.reservedFieldPolicy} for reserved
|
|
||||||
* fields found inside a metadata bag. Called by every write-path remap once
|
|
||||||
* the bag has been split and at least one reserved key is present.
|
|
||||||
*
|
|
||||||
* - `'throw'` (default): throw a clear Error naming every offending key and
|
|
||||||
* its correct write path. The caller never reaches the remap.
|
|
||||||
* - `'warn'`: emit a ONE-SHOT (per method+field, per process) warning for
|
|
||||||
* EVERY reserved key found — both the user-mutable fields that are about to
|
|
||||||
* be remapped and the system-managed fields that are about to be dropped —
|
|
||||||
* then fall through to the legacy remap.
|
|
||||||
* - `'remap'`: silent legacy remap, no warning.
|
|
||||||
*
|
|
||||||
* @param method - The public write method the bag arrived through.
|
|
||||||
* @param reserved - The reserved half of the split metadata bag (non-empty).
|
|
||||||
* @param reservedListName - `'RESERVED_ENTITY_FIELDS'` or
|
|
||||||
* `'RESERVED_RELATION_FIELDS'` — named in the thrown Error for discoverability.
|
|
||||||
* @returns `true` when the caller should proceed with the legacy remap
|
|
||||||
* (`'warn'` / `'remap'`); `'throw'` never returns (it throws first).
|
|
||||||
* @throws {Error} When the policy is `'throw'` and any reserved key is present.
|
|
||||||
*/
|
|
||||||
private enforceReservedPolicy(
|
|
||||||
method: 'add' | 'update' | 'relate' | 'updateRelation',
|
|
||||||
reserved: Partial<Record<string, unknown>>,
|
|
||||||
reservedListName: 'RESERVED_ENTITY_FIELDS' | 'RESERVED_RELATION_FIELDS'
|
|
||||||
): boolean {
|
|
||||||
const policy = this.config.reservedFieldPolicy ?? 'throw'
|
|
||||||
const keys = Object.keys(reserved)
|
|
||||||
if (keys.length === 0) return true
|
|
||||||
|
|
||||||
if (policy === 'throw') {
|
|
||||||
const detail = keys
|
|
||||||
.map((k) => {
|
|
||||||
const path = this.reservedWritePath(method, k)
|
|
||||||
// System-managed fields resolve to a "nothing — …" sentinel; phrase
|
|
||||||
// those as "is system-managed" rather than "pass it as the nothing".
|
|
||||||
return path.startsWith('nothing')
|
|
||||||
? `metadata.${k} is a reserved field (${path.replace(/^nothing\s*—\s*/, '')}) and cannot be set through ${method}()`
|
|
||||||
: `metadata.${k} is a reserved field — pass it as ${path} to ${method}()`
|
|
||||||
})
|
|
||||||
.join('; ')
|
|
||||||
throw new Error(
|
|
||||||
`${detail} (reserved: see ${reservedListName}). ` +
|
|
||||||
`Set reservedFieldPolicy:'remap' to opt into legacy remapping, ` +
|
|
||||||
`or reservedFieldPolicy:'warn' to remap with a warning.`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
if (policy === 'warn') {
|
|
||||||
// One-shot warning for EVERY reserved key (today only system-managed ones
|
|
||||||
// warn — this closes that gap so user-mutable remaps are visible too).
|
|
||||||
for (const k of keys) {
|
|
||||||
this.warnReservedRemapped(method, k, this.reservedWritePath(method, k))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// 'warn' and 'remap' both fall through to the legacy remap.
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description One-shot (per method+field, per process) warning that a
|
|
||||||
* reserved field arrived inside a metadata bag under the `'warn'` policy. The
|
|
||||||
* wording is neutral on "remapped vs dropped" — `reservedWritePath()` already
|
|
||||||
* tells the caller where the value goes (a dedicated param, or "nothing").
|
|
||||||
* @param method - The public write method the bag arrived through.
|
|
||||||
* @param field - The reserved field name found in the bag.
|
|
||||||
* @param rightPath - Guidance naming the correct write path.
|
|
||||||
*/
|
|
||||||
private warnReservedRemapped(method: string, field: string, rightPath: string): void {
|
|
||||||
const key = `${method}:${field}`
|
|
||||||
if (Brainy.warnedReservedFields.has(key)) return
|
|
||||||
Brainy.warnedReservedFields.add(key)
|
|
||||||
// System-managed fields resolve to a "nothing — …" sentinel; phrase the
|
|
||||||
// guidance so it reads cleanly in both the remapped and dropped cases.
|
|
||||||
const guidance = rightPath.startsWith('nothing')
|
|
||||||
? `it is ${rightPath.replace(/^nothing\s*—\s*/, '')} and was dropped`
|
|
||||||
: `set it via ${rightPath} instead`
|
|
||||||
prodLog.warn(
|
|
||||||
`[brainy] ${method}(): '${field}' is a reserved field and was found inside the ` +
|
|
||||||
`metadata bag — ${guidance}. (Legacy remap applied because ` +
|
|
||||||
`reservedFieldPolicy is 'warn'. This warning is shown once per field per process.)`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Normalize an `add()` params object with respect to
|
|
||||||
* Brainy-reserved fields arriving inside `metadata` (untyped callers only —
|
|
||||||
* the compile-time guard on `AddParams.metadata` stops TypeScript callers).
|
|
||||||
* Governed by {@link BrainyConfig.reservedFieldPolicy} (default `'throw'`):
|
|
||||||
* `'throw'` rejects the write naming the offending key(s); `'warn'`/`'remap'`
|
|
||||||
* fall through to the legacy remap, where fields with a dedicated `add()`
|
|
||||||
* param (`confidence`, `weight`, `subtype`, `visibility`, `service`,
|
|
||||||
* `createdBy`) are remapped to that param unless the caller also passed it
|
|
||||||
* explicitly (top-level wins) and system-managed fields (`noun`, `data`,
|
|
||||||
* `createdAt`, `updatedAt`, `_rev`) are dropped. A remapped `subtype` flows
|
|
||||||
* through subtype-pairing enforcement exactly like a top-level one.
|
|
||||||
* @param params - The caller's add params (not mutated).
|
|
||||||
* @returns Params with reserved fields normalized out of `metadata`.
|
|
||||||
* @throws {Error} When `reservedFieldPolicy` is `'throw'` and the bag carries a reserved key.
|
|
||||||
*/
|
|
||||||
private remapReservedAddMetadata(params: AddParams<T>): AddParams<T> {
|
|
||||||
const bag = params.metadata as Record<string, unknown> | undefined
|
|
||||||
if (!bag || typeof bag !== 'object') return params
|
|
||||||
const { reserved, custom } = splitNounMetadataRecord(bag)
|
|
||||||
if (Object.keys(reserved).length === 0) return params
|
|
||||||
|
|
||||||
// Policy gate: 'throw' (default) throws here; 'warn' warns once per key then
|
|
||||||
// remaps; 'remap' silently remaps. (Throw never returns.)
|
|
||||||
this.enforceReservedPolicy('add', reserved, 'RESERVED_ENTITY_FIELDS')
|
|
||||||
|
|
||||||
const createdBy = reserved.createdBy as { augmentation?: unknown; version?: unknown } | undefined
|
|
||||||
const createdByValid =
|
|
||||||
typeof createdBy === 'object' &&
|
|
||||||
createdBy !== null &&
|
|
||||||
typeof createdBy.augmentation === 'string' &&
|
|
||||||
typeof createdBy.version === 'string'
|
|
||||||
|
|
||||||
return {
|
|
||||||
...params,
|
|
||||||
metadata: custom as AddParams<T>['metadata'],
|
|
||||||
...(params.confidence === undefined &&
|
|
||||||
typeof reserved.confidence === 'number' && { confidence: reserved.confidence }),
|
|
||||||
...(params.weight === undefined &&
|
|
||||||
typeof reserved.weight === 'number' && { weight: reserved.weight }),
|
|
||||||
...(params.subtype === undefined &&
|
|
||||||
typeof reserved.subtype === 'string' && { subtype: reserved.subtype }),
|
|
||||||
...(params.visibility === undefined &&
|
|
||||||
(reserved.visibility === 'public' || reserved.visibility === 'internal') && {
|
|
||||||
visibility: reserved.visibility as 'public' | 'internal'
|
|
||||||
}),
|
|
||||||
...(params.service === undefined &&
|
|
||||||
typeof reserved.service === 'string' && { service: reserved.service }),
|
|
||||||
...(params.createdBy === undefined &&
|
|
||||||
createdByValid && { createdBy: createdBy as { augmentation: string; version: string } })
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Normalize an `update()` params object with respect to
|
|
||||||
* Brainy-reserved fields arriving inside the metadata patch — the `update()`
|
|
||||||
* mirror of {@link remapReservedAddMetadata}, closing the historical trap
|
|
||||||
* where `add({metadata:{confidence}})` lifted the field but
|
|
||||||
* `update({metadata:{confidence}})` silently dropped it (the patch value
|
|
||||||
* survived the merge and was then clobbered by the preserve-existing
|
|
||||||
* spread; a production consumer's confidence-evolution writes no-oped until
|
|
||||||
* read back). Governed by {@link BrainyConfig.reservedFieldPolicy} (default
|
|
||||||
* `'throw'`): `'throw'` rejects the write; `'warn'`/`'remap'` remap
|
|
||||||
* user-mutable fields (`confidence`, `weight`, `subtype`) to their dedicated
|
|
||||||
* param unless the caller also passed it (top-level wins) and drop everything
|
|
||||||
* else (`noun`, `data`, `createdAt`, `updatedAt`, `service`, `createdBy`,
|
|
||||||
* `_rev`) as system-managed or fixed at `add()` time.
|
|
||||||
* @param params - The caller's update params (not mutated).
|
|
||||||
* @returns Params with reserved fields normalized out of `metadata`.
|
|
||||||
* @throws {Error} When `reservedFieldPolicy` is `'throw'` and the bag carries a reserved key.
|
|
||||||
*/
|
|
||||||
private remapReservedUpdateMetadata(params: UpdateParams<T>): UpdateParams<T> {
|
|
||||||
const bag = params.metadata as Record<string, unknown> | undefined
|
|
||||||
if (!bag || typeof bag !== 'object') return params
|
|
||||||
const { reserved, custom } = splitNounMetadataRecord(bag)
|
|
||||||
if (Object.keys(reserved).length === 0) return params
|
|
||||||
|
|
||||||
// Policy gate: 'throw' (default) throws; 'warn' warns once per key then
|
|
||||||
// remaps; 'remap' silently remaps.
|
|
||||||
this.enforceReservedPolicy('update', reserved, 'RESERVED_ENTITY_FIELDS')
|
|
||||||
|
|
||||||
return {
|
|
||||||
...params,
|
|
||||||
metadata: custom as UpdateParams<T>['metadata'],
|
|
||||||
...(params.confidence === undefined &&
|
|
||||||
typeof reserved.confidence === 'number' && { confidence: reserved.confidence }),
|
|
||||||
...(params.weight === undefined &&
|
|
||||||
typeof reserved.weight === 'number' && { weight: reserved.weight }),
|
|
||||||
...(params.subtype === undefined &&
|
|
||||||
typeof reserved.subtype === 'string' && { subtype: reserved.subtype })
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Normalize a `relate()` params object with respect to
|
|
||||||
* Brainy-reserved fields arriving inside `metadata` — the relationship
|
|
||||||
* mirror of {@link remapReservedAddMetadata}. Governed by
|
|
||||||
* {@link BrainyConfig.reservedFieldPolicy} (default `'throw'`): `'throw'`
|
|
||||||
* rejects the write; `'warn'`/`'remap'` remap fields with a dedicated
|
|
||||||
* `relate()` param (`confidence`, `weight`, `subtype`, `visibility`,
|
|
||||||
* `service`) to that param (top-level wins) and drop system-managed fields
|
|
||||||
* (`verb`, `data`, `createdAt`, `updatedAt`, `createdBy`, `_rev`).
|
|
||||||
* @param params - The caller's relate params (not mutated).
|
|
||||||
* @returns Params with reserved fields normalized out of `metadata`.
|
|
||||||
* @throws {Error} When `reservedFieldPolicy` is `'throw'` and the bag carries a reserved key.
|
|
||||||
*/
|
|
||||||
private remapReservedRelateMetadata(params: RelateParams<T>): RelateParams<T> {
|
|
||||||
const bag = params.metadata as Record<string, unknown> | undefined
|
|
||||||
if (!bag || typeof bag !== 'object') return params
|
|
||||||
const { reserved, custom } = splitVerbMetadataRecord(bag)
|
|
||||||
if (Object.keys(reserved).length === 0) return params
|
|
||||||
|
|
||||||
// Policy gate: 'throw' (default) throws; 'warn' warns once per key then
|
|
||||||
// remaps; 'remap' silently remaps.
|
|
||||||
this.enforceReservedPolicy('relate', reserved, 'RESERVED_RELATION_FIELDS')
|
|
||||||
|
|
||||||
return {
|
|
||||||
...params,
|
|
||||||
metadata: custom as RelateParams<T>['metadata'],
|
|
||||||
...(params.confidence === undefined &&
|
|
||||||
typeof reserved.confidence === 'number' && { confidence: reserved.confidence }),
|
|
||||||
...(params.weight === undefined &&
|
|
||||||
typeof reserved.weight === 'number' && { weight: reserved.weight }),
|
|
||||||
...(params.subtype === undefined &&
|
|
||||||
typeof reserved.subtype === 'string' && { subtype: reserved.subtype }),
|
|
||||||
...(params.visibility === undefined &&
|
|
||||||
(reserved.visibility === 'public' || reserved.visibility === 'internal') && {
|
|
||||||
visibility: reserved.visibility as 'public' | 'internal'
|
|
||||||
}),
|
|
||||||
...(params.service === undefined &&
|
|
||||||
typeof reserved.service === 'string' && { service: reserved.service })
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Normalize an `updateRelation()` params object with respect
|
|
||||||
* to Brainy-reserved fields arriving inside the metadata patch — the
|
|
||||||
* relationship mirror of {@link remapReservedUpdateMetadata}. Governed by
|
|
||||||
* {@link BrainyConfig.reservedFieldPolicy} (default `'throw'`): `'throw'`
|
|
||||||
* rejects the write; `'warn'`/`'remap'` remap user-mutable fields
|
|
||||||
* (`confidence`, `weight`, `subtype`, `visibility`) to their dedicated param
|
|
||||||
* (top-level wins) and drop everything else.
|
|
||||||
* @param params - The caller's update-relation params (not mutated).
|
|
||||||
* @returns Params with reserved fields normalized out of `metadata`.
|
|
||||||
* @throws {Error} When `reservedFieldPolicy` is `'throw'` and the bag carries a reserved key.
|
|
||||||
*/
|
|
||||||
private remapReservedUpdateRelationMetadata(
|
|
||||||
params: UpdateRelationParams<T>
|
|
||||||
): UpdateRelationParams<T> {
|
|
||||||
const bag = params.metadata as Record<string, unknown> | undefined
|
|
||||||
if (!bag || typeof bag !== 'object') return params
|
|
||||||
const { reserved, custom } = splitVerbMetadataRecord(bag)
|
|
||||||
if (Object.keys(reserved).length === 0) return params
|
|
||||||
|
|
||||||
// Policy gate: 'throw' (default) throws; 'warn' warns once per key then
|
|
||||||
// remaps; 'remap' silently remaps.
|
|
||||||
this.enforceReservedPolicy('updateRelation', reserved, 'RESERVED_RELATION_FIELDS')
|
|
||||||
|
|
||||||
return {
|
|
||||||
...params,
|
|
||||||
metadata: custom as UpdateRelationParams<T>['metadata'],
|
|
||||||
...(params.confidence === undefined &&
|
|
||||||
typeof reserved.confidence === 'number' && { confidence: reserved.confidence }),
|
|
||||||
...(params.weight === undefined &&
|
|
||||||
typeof reserved.weight === 'number' && { weight: reserved.weight }),
|
|
||||||
...(params.subtype === undefined &&
|
|
||||||
typeof reserved.subtype === 'string' && { subtype: reserved.subtype }),
|
|
||||||
...(params.visibility === undefined &&
|
|
||||||
(reserved.visibility === 'public' || reserved.visibility === 'internal') && {
|
|
||||||
visibility: reserved.visibility as 'public' | 'internal'
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Update an existing entity
|
* Update an existing entity
|
||||||
|
|
@ -3006,12 +2708,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Reserved fields arriving via the metadata patch are remapped to their
|
// Reserved fields arriving via the metadata patch are remapped to their
|
||||||
// canonical top-level location, mirroring add()'s lift. Without this the
|
// canonical top-level location, mirroring add()'s lift. Without this the
|
||||||
// patch value survived the merge but was then clobbered by the
|
// patch value survived the merge but was then clobbered by the
|
||||||
// preserve-existing spreads below — a silent no-op consumers could only
|
|
||||||
// detect by reading values back. User-mutable fields (confidence,
|
|
||||||
// weight, subtype) remap unless the same field was also passed top-level
|
|
||||||
// (top-level wins); system-managed fields are dropped with a one-shot
|
|
||||||
// warning naming the right path.
|
|
||||||
params = this.remapReservedUpdateMetadata(params)
|
|
||||||
|
|
||||||
// Tracked-field vocabulary enforcement (Layer 2). Same as add() — the
|
// Tracked-field vocabulary enforcement (Layer 2). Same as add() — the
|
||||||
// metadata bag carries fields registered via trackField(), and subtype is
|
// metadata bag carries fields registered via trackField(), and subtype is
|
||||||
|
|
@ -3078,31 +2774,33 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
? { ...existing.metadata, ...params.metadata }
|
? { ...existing.metadata, ...params.metadata }
|
||||||
: params.metadata || existing.metadata
|
: params.metadata || existing.metadata
|
||||||
|
|
||||||
// Prepare updated metadata object
|
// Prepare the updated v2 nested-bag record: engine fields top-level,
|
||||||
// data is stored opaquely in the 'data' field - NOT spread into top-level metadata.
|
// the merged user bag nested verbatim (collider names stay the user's).
|
||||||
const updatedMetadata = {
|
const updatedMetadata = buildNounMetadataRecord(
|
||||||
...newMetadata,
|
{
|
||||||
data: params.data !== undefined ? params.data : existing.data,
|
data: params.data !== undefined ? params.data : existing.data,
|
||||||
noun: params.type || existing.type,
|
noun: params.type || existing.type,
|
||||||
service: existing.service,
|
service: existing.service,
|
||||||
createdAt: existing.createdAt,
|
createdAt: existing.createdAt,
|
||||||
updatedAt: Date.now(),
|
updatedAt: Date.now(),
|
||||||
_rev: currentRev + 1,
|
_rev: currentRev + 1,
|
||||||
// Update confidence and weight if provided, otherwise preserve existing
|
// Update confidence and weight if provided, otherwise preserve existing
|
||||||
...(params.confidence !== undefined && { confidence: params.confidence }),
|
...(params.confidence !== undefined && { confidence: params.confidence }),
|
||||||
...(params.weight !== undefined && { weight: params.weight }),
|
...(params.weight !== undefined && { weight: params.weight }),
|
||||||
...(params.confidence === undefined && existing.confidence !== undefined && { confidence: existing.confidence }),
|
...(params.confidence === undefined && existing.confidence !== undefined && { confidence: existing.confidence }),
|
||||||
...(params.weight === undefined && existing.weight !== undefined && { weight: existing.weight }),
|
...(params.weight === undefined && existing.weight !== undefined && { weight: existing.weight }),
|
||||||
// Update subtype if provided, otherwise preserve existing
|
// Update subtype if provided, otherwise preserve existing
|
||||||
...(params.subtype !== undefined && { subtype: params.subtype }),
|
...(params.subtype !== undefined && { subtype: params.subtype }),
|
||||||
...(params.subtype === undefined && existing.subtype !== undefined && { subtype: existing.subtype }),
|
...(params.subtype === undefined && existing.subtype !== undefined && { subtype: existing.subtype }),
|
||||||
// Visibility: take the new value if provided, else preserve existing. Stored only
|
// Visibility: take the new value if provided, else preserve existing. Stored only
|
||||||
// when the effective value is not 'public' (absent === public, keeps records lean).
|
// when the effective value is not 'public' (absent === public, keeps records lean).
|
||||||
// A change to 'public' therefore drops the field entirely.
|
// A change to 'public' therefore drops the field entirely.
|
||||||
...(((params.visibility ?? existing.visibility) ?? 'public') !== 'public' && {
|
...(((params.visibility ?? existing.visibility) ?? 'public') !== 'public' && {
|
||||||
visibility: params.visibility ?? existing.visibility
|
visibility: params.visibility ?? existing.visibility
|
||||||
})
|
})
|
||||||
}
|
},
|
||||||
|
newMetadata as Record<string, unknown>
|
||||||
|
)
|
||||||
|
|
||||||
// Build entity structure for metadata index (with top-level fields).
|
// Build entity structure for metadata index (with top-level fields).
|
||||||
// No `level`: engine plumbing never enters the indexing view (it
|
// No `level`: engine plumbing never enters the indexing view (it
|
||||||
|
|
@ -4043,9 +3741,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// engine-minted UUID — relation ids are never caller-supplied here.)
|
// engine-minted UUID — relation ids are never caller-supplied here.)
|
||||||
params = { ...params, from: resolveEntityId(params.from), to: resolveEntityId(params.to) }
|
params = { ...params, from: resolveEntityId(params.from), to: resolveEntityId(params.to) }
|
||||||
|
|
||||||
// Reserved fields arriving via the metadata bag are normalized to their
|
|
||||||
// canonical top-level params before enforcement — mirror of add()'s lift.
|
|
||||||
params = this.remapReservedRelateMetadata(params)
|
|
||||||
|
|
||||||
// Subtype pairing enforcement (Layer 3 — 7.30.0). Per-type rules registered
|
// Subtype pairing enforcement (Layer 3 — 7.30.0). Per-type rules registered
|
||||||
// via brain.requireSubtype() compose with the brain-wide strict-mode flag.
|
// via brain.requireSubtype() compose with the brain-wide strict-mode flag.
|
||||||
|
|
@ -4097,25 +3792,28 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
(v, i) => (v + toEntity.vector[i]) / 2
|
(v, i) => (v + toEntity.vector[i]) / 2
|
||||||
)
|
)
|
||||||
|
|
||||||
// Prepare verb metadata
|
// Prepare verb metadata: a v2 nested-bag record — engine fields
|
||||||
// User metadata spread FIRST, then system fields ALWAYS win (prevents collision)
|
// top-level, the user's edge bag nested verbatim (any name is the
|
||||||
|
// user's; the field-addressing law).
|
||||||
// One timestamp for both createdAt and updatedAt so a never-updated edge reports a
|
// One timestamp for both createdAt and updatedAt so a never-updated edge reports a
|
||||||
// stable updatedAt (=== createdAt) instead of a fresh Date.now() fabricated per read.
|
// stable updatedAt (=== createdAt) instead of a fresh Date.now() fabricated per read.
|
||||||
const relateTs = Date.now()
|
const relateTs = Date.now()
|
||||||
const verbMetadata = {
|
const verbMetadata = buildVerbMetadataRecord(
|
||||||
...(params.metadata || {}),
|
{
|
||||||
verb: params.type,
|
verb: params.type,
|
||||||
...(params.subtype !== undefined && { subtype: params.subtype }),
|
...(params.subtype !== undefined && { subtype: params.subtype }),
|
||||||
// visibility: stored only when not 'public' (absent === public, keeps records lean)
|
// visibility: stored only when not 'public' (absent === public, keeps records lean)
|
||||||
...(params.visibility !== undefined &&
|
...(params.visibility !== undefined &&
|
||||||
params.visibility !== 'public' && { visibility: params.visibility }),
|
params.visibility !== 'public' && { visibility: params.visibility }),
|
||||||
weight: params.weight ?? 1.0,
|
weight: params.weight ?? 1.0,
|
||||||
...(params.confidence !== undefined && { confidence: params.confidence }),
|
...(params.confidence !== undefined && { confidence: params.confidence }),
|
||||||
...(params.service !== undefined && { service: params.service }),
|
...(params.service !== undefined && { service: params.service }),
|
||||||
createdAt: relateTs,
|
createdAt: relateTs,
|
||||||
updatedAt: relateTs,
|
updatedAt: relateTs,
|
||||||
...(params.data !== undefined && { data: params.data })
|
...(params.data !== undefined && { data: params.data })
|
||||||
}
|
},
|
||||||
|
(params.metadata as Record<string, unknown>) || {}
|
||||||
|
)
|
||||||
|
|
||||||
// Save to storage (vector and metadata separately)
|
// Save to storage (vector and metadata separately)
|
||||||
const verb: GraphVerb = {
|
const verb: GraphVerb = {
|
||||||
|
|
@ -4347,9 +4045,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
|
|
||||||
validateUpdateRelationParams(params)
|
validateUpdateRelationParams(params)
|
||||||
|
|
||||||
// Reserved fields arriving via the metadata patch are remapped to their
|
|
||||||
// canonical top-level params — mirror of update()'s normalization.
|
|
||||||
params = this.remapReservedUpdateRelationMetadata(params)
|
|
||||||
|
|
||||||
const existing = await this.storage.getVerb(params.id)
|
const existing = await this.storage.getVerb(params.id)
|
||||||
if (!existing) {
|
if (!existing) {
|
||||||
|
|
@ -4378,32 +4073,36 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
? { ...(existingRec.metadata || {}), ...(params.metadata || {}) }
|
? { ...(existingRec.metadata || {}), ...(params.metadata || {}) }
|
||||||
: params.metadata || existingRec.metadata
|
: params.metadata || existingRec.metadata
|
||||||
|
|
||||||
// Build updated stored metadata. System fields ALWAYS win — same shape as relate().
|
// Build the updated stored record: v2 nested-bag — engine fields
|
||||||
const updatedMetadata = {
|
// top-level, the merged user bag nested verbatim (mirror of update()).
|
||||||
...newMetadata,
|
const updatedWeight = params.weight ?? existingRec.weight ?? 1.0
|
||||||
verb: newVerbType,
|
const updatedData =
|
||||||
...(params.subtype !== undefined
|
params.data !== undefined ? params.data : existingRec.data
|
||||||
? { subtype: params.subtype }
|
const updatedMetadata = buildVerbMetadataRecord(
|
||||||
: existingRec.subtype !== undefined && { subtype: existingRec.subtype }),
|
{
|
||||||
// Visibility: new value if provided, else preserve existing; stored only when the
|
verb: newVerbType,
|
||||||
// effective value is not 'public' (a change to 'public' drops the field).
|
...(params.subtype !== undefined
|
||||||
...(((params.visibility ?? existingRec.visibility) ?? 'public') !== 'public' && {
|
? { subtype: params.subtype }
|
||||||
visibility: params.visibility ?? existingRec.visibility
|
: existingRec.subtype !== undefined && { subtype: existingRec.subtype }),
|
||||||
}),
|
// Visibility: new value if provided, else preserve existing; stored only when the
|
||||||
weight: params.weight ?? existingRec.weight ?? 1.0,
|
// effective value is not 'public' (a change to 'public' drops the field).
|
||||||
...(params.confidence !== undefined
|
...(((params.visibility ?? existingRec.visibility) ?? 'public') !== 'public' && {
|
||||||
? { confidence: params.confidence }
|
visibility: params.visibility ?? existingRec.visibility
|
||||||
: existingRec.confidence !== undefined && { confidence: existingRec.confidence }),
|
}),
|
||||||
// service/createdBy are fixed at relate() time — always carried forward
|
weight: updatedWeight,
|
||||||
// (omitting them here silently erased them on every updateRelation()).
|
...(params.confidence !== undefined
|
||||||
...(existingRec.service !== undefined && { service: existingRec.service }),
|
? { confidence: params.confidence }
|
||||||
...(existingRec.createdBy !== undefined && { createdBy: existingRec.createdBy }),
|
: existingRec.confidence !== undefined && { confidence: existingRec.confidence }),
|
||||||
createdAt: existingRec.createdAt,
|
// service/createdBy are fixed at relate() time — always carried forward
|
||||||
updatedAt: Date.now(),
|
// (omitting them here silently erased them on every updateRelation()).
|
||||||
...(params.data !== undefined
|
...(existingRec.service !== undefined && { service: existingRec.service }),
|
||||||
? { data: params.data }
|
...(existingRec.createdBy !== undefined && { createdBy: existingRec.createdBy }),
|
||||||
: existingRec.data !== undefined && { data: existingRec.data })
|
createdAt: existingRec.createdAt,
|
||||||
}
|
updatedAt: Date.now(),
|
||||||
|
...(updatedData !== undefined && { data: updatedData })
|
||||||
|
},
|
||||||
|
newMetadata as Record<string, unknown>
|
||||||
|
)
|
||||||
|
|
||||||
// Build the verb view used by the graph index — top-level fields mirror relate()'s.
|
// Build the verb view used by the graph index — top-level fields mirror relate()'s.
|
||||||
const verbForIndex: GraphVerb = {
|
const verbForIndex: GraphVerb = {
|
||||||
|
|
@ -4419,9 +4118,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
...(((params.visibility ?? existingRec.visibility) ?? 'public') !== 'public' && {
|
...(((params.visibility ?? existingRec.visibility) ?? 'public') !== 'public' && {
|
||||||
visibility: params.visibility ?? existingRec.visibility
|
visibility: params.visibility ?? existingRec.visibility
|
||||||
}),
|
}),
|
||||||
weight: updatedMetadata.weight,
|
weight: updatedWeight,
|
||||||
metadata: newMetadata,
|
metadata: newMetadata,
|
||||||
data: updatedMetadata.data,
|
data: updatedData,
|
||||||
createdAt: existingRec.createdAt
|
createdAt: existingRec.createdAt
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -5619,7 +5318,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
this._aggregationIndex!.defineAggregate({
|
this._aggregationIndex!.defineAggregate({
|
||||||
name: aggregateName,
|
name: aggregateName,
|
||||||
source: {},
|
source: {},
|
||||||
groupBy: perType ? [name, 'noun'] : [name],
|
groupBy: perType ? [name, 'system.type'] : [name],
|
||||||
metrics: { count: { op: 'count' } }
|
metrics: { count: { op: 'count' } }
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
@ -5630,7 +5329,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
* and `counts.byField()` agree on the convention.
|
* and `counts.byField()` agree on the convention.
|
||||||
*/
|
*/
|
||||||
private fieldCountsAggregateName(name: string): string {
|
private fieldCountsAggregateName(name: string): string {
|
||||||
return `__fieldCounts__${name}`
|
// v2 suffix: the per-type dimension moved from the legacy 'noun' alias to
|
||||||
|
// 'system.type' under the addressing law — a NEW name makes the ensure
|
||||||
|
// block re-define and BACKFILL from canonical instead of silently serving
|
||||||
|
// the old-dim definition (whose 'noun' key now reads user metadata and
|
||||||
|
// would drift). The v1 rows are derived state, superseded not lost.
|
||||||
|
return `__fieldCounts_v2__${name}`
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -6022,8 +5726,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
): Promise<Set<string>> {
|
): Promise<Set<string>> {
|
||||||
const excluded = this.excludedVisibilityTiers(params)
|
const excluded = this.excludedVisibilityTiers(params)
|
||||||
if (!excluded) return new Set()
|
if (!excluded) return new Set()
|
||||||
|
// 'system.visibility' — the engine scalar's frozen address. A bare
|
||||||
|
// 'visibility' key would address the USER's metadata bag under the
|
||||||
|
// field-addressing law and silently hide nothing (VFS/system entities
|
||||||
|
// would leak into every default read).
|
||||||
const ids = await this.metadataIndex.getIdsForFilter({
|
const ids = await this.metadataIndex.getIdsForFilter({
|
||||||
visibility: excluded.length === 1 ? excluded[0] : { oneOf: excluded }
|
'system.visibility': excluded.length === 1 ? excluded[0] : { oneOf: excluded }
|
||||||
})
|
})
|
||||||
return new Set(ids)
|
return new Set(ids)
|
||||||
}
|
}
|
||||||
|
|
@ -6148,20 +5856,18 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Build filter for metadata index
|
// Build filter for metadata index
|
||||||
let filter: any = {}
|
let filter: any = {}
|
||||||
if (params.where) {
|
if (params.where) {
|
||||||
|
// Where keys pass through UNTOUCHED — the addressing law parses
|
||||||
|
// them at the index boundary. The old where.type→noun alias is
|
||||||
|
// dead: bare 'type' is the user's own field now.
|
||||||
Object.assign(filter, params.where)
|
Object.assign(filter, params.where)
|
||||||
// Alias: where.type → where.noun (storage field name for entity type)
|
|
||||||
if ('type' in filter && !('noun' in filter)) {
|
|
||||||
filter.noun = filter.type
|
|
||||||
delete filter.type
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if (params.service) filter.service = params.service
|
if (params.service) filter['system.service'] = params.service
|
||||||
|
|
||||||
// Subtype (top-level standard field — fast path, not metadata fallback).
|
// Subtype (top-level standard field — fast path, not metadata fallback).
|
||||||
// Must be assigned BEFORE the type-array expansion below so the spread
|
// Must be assigned BEFORE the type-array expansion below so the spread
|
||||||
// into each anyOf branch carries it through.
|
// into each anyOf branch carries it through.
|
||||||
if (params.subtype !== undefined) {
|
if (params.subtype !== undefined) {
|
||||||
filter.subtype = Array.isArray(params.subtype)
|
filter['system.subtype'] = Array.isArray(params.subtype)
|
||||||
? { oneOf: params.subtype }
|
? { oneOf: params.subtype }
|
||||||
: params.subtype
|
: params.subtype
|
||||||
}
|
}
|
||||||
|
|
@ -6169,11 +5875,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
if (params.type) {
|
if (params.type) {
|
||||||
const types = Array.isArray(params.type) ? params.type : [params.type]
|
const types = Array.isArray(params.type) ? params.type : [params.type]
|
||||||
if (types.length === 1) {
|
if (types.length === 1) {
|
||||||
filter.noun = types[0]
|
filter['system.type'] = types[0]
|
||||||
} else {
|
} else {
|
||||||
filter = {
|
filter = {
|
||||||
anyOf: types.map(type => ({
|
anyOf: types.map(type => ({
|
||||||
noun: type,
|
'system.type': type,
|
||||||
...filter
|
...filter
|
||||||
}))
|
}))
|
||||||
}
|
}
|
||||||
|
|
@ -9279,10 +8985,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
): Promise<string> {
|
): Promise<string> {
|
||||||
const { op: _discriminator, ...rawParams } = op
|
const { op: _discriminator, ...rawParams } = op
|
||||||
validateAddParams(rawParams as AddParams<T>)
|
validateAddParams(rawParams as AddParams<T>)
|
||||||
// Same reserved-field normalization as add() — the metadata bag is
|
const params = rawParams as AddParams<T>
|
||||||
// cleaned BEFORE enforcement so a remapped subtype participates in
|
|
||||||
// subtype-pairing enforcement and only custom fields reach the index.
|
|
||||||
const params = this.remapReservedAddMetadata(rawParams as AddParams<T>)
|
|
||||||
this.enforceTrackedFieldValues(params.metadata as Record<string, unknown> | undefined, 'metadata')
|
this.enforceTrackedFieldValues(params.metadata as Record<string, unknown> | undefined, 'metadata')
|
||||||
this.enforceTrackedFieldValues({ subtype: params.subtype } as Record<string, unknown>, 'top-level')
|
this.enforceTrackedFieldValues({ subtype: params.subtype } as Record<string, unknown>, 'top-level')
|
||||||
this.enforceSubtypeOnAdd('add', params.type, params.subtype, params.metadata)
|
this.enforceSubtypeOnAdd('add', params.type, params.subtype, params.metadata)
|
||||||
|
|
@ -9357,25 +9060,31 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
plan.createdNouns.add(id)
|
plan.createdNouns.add(id)
|
||||||
|
|
||||||
const now = Date.now()
|
const now = Date.now()
|
||||||
const storageMetadata = {
|
// v2 nested-bag record — mirror of add(): engine fields top-level, the
|
||||||
...params.metadata,
|
// user's bag nested verbatim (collider names stay the user's).
|
||||||
// Preserve the caller's original (non-UUID) id when normalized — mirror
|
const storageMetadata = buildNounMetadataRecord(
|
||||||
// of add(). A real UUID passes through with no _originalId.
|
{
|
||||||
...(originalId !== undefined && { [ORIGINAL_ID_KEY]: originalId }),
|
data: params.data,
|
||||||
data: params.data,
|
noun: params.type,
|
||||||
noun: params.type,
|
...(params.subtype !== undefined && { subtype: params.subtype }),
|
||||||
...(params.subtype !== undefined && { subtype: params.subtype }),
|
// visibility: stored only when not 'public' (absent === public, keeps records lean)
|
||||||
// visibility: stored only when not 'public' (absent === public, keeps records lean)
|
...(params.visibility !== undefined &&
|
||||||
...(params.visibility !== undefined &&
|
params.visibility !== 'public' && { visibility: params.visibility }),
|
||||||
params.visibility !== 'public' && { visibility: params.visibility }),
|
service: params.service,
|
||||||
service: params.service,
|
createdAt: now,
|
||||||
createdAt: now,
|
updatedAt: now,
|
||||||
updatedAt: now,
|
_rev: 1,
|
||||||
_rev: 1,
|
...(params.confidence !== undefined && { confidence: params.confidence }),
|
||||||
...(params.confidence !== undefined && { confidence: params.confidence }),
|
...(params.weight !== undefined && { weight: params.weight }),
|
||||||
...(params.weight !== undefined && { weight: params.weight }),
|
...(params.createdBy && { createdBy: params.createdBy })
|
||||||
...(params.createdBy && { createdBy: params.createdBy })
|
},
|
||||||
}
|
{
|
||||||
|
...params.metadata,
|
||||||
|
// Preserve the caller's original (non-UUID) id when normalized — mirror
|
||||||
|
// of add(). A real UUID passes through with no _originalId.
|
||||||
|
...(originalId !== undefined && { [ORIGINAL_ID_KEY]: originalId })
|
||||||
|
}
|
||||||
|
)
|
||||||
const entityForIndexing = {
|
const entityForIndexing = {
|
||||||
id,
|
id,
|
||||||
vector,
|
vector,
|
||||||
|
|
@ -9438,10 +9147,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
): Promise<string> {
|
): Promise<string> {
|
||||||
const { op: _discriminator, ...rawParams } = op
|
const { op: _discriminator, ...rawParams } = op
|
||||||
validateUpdateParams(rawParams as UpdateParams<T>)
|
validateUpdateParams(rawParams as UpdateParams<T>)
|
||||||
// Same reserved-field normalization as update() — user-mutable fields
|
const params = rawParams as UpdateParams<T>
|
||||||
// remap to their dedicated param (top-level wins), system-managed fields
|
|
||||||
// drop with a one-shot warning.
|
|
||||||
const params = this.remapReservedUpdateMetadata(rawParams as UpdateParams<T>)
|
|
||||||
// Id normalization (8.0) — mirror of update(): a natural key resolves to the
|
// Id normalization (8.0) — mirror of update(): a natural key resolves to the
|
||||||
// canonical UUID add() stored. A real UUID passes through.
|
// canonical UUID add() stored. A real UUID passes through.
|
||||||
params.id = resolveEntityId(params.id)
|
params.id = resolveEntityId(params.id)
|
||||||
|
|
@ -9493,29 +9199,33 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
? { ...existing.metadata, ...params.metadata }
|
? { ...existing.metadata, ...params.metadata }
|
||||||
: params.metadata || existing.metadata
|
: params.metadata || existing.metadata
|
||||||
const now = Date.now()
|
const now = Date.now()
|
||||||
const updatedMetadata = {
|
// v2 nested-bag record — mirror of update(): engine fields top-level,
|
||||||
...newMetadata,
|
// the merged user bag nested verbatim.
|
||||||
data: params.data !== undefined ? params.data : existing.data,
|
const updatedMetadata = buildNounMetadataRecord(
|
||||||
noun: params.type || existing.type,
|
{
|
||||||
service: existing.service,
|
data: params.data !== undefined ? params.data : existing.data,
|
||||||
createdAt: existing.createdAt,
|
noun: params.type || existing.type,
|
||||||
updatedAt: now,
|
service: existing.service,
|
||||||
_rev: currentRev + 1,
|
createdAt: existing.createdAt,
|
||||||
...(params.confidence !== undefined && { confidence: params.confidence }),
|
updatedAt: now,
|
||||||
...(params.weight !== undefined && { weight: params.weight }),
|
_rev: currentRev + 1,
|
||||||
...(params.confidence === undefined &&
|
...(params.confidence !== undefined && { confidence: params.confidence }),
|
||||||
existing.confidence !== undefined && { confidence: existing.confidence }),
|
...(params.weight !== undefined && { weight: params.weight }),
|
||||||
...(params.weight === undefined &&
|
...(params.confidence === undefined &&
|
||||||
existing.weight !== undefined && { weight: existing.weight }),
|
existing.confidence !== undefined && { confidence: existing.confidence }),
|
||||||
...(params.subtype !== undefined && { subtype: params.subtype }),
|
...(params.weight === undefined &&
|
||||||
...(params.subtype === undefined &&
|
existing.weight !== undefined && { weight: existing.weight }),
|
||||||
existing.subtype !== undefined && { subtype: existing.subtype }),
|
...(params.subtype !== undefined && { subtype: params.subtype }),
|
||||||
// Visibility: new value if provided, else preserve existing; stored only when the
|
...(params.subtype === undefined &&
|
||||||
// effective value is not 'public' (a change to 'public' drops the field).
|
existing.subtype !== undefined && { subtype: existing.subtype }),
|
||||||
...(((params.visibility ?? existing.visibility) ?? 'public') !== 'public' && {
|
// Visibility: new value if provided, else preserve existing; stored only when the
|
||||||
visibility: params.visibility ?? existing.visibility
|
// effective value is not 'public' (a change to 'public' drops the field).
|
||||||
})
|
...(((params.visibility ?? existing.visibility) ?? 'public') !== 'public' && {
|
||||||
}
|
visibility: params.visibility ?? existing.visibility
|
||||||
|
})
|
||||||
|
},
|
||||||
|
newMetadata as Record<string, unknown>
|
||||||
|
)
|
||||||
|
|
||||||
// Register for the authoritative under-mutex CAS re-verify + rev re-stamp
|
// Register for the authoritative under-mutex CAS re-verify + rev re-stamp
|
||||||
// (see PlannedTransact.casUpdates). The staged UpdateNounMetadataOperation
|
// (see PlannedTransact.casUpdates). The staged UpdateNounMetadataOperation
|
||||||
|
|
@ -9736,8 +9446,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
): Promise<string> {
|
): Promise<string> {
|
||||||
const { op: _discriminator, ...rawParams } = op
|
const { op: _discriminator, ...rawParams } = op
|
||||||
validateRelateParams(rawParams as RelateParams<T>)
|
validateRelateParams(rawParams as RelateParams<T>)
|
||||||
// Same reserved-field normalization as relate().
|
const params = rawParams as RelateParams<T>
|
||||||
const params = this.remapReservedRelateMetadata(rawParams as RelateParams<T>)
|
|
||||||
// Id normalization (8.0) — mirror of relate(): resolve BOTH endpoints to the
|
// Id normalization (8.0) — mirror of relate(): resolve BOTH endpoints to the
|
||||||
// canonical UUID add() stored, so a relate op may reference either side by
|
// canonical UUID add() stored, so a relate op may reference either side by
|
||||||
// natural key. Real UUIDs pass through. (Relationship ids are engine-minted.)
|
// natural key. Real UUIDs pass through. (Relationship ids are engine-minted.)
|
||||||
|
|
@ -9787,19 +9496,23 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
const id = uuidv4()
|
const id = uuidv4()
|
||||||
const relationVector = fromEntity.vector.map((v, i) => (v + toEntity.vector[i]) / 2)
|
const relationVector = fromEntity.vector.map((v, i) => (v + toEntity.vector[i]) / 2)
|
||||||
const now = Date.now()
|
const now = Date.now()
|
||||||
const verbMetadata = {
|
// v2 nested-bag record — mirror of relate(): engine fields top-level,
|
||||||
...(params.metadata || {}),
|
// the user's edge bag nested verbatim.
|
||||||
verb: params.type,
|
const verbMetadata = buildVerbMetadataRecord(
|
||||||
...(params.subtype !== undefined && { subtype: params.subtype }),
|
{
|
||||||
// visibility: stored only when not 'public' (absent === public, keeps records lean)
|
verb: params.type,
|
||||||
...(params.visibility !== undefined &&
|
...(params.subtype !== undefined && { subtype: params.subtype }),
|
||||||
params.visibility !== 'public' && { visibility: params.visibility }),
|
// visibility: stored only when not 'public' (absent === public, keeps records lean)
|
||||||
weight: params.weight ?? 1.0,
|
...(params.visibility !== undefined &&
|
||||||
...(params.confidence !== undefined && { confidence: params.confidence }),
|
params.visibility !== 'public' && { visibility: params.visibility }),
|
||||||
...(params.service !== undefined && { service: params.service }),
|
weight: params.weight ?? 1.0,
|
||||||
createdAt: now,
|
...(params.confidence !== undefined && { confidence: params.confidence }),
|
||||||
...(params.data !== undefined && { data: params.data })
|
...(params.service !== undefined && { service: params.service }),
|
||||||
}
|
createdAt: now,
|
||||||
|
...(params.data !== undefined && { data: params.data })
|
||||||
|
},
|
||||||
|
(params.metadata as Record<string, unknown>) || {}
|
||||||
|
)
|
||||||
const verb: GraphVerb = {
|
const verb: GraphVerb = {
|
||||||
id,
|
id,
|
||||||
vector: relationVector,
|
vector: relationVector,
|
||||||
|
|
@ -11388,27 +11101,26 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
if (params.where || params.subtype || params.service) {
|
if (params.where || params.subtype || params.service) {
|
||||||
let filter: any = {}
|
let filter: any = {}
|
||||||
if (params.where) {
|
if (params.where) {
|
||||||
|
// Where keys pass through UNTOUCHED — the one addressing law
|
||||||
|
// parses them at the index boundary (bare = user metadata,
|
||||||
|
// system.* = engine scalars). The old where.type→noun alias is
|
||||||
|
// dead: a bare 'type' is the user's own field now.
|
||||||
Object.assign(filter, params.where)
|
Object.assign(filter, params.where)
|
||||||
// Alias: where.type → where.noun (storage field name for entity type)
|
|
||||||
if ('type' in filter && !('noun' in filter)) {
|
|
||||||
filter.noun = filter.type
|
|
||||||
delete filter.type
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if (params.service) filter.service = params.service
|
if (params.service) filter['system.service'] = params.service
|
||||||
if (params.subtype !== undefined) {
|
if (params.subtype !== undefined) {
|
||||||
filter.subtype = Array.isArray(params.subtype)
|
filter['system.subtype'] = Array.isArray(params.subtype)
|
||||||
? { oneOf: params.subtype }
|
? { oneOf: params.subtype }
|
||||||
: params.subtype
|
: params.subtype
|
||||||
}
|
}
|
||||||
if (params.type) {
|
if (params.type) {
|
||||||
const types = Array.isArray(params.type) ? params.type : [params.type]
|
const types = Array.isArray(params.type) ? params.type : [params.type]
|
||||||
if (types.length === 1) {
|
if (types.length === 1) {
|
||||||
filter.noun = types[0]
|
filter['system.type'] = types[0]
|
||||||
} else {
|
} else {
|
||||||
const baseFilter = { ...filter }
|
const baseFilter = { ...filter }
|
||||||
filter = {
|
filter = {
|
||||||
anyOf: types.map(type => ({ noun: type, ...baseFilter }))
|
anyOf: types.map(type => ({ 'system.type': type, ...baseFilter }))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -11458,27 +11170,24 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Use MetadataIndexManager for efficient filtered streaming
|
// Use MetadataIndexManager for efficient filtered streaming
|
||||||
let filterObj: any = {}
|
let filterObj: any = {}
|
||||||
if (filter.where) {
|
if (filter.where) {
|
||||||
|
// Where keys pass through — the addressing law parses them at
|
||||||
|
// the index boundary; the type→noun alias is dead.
|
||||||
Object.assign(filterObj, filter.where)
|
Object.assign(filterObj, filter.where)
|
||||||
// Alias: where.type → where.noun (storage field name for entity type)
|
|
||||||
if ('type' in filterObj && !('noun' in filterObj)) {
|
|
||||||
filterObj.noun = filterObj.type
|
|
||||||
delete filterObj.type
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if (filter.service) filterObj.service = filter.service
|
if (filter.service) filterObj['system.service'] = filter.service
|
||||||
if (filter.subtype !== undefined) {
|
if (filter.subtype !== undefined) {
|
||||||
filterObj.subtype = Array.isArray(filter.subtype)
|
filterObj['system.subtype'] = Array.isArray(filter.subtype)
|
||||||
? { oneOf: filter.subtype }
|
? { oneOf: filter.subtype }
|
||||||
: filter.subtype
|
: filter.subtype
|
||||||
}
|
}
|
||||||
if (filter.type) {
|
if (filter.type) {
|
||||||
const types = Array.isArray(filter.type) ? filter.type : [filter.type]
|
const types = Array.isArray(filter.type) ? filter.type : [filter.type]
|
||||||
if (types.length === 1) {
|
if (types.length === 1) {
|
||||||
filterObj.noun = types[0]
|
filterObj['system.type'] = types[0]
|
||||||
} else {
|
} else {
|
||||||
const baseFilterObj = { ...filterObj }
|
const baseFilterObj = { ...filterObj }
|
||||||
filterObj = {
|
filterObj = {
|
||||||
anyOf: types.map(type => ({ noun: type, ...baseFilterObj }))
|
anyOf: types.map(type => ({ 'system.type': type, ...baseFilterObj }))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -11858,7 +11567,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// don't have the tracked field at all (e.g. the VFS root) bucket under
|
// don't have the tracked field at all (e.g. the VFS root) bucket under
|
||||||
// '__null__' and would otherwise pollute the count map.
|
// '__null__' and would otherwise pollute the count map.
|
||||||
if (value === undefined || value === null || value === '__null__') continue
|
if (value === undefined || value === null || value === '__null__') continue
|
||||||
if (options?.type !== undefined && row.groupKey?.['noun'] !== options.type) continue
|
if (options?.type !== undefined && row.groupKey?.['system.type'] !== options.type) continue
|
||||||
const key = String(value)
|
const key = String(value)
|
||||||
result[key] = (result[key] || 0) + (typeof row.metrics?.count === 'number' ? row.metrics.count : row.count)
|
result[key] = (result[key] || 0) + (typeof row.metrics?.count === 'number' ? row.metrics.count : row.count)
|
||||||
}
|
}
|
||||||
|
|
@ -13605,14 +13314,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
}
|
}
|
||||||
let filter: any = {}
|
let filter: any = {}
|
||||||
if (params.where) {
|
if (params.where) {
|
||||||
|
// Where keys pass through UNTOUCHED — the one addressing law parses
|
||||||
|
// them at the index boundary (bare = user metadata, system.* = engine
|
||||||
|
// scalars, typed refusal otherwise). The old type→noun alias is dead.
|
||||||
Object.assign(filter, params.where)
|
Object.assign(filter, params.where)
|
||||||
// Alias: where.type → where.noun (storage field name for entity type)
|
|
||||||
if ('type' in filter && !('noun' in filter)) {
|
|
||||||
filter.noun = filter.type
|
|
||||||
delete filter.type
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if (params.service) filter.service = params.service
|
if (params.service) filter['system.service'] = params.service
|
||||||
if (params.excludeVFS === true) {
|
if (params.excludeVFS === true) {
|
||||||
filter.vfsType = { exists: false }
|
filter.vfsType = { exists: false }
|
||||||
filter.isVFSEntity = { ne: true }
|
filter.isVFSEntity = { ne: true }
|
||||||
|
|
@ -13620,14 +13327,14 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Subtype (top-level standard field — fast path). Assigned BEFORE the type-array
|
// Subtype (top-level standard field — fast path). Assigned BEFORE the type-array
|
||||||
// expansion below so the spread into each anyOf branch carries it through.
|
// expansion below so the spread into each anyOf branch carries it through.
|
||||||
if (params.subtype !== undefined) {
|
if (params.subtype !== undefined) {
|
||||||
filter.subtype = Array.isArray(params.subtype) ? { oneOf: params.subtype } : params.subtype
|
filter['system.subtype'] = Array.isArray(params.subtype) ? { oneOf: params.subtype } : params.subtype
|
||||||
}
|
}
|
||||||
if (params.type) {
|
if (params.type) {
|
||||||
const types = Array.isArray(params.type) ? params.type : [params.type]
|
const types = Array.isArray(params.type) ? params.type : [params.type]
|
||||||
if (types.length === 1) {
|
if (types.length === 1) {
|
||||||
filter.noun = types[0]
|
filter['system.type'] = types[0]
|
||||||
} else {
|
} else {
|
||||||
filter = { anyOf: types.map((type) => ({ noun: type, ...filter })) }
|
filter = { anyOf: types.map((type) => ({ 'system.type': type, ...filter })) }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return filter
|
return filter
|
||||||
|
|
@ -15118,12 +14825,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
requireSubtype: config?.requireSubtype ?? true,
|
requireSubtype: config?.requireSubtype ?? true,
|
||||||
// Multi-process safety
|
// Multi-process safety
|
||||||
mode: config?.mode ?? 'writer',
|
mode: config?.mode ?? 'writer',
|
||||||
force: config?.force ?? false,
|
force: config?.force ?? false
|
||||||
// Reserved-field-in-metadata-bag policy (8.0 — no silent failures).
|
|
||||||
// Default 'throw': an untyped caller that smuggles a reserved key past
|
|
||||||
// the compile guard gets a loud Error naming the correct write path.
|
|
||||||
// 'warn' = remap + one-shot warning per key; 'remap' = legacy silent remap.
|
|
||||||
reservedFieldPolicy: config?.reservedFieldPolicy ?? 'throw'
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
67
src/db/db.ts
67
src/db/db.ts
|
|
@ -59,10 +59,6 @@ import type {
|
||||||
import type { StorageAdapter } from '../coreTypes.js'
|
import type { StorageAdapter } from '../coreTypes.js'
|
||||||
import { exportGraph } from './portableGraph.js'
|
import { exportGraph } from './portableGraph.js'
|
||||||
import type { ExportSelector, ExportOptions, PortableGraph } from './portableGraph.js'
|
import type { ExportSelector, ExportOptions, PortableGraph } from './portableGraph.js'
|
||||||
import {
|
|
||||||
splitNounMetadataRecord,
|
|
||||||
splitVerbMetadataRecord
|
|
||||||
} from '../types/reservedFields.js'
|
|
||||||
import { v4 as uuidv4 } from '../universal/uuid.js'
|
import { v4 as uuidv4 } from '../universal/uuid.js'
|
||||||
import { coerceNewEntityId, resolveEntityId, ORIGINAL_ID_KEY } from '../utils/idNormalization.js'
|
import { coerceNewEntityId, resolveEntityId, ORIGINAL_ID_KEY } from '../utils/idNormalization.js'
|
||||||
import { EntityNotFoundError } from '../errors/notFound.js'
|
import { EntityNotFoundError } from '../errors/notFound.js'
|
||||||
|
|
@ -705,23 +701,15 @@ export class Db<T = any> {
|
||||||
for (const op of ops) {
|
for (const op of ops) {
|
||||||
switch (op.op) {
|
switch (op.op) {
|
||||||
case 'add': {
|
case 'add': {
|
||||||
// Reserved-field normalization — mirror of the brain.transact()
|
// Field-addressing law: the metadata bag is the user's, VERBATIM —
|
||||||
// write path: user-settable fields lift to their dedicated field
|
// no reserved-name lift, no drops. Engine scalars come ONLY from
|
||||||
// (top-level wins), system-managed fields drop, and the entity's
|
// their dedicated op fields; a bag field named `confidence` is an
|
||||||
// metadata bag carries ONLY custom fields. Speculative views skip
|
// ordinary user field, exactly as on the committed write path.
|
||||||
// the one-shot warnings — committing the same ops through
|
const custom = { ...(op.metadata as Record<string, unknown> | undefined) }
|
||||||
// `brain.transact()` warns on the real write path.
|
const confidence = op.confidence
|
||||||
const { reserved, custom } = splitNounMetadataRecord(
|
const weight = op.weight
|
||||||
op.metadata as Record<string, unknown> | undefined
|
const subtype = op.subtype
|
||||||
)
|
const service = op.service
|
||||||
const confidence =
|
|
||||||
op.confidence ?? (typeof reserved.confidence === 'number' ? reserved.confidence : undefined)
|
|
||||||
const weight =
|
|
||||||
op.weight ?? (typeof reserved.weight === 'number' ? reserved.weight : undefined)
|
|
||||||
const subtype =
|
|
||||||
op.subtype ?? (typeof reserved.subtype === 'string' ? reserved.subtype : undefined)
|
|
||||||
const service =
|
|
||||||
op.service ?? (typeof reserved.service === 'string' ? reserved.service : undefined)
|
|
||||||
|
|
||||||
// Id normalization (8.0) — mirror of the committed transact() add
|
// Id normalization (8.0) — mirror of the committed transact() add
|
||||||
// path: a natural key coerces to a STABLE UUID (v5), preserving the
|
// path: a natural key coerces to a STABLE UUID (v5), preserving the
|
||||||
|
|
@ -759,16 +747,12 @@ export class Db<T = any> {
|
||||||
`with(): entity ${updateId} not found at generation ${this.gen}`
|
`with(): entity ${updateId} not found at generation ${this.gen}`
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
// Same reserved-field normalization as the committed update path.
|
// Field-addressing law — mirror of the add case: the patch bag is
|
||||||
const { reserved, custom } = splitNounMetadataRecord(
|
// the user's verbatim; engine scalars only from dedicated op fields.
|
||||||
op.metadata as Record<string, unknown> | undefined
|
const custom = { ...(op.metadata as Record<string, unknown> | undefined) }
|
||||||
)
|
const confidence = op.confidence
|
||||||
const confidence =
|
const weight = op.weight
|
||||||
op.confidence ?? (typeof reserved.confidence === 'number' ? reserved.confidence : undefined)
|
const subtype = op.subtype
|
||||||
const weight =
|
|
||||||
op.weight ?? (typeof reserved.weight === 'number' ? reserved.weight : undefined)
|
|
||||||
const subtype =
|
|
||||||
op.subtype ?? (typeof reserved.subtype === 'string' ? reserved.subtype : undefined)
|
|
||||||
const mergedMetadata =
|
const mergedMetadata =
|
||||||
op.merge !== false
|
op.merge !== false
|
||||||
? ({ ...(base.metadata as object), ...custom } as T)
|
? ({ ...(base.metadata as object), ...custom } as T)
|
||||||
|
|
@ -830,19 +814,14 @@ export class Db<T = any> {
|
||||||
}
|
}
|
||||||
if (duplicate) break
|
if (duplicate) break
|
||||||
|
|
||||||
// Reserved-field normalization — relationship mirror of the add
|
// Field-addressing law — relationship mirror of the add case: the
|
||||||
// op above (and of the committed relate() path).
|
// edge bag is the user's verbatim; engine scalars only from
|
||||||
const { reserved, custom } = splitVerbMetadataRecord(
|
// dedicated op fields.
|
||||||
op.metadata as Record<string, unknown> | undefined
|
const custom = { ...(op.metadata as Record<string, unknown> | undefined) }
|
||||||
)
|
const confidence = op.confidence
|
||||||
const confidence =
|
const weight = op.weight
|
||||||
op.confidence ?? (typeof reserved.confidence === 'number' ? reserved.confidence : undefined)
|
const subtype = op.subtype
|
||||||
const weight =
|
const service = op.service
|
||||||
op.weight ?? (typeof reserved.weight === 'number' ? reserved.weight : undefined)
|
|
||||||
const subtype =
|
|
||||||
op.subtype ?? (typeof reserved.subtype === 'string' ? reserved.subtype : undefined)
|
|
||||||
const service =
|
|
||||||
op.service ?? (typeof reserved.service === 'string' ? reserved.service : undefined)
|
|
||||||
|
|
||||||
const id = uuidv4()
|
const id = uuidv4()
|
||||||
overlay.verbs.set(id, {
|
overlay.verbs.set(id, {
|
||||||
|
|
|
||||||
323
src/db/fieldAddressing.ts
Normal file
323
src/db/fieldAddressing.ts
Normal file
|
|
@ -0,0 +1,323 @@
|
||||||
|
/**
|
||||||
|
* @module db/fieldAddressing
|
||||||
|
* @description The one field-addressing law for every query surface (find()'s
|
||||||
|
* `where` / `orderBy` / `groupBy`, aggregation `source.where`), ruled
|
||||||
|
* 2026-08-03 after a production incident in which a user metadata field
|
||||||
|
* named `level` was silently shadowed by the engine's internal HNSW node
|
||||||
|
* layer (VENUE-BRAINY-ORDERBY-NOOP — thread id kept verbatim as the audit
|
||||||
|
* key; it names no product):
|
||||||
|
*
|
||||||
|
* 1. A BARE field name addresses the user's metadata field. Always.
|
||||||
|
* No priority resolution, no fallback chain — `orderBy: 'level'`
|
||||||
|
* reads `entity.metadata.level`, full stop.
|
||||||
|
* 2. `system.<field>` addresses an engine scalar, reachable ONLY with the
|
||||||
|
* explicit prefix. The entity map is exactly ten scalars; the relation
|
||||||
|
* map mirrors it with `verb`/`sourceId`/`targetId` as the structural
|
||||||
|
* members.
|
||||||
|
* 3. Engine plumbing (`vector`, `connections`, `level`, `data`, `_rev`) is
|
||||||
|
* INVISIBLE to the query surface in either spelling — `system.level`
|
||||||
|
* refuses; bare `level` is the user's field.
|
||||||
|
* 4. `metadata.<field>` is the explicit spelling of the bare form —
|
||||||
|
* identical semantics on every path.
|
||||||
|
* 5. Anything unresolvable refuses with a TYPED error naming both
|
||||||
|
* candidate spellings — an accepted name either works or refuses;
|
||||||
|
* there is no third state.
|
||||||
|
*
|
||||||
|
* This module is the SINGLE source of truth for the law: parsing, the maps,
|
||||||
|
* and the refusal builders live here so the JS engine, the provider seams,
|
||||||
|
* and the cross-engine conformance suite can never drift on the contract.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import type { HNSWNounWithMetadata, HNSWVerbWithMetadata } from '../coreTypes.js'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description The entity-side `system.*` map — EXACTLY the ten engine
|
||||||
|
* scalars David ruled queryable (2026-08-03). Adding a name here is a
|
||||||
|
* cross-engine contract change: the native accelerator's conformance suite
|
||||||
|
* pins this list verbatim, so any edit must ship as a paired release.
|
||||||
|
*/
|
||||||
|
export const SYSTEM_ENTITY_SCALARS: ReadonlySet<string> = new Set([
|
||||||
|
'id',
|
||||||
|
'type',
|
||||||
|
'subtype',
|
||||||
|
'createdAt',
|
||||||
|
'updatedAt',
|
||||||
|
'confidence',
|
||||||
|
'weight',
|
||||||
|
'visibility',
|
||||||
|
'service',
|
||||||
|
'createdBy'
|
||||||
|
])
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description The relation-side `system.*` map — the verb mirror of
|
||||||
|
* {@link SYSTEM_ENTITY_SCALARS}: `verb`, `sourceId`, `targetId` are the
|
||||||
|
* structural members beside the eight shared scalars. Same one law, same
|
||||||
|
* pairing rule for edits.
|
||||||
|
*/
|
||||||
|
export const SYSTEM_RELATION_SCALARS: ReadonlySet<string> = new Set([
|
||||||
|
'verb',
|
||||||
|
'sourceId',
|
||||||
|
'targetId',
|
||||||
|
'subtype',
|
||||||
|
'createdAt',
|
||||||
|
'updatedAt',
|
||||||
|
'confidence',
|
||||||
|
'weight',
|
||||||
|
'visibility',
|
||||||
|
'service',
|
||||||
|
'createdBy'
|
||||||
|
])
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Engine plumbing — never addressable from the query surface in
|
||||||
|
* ANY spelling. `level` is the HNSW node layer (the incident field: listing
|
||||||
|
* it as resolvable shadowed real user data); `data` is the payload container,
|
||||||
|
* not a scalar — content is reached through the content/text-search APIs,
|
||||||
|
* and addressing it as a sortable field would lie about its shape.
|
||||||
|
*/
|
||||||
|
export const PLUMBING_FIELDS: ReadonlySet<string> = new Set([
|
||||||
|
'vector',
|
||||||
|
'connections',
|
||||||
|
'level',
|
||||||
|
'data',
|
||||||
|
'_rev'
|
||||||
|
])
|
||||||
|
|
||||||
|
/** @description Which record kind a field address is being resolved against. */
|
||||||
|
export type FieldAddressKind = 'entity' | 'relation'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description A parsed, law-valid field address. `scope` says which side of
|
||||||
|
* the record the name lives on; `field` is the unprefixed name to read.
|
||||||
|
*/
|
||||||
|
export interface FieldAddress {
|
||||||
|
/** 'metadata' = the user's field (bare or `metadata.`-prefixed); 'system' = an engine scalar. */
|
||||||
|
scope: 'metadata' | 'system'
|
||||||
|
/** The field name with any scope prefix removed. */
|
||||||
|
field: string
|
||||||
|
/** The exact spelling the caller used — preserved for error text and telemetry. */
|
||||||
|
raw: string
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Parse a query-surface field name under the one law. Pure and data-blind:
|
||||||
|
* this validates the ADDRESS (spelling + map membership), not whether any
|
||||||
|
* row actually carries the field — data-aware refusals (the did-you-mean
|
||||||
|
* for a bare system-scalar name no row carries) belong to the query layer,
|
||||||
|
* which calls {@link buildUnresolvableMessage} with index knowledge.
|
||||||
|
*
|
||||||
|
* @param raw - The field name as the caller wrote it (`level`,
|
||||||
|
* `metadata.level`, `system.createdAt`, …)
|
||||||
|
* @param kind - Entity or relation resolution (selects the system map)
|
||||||
|
* @returns The parsed {@link FieldAddress}
|
||||||
|
* @throws {InvalidFieldAddressError} for a `system.*` name outside the ruled
|
||||||
|
* map (including every plumbing field) or a malformed spelling — the error
|
||||||
|
* text enumerates the valid system scalars so the fix is in the message.
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* parseFieldAddress('level', 'entity') // { scope: 'metadata', field: 'level' }
|
||||||
|
* parseFieldAddress('metadata.level', 'entity') // { scope: 'metadata', field: 'level' }
|
||||||
|
* parseFieldAddress('system.createdAt', 'entity') // { scope: 'system', field: 'createdAt' }
|
||||||
|
* parseFieldAddress('system.level', 'entity') // throws — plumbing is invisible
|
||||||
|
*/
|
||||||
|
export function parseFieldAddress(
|
||||||
|
raw: string,
|
||||||
|
kind: FieldAddressKind
|
||||||
|
): FieldAddress {
|
||||||
|
const systemMap =
|
||||||
|
kind === 'entity' ? SYSTEM_ENTITY_SCALARS : SYSTEM_RELATION_SCALARS
|
||||||
|
|
||||||
|
if (raw.startsWith('system.')) {
|
||||||
|
const field = raw.slice('system.'.length)
|
||||||
|
if (!systemMap.has(field)) {
|
||||||
|
throw new InvalidFieldAddressError(raw, kind, systemMap)
|
||||||
|
}
|
||||||
|
return { scope: 'system', field, raw }
|
||||||
|
}
|
||||||
|
|
||||||
|
if (raw.startsWith('metadata.')) {
|
||||||
|
const field = raw.slice('metadata.'.length)
|
||||||
|
if (field.length === 0) {
|
||||||
|
throw new InvalidFieldAddressError(raw, kind, systemMap)
|
||||||
|
}
|
||||||
|
return { scope: 'metadata', field, raw }
|
||||||
|
}
|
||||||
|
|
||||||
|
if (raw.length === 0) {
|
||||||
|
throw new InvalidFieldAddressError(raw, kind, systemMap)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Bare name = the user's metadata field. Always. Even when the same name
|
||||||
|
// exists in the system map — `confidence` as a bare name is the user's
|
||||||
|
// metadata field named confidence; the engine scalar is system.confidence.
|
||||||
|
return { scope: 'metadata', field: raw, raw }
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Read the addressed value off an entity. The ONLY sanctioned way a query
|
||||||
|
* surface turns a {@link FieldAddress} into a value — direct property reads
|
||||||
|
* against records re-create the shadow class this module exists to kill.
|
||||||
|
*
|
||||||
|
* @returns The value, or `undefined` when the record does not carry it
|
||||||
|
* (missing values sort LAST in both directions per the ordering contract —
|
||||||
|
* they are never grounds for dropping a row).
|
||||||
|
*/
|
||||||
|
export function readEntityFieldAddress(
|
||||||
|
entity: HNSWNounWithMetadata,
|
||||||
|
address: FieldAddress
|
||||||
|
): unknown {
|
||||||
|
const rec = entity as unknown as Record<string, unknown>
|
||||||
|
const bag =
|
||||||
|
rec.metadata && typeof rec.metadata === 'object'
|
||||||
|
? (rec.metadata as Record<string, unknown>)
|
||||||
|
: null
|
||||||
|
|
||||||
|
if (address.scope === 'system') {
|
||||||
|
// System scalars live at the record's top level, NEVER in the user's
|
||||||
|
// bag — a user field named `confidence` must be unreachable from
|
||||||
|
// system.confidence (and vice versa). Entity views carry the scalars
|
||||||
|
// top-level directly; record-derived views spell the type `noun`.
|
||||||
|
const top = rec[address.field]
|
||||||
|
if (top !== undefined) return top
|
||||||
|
if (address.field === 'type') return rec.noun
|
||||||
|
return undefined
|
||||||
|
}
|
||||||
|
|
||||||
|
// User scope: the bag IS the user's namespace, authoritative — EVERY name
|
||||||
|
// reads from it, engine spellings included (`bag.confidence` is the user's
|
||||||
|
// confidence field under the field-addressing law).
|
||||||
|
if (bag) return bag[address.field]
|
||||||
|
|
||||||
|
// No bag at all: a LEGACY flat record (pre-nested-bag storage). Its keys
|
||||||
|
// matching system/plumbing names are the ENGINE's — the pre-law write door
|
||||||
|
// refused user colliders — so a bare system name reads as ABSENT rather
|
||||||
|
// than resurrecting the shadow this module exists to kill. Same for the
|
||||||
|
// legacy 'noun' spelling.
|
||||||
|
if (
|
||||||
|
SYSTEM_ENTITY_SCALARS.has(address.field) ||
|
||||||
|
PLUMBING_FIELDS.has(address.field) ||
|
||||||
|
address.field === 'noun'
|
||||||
|
) {
|
||||||
|
return undefined
|
||||||
|
}
|
||||||
|
return rec[address.field]
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Relation twin of {@link readEntityFieldAddress}. The stored flat record
|
||||||
|
* keys the relation type under `verb`; public Relation shapes may carry it
|
||||||
|
* as `type` — both spellings of the record are read, the ADDRESS is always
|
||||||
|
* `system.verb`.
|
||||||
|
*/
|
||||||
|
export function readRelationFieldAddress(
|
||||||
|
verb: HNSWVerbWithMetadata,
|
||||||
|
address: FieldAddress
|
||||||
|
): unknown {
|
||||||
|
if (address.scope === 'system') {
|
||||||
|
const rec = verb as unknown as Record<string, unknown>
|
||||||
|
if (address.field === 'verb') return rec.verb ?? rec.type
|
||||||
|
return rec[address.field]
|
||||||
|
}
|
||||||
|
return verb.metadata?.[address.field]
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Build the ruled did-you-mean refusal text for a bare name that resolved to
|
||||||
|
* metadata but is UNKNOWN to the index — the data-aware half of the law,
|
||||||
|
* called by the query layer once it has consulted the known-field set:
|
||||||
|
*
|
||||||
|
* "no metadata field 'createdAt' — did you mean system.createdAt or
|
||||||
|
* metadata.createdAt?"
|
||||||
|
*
|
||||||
|
* When the bare name is NOT a system scalar the system candidate is omitted
|
||||||
|
* (there is only one thing the caller could have meant; the refusal exists
|
||||||
|
* because refusing beats silently sorting nothing).
|
||||||
|
*/
|
||||||
|
export function buildUnresolvableMessage(
|
||||||
|
raw: string,
|
||||||
|
kind: FieldAddressKind
|
||||||
|
): string {
|
||||||
|
const systemMap =
|
||||||
|
kind === 'entity' ? SYSTEM_ENTITY_SCALARS : SYSTEM_RELATION_SCALARS
|
||||||
|
if (systemMap.has(raw)) {
|
||||||
|
return (
|
||||||
|
`no metadata field '${raw}' — did you mean system.${raw} or metadata.${raw}? ` +
|
||||||
|
`(bare names always address your metadata; engine fields need the system. prefix)`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return (
|
||||||
|
`no metadata field '${raw}' on this store — nothing carries it, so an ordered or ` +
|
||||||
|
`filtered read against it cannot mean anything. Spell it metadata.${raw} once the ` +
|
||||||
|
`field exists, or check the field name (system.${raw} is NOT valid — '${raw}' is ` +
|
||||||
|
`not one of the engine's system scalars).`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Refusal for a syntactically valid address that resolves to
|
||||||
|
* NOTHING — a bare name no user field carries. Carries the did-you-mean
|
||||||
|
* (both candidate spellings when the name collides with a system scalar) so
|
||||||
|
* the fix ships inside the error. Thrown by the query layer with index
|
||||||
|
* knowledge, never by the pure parser.
|
||||||
|
*/
|
||||||
|
export class UnresolvableFieldError extends Error {
|
||||||
|
public readonly raw: string
|
||||||
|
public readonly kind: FieldAddressKind
|
||||||
|
|
||||||
|
constructor(raw: string, kind: FieldAddressKind, messageOverride?: string) {
|
||||||
|
super(messageOverride ?? buildUnresolvableMessage(raw, kind))
|
||||||
|
this.name = 'UnresolvableFieldError'
|
||||||
|
this.raw = raw
|
||||||
|
this.kind = kind
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Refusal for a malformed or out-of-map field ADDRESS —
|
||||||
|
* `system.<anything-not-in-the-map>` (including all plumbing), an empty
|
||||||
|
* name, or a bare `metadata.` prefix. The message carries the full valid
|
||||||
|
* system map so the fix never needs a docs lookup.
|
||||||
|
*/
|
||||||
|
export class InvalidFieldAddressError extends UnresolvableFieldError {
|
||||||
|
constructor(raw: string, kind: FieldAddressKind, systemMap: ReadonlySet<string>) {
|
||||||
|
const valid = [...systemMap].map((f) => `system.${f}`).join(', ')
|
||||||
|
super(
|
||||||
|
raw,
|
||||||
|
kind,
|
||||||
|
`'${raw}' is not an addressable ${kind} field. Bare names address your own ` +
|
||||||
|
`metadata fields; engine fields are exactly: ${valid}. Engine plumbing ` +
|
||||||
|
`(vector, connections, level, data, _rev) is not part of the query surface.`
|
||||||
|
)
|
||||||
|
this.name = 'InvalidFieldAddressError'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Refusal for a find() option that is accepted by the type
|
||||||
|
* surface but NOT implemented — an accepted option must work or refuse;
|
||||||
|
* accepted-and-ignored died as a class (sealed 2026-08-03). Names the
|
||||||
|
* option and the honest state so nobody discovers a no-op by measurement.
|
||||||
|
*/
|
||||||
|
export class UnsupportedFindOptionError extends Error {
|
||||||
|
public readonly option: string
|
||||||
|
|
||||||
|
constructor(option: string) {
|
||||||
|
super(
|
||||||
|
`find() option '${option}' is not implemented — it used to be silently ` +
|
||||||
|
`ignored, which read as working. Remove it from the call (or track the ` +
|
||||||
|
`feature request); it will be honored or refused, never swallowed.`
|
||||||
|
)
|
||||||
|
this.name = 'UnsupportedFindOptionError'
|
||||||
|
this.option = option
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description The capability signal both engines' conformance suites arm on
|
||||||
|
* (never a version guess): its presence at the package root means the one
|
||||||
|
* field-addressing law is LIVE on every query surface — bare = user metadata,
|
||||||
|
* `system.*` = the ruled scalars, plumbing invisible, refusals typed.
|
||||||
|
*/
|
||||||
|
export const FIELD_ADDRESSING_CAPABILITY = 'field-addressing/v1'
|
||||||
|
|
@ -61,41 +61,44 @@ export class UnsupportedWhereOperatorError extends Error {
|
||||||
* @returns The field's value, or `undefined` when absent.
|
* @returns The field's value, or `undefined` when absent.
|
||||||
*/
|
*/
|
||||||
export function resolveEntityField(entity: Entity, field: string): unknown {
|
export function resolveEntityField(entity: Entity, field: string): unknown {
|
||||||
switch (field) {
|
// THE ONE ADDRESSING LAW (sealed 2026-08-03): `system.<field>` reads the
|
||||||
case 'noun':
|
// entity scalar; bare and `metadata.`-prefixed names read the user's
|
||||||
case 'type':
|
// metadata bag (dotted paths traverse INSIDE the bag). The old bare-name
|
||||||
return entity.type
|
// switch over system fields is dead — bare `createdAt` is the user's own
|
||||||
case 'subtype':
|
// field now; the engine scalar is `system.createdAt`. Plumbing (vector,
|
||||||
return entity.subtype
|
// connections, level, data, _rev) is invisible: no spelling reaches it.
|
||||||
case 'id':
|
if (field.startsWith('system.')) {
|
||||||
return entity.id
|
switch (field.slice('system.'.length)) {
|
||||||
case 'createdAt':
|
case 'type':
|
||||||
return entity.createdAt
|
return entity.type
|
||||||
case 'updatedAt':
|
case 'subtype':
|
||||||
return entity.updatedAt
|
return entity.subtype
|
||||||
case 'service':
|
case 'id':
|
||||||
return entity.service
|
return entity.id
|
||||||
case 'createdBy':
|
case 'createdAt':
|
||||||
return entity.createdBy
|
return entity.createdAt
|
||||||
case 'confidence':
|
case 'updatedAt':
|
||||||
return entity.confidence
|
return entity.updatedAt
|
||||||
case 'weight':
|
case 'service':
|
||||||
return entity.weight
|
return entity.service
|
||||||
case '_rev':
|
case 'createdBy':
|
||||||
return entity._rev
|
return entity.createdBy
|
||||||
case 'data':
|
case 'confidence':
|
||||||
return entity.data
|
return entity.confidence
|
||||||
|
case 'weight':
|
||||||
|
return entity.weight
|
||||||
|
case 'visibility':
|
||||||
|
return (entity as unknown as Record<string, unknown>).visibility
|
||||||
|
}
|
||||||
|
// Out-of-map system spelling: parse refuses these upstream with a typed
|
||||||
|
// error; reaching here (internal callers only) reads as absent.
|
||||||
|
return undefined
|
||||||
}
|
}
|
||||||
|
|
||||||
if (field.includes('.')) {
|
const path = field.startsWith('metadata.') ? field.slice('metadata.'.length) : field
|
||||||
// Dotted path: resolve against the whole entity first (`metadata.x`),
|
const bag = (entity.metadata ?? {}) as Record<string, unknown>
|
||||||
// then against the metadata bag (`address.city` on nested metadata).
|
if (!path.includes('.')) return bag[path]
|
||||||
const fromEntity = resolvePath(entity as unknown as Record<string, unknown>, field)
|
return resolvePath(bag, path)
|
||||||
if (fromEntity !== undefined) return fromEntity
|
|
||||||
return resolvePath((entity.metadata ?? {}) as Record<string, unknown>, field)
|
|
||||||
}
|
|
||||||
|
|
||||||
return ((entity.metadata ?? {}) as Record<string, unknown>)[field]
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Walk a dotted path through nested plain objects. */
|
/** Walk a dotted path through nested plain objects. */
|
||||||
|
|
|
||||||
|
|
@ -22,7 +22,6 @@ import { SmartYAMLImporter } from '../importers/SmartYAMLImporter.js'
|
||||||
import { SmartDOCXImporter } from '../importers/SmartDOCXImporter.js'
|
import { SmartDOCXImporter } from '../importers/SmartDOCXImporter.js'
|
||||||
import { VFSStructureGenerator } from '../importers/VFSStructureGenerator.js'
|
import { VFSStructureGenerator } from '../importers/VFSStructureGenerator.js'
|
||||||
import { NounType, VerbType } from '../types/graphTypes.js'
|
import { NounType, VerbType } from '../types/graphTypes.js'
|
||||||
import { splitNounMetadataRecord, splitVerbMetadataRecord } from '../types/reservedFields.js'
|
|
||||||
import { v4 as uuidv4 } from '../universal/uuid.js'
|
import { v4 as uuidv4 } from '../universal/uuid.js'
|
||||||
import * as fs from 'fs'
|
import * as fs from 'fs'
|
||||||
import * as path from 'path'
|
import * as path from 'path'
|
||||||
|
|
@ -871,35 +870,18 @@ export class ImportCoordinator {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Strip Brainy-reserved entity keys out of an extractor-supplied metadata bag.
|
* Normalize an extractor/consumer metadata bag for spreading — the
|
||||||
*
|
* field-addressing law: the bag is the user's, VERBATIM. No name is
|
||||||
* Extractors (and consumer `customMetadata`) can carry reserved keys
|
* reserved anymore ('confidence', 'subtype', 'type', … in a source bag
|
||||||
* (`confidence`, `subtype`, `weight`, …) inside `metadata`. Brainy 8.0's
|
* import as ordinary user fields); the old reserved-key strip was data
|
||||||
* default `reservedFieldPolicy` is `'throw'`, so spreading such a bag into
|
* loss under the law and is gone. A forged 'system.'-prefixed key still
|
||||||
* `add({ metadata })` would reject the whole import. The import pipeline owns
|
* refuses loudly at the write door (`rejectForgedSystemKeys`).
|
||||||
* the correct write path: user-mutable reserved values are passed as dedicated
|
|
||||||
* `AddParams` params (see the call sites), so here we simply drop the reserved
|
|
||||||
* half of the bag and keep only the custom fields that belong in `metadata`.
|
|
||||||
*
|
|
||||||
* @param bag - The extractor/consumer metadata bag (may be undefined).
|
* @param bag - The extractor/consumer metadata bag (may be undefined).
|
||||||
* @returns The custom-only metadata (reserved keys removed).
|
* @returns The bag itself, or `{}` for non-object inputs.
|
||||||
*/
|
*/
|
||||||
private stripReservedFromBag(bag: Record<string, any> | undefined | null): Record<string, any> {
|
private bagVerbatim(bag: Record<string, any> | undefined | null): Record<string, any> {
|
||||||
if (!bag || typeof bag !== 'object') return {}
|
if (!bag || typeof bag !== 'object') return {}
|
||||||
return splitNounMetadataRecord(bag).custom
|
return bag
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Relationship mirror of {@link stripReservedFromBag} — strips reserved verb
|
|
||||||
* keys (`verb`, `confidence`, `weight`, `subtype`, …) out of an edge metadata
|
|
||||||
* bag so it carries only custom fields. Reserved values that have a dedicated
|
|
||||||
* `RelateParams` param are passed there by the call site instead.
|
|
||||||
* @param bag - The extractor/consumer edge metadata bag (may be undefined).
|
|
||||||
* @returns The custom-only edge metadata (reserved keys removed).
|
|
||||||
*/
|
|
||||||
private stripReservedFromRelationBag(bag: Record<string, any> | undefined | null): Record<string, any> {
|
|
||||||
if (!bag || typeof bag !== 'object') return {}
|
|
||||||
return splitVerbMetadataRecord(bag).custom
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -1017,7 +999,7 @@ export class ImportCoordinator {
|
||||||
importedAt: trackingContext.importedAt,
|
importedAt: trackingContext.importedAt,
|
||||||
importFormat: trackingContext.importFormat,
|
importFormat: trackingContext.importFormat,
|
||||||
importSource: trackingContext.importSource,
|
importSource: trackingContext.importSource,
|
||||||
...this.stripReservedFromBag(trackingContext.customMetadata)
|
...this.bagVerbatim(trackingContext.customMetadata)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
@ -1045,13 +1027,11 @@ export class ImportCoordinator {
|
||||||
data: entity.description || entity.name,
|
data: entity.description || entity.name,
|
||||||
type: entity.type,
|
type: entity.type,
|
||||||
subtype: entity.subtype ?? options.defaultSubtype ?? 'imported',
|
subtype: entity.subtype ?? options.defaultSubtype ?? 'imported',
|
||||||
// `confidence` is a reserved field — pass it as the dedicated param,
|
// Engine confidence rides its dedicated param; the bag below is
|
||||||
// never inside the metadata bag (8.0 reservedFieldPolicy defaults to 'throw').
|
// the user's verbatim (no name is reserved — field-addressing law).
|
||||||
confidence: entity.confidence,
|
confidence: entity.confidence,
|
||||||
metadata: {
|
metadata: {
|
||||||
// Extractor/consumer bags may smuggle reserved keys — strip them so
|
...this.bagVerbatim(entity.metadata),
|
||||||
// the bag carries only custom fields.
|
|
||||||
...this.stripReservedFromBag(entity.metadata),
|
|
||||||
name: entity.name,
|
name: entity.name,
|
||||||
vfsPath: vfsFile?.path,
|
vfsPath: vfsFile?.path,
|
||||||
importedFrom: 'import-coordinator',
|
importedFrom: 'import-coordinator',
|
||||||
|
|
@ -1064,7 +1044,7 @@ export class ImportCoordinator {
|
||||||
importSource: trackingContext.importSource,
|
importSource: trackingContext.importSource,
|
||||||
sourceRow: row.rowNumber,
|
sourceRow: row.rowNumber,
|
||||||
sourceSheet: row.sheet,
|
sourceSheet: row.sheet,
|
||||||
...this.stripReservedFromBag(trackingContext.customMetadata)
|
...this.bagVerbatim(trackingContext.customMetadata)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -1145,7 +1125,7 @@ export class ImportCoordinator {
|
||||||
importIds: [trackingContext.importId],
|
importIds: [trackingContext.importId],
|
||||||
projectId: trackingContext.projectId,
|
projectId: trackingContext.projectId,
|
||||||
importFormat: trackingContext.importFormat,
|
importFormat: trackingContext.importFormat,
|
||||||
...this.stripReservedFromRelationBag(trackingContext.customMetadata)
|
...this.bagVerbatim(trackingContext.customMetadata)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -1180,7 +1160,7 @@ export class ImportCoordinator {
|
||||||
confidence: entity.confidence,
|
confidence: entity.confidence,
|
||||||
metadata: {
|
metadata: {
|
||||||
// Strip any reserved keys an extractor smuggled into the bag.
|
// Strip any reserved keys an extractor smuggled into the bag.
|
||||||
...this.stripReservedFromBag(entity.metadata),
|
...this.bagVerbatim(entity.metadata),
|
||||||
name: entity.name,
|
name: entity.name,
|
||||||
vfsPath: vfsFile?.path,
|
vfsPath: vfsFile?.path,
|
||||||
importedFrom: 'import-coordinator',
|
importedFrom: 'import-coordinator',
|
||||||
|
|
@ -1194,7 +1174,7 @@ export class ImportCoordinator {
|
||||||
importSource: trackingContext.importSource,
|
importSource: trackingContext.importSource,
|
||||||
sourceRow: row.rowNumber,
|
sourceRow: row.rowNumber,
|
||||||
sourceSheet: row.sheet,
|
sourceSheet: row.sheet,
|
||||||
...this.stripReservedFromBag(trackingContext.customMetadata)
|
...this.bagVerbatim(trackingContext.customMetadata)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
@ -1234,7 +1214,7 @@ export class ImportCoordinator {
|
||||||
importIds: [trackingContext.importId],
|
importIds: [trackingContext.importId],
|
||||||
projectId: trackingContext.projectId,
|
projectId: trackingContext.projectId,
|
||||||
importFormat: trackingContext.importFormat,
|
importFormat: trackingContext.importFormat,
|
||||||
...this.stripReservedFromRelationBag(trackingContext.customMetadata)
|
...this.bagVerbatim(trackingContext.customMetadata)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
@ -1289,7 +1269,7 @@ export class ImportCoordinator {
|
||||||
projectId: trackingContext.projectId,
|
projectId: trackingContext.projectId,
|
||||||
importedAt: trackingContext.importedAt,
|
importedAt: trackingContext.importedAt,
|
||||||
importFormat: trackingContext.importFormat,
|
importFormat: trackingContext.importFormat,
|
||||||
...this.stripReservedFromBag(trackingContext.customMetadata)
|
...this.bagVerbatim(trackingContext.customMetadata)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
@ -1319,7 +1299,7 @@ export class ImportCoordinator {
|
||||||
projectId: trackingContext.projectId,
|
projectId: trackingContext.projectId,
|
||||||
importedAt: trackingContext.importedAt,
|
importedAt: trackingContext.importedAt,
|
||||||
importFormat: trackingContext.importFormat,
|
importFormat: trackingContext.importFormat,
|
||||||
...this.stripReservedFromRelationBag(trackingContext.customMetadata)
|
...this.bagVerbatim(trackingContext.customMetadata)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
@ -1422,7 +1402,7 @@ export class ImportCoordinator {
|
||||||
...(typeof (rel as any).confidence === 'number' && { confidence: (rel as any).confidence }),
|
...(typeof (rel as any).confidence === 'number' && { confidence: (rel as any).confidence }),
|
||||||
...(typeof (rel as any).weight === 'number' && { weight: (rel as any).weight }),
|
...(typeof (rel as any).weight === 'number' && { weight: (rel as any).weight }),
|
||||||
metadata: {
|
metadata: {
|
||||||
...this.stripReservedFromRelationBag(rel.metadata),
|
...this.bagVerbatim(rel.metadata),
|
||||||
relationshipType: 'semantic', // Distinguish from VFS/provenance
|
relationshipType: 'semantic', // Distinguish from VFS/provenance
|
||||||
inferredType: verbType !== rel.type, // Track if type was enhanced
|
inferredType: verbType !== rel.type, // Track if type was enhanced
|
||||||
originalType: rel.type
|
originalType: rel.type
|
||||||
|
|
|
||||||
26
src/index.ts
26
src/index.ts
|
|
@ -89,7 +89,12 @@ export {
|
||||||
RESERVED_ENTITY_FIELDS,
|
RESERVED_ENTITY_FIELDS,
|
||||||
RESERVED_RELATION_FIELDS,
|
RESERVED_RELATION_FIELDS,
|
||||||
splitNounMetadataRecord,
|
splitNounMetadataRecord,
|
||||||
splitVerbMetadataRecord
|
splitVerbMetadataRecord,
|
||||||
|
buildNounMetadataRecord,
|
||||||
|
buildVerbMetadataRecord,
|
||||||
|
isNestedBagRecord,
|
||||||
|
METADATA_RECORD_FORMAT_KEY,
|
||||||
|
NESTED_BAG_FORMAT
|
||||||
} from './types/reservedFields.js'
|
} from './types/reservedFields.js'
|
||||||
export type {
|
export type {
|
||||||
ReservedEntityField,
|
ReservedEntityField,
|
||||||
|
|
@ -106,6 +111,25 @@ export type {
|
||||||
// Export Aggregation Engine
|
// Export Aggregation Engine
|
||||||
export { AggregationIndex, AggregateMaterializer, bucketTimestamp, parseBucketRange } from './aggregation/index.js'
|
export { AggregationIndex, AggregateMaterializer, bucketTimestamp, parseBucketRange } from './aggregation/index.js'
|
||||||
|
|
||||||
|
// THE ONE FIELD-ADDRESSING LAW (sealed 2026-08-03) — the arming surface both
|
||||||
|
// engines' conformance suites detect: bare names = user metadata, system.* =
|
||||||
|
// the ten ruled scalars, plumbing invisible, refusals typed with the fix in
|
||||||
|
// the message. See docs/concepts/field-addressing.md.
|
||||||
|
export {
|
||||||
|
FIELD_ADDRESSING_CAPABILITY,
|
||||||
|
SYSTEM_ENTITY_SCALARS,
|
||||||
|
SYSTEM_RELATION_SCALARS,
|
||||||
|
PLUMBING_FIELDS,
|
||||||
|
parseFieldAddress,
|
||||||
|
readEntityFieldAddress,
|
||||||
|
readRelationFieldAddress,
|
||||||
|
buildUnresolvableMessage,
|
||||||
|
InvalidFieldAddressError,
|
||||||
|
UnresolvableFieldError,
|
||||||
|
UnsupportedFindOptionError
|
||||||
|
} from './db/fieldAddressing.js'
|
||||||
|
export type { FieldAddress, FieldAddressKind } from './db/fieldAddressing.js'
|
||||||
|
|
||||||
// Export Neural Import (AI data understanding)
|
// Export Neural Import (AI data understanding)
|
||||||
export { NeuralImport } from './neural/neuralImport.js'
|
export { NeuralImport } from './neural/neuralImport.js'
|
||||||
export type {
|
export type {
|
||||||
|
|
|
||||||
|
|
@ -9,6 +9,67 @@ import type { BaseStorage } from '../storage/baseStorage.js'
|
||||||
import type { NounMetadata, VerbMetadata } from '../coreTypes.js'
|
import type { NounMetadata, VerbMetadata } from '../coreTypes.js'
|
||||||
import type { Migration, MigrationState, MigrationPreview, MigrationResult, MigrateOptions, MigrationError } from './types.js'
|
import type { Migration, MigrationState, MigrationPreview, MigrationResult, MigrateOptions, MigrationError } from './types.js'
|
||||||
import { MIGRATIONS } from './migrations.js'
|
import { MIGRATIONS } from './migrations.js'
|
||||||
|
import {
|
||||||
|
splitNounMetadataRecord,
|
||||||
|
splitVerbMetadataRecord,
|
||||||
|
buildNounMetadataRecord,
|
||||||
|
buildVerbMetadataRecord,
|
||||||
|
RESERVED_ENTITY_FIELDS,
|
||||||
|
RESERVED_RELATION_FIELDS
|
||||||
|
} from '../types/reservedFields.js'
|
||||||
|
|
||||||
|
const RESERVED_NOUN_SET: ReadonlySet<string> = new Set(RESERVED_ENTITY_FIELDS)
|
||||||
|
const RESERVED_VERB_SET: ReadonlySet<string> = new Set(RESERVED_RELATION_FIELDS)
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Normalize a stored record (either era: legacy flat OR v2 nested-bag) into
|
||||||
|
* THE transform view — the one shape every migration transform receives:
|
||||||
|
* engine fields top-level, the user's metadata bag nested under `metadata`.
|
||||||
|
* Transforms never see the storage era; a migration written today works on
|
||||||
|
* a brain of any age.
|
||||||
|
*/
|
||||||
|
function toTransformView(
|
||||||
|
record: Record<string, unknown>,
|
||||||
|
kind: 'noun' | 'verb'
|
||||||
|
): Record<string, unknown> {
|
||||||
|
const { reserved, custom } =
|
||||||
|
kind === 'noun' ? splitNounMetadataRecord(record) : splitVerbMetadataRecord(record)
|
||||||
|
return { ...reserved, metadata: { ...custom } }
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert a transform's returned view back into a stamped v2 stored record.
|
||||||
|
* LOUD CONTRACT: user fields belong inside `.metadata` — a stray top-level
|
||||||
|
* key that is not an engine field is a migration bug under the
|
||||||
|
* field-addressing law (pre-law transforms wrote user fields flat), and it
|
||||||
|
* refuses with the fix in the message rather than silently dropping or
|
||||||
|
* silently storing it as an engine key.
|
||||||
|
*/
|
||||||
|
function fromTransformView(
|
||||||
|
view: Record<string, unknown>,
|
||||||
|
kind: 'noun' | 'verb'
|
||||||
|
): Record<string, unknown> {
|
||||||
|
const reservedSet = kind === 'noun' ? RESERVED_NOUN_SET : RESERVED_VERB_SET
|
||||||
|
const engine: Record<string, unknown> = {}
|
||||||
|
for (const [key, value] of Object.entries(view)) {
|
||||||
|
if (key === 'metadata') continue
|
||||||
|
if (!reservedSet.has(key)) {
|
||||||
|
throw new Error(
|
||||||
|
`migration transform returned a top-level key '${key}' that is not an ` +
|
||||||
|
`engine field — under the field-addressing law user fields live inside ` +
|
||||||
|
`.metadata (return { ...view, metadata: { ...view.metadata, ${key}: … } }).`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
engine[key] = value
|
||||||
|
}
|
||||||
|
const bag =
|
||||||
|
view.metadata && typeof view.metadata === 'object' && !Array.isArray(view.metadata)
|
||||||
|
? (view.metadata as Record<string, unknown>)
|
||||||
|
: {}
|
||||||
|
return kind === 'noun'
|
||||||
|
? buildNounMetadataRecord(engine, bag)
|
||||||
|
: buildVerbMetadataRecord(engine, bag)
|
||||||
|
}
|
||||||
|
|
||||||
const MIGRATION_STATE_KEY = '__migration_state__'
|
const MIGRATION_STATE_KEY = '__migration_state__'
|
||||||
const PREVIEW_SAMPLE_SIZE = 5
|
const PREVIEW_SAMPLE_SIZE = 5
|
||||||
|
|
@ -125,14 +186,16 @@ export class MigrationRunner {
|
||||||
const entityMeta = metadataBatch.get(entity.id)
|
const entityMeta = metadataBatch.get(entity.id)
|
||||||
if (!entityMeta) continue
|
if (!entityMeta) continue
|
||||||
|
|
||||||
const metadata = entityMeta as Record<string, unknown>
|
// Transforms see THE view (engine fields + nested user bag),
|
||||||
const result = this.applyTransforms(metadata, nounMigrations)
|
// never the raw storage era.
|
||||||
|
const view = toTransformView(entityMeta as Record<string, unknown>, 'noun')
|
||||||
|
const result = this.applyTransforms(view, nounMigrations)
|
||||||
if (result !== null) {
|
if (result !== null) {
|
||||||
affectedEntities++
|
affectedEntities++
|
||||||
if (sampleChanges.length < PREVIEW_SAMPLE_SIZE) {
|
if (sampleChanges.length < PREVIEW_SAMPLE_SIZE) {
|
||||||
sampleChanges.push({
|
sampleChanges.push({
|
||||||
id: entity.id,
|
id: entity.id,
|
||||||
before: { ...metadata },
|
before: view,
|
||||||
after: result
|
after: result
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
@ -157,14 +220,14 @@ export class MigrationRunner {
|
||||||
const verbMeta = await this.storage.getVerbMetadata(verb.id)
|
const verbMeta = await this.storage.getVerbMetadata(verb.id)
|
||||||
if (!verbMeta) continue
|
if (!verbMeta) continue
|
||||||
|
|
||||||
const metadata = verbMeta as Record<string, unknown>
|
const view = toTransformView(verbMeta as Record<string, unknown>, 'verb')
|
||||||
const result = this.applyTransforms(metadata, verbMigrations)
|
const result = this.applyTransforms(view, verbMigrations)
|
||||||
if (result !== null) {
|
if (result !== null) {
|
||||||
affectedEntities++
|
affectedEntities++
|
||||||
if (sampleChanges.length < PREVIEW_SAMPLE_SIZE) {
|
if (sampleChanges.length < PREVIEW_SAMPLE_SIZE) {
|
||||||
sampleChanges.push({
|
sampleChanges.push({
|
||||||
id: verb.id,
|
id: verb.id,
|
||||||
before: { ...metadata },
|
before: view,
|
||||||
after: result
|
after: result
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
@ -289,9 +352,16 @@ export class MigrationRunner {
|
||||||
if (!entityMeta) continue
|
if (!entityMeta) continue
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const transformed = migration.transform(entityMeta as Record<string, unknown>)
|
const transformed = migration.transform(
|
||||||
|
toTransformView(entityMeta as Record<string, unknown>, 'noun')
|
||||||
|
)
|
||||||
if (transformed !== null) {
|
if (transformed !== null) {
|
||||||
await this.storage.saveNounMetadata(entity.id, transformed as NounMetadata)
|
// Re-stamp as a v2 record (also upgrades legacy records touched
|
||||||
|
// by a migration onto the nested-bag shape).
|
||||||
|
await this.storage.saveNounMetadata(
|
||||||
|
entity.id,
|
||||||
|
fromTransformView(transformed, 'noun') as NounMetadata
|
||||||
|
)
|
||||||
modified++
|
modified++
|
||||||
}
|
}
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
|
|
@ -357,9 +427,14 @@ export class MigrationRunner {
|
||||||
if (!metadata) continue
|
if (!metadata) continue
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const transformed = migration.transform(metadata as Record<string, unknown>)
|
const transformed = migration.transform(
|
||||||
|
toTransformView(metadata as Record<string, unknown>, 'verb')
|
||||||
|
)
|
||||||
if (transformed !== null) {
|
if (transformed !== null) {
|
||||||
await this.storage.saveVerbMetadata(verb.id, transformed as VerbMetadata)
|
await this.storage.saveVerbMetadata(
|
||||||
|
verb.id,
|
||||||
|
fromTransformView(transformed, 'verb') as VerbMetadata
|
||||||
|
)
|
||||||
modified++
|
modified++
|
||||||
}
|
}
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
|
|
|
||||||
|
|
@ -14,7 +14,19 @@ export interface Migration {
|
||||||
description: string
|
description: string
|
||||||
/** Which entity types this migration applies to */
|
/** Which entity types this migration applies to */
|
||||||
applies: 'nouns' | 'verbs' | 'both'
|
applies: 'nouns' | 'verbs' | 'both'
|
||||||
/** Return transformed metadata, or null if no change needed */
|
/**
|
||||||
|
* Return the transformed record view, or null if no change needed.
|
||||||
|
*
|
||||||
|
* THE VIEW CONTRACT (field-addressing law): the transform receives ONE
|
||||||
|
* normalized shape regardless of how old the stored record is — engine
|
||||||
|
* fields top-level (`noun`/`verb`, `subtype`, `confidence`, `weight`,
|
||||||
|
* timestamps, `_rev`, …) and the USER's metadata bag nested under
|
||||||
|
* `metadata` (where every name is the user's, engine spellings included).
|
||||||
|
* Return the same shape: user-field changes go inside `.metadata`; a
|
||||||
|
* stray non-engine top-level key in the returned object refuses loudly
|
||||||
|
* (it is the pre-law flat habit, and silently guessing its namespace
|
||||||
|
* would corrupt data).
|
||||||
|
*/
|
||||||
transform: (metadata: Record<string, unknown>) => Record<string, unknown> | null
|
transform: (metadata: Record<string, unknown>) => Record<string, unknown> | null
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,6 @@
|
||||||
|
|
||||||
import { Brainy } from '../brainy.js'
|
import { Brainy } from '../brainy.js'
|
||||||
import { NounType, VerbType } from '../types/graphTypes.js'
|
import { NounType, VerbType } from '../types/graphTypes.js'
|
||||||
import { splitNounMetadataRecord, splitVerbMetadataRecord } from '../types/reservedFields.js'
|
|
||||||
import * as fs from '../universal/fs.js'
|
import * as fs from '../universal/fs.js'
|
||||||
import * as path from '../universal/path.js'
|
import * as path from '../universal/path.js'
|
||||||
// @ts-ignore
|
// @ts-ignore
|
||||||
|
|
@ -803,12 +802,14 @@ export class NeuralImport {
|
||||||
data: this.extractMainText(entity.originalData),
|
data: this.extractMainText(entity.originalData),
|
||||||
type: entity.nounType as NounType,
|
type: entity.nounType as NounType,
|
||||||
subtype: entity.subtype ?? options.defaultSubtype ?? 'extracted',
|
subtype: entity.subtype ?? options.defaultSubtype ?? 'extracted',
|
||||||
// `confidence` is a reserved field — dedicated param, not metadata
|
// Engine confidence rides its dedicated param; the source object
|
||||||
// (8.0 reservedFieldPolicy defaults to 'throw').
|
// imports as the user's bag VERBATIM — no name is reserved
|
||||||
|
// (field-addressing law).
|
||||||
confidence: entity.confidence,
|
confidence: entity.confidence,
|
||||||
metadata: {
|
metadata: {
|
||||||
// Strip any reserved keys the source data smuggled into the bag.
|
...(typeof entity.originalData === 'object' && entity.originalData !== null
|
||||||
...splitNounMetadataRecord(entity.originalData).custom,
|
? entity.originalData
|
||||||
|
: {}),
|
||||||
id: entity.suggestedId
|
id: entity.suggestedId
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
@ -822,11 +823,13 @@ export class NeuralImport {
|
||||||
type: relationship.verbType as VerbType,
|
type: relationship.verbType as VerbType,
|
||||||
subtype: relationship.subtype ?? options.defaultSubtype ?? 'extracted',
|
subtype: relationship.subtype ?? options.defaultSubtype ?? 'extracted',
|
||||||
weight: relationship.weight,
|
weight: relationship.weight,
|
||||||
confidence: relationship.confidence, // reserved field — dedicated param, not metadata
|
confidence: relationship.confidence, // engine confidence — dedicated param
|
||||||
metadata: {
|
metadata: {
|
||||||
context: relationship.context,
|
context: relationship.context,
|
||||||
// Strip any reserved keys smuggled into the edge metadata bag.
|
// The edge bag imports verbatim — no name is reserved.
|
||||||
...splitVerbMetadataRecord(relationship.metadata).custom
|
...(typeof relationship.metadata === 'object' && relationship.metadata !== null
|
||||||
|
? relationship.metadata
|
||||||
|
: {})
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -36,7 +36,8 @@ import { BrainyError, ProtectedArtifactError, DerivedArtifactMissingError } from
|
||||||
import { MetadataWriteBuffer } from '../utils/metadataWriteBuffer.js'
|
import { MetadataWriteBuffer } from '../utils/metadataWriteBuffer.js'
|
||||||
import {
|
import {
|
||||||
splitNounMetadataRecord,
|
splitNounMetadataRecord,
|
||||||
splitVerbMetadataRecord
|
splitVerbMetadataRecord,
|
||||||
|
isNestedBagRecord
|
||||||
} from '../types/reservedFields.js'
|
} from '../types/reservedFields.js'
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -1013,8 +1014,14 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
||||||
const hashes: string[] = []
|
const hashes: string[] = []
|
||||||
for (const record of records) {
|
for (const record of records) {
|
||||||
if (record.kind !== 'noun') continue
|
if (record.kind !== 'noun') continue
|
||||||
const storage = (record.metadata as { storage?: { type?: string; hash?: unknown } } | null)
|
// The VFS blob pointer (`storage: {type:'blob', hash}`) is a USER-bag
|
||||||
?.storage
|
// field: in a v2 nested-bag record it lives inside `metadata`, in a
|
||||||
|
// legacy flat record it sits at the top level — read shape-aware.
|
||||||
|
const raw = record.metadata as Record<string, unknown> | null
|
||||||
|
const bag = isNestedBagRecord(raw)
|
||||||
|
? (raw!.metadata as Record<string, unknown>)
|
||||||
|
: raw
|
||||||
|
const storage = (bag as { storage?: { type?: string; hash?: unknown } } | null)?.storage
|
||||||
if (storage?.type === 'blob' && typeof storage.hash === 'string') {
|
if (storage?.type === 'blob' && typeof storage.hash === 'string') {
|
||||||
hashes.push(storage.hash)
|
hashes.push(storage.hash)
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -69,12 +69,15 @@ export const BRAIN_FORMAT_PATH = '_system/brain-format.json'
|
||||||
* (the 8.0 GA baseline). An on-disk `indexEpoch` that differs from this — or an
|
* (the 8.0 GA baseline). An on-disk `indexEpoch` that differs from this — or an
|
||||||
* absent marker — triggers a full derived-index rebuild on open.
|
* absent marker — triggers a full derived-index rebuild on open.
|
||||||
*/
|
*/
|
||||||
// Epoch 2 (2026-08-03, paired with the native accelerator's same-day release):
|
// Epoch 3 (2026-08-03, the namespace-law pair): the index key format split
|
||||||
// user metadata fields named `level` become indexable on both engines — the
|
// the two namespaces — user fields keep bare flattened keys, the ten system
|
||||||
// derived posting set changed, so every pre-fix brain must rebuild its
|
// scalars moved to literal 'system.<field>' keys (the legacy 'noun' column
|
||||||
// metadata index from canonical at first open (poisoned multi-valued `level`
|
// spelling died with them). Every brain rebuilds its derived indexes from
|
||||||
// columns heal through this rebuild; no bespoke heal path).
|
// canonical at first open onto the frozen keys.
|
||||||
export const EXPECTED_INDEX_EPOCH = 2
|
// Epoch 2 (2026-08-03, same day, the interim pair): user metadata fields
|
||||||
|
// named `level` became indexable on both engines; poisoned multi-valued
|
||||||
|
// `level` columns healed through the rebuild.
|
||||||
|
export const EXPECTED_INDEX_EPOCH = 3
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description The data-layer format string this build writes and runs as.
|
* @description The data-layer format string this build writes and runs as.
|
||||||
|
|
|
||||||
|
|
@ -77,8 +77,27 @@ export class SaveNounOperation implements Operation {
|
||||||
? null
|
? null
|
||||||
: await this.storage.getNoun(this.noun.id)
|
: await this.storage.getNoun(this.noun.id)
|
||||||
|
|
||||||
// Save new noun
|
// PRESERVE stored graph state on updates. Callers stage this op with
|
||||||
await this.storage.saveNoun(this.noun)
|
// placeholder adjacency ({connections: empty, level: 0}) because the
|
||||||
|
// vector index owns those values and persists them at flush. Codec-era
|
||||||
|
// records (2.4.0+) carry an empty connections field by design (adjacency
|
||||||
|
// lives in a separate compressed blob — the placeholder is harmless), but
|
||||||
|
// LEGACY pre-codec records store adjacency INLINE: writing the
|
||||||
|
// placeholder over one stamped out its stored connections, leaving a
|
||||||
|
// crash window (until the next flush) where a reload found the node
|
||||||
|
// unreachable. Stale adjacency in that window is tolerable — HNSW
|
||||||
|
// self-corrects at the reindex flush; EMPTY adjacency is silent recall
|
||||||
|
// loss. The read above is already paid for rollback; preservation is free.
|
||||||
|
const toSave: HNSWNoun =
|
||||||
|
previousNoun && this.noun.connections.size === 0
|
||||||
|
? {
|
||||||
|
...this.noun,
|
||||||
|
connections: previousNoun.connections || this.noun.connections,
|
||||||
|
level: previousNoun.level ?? this.noun.level
|
||||||
|
}
|
||||||
|
: this.noun
|
||||||
|
|
||||||
|
await this.storage.saveNoun(toSave)
|
||||||
|
|
||||||
// Return rollback action
|
// Return rollback action
|
||||||
return async () => {
|
return async () => {
|
||||||
|
|
|
||||||
|
|
@ -320,15 +320,18 @@ export interface AddParams<T = any> {
|
||||||
*/
|
*/
|
||||||
visibility?: 'public' | 'internal'
|
visibility?: 'public' | 'internal'
|
||||||
/**
|
/**
|
||||||
* Structured queryable fields — indexed by MetadataIndex, used in `where` filters.
|
* Structured queryable fields — indexed by MetadataIndex, used in `where`
|
||||||
|
* filters, `orderBy`, and aggregation.
|
||||||
*
|
*
|
||||||
* Reserved entity fields (`RESERVED_ENTITY_FIELDS` — `noun`, `subtype`, `visibility`,
|
* THE FIELD-ADDRESSING LAW: every name here is YOURS. There are no
|
||||||
* `createdAt`, `updatedAt`, `confidence`, `weight`, `service`, `data`, `createdBy`,
|
* reserved metadata names — `confidence`, `type`, `id`, `level`, `data`,
|
||||||
* `_rev`) may NOT appear here — they have dedicated top-level params and the type makes
|
* `content`, … are ordinary user fields that index, filter, sort, and
|
||||||
* a literal reserved key a compile error. Untyped (JavaScript) callers that pass one
|
* aggregate like any other, and survive faithfully across restarts and
|
||||||
* anyway are normalized at write time: user-settable fields remap to their top-level
|
* rebuilds. Engine scalars are set only via their dedicated params
|
||||||
* param (top-level wins when both are supplied), system-managed fields are dropped with
|
* (`confidence`, `weight`, `subtype`, …) and are queried explicitly as
|
||||||
* a one-shot warning.
|
* `system.<field>` (`where: { 'system.confidence': … }`). The ONE illegal
|
||||||
|
* spelling is a key starting `'system.'` — the engine's explicit address
|
||||||
|
* namespace cannot be forged; such a write refuses with a typed error.
|
||||||
*/
|
*/
|
||||||
metadata?: EntityMetadataInput<T>
|
metadata?: EntityMetadataInput<T>
|
||||||
/** Custom entity ID. When omitted, a time-ordered UUID v7 is generated; a supplied natural-key string is normalized to a stable UUID v5. */
|
/** Custom entity ID. When omitted, a time-ordered UUID v7 is generated; a supplied natural-key string is normalized to a stable UUID v5. */
|
||||||
|
|
@ -386,12 +389,11 @@ export interface UpdateParams<T = any> {
|
||||||
*/
|
*/
|
||||||
visibility?: EntityVisibility
|
visibility?: EntityVisibility
|
||||||
/**
|
/**
|
||||||
* Metadata fields to merge (or replace when `merge: false`). Reserved entity
|
* Metadata fields to merge (or replace when `merge: false`). Every name is
|
||||||
* fields (`RESERVED_ENTITY_FIELDS`) may NOT appear here — `confidence` /
|
* the user's (the field-addressing law) — a patch field named `confidence`
|
||||||
* `weight` / `subtype` / `visibility` have dedicated params on this call, and the rest
|
* updates YOUR field of that name, never the engine scalar (use the
|
||||||
* are system-managed. A literal reserved key is a compile error; untyped callers
|
* dedicated `confidence` param for that). Keys spelled `'system.…'` refuse
|
||||||
* are normalized at write time (remap user-settable, drop system-managed
|
* with a typed error (namespace forgery).
|
||||||
* with a one-shot warning).
|
|
||||||
*/
|
*/
|
||||||
metadata?: EntityMetadataPatch<T>
|
metadata?: EntityMetadataPatch<T>
|
||||||
merge?: boolean // Merge or replace metadata (default: true)
|
merge?: boolean // Merge or replace metadata (default: true)
|
||||||
|
|
@ -444,11 +446,11 @@ export interface RelateParams<T = any> {
|
||||||
/** Content for the relationship (optional — overrides auto-computed vector) */
|
/** Content for the relationship (optional — overrides auto-computed vector) */
|
||||||
data?: any
|
data?: any
|
||||||
/**
|
/**
|
||||||
* Structured queryable fields on the edge. Reserved relationship fields
|
* Structured queryable fields on the edge. Every name is the user's (the
|
||||||
* (`RESERVED_RELATION_FIELDS` — `verb`, `subtype`, `visibility`, `createdAt`,
|
* field-addressing law) — `verb`, `confidence`, `weight`, … in this bag are
|
||||||
* `updatedAt`, `confidence`, `weight`, `service`, `data`, `createdBy`, `_rev`) may NOT
|
* ordinary user fields; engine scalars ride their dedicated params and are
|
||||||
* appear here — they have dedicated params. A literal reserved key is a
|
* addressed as `system.<field>`. Keys spelled `'system.…'` refuse with a
|
||||||
* compile error; untyped callers are normalized at write time.
|
* typed error (namespace forgery).
|
||||||
*/
|
*/
|
||||||
metadata?: RelationMetadataInput<T>
|
metadata?: RelationMetadataInput<T>
|
||||||
/** Create reverse edge too (default: false) */
|
/** Create reverse edge too (default: false) */
|
||||||
|
|
@ -478,10 +480,9 @@ export interface UpdateRelationParams<T = any> {
|
||||||
confidence?: number // New confidence (0-1)
|
confidence?: number // New confidence (0-1)
|
||||||
data?: any // New content
|
data?: any // New content
|
||||||
/**
|
/**
|
||||||
* Metadata fields to merge (or replace when `merge: false`). Reserved
|
* Metadata fields to merge (or replace when `merge: false`). Every name is
|
||||||
* relationship fields (`RESERVED_RELATION_FIELDS`) may NOT appear here —
|
* the user's (the field-addressing law); engine scalars ride their
|
||||||
* a literal reserved key is a compile error; untyped callers are
|
* dedicated params. Keys spelled `'system.…'` refuse with a typed error.
|
||||||
* normalized at write time.
|
|
||||||
*/
|
*/
|
||||||
metadata?: RelationMetadataPatch<T>
|
metadata?: RelationMetadataPatch<T>
|
||||||
merge?: boolean // Merge or replace metadata
|
merge?: boolean // Merge or replace metadata
|
||||||
|
|
@ -498,6 +499,43 @@ export interface UpdateRelationParams<T = any> {
|
||||||
* - **Graph:** `connected` for relationship traversal (via GraphAdjacencyIndex)
|
* - **Graph:** `connected` for relationship traversal (via GraphAdjacencyIndex)
|
||||||
*
|
*
|
||||||
* See also: [Query Operators](../../docs/QUERY_OPERATORS.md) for all `where` operators.
|
* See also: [Query Operators](../../docs/QUERY_OPERATORS.md) for all `where` operators.
|
||||||
|
*
|
||||||
|
* @remarks
|
||||||
|
* **Field-addressing law.** Governs every query-surface field name — `where`
|
||||||
|
* and `orderBy` on this interface, plus `AggregateSource.where` and
|
||||||
|
* `AggregateDefinition.groupBy` in the aggregation engine:
|
||||||
|
*
|
||||||
|
* 1. A bare name (e.g. `'level'`, `'rank'`, `'score'`) always means the
|
||||||
|
* caller's own metadata field — it reads `entity.metadata.<name>`. There
|
||||||
|
* is no fallback to an engine-internal field of the same name and no
|
||||||
|
* priority resolution between the two; metadata wins unconditionally.
|
||||||
|
* 2. `system.<field>` reaches an engine scalar, explicitly, and only for
|
||||||
|
* these ten: `id`, `type`, `subtype`, `createdAt`, `updatedAt`,
|
||||||
|
* `confidence`, `weight`, `visibility`, `service`, `createdBy`.
|
||||||
|
* 3. `vector`, `connections`, `level` (the engine-internal node field — a
|
||||||
|
* different thing from a user metadata field also named `level`),
|
||||||
|
* `data`, and `_rev` are invisible plumbing: neither spelling can
|
||||||
|
* address them from a query surface.
|
||||||
|
* 4. `metadata.<field>` is the explicit spelling of the bare form and means
|
||||||
|
* exactly the same thing as rule 1.
|
||||||
|
* 5. A name that matches none of the above — most often a bare name that
|
||||||
|
* collides with one of the ten system-scalar names in rule 2 — REFUSES
|
||||||
|
* with a typed {@link UnresolvableFieldError} naming both candidates,
|
||||||
|
* e.g. `no metadata field 'createdAt' — did you mean system.createdAt or
|
||||||
|
* metadata.createdAt?`. The same loud-refusal principle covers whole
|
||||||
|
* options: the previously accepted-and-silently-ignored `cursor`,
|
||||||
|
* `includeRelations`, and `writeOnly` now throw
|
||||||
|
* {@link UnsupportedFindOptionError} instead of doing nothing.
|
||||||
|
* 6. **Ordering contract** (identical on the pure-JS engine and the native
|
||||||
|
* accelerator): rows missing or `null` on the `orderBy` field sort LAST
|
||||||
|
* in BOTH `asc` and `desc` order and are never dropped from the result;
|
||||||
|
* ties break by `id` ascending.
|
||||||
|
*
|
||||||
|
* Migration note: a call site written against the old rule — e.g.
|
||||||
|
* `orderBy: 'createdAt'` or `where: { visibility: 'internal' }` meaning the
|
||||||
|
* engine scalar — now refuses instead of silently reading the wrong field.
|
||||||
|
* The thrown error names the exact fix (`system.createdAt`). A loud
|
||||||
|
* refusal with the fix in hand beats a silent behavior flip.
|
||||||
*/
|
*/
|
||||||
export interface FindParams<T = any> {
|
export interface FindParams<T = any> {
|
||||||
// Vector Intelligence
|
// Vector Intelligence
|
||||||
|
|
@ -516,7 +554,18 @@ export interface FindParams<T = any> {
|
||||||
* `{ exists: true }`, `{ missing: true }`) use `where: { subtype: { …operators… } }`.
|
* `{ exists: true }`, `{ missing: true }`) use `where: { subtype: { …operators… } }`.
|
||||||
*/
|
*/
|
||||||
subtype?: string | string[]
|
subtype?: string | string[]
|
||||||
/** Metadata filters using BFO operators (e.g., `{ year: { greaterThan: 2020 } }`) */
|
/**
|
||||||
|
* Metadata filters using BFO operators (e.g., `{ year: { greaterThan: 2020 } }`).
|
||||||
|
* Field names follow the field-addressing law — see the `@remarks` on
|
||||||
|
* {@link FindParams}: a bare key is always the caller's metadata field;
|
||||||
|
* an engine scalar needs the explicit `system.<field>` form.
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* ```typescript
|
||||||
|
* await brain.find({ where: { level: { greaterThan: 5 } } }) // metadata.level
|
||||||
|
* await brain.find({ where: { 'system.visibility': 'internal' } }) // engine scalar
|
||||||
|
* ```
|
||||||
|
*/
|
||||||
where?: Partial<T>
|
where?: Partial<T>
|
||||||
|
|
||||||
// Visibility
|
// Visibility
|
||||||
|
|
@ -548,29 +597,49 @@ export interface FindParams<T = any> {
|
||||||
// Control options
|
// Control options
|
||||||
limit?: number // Max results (default: 10)
|
limit?: number // Max results (default: 10)
|
||||||
offset?: number // Skip N results
|
offset?: number // Skip N results
|
||||||
|
/**
|
||||||
|
* @deprecated Not implemented. Passing `cursor` throws
|
||||||
|
* {@link UnsupportedFindOptionError} — it used to be accepted and
|
||||||
|
* silently ignored, which masked that no cursor pagination ever ran. Use
|
||||||
|
* `offset` / `limit` until cursor pagination ships.
|
||||||
|
*/
|
||||||
cursor?: string // Cursor-based pagination
|
cursor?: string // Cursor-based pagination
|
||||||
|
|
||||||
// Sorting
|
// Sorting
|
||||||
/**
|
/**
|
||||||
* Field to sort by. User metadata fields sort by their stored values —
|
* Field to sort by. Follows the field-addressing law (see the `@remarks`
|
||||||
* including natural names like `level`, `rank`, or `score` (an engine-internal
|
* on {@link FindParams}): a bare name (`'level'`, `'rank'`, `'score'`, …)
|
||||||
* field can never shadow your metadata; fixed 2026-08 after a production
|
* always sorts by that metadata field; the ten engine scalars sort only
|
||||||
* report). System timestamps (`createdAt`, `updatedAt`) sort by entity age.
|
* via the explicit `system.<field>` form (e.g. `'system.createdAt'`); a
|
||||||
|
* name that resolves to neither throws {@link UnresolvableFieldError}
|
||||||
|
* naming the fix.
|
||||||
*
|
*
|
||||||
* Ordering contract (identical on the pure-JS engine and the native
|
* Ordering contract (identical on the pure-JS engine and the native
|
||||||
* accelerator): entities missing the field sort LAST in both directions —
|
* accelerator): rows missing or `null` on this field sort LAST in BOTH
|
||||||
* they are never dropped from the result; ties break deterministically.
|
* `asc` and `desc` order and are never dropped from the result; ties
|
||||||
|
* break by `id` ascending.
|
||||||
*
|
*
|
||||||
* NOTE — the field-addressing law is changing (announced 2026-08): bare
|
* @example
|
||||||
* names will mean user metadata ALWAYS, and system fields will be reached
|
* ```typescript
|
||||||
* explicitly as `system.<field>` (e.g. `system.createdAt`), with typed
|
* await brain.find({ orderBy: 'level', order: 'desc' }) // metadata.level
|
||||||
* refusals for unresolvable names. Until that release, bare `createdAt`
|
* await brain.find({ orderBy: 'system.createdAt', order: 'desc' }) // engine scalar
|
||||||
* and friends keep resolving to the system fields as documented above.
|
* ```
|
||||||
*/
|
*/
|
||||||
orderBy?: string
|
orderBy?: string
|
||||||
|
/**
|
||||||
|
* Sort direction: `'asc'` (default) or `'desc'`. Per the ordering
|
||||||
|
* contract on `orderBy`, rows missing/`null` on the sorted field sort
|
||||||
|
* LAST in both directions — `order` never moves them to the front.
|
||||||
|
*/
|
||||||
order?: 'asc' | 'desc' // Sort direction: 'asc' (default) or 'desc'
|
order?: 'asc' | 'desc' // Sort direction: 'asc' (default) or 'desc'
|
||||||
|
|
||||||
// Advanced options
|
// Advanced options
|
||||||
|
/**
|
||||||
|
* @deprecated Not implemented. Passing `includeRelations` throws
|
||||||
|
* {@link UnsupportedFindOptionError} — it used to be accepted and
|
||||||
|
* silently ignored, so no relationships were ever attached. Fetch
|
||||||
|
* relationships separately via `brain.related()`.
|
||||||
|
*/
|
||||||
includeRelations?: boolean // Include entity relationships
|
includeRelations?: boolean // Include entity relationships
|
||||||
excludeVFS?: boolean // Exclude VFS entities from results (default: false - VFS included)
|
excludeVFS?: boolean // Exclude VFS entities from results (default: false - VFS included)
|
||||||
service?: string // Multi-tenancy filter
|
service?: string // Multi-tenancy filter
|
||||||
|
|
@ -603,6 +672,11 @@ export interface FindParams<T = any> {
|
||||||
}
|
}
|
||||||
|
|
||||||
// Performance options
|
// Performance options
|
||||||
|
/**
|
||||||
|
* @deprecated Not implemented. Passing `writeOnly` throws
|
||||||
|
* {@link UnsupportedFindOptionError} — it used to be accepted and
|
||||||
|
* silently ignored, so validation was never actually skipped.
|
||||||
|
*/
|
||||||
writeOnly?: boolean // Skip validation for high-speed ingestion
|
writeOnly?: boolean // Skip validation for high-speed ingestion
|
||||||
|
|
||||||
// Aggregation
|
// Aggregation
|
||||||
|
|
@ -1352,7 +1426,10 @@ export type GroupByDimension =
|
||||||
export interface AggregateSource {
|
export interface AggregateSource {
|
||||||
/** Filter by entity type(s) */
|
/** Filter by entity type(s) */
|
||||||
type?: NounType | NounType[]
|
type?: NounType | NounType[]
|
||||||
/** Metadata filter (same syntax as find({ where })) */
|
/**
|
||||||
|
* Metadata filter — same syntax and field-addressing law as find()'s
|
||||||
|
* `where` (see the `@remarks` on {@link FindParams}).
|
||||||
|
*/
|
||||||
where?: Record<string, unknown>
|
where?: Record<string, unknown>
|
||||||
/** Multi-tenancy service filter */
|
/** Multi-tenancy service filter */
|
||||||
service?: string
|
service?: string
|
||||||
|
|
@ -1366,7 +1443,11 @@ export interface AggregateDefinition {
|
||||||
name: string
|
name: string
|
||||||
/** Which entities contribute to this aggregate */
|
/** Which entities contribute to this aggregate */
|
||||||
source: AggregateSource
|
source: AggregateSource
|
||||||
/** Dimensions to group by */
|
/**
|
||||||
|
* Dimensions to group by — field names follow the same field-addressing
|
||||||
|
* law as find()'s `where` / `orderBy` (see the `@remarks` on
|
||||||
|
* {@link FindParams}).
|
||||||
|
*/
|
||||||
groupBy: GroupByDimension[]
|
groupBy: GroupByDimension[]
|
||||||
/** Named metrics to compute */
|
/** Named metrics to compute */
|
||||||
metrics: Record<string, AggregateMetricDef>
|
metrics: Record<string, AggregateMetricDef>
|
||||||
|
|
@ -1425,16 +1506,25 @@ export interface AggregateGroupState {
|
||||||
export interface AggregateQueryParams {
|
export interface AggregateQueryParams {
|
||||||
/** Name of the aggregate to query */
|
/** Name of the aggregate to query */
|
||||||
name: string
|
name: string
|
||||||
/** Filter aggregate groups by their key values */
|
/**
|
||||||
|
* Filter aggregate groups by their key values — same field-addressing
|
||||||
|
* law as find() (see the `@remarks` on {@link FindParams}).
|
||||||
|
*/
|
||||||
where?: Record<string, unknown>
|
where?: Record<string, unknown>
|
||||||
/**
|
/**
|
||||||
* Filter groups by their computed METRIC values (SQL HAVING). Same BFO operators as
|
* Filter groups by their computed METRIC values (SQL HAVING). Same BFO operators as
|
||||||
* `where`, but applied to the derived metric results plus `count`, e.g.
|
* `where`, but applied to the derived metric results plus `count`, e.g.
|
||||||
* `{ revenue: { greaterThan: 1000 } }`. Evaluated per group (O(groups), independent of
|
* `{ revenue: { greaterThan: 1000 } }`. Evaluated per group (O(groups), independent of
|
||||||
* entity count), before sort/pagination.
|
* entity count), before sort/pagination. Metric names and `count` are looked up
|
||||||
|
* directly, not field-addressed; a group-KEY field used here follows the same
|
||||||
|
* field-addressing law as find() (see the `@remarks` on {@link FindParams}).
|
||||||
*/
|
*/
|
||||||
having?: Record<string, unknown>
|
having?: Record<string, unknown>
|
||||||
/** Sort by metric name or group key field */
|
/**
|
||||||
|
* Sort by metric name (a key from `metrics`, looked up directly) or by a
|
||||||
|
* group key field — a group key field follows the same field-addressing
|
||||||
|
* law as find()'s `orderBy` (see the `@remarks` on {@link FindParams}).
|
||||||
|
*/
|
||||||
orderBy?: string
|
orderBy?: string
|
||||||
/** Sort direction */
|
/** Sort direction */
|
||||||
order?: 'asc' | 'desc'
|
order?: 'asc' | 'desc'
|
||||||
|
|
@ -1938,32 +2028,6 @@ export interface BrainyConfig {
|
||||||
*/
|
*/
|
||||||
force?: boolean
|
force?: boolean
|
||||||
|
|
||||||
/**
|
|
||||||
* How write paths react when an untyped (JavaScript) caller smuggles a
|
|
||||||
* Brainy-reserved field (`RESERVED_ENTITY_FIELDS` / `RESERVED_RELATION_FIELDS`
|
|
||||||
* — `confidence`, `weight`, `subtype`, `visibility`, `service`, `createdBy`,
|
|
||||||
* `noun`/`verb`, `data`, `createdAt`, `updatedAt`, `_rev`) **inside the
|
|
||||||
* `metadata` bag** of `add()` / `update()` / `relate()` / `updateRelation()`
|
|
||||||
* (and their `transact()` / `with()` mirrors). TypeScript callers can't write
|
|
||||||
* these shapes at all — the compile-time guard on the metadata param types
|
|
||||||
* (`NoReservedEntityKeys` / `NoReservedRelationKeys`) rejects a literal
|
|
||||||
* reserved key — so this policy only governs untyped callers that slip one
|
|
||||||
* past the compiler.
|
|
||||||
*
|
|
||||||
* - `'throw'` (**default, 8.0**): a reserved key in the bag throws a clear
|
|
||||||
* `Error` naming the offending key(s) and the correct write path. No silent
|
|
||||||
* remap, no data loss, no surprise. This is the 8.0 "no silent failures"
|
|
||||||
* contract.
|
|
||||||
* - `'warn'`: legacy remapping with a loud, one-shot (per key, per process)
|
|
||||||
* warning for EVERY reserved key found — user-mutable fields are remapped to
|
|
||||||
* their dedicated top-level param (top-level wins when both are supplied),
|
|
||||||
* system-managed fields are dropped. Use while migrating untyped call sites.
|
|
||||||
* - `'remap'`: the pre-8.0 silent remapping, no warning. Last-resort
|
|
||||||
* compatibility hatch for code that intentionally relies on the bag path.
|
|
||||||
*
|
|
||||||
* @default 'throw'
|
|
||||||
*/
|
|
||||||
reservedFieldPolicy?: 'throw' | 'warn' | 'remap'
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ============= Neural API Types =============
|
// ============= Neural API Types =============
|
||||||
|
|
|
||||||
|
|
@ -1,35 +1,54 @@
|
||||||
/**
|
/**
|
||||||
* @module types/reservedFields
|
* @module types/reservedFields
|
||||||
* @description The canonical reserved-field contract — ONE place that defines
|
* @description The stored-record layout contract — ONE place that defines
|
||||||
* which keys belong to Brainy (top-level entity/relationship fields) and may
|
* which keys of a persisted metadata record belong to the ENGINE (top-level
|
||||||
* therefore never live inside a `metadata` bag.
|
* entity/relationship fields) and how the USER's metadata bag is kept apart
|
||||||
|
* from them, faithfully, across flush / reopen / rebuild / time travel.
|
||||||
*
|
*
|
||||||
* Three layers enforce the contract, all driven by the constants below:
|
* THE FIELD-ADDRESSING LAW (ruled 2026-08-03, VENUE-BRAINY-ORDERBY-NOOP):
|
||||||
|
* data is either in main space — where developers can use ANY name, and it
|
||||||
|
* all works with every database function — or it is in `system.*`. There are
|
||||||
|
* NO reserved user-facing metadata names anymore: `confidence`, `type`,
|
||||||
|
* `level`, `data`, `id`, `content` … inside a metadata bag are ordinary user
|
||||||
|
* fields. The only refused write is a user metadata key literally starting
|
||||||
|
* with `'system.'` (namespace forgery — see `rejectForgedSystemKeys`).
|
||||||
*
|
*
|
||||||
* 1. **Compile time** — `AddParams.metadata`, `UpdateParams.metadata`,
|
* That law makes name-based storage discrimination unsound for NEW records
|
||||||
* `RelateParams.metadata` and `UpdateRelationParams.metadata` are typed so
|
* (a user field named `confidence` may now legally sit beside the engine's
|
||||||
* a literal reserved key is a TypeScript error (see
|
* confidence scalar), so persisted metadata records carry the user bag
|
||||||
* {@link EntityMetadataInput} / {@link RelationMetadataInput}).
|
* NESTED, shape-discriminated by a format stamp:
|
||||||
* 2. **Write time** — for untyped (JavaScript) callers that smuggle a
|
|
||||||
* reserved key past the compiler anyway, every write path normalizes the
|
|
||||||
* bag: user-mutable fields are remapped to their dedicated top-level
|
|
||||||
* param (top-level wins when both are supplied) and system-managed fields
|
|
||||||
* are dropped with a one-shot warning naming the correct write path.
|
|
||||||
* 3. **Read time** — every read path splits the stored flat record through
|
|
||||||
* {@link splitNounMetadataRecord} / {@link splitVerbMetadataRecord}, so a
|
|
||||||
* reserved field is surfaced ONLY at top level and `entity.metadata` /
|
|
||||||
* `relation.metadata` contain ONLY custom fields, always — live reads,
|
|
||||||
* batch reads, and historical (`asOf`) reads alike.
|
|
||||||
*
|
*
|
||||||
* Documented for consumers in `docs/concepts/consistency-model.md`
|
* - **v2 (nested-bag)** — `{ …engine fields…, [METADATA_RECORD_FORMAT_KEY]:
|
||||||
* ("Reserved fields").
|
* NESTED_BAG_FORMAT, metadata: { …user bag, verbatim… } }`. Built ONLY by
|
||||||
|
* {@link buildNounMetadataRecord} / {@link buildVerbMetadataRecord}; the
|
||||||
|
* engine half and the user bag can never collide because they never share
|
||||||
|
* a level.
|
||||||
|
* - **legacy (flat)** — engine fields and user fields mixed at one level,
|
||||||
|
* discriminated BY NAME through the RESERVED_* lists. Sound for legacy
|
||||||
|
* records precisely because the pre-law write door REFUSED user metadata
|
||||||
|
* carrying those names — a flat key matching a reserved name IS the
|
||||||
|
* engine's value in any record the old door admitted.
|
||||||
|
*
|
||||||
|
* {@link splitNounMetadataRecord} / {@link splitVerbMetadataRecord} read
|
||||||
|
* BOTH shapes (stamp first, name split as the legacy fallback) and are the
|
||||||
|
* single read-side choke point for live, batch, AND historical (`asOf`)
|
||||||
|
* reads — the generation store snapshots whole records, so time travel
|
||||||
|
* rides the same split.
|
||||||
|
*
|
||||||
|
* The RESERVED_* lists therefore no longer describe a user-facing ban — they
|
||||||
|
* describe the ENGINE HALF of the stored record layout (and drive the legacy
|
||||||
|
* split). The write-door remap machinery and the compile-time metadata key
|
||||||
|
* bans that used to enforce the old contract are gone.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description Entity (noun) field names reserved by Brainy. These keys are
|
* @description Entity (noun) field names owned by the ENGINE in a stored
|
||||||
* stored in the flat per-entity metadata record alongside custom fields, but
|
* metadata record. In v2 (nested-bag) records these are the legal TOP-LEVEL
|
||||||
* they belong to Brainy: every read path extracts them to top-level
|
* keys beside the nested `metadata` bag; in legacy flat records they drive
|
||||||
* `Entity` fields, and no write path accepts them inside `metadata`.
|
* the by-name split. They are NOT a user-facing ban list: since the
|
||||||
|
* field-addressing law, a user metadata field may carry any of these names
|
||||||
|
* and remains the user's — it lives inside the nested bag, never at the
|
||||||
|
* record's top level.
|
||||||
*
|
*
|
||||||
* | Key | Canonical write path |
|
* | Key | Canonical write path |
|
||||||
* |-----|----------------------|
|
* |-----|----------------------|
|
||||||
|
|
@ -119,68 +138,54 @@ export type ReservedRelationField = (typeof RESERVED_RELATION_FIELDS)[number]
|
||||||
type IsAny<T> = 0 extends 1 & T ? true : false
|
type IsAny<T> = 0 extends 1 & T ? true : false
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description Compile-time tripwire: marks every reserved entity key as
|
* @deprecated The compile-time reserved-key ban died with the
|
||||||
* `never` so an object literal carrying one fails to type-check. Keys that
|
* field-addressing law: every name is legal user metadata now. Kept as an
|
||||||
* `T` itself declares (including via an index signature, where
|
* empty (no-op) guard so external type references keep compiling; it bans
|
||||||
* `keyof T = string`) are exempted — a consumer who *explicitly* types a
|
* nothing.
|
||||||
* reserved key into their metadata shape keeps a working (if unwise) type,
|
|
||||||
* and index-signature metadata types remain assignable.
|
|
||||||
*/
|
*/
|
||||||
export type NoReservedEntityKeys<T> = {
|
export type NoReservedEntityKeys<T> = unknown
|
||||||
readonly [K in ReservedEntityField as K extends keyof T ? never : K]?: never
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description Relationship mirror of {@link NoReservedEntityKeys}.
|
* @deprecated Relationship mirror of {@link NoReservedEntityKeys} — no-op
|
||||||
|
* for the same reason.
|
||||||
*/
|
*/
|
||||||
export type NoReservedRelationKeys<T> = {
|
export type NoReservedRelationKeys<T> = unknown
|
||||||
readonly [K in ReservedRelationField as K extends keyof T ? never : K]?: never
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description The metadata bag shape for untyped brains (`T = any`): an
|
|
||||||
* open index signature (any custom key, any value — exactly the pre-8.0
|
|
||||||
* latitude) intersected with the reserved-key guard, whose declared
|
|
||||||
* `?: never` properties take precedence over the index signature so a
|
|
||||||
* literal reserved key is still a compile error.
|
|
||||||
*/
|
|
||||||
type OpenBag<Guard> = { [key: string]: any } & Guard
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description The type of `AddParams.metadata`: the consumer's metadata
|
* @description The type of `AddParams.metadata`: the consumer's metadata
|
||||||
* shape `T` with reserved entity keys forbidden at compile time. For untyped
|
* shape `T`, open. Under the field-addressing law EVERY key is a legal user
|
||||||
* brains (`T = any`) the bag stays open ({@link OpenBag}), so arbitrary
|
* field (engine scalars are written only via their dedicated params and read
|
||||||
* custom fields remain legal while literal reserved keys still error.
|
* at `system.*`), so no name is banned at compile time. The one illegal
|
||||||
|
* spelling — a key starting `'system.'` — cannot be expressed as a mapped
|
||||||
|
* type ban and is refused at runtime (`rejectForgedSystemKeys`).
|
||||||
*/
|
*/
|
||||||
export type EntityMetadataInput<T> = IsAny<T> extends true
|
export type EntityMetadataInput<T> = IsAny<T> extends true
|
||||||
? OpenBag<NoReservedEntityKeys<object>>
|
? { [key: string]: any }
|
||||||
: T & NoReservedEntityKeys<T>
|
: T
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description The type of `UpdateParams.metadata`: a partial patch of the
|
* @description The type of `UpdateParams.metadata`: a partial patch of the
|
||||||
* consumer's metadata shape with reserved entity keys forbidden at compile
|
* consumer's metadata shape. Same openness as {@link EntityMetadataInput}.
|
||||||
* time. Same `T = any` handling as {@link EntityMetadataInput}.
|
|
||||||
*/
|
*/
|
||||||
export type EntityMetadataPatch<T> = IsAny<T> extends true
|
export type EntityMetadataPatch<T> = IsAny<T> extends true
|
||||||
? OpenBag<NoReservedEntityKeys<object>>
|
? { [key: string]: any }
|
||||||
: Partial<T> & NoReservedEntityKeys<T>
|
: Partial<T>
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description The type of `RelateParams.metadata`: the consumer's edge
|
* @description The type of `RelateParams.metadata`: the consumer's edge
|
||||||
* metadata shape with reserved relationship keys forbidden at compile time.
|
* metadata shape, open — the relation mirror of {@link EntityMetadataInput}.
|
||||||
*/
|
*/
|
||||||
export type RelationMetadataInput<T> = IsAny<T> extends true
|
export type RelationMetadataInput<T> = IsAny<T> extends true
|
||||||
? OpenBag<NoReservedRelationKeys<object>>
|
? { [key: string]: any }
|
||||||
: T & NoReservedRelationKeys<T>
|
: T
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description The type of `UpdateRelationParams.metadata`: a partial patch
|
* @description The type of `UpdateRelationParams.metadata`: a partial patch
|
||||||
* of the consumer's edge metadata shape with reserved relationship keys
|
* of the consumer's edge metadata shape, open.
|
||||||
* forbidden at compile time.
|
|
||||||
*/
|
*/
|
||||||
export type RelationMetadataPatch<T> = IsAny<T> extends true
|
export type RelationMetadataPatch<T> = IsAny<T> extends true
|
||||||
? OpenBag<NoReservedRelationKeys<object>>
|
? { [key: string]: any }
|
||||||
: Partial<T> & NoReservedRelationKeys<T>
|
: Partial<T>
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description Result of splitting a stored flat metadata record into its
|
* @description Result of splitting a stored flat metadata record into its
|
||||||
|
|
@ -196,6 +201,103 @@ export interface SplitMetadataRecord<F extends string> {
|
||||||
const RESERVED_ENTITY_SET: ReadonlySet<string> = new Set(RESERVED_ENTITY_FIELDS)
|
const RESERVED_ENTITY_SET: ReadonlySet<string> = new Set(RESERVED_ENTITY_FIELDS)
|
||||||
const RESERVED_RELATION_SET: ReadonlySet<string> = new Set(RESERVED_RELATION_FIELDS)
|
const RESERVED_RELATION_SET: ReadonlySet<string> = new Set(RESERVED_RELATION_FIELDS)
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description The format-stamp key of a persisted metadata record. Its
|
||||||
|
* presence with the exact value {@link NESTED_BAG_FORMAT} marks a v2
|
||||||
|
* (nested-bag) record; its absence marks a legacy flat record. The stamp is
|
||||||
|
* what makes the shape check collision-proof against legacy user data: a
|
||||||
|
* pre-law record COULD carry a user field named `metadata` (the name was
|
||||||
|
* never reserved), but it cannot also carry this engine-written stamp.
|
||||||
|
*/
|
||||||
|
export const METADATA_RECORD_FORMAT_KEY = '_fmt'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description The nested-bag record format stamp (v2, the field-addressing
|
||||||
|
* law's storage shape, 2026-08-03): engine fields at top level, the user's
|
||||||
|
* metadata bag NESTED verbatim under `metadata`. Cross-engine: the native
|
||||||
|
* provider discriminates record shapes by the same stamp.
|
||||||
|
*/
|
||||||
|
export const NESTED_BAG_FORMAT = 2
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description `true` when a persisted record carries the v2 nested-bag
|
||||||
|
* stamp (and a structurally valid nested bag).
|
||||||
|
*/
|
||||||
|
export function isNestedBagRecord(
|
||||||
|
record: Record<string, unknown> | null | undefined
|
||||||
|
): boolean {
|
||||||
|
return (
|
||||||
|
record !== null &&
|
||||||
|
record !== undefined &&
|
||||||
|
typeof record === 'object' &&
|
||||||
|
record[METADATA_RECORD_FORMAT_KEY] === NESTED_BAG_FORMAT &&
|
||||||
|
typeof record.metadata === 'object' &&
|
||||||
|
record.metadata !== null &&
|
||||||
|
!Array.isArray(record.metadata)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Build a v2 (nested-bag) entity metadata record — THE only
|
||||||
|
* sanctioned way to construct a persisted noun metadata record. The engine
|
||||||
|
* half goes top-level; the user bag nests verbatim under `metadata`; the
|
||||||
|
* format stamp seals the shape. Because the two halves never share a level,
|
||||||
|
* a user field named `confidence` (or any other engine spelling) survives
|
||||||
|
* flush / reopen / rebuild / time travel exactly as written.
|
||||||
|
* @param engineFields - The engine-owned half (keys from
|
||||||
|
* {@link RESERVED_ENTITY_FIELDS} — `noun`, timestamps, `_rev`, …).
|
||||||
|
* @param userBag - The consumer's metadata bag, stored verbatim.
|
||||||
|
* @returns The stamped v2 record.
|
||||||
|
*/
|
||||||
|
export function buildNounMetadataRecord(
|
||||||
|
engineFields: Partial<Record<ReservedEntityField, unknown>>,
|
||||||
|
userBag: Record<string, unknown> | undefined
|
||||||
|
): Record<string, unknown> {
|
||||||
|
return {
|
||||||
|
...engineFields,
|
||||||
|
[METADATA_RECORD_FORMAT_KEY]: NESTED_BAG_FORMAT,
|
||||||
|
metadata: { ...(userBag ?? {}) }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Build a v2 (nested-bag) relationship metadata record — the
|
||||||
|
* verb mirror of {@link buildNounMetadataRecord}.
|
||||||
|
* @param engineFields - The engine-owned half (keys from
|
||||||
|
* {@link RESERVED_RELATION_FIELDS} — `verb`, `weight`, timestamps, …).
|
||||||
|
* @param userBag - The consumer's edge metadata bag, stored verbatim.
|
||||||
|
* @returns The stamped v2 record.
|
||||||
|
*/
|
||||||
|
export function buildVerbMetadataRecord(
|
||||||
|
engineFields: Partial<Record<ReservedRelationField, unknown>>,
|
||||||
|
userBag: Record<string, unknown> | undefined
|
||||||
|
): Record<string, unknown> {
|
||||||
|
return {
|
||||||
|
...engineFields,
|
||||||
|
[METADATA_RECORD_FORMAT_KEY]: NESTED_BAG_FORMAT,
|
||||||
|
metadata: { ...(userBag ?? {}) }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Shape-first split of a v2 record: the engine half is the top
|
||||||
|
* level filtered through the reserved list (belt — the builders only ever
|
||||||
|
* write reserved names there), the user bag is `record.metadata` verbatim.
|
||||||
|
*/
|
||||||
|
function splitNestedRecord<F extends string>(
|
||||||
|
record: Record<string, unknown>,
|
||||||
|
reservedSet: ReadonlySet<string>
|
||||||
|
): SplitMetadataRecord<F> {
|
||||||
|
const reserved: Record<string, unknown> = {}
|
||||||
|
for (const [key, value] of Object.entries(record)) {
|
||||||
|
if (reservedSet.has(key)) reserved[key] = value
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
reserved: reserved as Partial<Record<F, unknown>>,
|
||||||
|
custom: { ...(record.metadata as Record<string, unknown>) }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description Shared splitter — partitions a record's keys against a
|
* @description Shared splitter — partitions a record's keys against a
|
||||||
* reserved-name set. `null`/`undefined` records split to two empty objects.
|
* reserved-name set. `null`/`undefined` records split to two empty objects.
|
||||||
|
|
@ -222,33 +324,45 @@ function splitRecord<F extends string>(
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description Split a stored entity (noun) flat metadata record into
|
* @description Split a stored entity (noun) metadata record into engine
|
||||||
* reserved fields and custom metadata — THE canonical read-side split. Every
|
* fields and the user's metadata bag — THE canonical read-side split, shape
|
||||||
* entity read path (live `get()`, batch reads, paginated listings, and
|
* aware. v2 (nested-bag) records split by SHAPE: engine half top-level, bag
|
||||||
* historical `asOf()` materialization) goes through this function, so the
|
* = `record.metadata` verbatim (user collider names survive faithfully).
|
||||||
* reserved list can never drift between read paths.
|
* Legacy flat records split BY NAME through the reserved list — sound for
|
||||||
* @param record - The stored flat metadata record.
|
* them because the pre-law write door refused user metadata carrying those
|
||||||
* @returns `reserved` (Brainy-owned fields) and `custom` (the consumer's metadata bag).
|
* names. Every entity read path (live `get()`, batch reads, paginated
|
||||||
|
* listings, and historical `asOf()` materialization — the generation store
|
||||||
|
* snapshots whole records) goes through this function, so the two shapes
|
||||||
|
* can never drift between read paths.
|
||||||
|
* @param record - The stored metadata record (either shape).
|
||||||
|
* @returns `reserved` (engine-owned fields) and `custom` (the consumer's metadata bag).
|
||||||
* @example
|
* @example
|
||||||
* const { reserved, custom } = splitNounMetadataRecord(stored)
|
* const { reserved, custom } = splitNounMetadataRecord(stored)
|
||||||
* // reserved.noun → entity.type, reserved.confidence → entity.confidence, …
|
* // reserved.noun → entity.type, reserved.confidence → entity.confidence, …
|
||||||
* // custom → entity.metadata (custom fields only, always)
|
* // custom → entity.metadata (the user's fields only, always — ANY names)
|
||||||
*/
|
*/
|
||||||
export function splitNounMetadataRecord(
|
export function splitNounMetadataRecord(
|
||||||
record: Record<string, unknown> | null | undefined
|
record: Record<string, unknown> | null | undefined
|
||||||
): SplitMetadataRecord<ReservedEntityField> {
|
): SplitMetadataRecord<ReservedEntityField> {
|
||||||
|
if (isNestedBagRecord(record)) {
|
||||||
|
return splitNestedRecord(record as Record<string, unknown>, RESERVED_ENTITY_SET)
|
||||||
|
}
|
||||||
return splitRecord(record, RESERVED_ENTITY_SET)
|
return splitRecord(record, RESERVED_ENTITY_SET)
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description Split a stored relationship (verb) flat metadata record into
|
* @description Split a stored relationship (verb) metadata record into
|
||||||
* reserved fields and custom metadata — the verb mirror of
|
* engine fields and the user's edge metadata bag — the verb mirror of
|
||||||
* {@link splitNounMetadataRecord}, used by every relationship read path.
|
* {@link splitNounMetadataRecord}, shape aware, used by every relationship
|
||||||
* @param record - The stored flat metadata record.
|
* read path.
|
||||||
* @returns `reserved` (Brainy-owned fields) and `custom` (the consumer's metadata bag).
|
* @param record - The stored metadata record (either shape).
|
||||||
|
* @returns `reserved` (engine-owned fields) and `custom` (the consumer's metadata bag).
|
||||||
*/
|
*/
|
||||||
export function splitVerbMetadataRecord(
|
export function splitVerbMetadataRecord(
|
||||||
record: Record<string, unknown> | null | undefined
|
record: Record<string, unknown> | null | undefined
|
||||||
): SplitMetadataRecord<ReservedRelationField> {
|
): SplitMetadataRecord<ReservedRelationField> {
|
||||||
|
if (isNestedBagRecord(record)) {
|
||||||
|
return splitNestedRecord(record as Record<string, unknown>, RESERVED_RELATION_SET)
|
||||||
|
}
|
||||||
return splitRecord(record, RESERVED_RELATION_SET)
|
return splitRecord(record, RESERVED_RELATION_SET)
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -5,6 +5,7 @@
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { StorageAdapter, resolveEntityField, NounMetadata, VerbMetadata } from '../coreTypes.js'
|
import { StorageAdapter, resolveEntityField, NounMetadata, VerbMetadata } from '../coreTypes.js'
|
||||||
|
import { SYSTEM_ENTITY_SCALARS, parseFieldAddress, UnresolvableFieldError } from '../db/fieldAddressing.js'
|
||||||
import { ColumnStore } from '../indexes/columnStore/ColumnStore.js'
|
import { ColumnStore } from '../indexes/columnStore/ColumnStore.js'
|
||||||
import type { MetadataIndexProvider } from '../plugin.js'
|
import type { MetadataIndexProvider } from '../plugin.js'
|
||||||
import { MetadataIndexCache, MetadataIndexCacheConfig } from './metadataIndexCache.js'
|
import { MetadataIndexCache, MetadataIndexCacheConfig } from './metadataIndexCache.js'
|
||||||
|
|
@ -43,8 +44,8 @@ import { BrainyError } from '../errors/brainyError.js'
|
||||||
* bucketed field is added (e.g. a compressed float), add it here too.
|
* bucketed field is added (e.g. a compressed float), add it here too.
|
||||||
*/
|
*/
|
||||||
const BUCKETED_INDEX_FIELDS: ReadonlySet<string> = new Set([
|
const BUCKETED_INDEX_FIELDS: ReadonlySet<string> = new Set([
|
||||||
'createdAt',
|
'system.createdAt',
|
||||||
'updatedAt'
|
'system.updatedAt'
|
||||||
])
|
])
|
||||||
|
|
||||||
export interface MetadataIndexEntry {
|
export interface MetadataIndexEntry {
|
||||||
|
|
@ -72,8 +73,11 @@ export interface MetadataIndexConfig {
|
||||||
maxIndexSize?: number // Max number of entries per field value (default: 10000)
|
maxIndexSize?: number // Max number of entries per field value (default: 10000)
|
||||||
rebuildThreshold?: number // Rebuild if index is this % stale (default: 0.1)
|
rebuildThreshold?: number // Rebuild if index is this % stale (default: 0.1)
|
||||||
autoOptimize?: boolean // Auto-cleanup unused entries (default: true)
|
autoOptimize?: boolean // Auto-cleanup unused entries (default: true)
|
||||||
indexedFields?: string[] // Only index these fields (default: all)
|
// NOTE: the name-based indexedFields/excludeFields knobs died with the
|
||||||
excludeFields?: string[] // Never index these fields
|
// field-addressing law ("no special names"): EVERY user field indexes,
|
||||||
|
// whatever its name. Bulk-payload protection is value-SHAPE based and
|
||||||
|
// uniform across all names (large arrays never become posting scalars;
|
||||||
|
// long values index hashed) — shape is not a name carve-out.
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface MetadataIndexOptions {
|
export interface MetadataIndexOptions {
|
||||||
|
|
@ -184,31 +188,12 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
this.config = {
|
this.config = {
|
||||||
maxIndexSize: config.maxIndexSize ?? 10000,
|
maxIndexSize: config.maxIndexSize ?? 10000,
|
||||||
rebuildThreshold: config.rebuildThreshold ?? 0.1,
|
rebuildThreshold: config.rebuildThreshold ?? 0.1,
|
||||||
autoOptimize: config.autoOptimize ?? true,
|
autoOptimize: config.autoOptimize ?? true
|
||||||
indexedFields: config.indexedFields ?? [],
|
// No name-based exclude/allow lists — the field-addressing law: every
|
||||||
excludeFields: config.excludeFields ?? [
|
// user field indexes, whatever its name ('content', 'data', 'id',
|
||||||
// ONLY exclude truly un-indexable fields (binary data, large content)
|
// 'vector', … included). Bulk payloads are kept out by uniform value-
|
||||||
// Timestamps are NOW indexed with automatic bucketing (prevents pollution)
|
// SHAPE rules in extractIndexableFields (arrays >10 never become
|
||||||
|
// posting scalars; >100-char values index hashed), never by name.
|
||||||
// Vectors and embeddings (binary data, already have HNSW indexes)
|
|
||||||
'embedding',
|
|
||||||
'vector',
|
|
||||||
'embeddings',
|
|
||||||
'vectors',
|
|
||||||
|
|
||||||
// Large content fields (too large for metadata indexing)
|
|
||||||
'content',
|
|
||||||
'data',
|
|
||||||
'originalData',
|
|
||||||
'_data',
|
|
||||||
|
|
||||||
// Primary keys (use direct lookups instead)
|
|
||||||
'id'
|
|
||||||
|
|
||||||
// NOTE: 'accessed', 'modified', 'createdAt', etc. are NO LONGER excluded!
|
|
||||||
// They are now indexed with automatic 1-minute bucketing to prevent file pollution
|
|
||||||
// This enables range queries like: modified > yesterday
|
|
||||||
]
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Initialize metadata cache with similar config to search cache
|
// Initialize metadata cache with similar config to search cache
|
||||||
|
|
@ -300,7 +285,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
}
|
}
|
||||||
|
|
||||||
// Warm the cache with common fields (lazy loading optimization)
|
// Warm the cache with common fields (lazy loading optimization)
|
||||||
// This loads the 'noun' sparse index which is needed for type counts
|
// This loads the type column ('system.type') needed for type counts
|
||||||
await this.warmCache()
|
await this.warmCache()
|
||||||
|
|
||||||
// Load type counts AFTER warmCache (sparse index is now cached)
|
// Load type counts AFTER warmCache (sparse index is now cached)
|
||||||
|
|
@ -349,8 +334,9 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
* Target: >80% cache hit rate for typical workloads
|
* Target: >80% cache hit rate for typical workloads
|
||||||
*/
|
*/
|
||||||
async warmCache(): Promise<void> {
|
async warmCache(): Promise<void> {
|
||||||
// Common fields used in most queries
|
// Common columns used in most queries — the frozen system keys, plus
|
||||||
const commonFields = ['noun', 'type', 'service', 'createdAt']
|
// legacy spellings for a pre-epoch-3 brain read before its rebuild runs.
|
||||||
|
const commonFields = ['system.type', 'system.service', 'system.createdAt', 'noun']
|
||||||
|
|
||||||
prodLog.debug(`🔥 Warming metadata cache with common fields: ${commonFields.join(', ')}`)
|
prodLog.debug(`🔥 Warming metadata cache with common fields: ${commonFields.join(', ')}`)
|
||||||
|
|
||||||
|
|
@ -536,9 +522,11 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Lazy load entity counts from the 'noun' field sparse index (O(n) where n = number of types)
|
* Lazy load entity counts from the type column (O(n) where n = number of
|
||||||
|
* types). The frozen key is 'system.type' (epoch 3); the legacy 'noun'
|
||||||
|
* column is read as a fallback for a pre-epoch-3 brain observed before its
|
||||||
|
* rebuild has run (e.g. a reader-mode open against an old writer).
|
||||||
* FIX: Previously read from stats.nounCount which was SERVICE-keyed, not TYPE-keyed
|
* FIX: Previously read from stats.nounCount which was SERVICE-keyed, not TYPE-keyed
|
||||||
* Now computes counts from the sparse index which has the correct type information
|
|
||||||
*/
|
*/
|
||||||
private async lazyLoadCounts(): Promise<void> {
|
private async lazyLoadCounts(): Promise<void> {
|
||||||
try {
|
try {
|
||||||
|
|
@ -548,23 +536,31 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
this.entityCountsByTypeFixed.fill(0)
|
this.entityCountsByTypeFixed.fill(0)
|
||||||
this.verbCountsByTypeFixed.fill(0)
|
this.verbCountsByTypeFixed.fill(0)
|
||||||
|
|
||||||
// PRIMARY (8.0+): rehydrate per-type counts from the column store's 'noun'
|
// PRIMARY (8.0+): rehydrate per-type counts from the column store's
|
||||||
// field — the authoritative on-disk source after a cold reopen.
|
// type column — the authoritative on-disk source after a cold reopen.
|
||||||
|
// Frozen key first ('system.type', epoch 3), legacy 'noun' as the
|
||||||
|
// pre-rebuild fallback.
|
||||||
//
|
//
|
||||||
// The chunked sparse-index WRITE path was removed in 7.20.0 (commit
|
// The chunked sparse-index WRITE path was removed in 7.20.0 (commit
|
||||||
// 11be039): new workspaces persist the 'noun' field ONLY to the column
|
// 11be039): new workspaces persist the type column ONLY to the column
|
||||||
// store, never to a `__sparse_index__noun` blob. So the legacy sparse
|
// store, never to a sparse-index blob. So the legacy sparse path below
|
||||||
// path below finds nothing and leaves every count at 0 — which is exactly
|
// finds nothing and leaves every count at 0 — which is exactly why
|
||||||
// why counts.byType/byTypeEnum/topTypes/allNounTypeCounts all read empty
|
// counts.byType/byTypeEnum/topTypes/allNounTypeCounts all read empty
|
||||||
// after close()+reopen while find()/getNounCount() (different sources)
|
// after close()+reopen while find()/getNounCount() (different sources)
|
||||||
// stay correct. The column store's per-value cardinality matches the warm
|
// stay correct. The column store's per-value cardinality matches the warm
|
||||||
// `updateTypeFieldAffinity` counts EXACTLY because both are driven from the
|
// `updateTypeFieldAffinity` counts EXACTLY because both are driven from the
|
||||||
// same `addToIndex` field set, in lockstep, with no visibility gate on
|
// same `addToIndex` field set, in lockstep, with no visibility gate on
|
||||||
// either — so this rehydration reproduces the warm values precisely.
|
// either — so this rehydration reproduces the warm values precisely.
|
||||||
if (this.columnStore && this.columnStore.getIndexedFields().includes('noun')) {
|
const indexedCols = this.columnStore ? this.columnStore.getIndexedFields() : []
|
||||||
const nounValues = await this.columnStore.getFilterValues('noun')
|
const typeCol = indexedCols.includes('system.type')
|
||||||
|
? 'system.type'
|
||||||
|
: indexedCols.includes('noun')
|
||||||
|
? 'noun'
|
||||||
|
: null
|
||||||
|
if (this.columnStore && typeCol) {
|
||||||
|
const nounValues = await this.columnStore.getFilterValues(typeCol)
|
||||||
for (const value of nounValues) {
|
for (const value of nounValues) {
|
||||||
const bitmap = await this.columnStore.filter('noun', value)
|
const bitmap = await this.columnStore.filter(typeCol, value)
|
||||||
if (bitmap.size > 0) {
|
if (bitmap.size > 0) {
|
||||||
// Use the stored value directly as the key (the legacy sparse path
|
// Use the stored value directly as the key (the legacy sparse path
|
||||||
// did the same): it is already the normalized type string that
|
// did the same): it is already the normalized type string that
|
||||||
|
|
@ -579,16 +575,17 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
}
|
}
|
||||||
|
|
||||||
// LEGACY FALLBACK (pre-7.20.0 workspaces still on the chunked sparse index).
|
// LEGACY FALLBACK (pre-7.20.0 workspaces still on the chunked sparse index).
|
||||||
const nounSparseIndex = await this.loadSparseIndex('noun')
|
const sparseCol = (await this.loadSparseIndex('system.type')) ? 'system.type' : 'noun'
|
||||||
|
const nounSparseIndex = await this.loadSparseIndex(sparseCol)
|
||||||
if (!nounSparseIndex) {
|
if (!nounSparseIndex) {
|
||||||
// No column-store 'noun' field and no sparse index yet — counts will be
|
// No column-store type column and no sparse index yet — counts will be
|
||||||
// populated as entities are added.
|
// populated as entities are added.
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// Iterate through all chunks and sum up bitmap sizes by type
|
// Iterate through all chunks and sum up bitmap sizes by type
|
||||||
for (const chunkId of nounSparseIndex.getAllChunkIds()) {
|
for (const chunkId of nounSparseIndex.getAllChunkIds()) {
|
||||||
const chunk = await this.chunkManager.loadChunk('noun', chunkId)
|
const chunk = await this.chunkManager.loadChunk(sparseCol, chunkId)
|
||||||
if (chunk) {
|
if (chunk) {
|
||||||
for (const [type, bitmap] of chunk.entries) {
|
for (const [type, bitmap] of chunk.entries) {
|
||||||
const currentCount = this.totalEntitiesByType.get(type) || 0
|
const currentCount = this.totalEntitiesByType.get(type) || 0
|
||||||
|
|
@ -1178,76 +1175,86 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
return `__HASH_${Math.abs(hash).toString(36)}`
|
return `__HASH_${Math.abs(hash).toString(36)}`
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Check if field should be indexed
|
|
||||||
*/
|
|
||||||
private shouldIndexField(field: string): boolean {
|
|
||||||
if (this.config.excludeFields.includes(field)) return false
|
|
||||||
if (this.config.indexedFields.length > 0) {
|
|
||||||
return this.config.indexedFields.includes(field)
|
|
||||||
}
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Extract indexable field-value pairs from entity or metadata
|
* Extract indexable field-value pairs from entity or metadata
|
||||||
*
|
*
|
||||||
* Now handles BOTH entity structure (with top-level fields) AND plain metadata
|
* Handles BOTH entity structure (with top-level fields) AND record shapes
|
||||||
* - Extracts from top-level fields (confidence, weight, timestamps, type, service, etc.)
|
* - Record-frame system scalars index under literal 'system.<field>' keys
|
||||||
* - Also extracts from nested metadata field (custom user fields)
|
* - The user's metadata bag indexes under bare keys — EVERY name (the
|
||||||
* - Skips HNSW-specific fields (vector, connections, level, id)
|
* field-addressing law: no special names; 'level', 'data', 'id',
|
||||||
* - Maps 'type' → 'noun' for backward compatibility with existing indexes
|
* 'content', 'vector' in a bag are ordinary user fields)
|
||||||
*
|
* - Record-frame plumbing (vector, connections, level, data, _rev, id)
|
||||||
* BUG FIX: Exclude vector embeddings and large arrays from indexing
|
* never indexes — that is namespace routing, not a name carve-out
|
||||||
* BUG FIX: Also exclude purely numeric field names (array indices)
|
* - Value-SHAPE rules apply uniformly to all names: arrays >10 never
|
||||||
* - Vector fields (384+ dimensions) were creating 825K chunk files for 1,144 entities
|
* become posting scalars; purely numeric key names (array indices)
|
||||||
* - Arrays converted to objects with numeric keys were still being indexed
|
* skip; >100-char values index hashed (normalizeValue)
|
||||||
*/
|
*/
|
||||||
private extractIndexableFields(data: any): Array<{ field: string, value: any }> {
|
private extractIndexableFields(data: any): Array<{ field: string, value: any }> {
|
||||||
const fields: Array<{ field: string, value: any }> = []
|
const fields: Array<{ field: string, value: any }> = []
|
||||||
|
|
||||||
// Fields that should NEVER be indexed: bulk structural payloads that would
|
// RECORD-FRAME-ONLY plumbing guard: on an entity/stored-record frame
|
||||||
// blow up the index (the 384-dim vector, embeddings, the adjacency list).
|
// these keys are the engine's structural payloads (the 384-dim vector,
|
||||||
// These are also caught by the array-size guard below, but naming them is
|
// embeddings, the adjacency list, the identity field) and never index.
|
||||||
// belt-and-suspenders. NOTE: `level` was previously here (an HNSW node's
|
// This set is NEVER applied inside the user's metadata bag — under the
|
||||||
// layer) but it never actually reaches this path — every caller passes a
|
// field-addressing law every user name indexes; a real vector-sized
|
||||||
// metadata bag or Entity record, neither of which carries the node's
|
// value in a bag is kept out by the uniform array-size shape guard, not
|
||||||
// `level` — so its only effect was to silently drop a legitimate USER
|
// by its name.
|
||||||
// metadata field named `level` (log level, skill level, access level…),
|
const RECORD_PLUMBING = new Set(['vector', 'embedding', 'embeddings', 'connections', 'id'])
|
||||||
// making `where: { level: … }` return nothing. Removed. (`id` stays: it is
|
|
||||||
// the reserved entity-identity field, resolved specially by find().)
|
|
||||||
const NEVER_INDEX = new Set(['vector', 'embedding', 'embeddings', 'connections', 'id'])
|
|
||||||
|
|
||||||
const extract = (obj: any, prefix = ''): void => {
|
// THE FROZEN INDEX KEY FORMAT (cross-engine, sealed 2026-08-03; the native
|
||||||
|
// accelerator keys identically — epoch 3 rebuilds every brain onto it):
|
||||||
|
// user fields index under BARE keys exactly as the caller wrote them;
|
||||||
|
// the ten system scalars index under literal 'system.<field>' keys — the
|
||||||
|
// key IS the query address, so the two namespaces can never collide
|
||||||
|
// inside the index again.
|
||||||
|
// Frame kinds: 'entity-record' = entityForIndexing shape / v2 nested-bag
|
||||||
|
// stored record (user fields nested under `metadata`; stray top-level
|
||||||
|
// keys are DROPPED, not guessed); 'flat-record' = the LEGACY stored
|
||||||
|
// metadata-record shape (user fields flat beside the engine's — sound to
|
||||||
|
// split by name because the pre-law write door refused user metadata
|
||||||
|
// carrying engine names, so a flat key matching a system name IS the
|
||||||
|
// system value); 'user' = inside the metadata bag, where EVERY key is
|
||||||
|
// the user's and indexes bare — collider names included.
|
||||||
|
type Frame = 'entity-record' | 'flat-record' | 'user'
|
||||||
|
const extract = (obj: any, prefix = '', frame: Frame = 'entity-record'): void => {
|
||||||
for (const [key, value] of Object.entries(obj)) {
|
for (const [key, value] of Object.entries(obj)) {
|
||||||
const fullKey = prefix ? `${prefix}.${key}` : key
|
let fullKey = prefix ? `${prefix}.${key}` : key
|
||||||
|
|
||||||
// Skip fields in never-index list (CRITICAL: prevents vector indexing bug + HNSW fields)
|
if (!prefix && frame !== 'user') {
|
||||||
if (!prefix && NEVER_INDEX.has(key)) continue
|
if (key === 'metadata' && typeof value === 'object' && value !== null && !Array.isArray(value)) {
|
||||||
|
extract(value, '', 'user') // the user's namespace: bare keys
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if (key === 'type' || key === 'noun') {
|
||||||
|
fullKey = 'system.type' // legacy 'noun' spelling folds into the frozen key
|
||||||
|
} else if (SYSTEM_ENTITY_SCALARS.has(key) && key !== 'id') {
|
||||||
|
fullKey = `system.${key}`
|
||||||
|
} else if (
|
||||||
|
key === 'data' || key === '_rev' || key === 'level' || key === '_fmt' ||
|
||||||
|
RECORD_PLUMBING.has(key)
|
||||||
|
) {
|
||||||
|
continue // plumbing / identity / format stamp — never indexed from a record frame
|
||||||
|
} else if (frame === 'entity-record') {
|
||||||
|
continue // stray entity-frame key: dropped, not guessed
|
||||||
|
}
|
||||||
|
// flat-record fallthrough: a non-system, non-plumbing key IS a user
|
||||||
|
// field (flat beside the engine's, legacy shape) — indexes bare.
|
||||||
|
}
|
||||||
|
// User frame: NO name-based skips — every user field indexes, whatever
|
||||||
|
// its name (the field-addressing law). Only the uniform value-shape
|
||||||
|
// guards below apply.
|
||||||
|
|
||||||
// Skip purely numeric field names (array indices converted to object keys)
|
// Skip purely numeric field names (array indices converted to object keys)
|
||||||
// Legitimate field names should never be purely numeric
|
// Legitimate field names should never be purely numeric
|
||||||
// This catches vectors stored as objects: {0: 0.1, 1: 0.2, ...}
|
// This catches vectors stored as objects: {0: 0.1, 1: 0.2, ...}
|
||||||
if (/^\d+$/.test(key)) continue
|
if (/^\d+$/.test(key)) continue
|
||||||
|
|
||||||
// Skip fields based on user configuration
|
|
||||||
if (!this.shouldIndexField(fullKey)) continue
|
|
||||||
|
|
||||||
// Special handling for metadata field at top level
|
|
||||||
// Flatten metadata fields to top-level (no prefix) for cleaner queries
|
|
||||||
// Standard fields are already at top-level, custom fields go in metadata
|
|
||||||
// By flattening here, queries can use { category: 'B' } instead of { 'metadata.category': 'B' }
|
|
||||||
if (key === 'metadata' && !prefix && typeof value === 'object' && !Array.isArray(value)) {
|
|
||||||
extract(value, '') // Flatten to top-level, no prefix
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
// Skip large arrays (> 10 elements) - likely vectors or bulk data
|
// Skip large arrays (> 10 elements) - likely vectors or bulk data
|
||||||
if (Array.isArray(value) && value.length > 10) continue
|
if (Array.isArray(value) && value.length > 10) continue
|
||||||
|
|
||||||
if (value && typeof value === 'object' && !Array.isArray(value)) {
|
if (value && typeof value === 'object' && !Array.isArray(value)) {
|
||||||
// Recurse into nested objects (but not arrays)
|
// Recurse into nested objects (but not arrays), keeping the frame
|
||||||
extract(value, fullKey)
|
extract(value, fullKey, frame)
|
||||||
} else if (Array.isArray(value) && value.length <= 10) {
|
} else if (Array.isArray(value) && value.length <= 10) {
|
||||||
// Small arrays: index as multi-value field (all with same field name)
|
// Small arrays: index as multi-value field (all with same field name)
|
||||||
// Example: tags: ["javascript", "node"] → field="tags", value="javascript" + field="tags", value="node"
|
// Example: tags: ["javascript", "node"] → field="tags", value="javascript" + field="tags", value="node"
|
||||||
|
|
@ -1258,16 +1265,21 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
// Primitive value: index it
|
// Primitive value: index it under the frozen key computed above.
|
||||||
// Map 'type' → 'noun' for backward compatibility
|
// (The legacy 'type'→'noun' remap is gone — 'noun' columns die at
|
||||||
const indexField = (!prefix && key === 'type') ? 'noun' : fullKey
|
// the epoch-3 rebuild; system.type is the one spelling.)
|
||||||
fields.push({ field: indexField, value })
|
fields.push({ field: fullKey, value })
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (data && typeof data === 'object') {
|
if (data && typeof data === 'object') {
|
||||||
extract(data)
|
// Shape detection for the top frame: an object carrying a nested
|
||||||
|
// `metadata` bag is the entityForIndexing shape; anything else is the
|
||||||
|
// flat stored-record shape (user fields flat beside reserved ones).
|
||||||
|
const entityShaped =
|
||||||
|
'metadata' in data && typeof data.metadata === 'object' && data.metadata !== null
|
||||||
|
extract(data, '', entityShaped ? 'entity-record' : 'flat-record')
|
||||||
}
|
}
|
||||||
|
|
||||||
// Extract words for hybrid text search
|
// Extract words for hybrid text search
|
||||||
|
|
@ -1469,10 +1481,11 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
prodLog.debug(`Entity ${id} has ${wordFields.length} indexed words (large document)`)
|
prodLog.debug(`Entity ${id} has ${wordFields.length} indexed words (large document)`)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Sort fields to process 'noun' field first for type-field affinity tracking
|
// Sort fields to process the type column first for type-field affinity
|
||||||
|
// tracking ('system.type' is the frozen key; 'noun' died at epoch 3).
|
||||||
fields.sort((a, b) => {
|
fields.sort((a, b) => {
|
||||||
if (a.field === 'noun') return -1
|
if (a.field === 'system.type') return -1
|
||||||
if (b.field === 'noun') return 1
|
if (b.field === 'system.type') return 1
|
||||||
return 0
|
return 0
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|
@ -1911,22 +1924,15 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
// Skip logical operators
|
// Skip logical operators
|
||||||
if (rawField === 'allOf' || rawField === 'anyOf' || rawField === 'not') continue
|
if (rawField === 'allOf' || rawField === 'anyOf' || rawField === 'not') continue
|
||||||
|
|
||||||
// Metadata is FLATTENED at index time (metadata.entry.title indexes as
|
// THE ONE ADDRESSING LAW (sealed 2026-08-03): every filter key routes
|
||||||
// entry.title), so a `metadata.`-prefixed where key is almost always
|
// through parseFieldAddress — bare and 'metadata.'-prefixed spellings
|
||||||
// the caller spelling the STORAGE shape rather than the index shape.
|
// address the user's fields (indexed under BARE keys), 'system.<field>'
|
||||||
// Accept both spellings: when the key as spelled is unindexed but its
|
// addresses the ten engine scalars (indexed under their literal
|
||||||
// stripped spelling is, query the stripped one. A literal nested
|
// 'system.<field>' keys). A malformed address (system.<not-in-map>,
|
||||||
// custom key named `metadata` still wins when indexed as spelled
|
// plumbing in the system spelling) throws typed BEFORE any index read —
|
||||||
// (checked first), so that rare shape keeps working.
|
// an accepted name either works or refuses.
|
||||||
let field = rawField
|
const address = parseFieldAddress(rawField, 'entity')
|
||||||
if (
|
const field = address.scope === 'system' ? `system.${address.field}` : address.field
|
||||||
rawField.startsWith('metadata.') &&
|
|
||||||
this.columnStore &&
|
|
||||||
!this.columnStore.hasField(rawField) &&
|
|
||||||
this.columnStore.hasField(rawField.slice('metadata.'.length))
|
|
||||||
) {
|
|
||||||
field = rawField.slice('metadata.'.length)
|
|
||||||
}
|
|
||||||
|
|
||||||
let fieldResults: string[] = []
|
let fieldResults: string[] = []
|
||||||
|
|
||||||
|
|
@ -2207,9 +2213,30 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
order: 'asc' | 'desc' = 'asc',
|
order: 'asc' | 'desc' = 'asc',
|
||||||
topK?: number
|
topK?: number
|
||||||
): Promise<string[]> {
|
): Promise<string[]> {
|
||||||
|
// THE ONE ADDRESSING LAW — the orderBy address routes through the same
|
||||||
|
// parse the filter path uses (the historical asymmetry where the filter
|
||||||
|
// path understood 'metadata.' but the sorted path never did is dead).
|
||||||
|
// Bare / 'metadata.' → the user's bare index key; 'system.<field>' → the
|
||||||
|
// literal frozen key; malformed addresses throw typed before any read.
|
||||||
|
const orderAddress = parseFieldAddress(orderBy, 'entity')
|
||||||
|
const orderKey =
|
||||||
|
orderAddress.scope === 'system' ? `system.${orderAddress.field}` : orderAddress.field
|
||||||
|
|
||||||
|
// DATA-AWARE REFUSAL (the did-you-mean): a bare address no user field
|
||||||
|
// carries cannot mean anything as a sort key — and when the name collides
|
||||||
|
// with a system scalar the caller almost certainly meant system.<field>.
|
||||||
|
// Refusing loudly with both candidates beats silently sorting nothing.
|
||||||
|
if (
|
||||||
|
orderAddress.scope === 'metadata' &&
|
||||||
|
!(this.columnStore && this.columnStore.hasField(orderKey)) &&
|
||||||
|
!(await this.loadSparseIndex(orderKey))
|
||||||
|
) {
|
||||||
|
throw new UnresolvableFieldError(orderAddress.raw, 'entity')
|
||||||
|
}
|
||||||
|
|
||||||
// Column store path: O(K log S) sort via k-way merge across segments.
|
// Column store path: O(K log S) sort via k-way merge across segments.
|
||||||
// No per-entity storage reads, no precision loss from bucketing.
|
// No per-entity storage reads, no precision loss from bucketing.
|
||||||
if (this.columnStore && this.columnStore.hasField(orderBy)) {
|
if (this.columnStore && this.columnStore.hasField(orderKey)) {
|
||||||
// Get filtered IDs from existing roaring bitmap path
|
// Get filtered IDs from existing roaring bitmap path
|
||||||
const hasFilter = filter && Object.keys(filter).length > 0
|
const hasFilter = filter && Object.keys(filter).length > 0
|
||||||
const filteredIds = hasFilter ? await this.getIdsForFilter(filter) : []
|
const filteredIds = hasFilter ? await this.getIdsForFilter(filter) : []
|
||||||
|
|
@ -2229,20 +2256,41 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
// log K) heap, not a full sort materialization.
|
// log K) heap, not a full sort materialization.
|
||||||
const k = topK !== undefined ? Math.min(topK, filteredIds.length) : filteredIds.length
|
const k = topK !== undefined ? Math.min(topK, filteredIds.length) : filteredIds.length
|
||||||
sortedIntIds = await this.columnStore.filteredSortTopK(
|
sortedIntIds = await this.columnStore.filteredSortTopK(
|
||||||
filterBitmap, orderBy, order, k
|
filterBitmap, orderKey, order, k
|
||||||
)
|
)
|
||||||
} else {
|
} else {
|
||||||
// Unfiltered sort — column store handles the full entity set efficiently
|
// Unfiltered sort — column store handles the full entity set efficiently
|
||||||
sortedIntIds = await this.columnStore.sortTopK(
|
sortedIntIds = await this.columnStore.sortTopK(
|
||||||
orderBy, order, topK !== undefined ? Math.min(topK, this.idMapper.size) : this.idMapper.size
|
orderKey, order, topK !== undefined ? Math.min(topK, this.idMapper.size) : this.idMapper.size
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Convert int IDs back to UUIDs. Number() narrowing is lossless — the
|
// Convert int IDs back to UUIDs. Number() narrowing is lossless — the
|
||||||
// shipped EntityIdSpaceExceeded guard caps the JS mapper at u32.
|
// shipped EntityIdSpaceExceeded guard caps the JS mapper at u32.
|
||||||
return sortedIntIds
|
const sortedUuids = sortedIntIds
|
||||||
.map(intId => this.idMapper.getUuid(Number(intId)))
|
.map(intId => this.idMapper.getUuid(Number(intId)))
|
||||||
.filter((uuid): uuid is string => uuid !== undefined)
|
.filter((uuid): uuid is string => uuid !== undefined)
|
||||||
|
|
||||||
|
// ORDERING CONTRACT (cross-engine, sealed): rows missing the field are
|
||||||
|
// NEVER dropped — they sort LAST in both directions — and ties break by
|
||||||
|
// id ascending. The column only contains rows that HAVE the field, so
|
||||||
|
// (1) re-sort the page deterministically (value, then id) with K cheap
|
||||||
|
// value reads, and (2) append the filtered rows the column omitted,
|
||||||
|
// id-ascending, filling any remaining page budget.
|
||||||
|
const page = await Promise.all(
|
||||||
|
sortedUuids.map(async id => ({ id, value: await this.getFieldValueForEntity(id, orderKey) }))
|
||||||
|
)
|
||||||
|
page.sort((a, b) => this.compareAddressedValues(a.value, b.value, a.id, b.id, order))
|
||||||
|
let result = page.map(p => p.id)
|
||||||
|
|
||||||
|
if (hasFilter) {
|
||||||
|
const present = new Set(sortedUuids)
|
||||||
|
if (topK === undefined || result.length < topK) {
|
||||||
|
const missing = filteredIds.filter(id => !present.has(id)).sort()
|
||||||
|
result = result.concat(missing)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return topK !== undefined ? result.slice(0, topK) : result
|
||||||
}
|
}
|
||||||
|
|
||||||
// Fallback: sparse index path (for fields not yet in column store).
|
// Fallback: sparse index path (for fields not yet in column store).
|
||||||
|
|
@ -2255,27 +2303,11 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
|
|
||||||
const idValuePairs: Array<{ id: string, value: any }> = []
|
const idValuePairs: Array<{ id: string, value: any }> = []
|
||||||
for (const id of filteredIds) {
|
for (const id of filteredIds) {
|
||||||
const value = await this.getFieldValueForEntity(id, orderBy)
|
const value = await this.getFieldValueForEntity(id, orderKey)
|
||||||
idValuePairs.push({ id, value })
|
idValuePairs.push({ id, value })
|
||||||
}
|
}
|
||||||
|
|
||||||
idValuePairs.sort((a, b) => {
|
idValuePairs.sort((a, b) => this.compareAddressedValues(a.value, b.value, a.id, b.id, order))
|
||||||
if (a.value == null && b.value == null) return 0
|
|
||||||
if (a.value == null) return order === 'asc' ? 1 : -1
|
|
||||||
if (b.value == null) return order === 'asc' ? -1 : 1
|
|
||||||
if (a.value === b.value) return 0
|
|
||||||
// Numbers compare numerically; everything else by code-point (UTF-8 byte) order.
|
|
||||||
// This makes the JS fallback sort match cor's native column store exactly
|
|
||||||
// (numeric i64/f64 vs code-point strings) and stay deterministic across
|
|
||||||
// environments, unlike the `<` operator's UTF-16 ordering for strings.
|
|
||||||
let comparison: number
|
|
||||||
if (typeof a.value === 'number' && typeof b.value === 'number') {
|
|
||||||
comparison = a.value < b.value ? -1 : 1
|
|
||||||
} else {
|
|
||||||
comparison = compareCodePoints(String(a.value), String(b.value))
|
|
||||||
}
|
|
||||||
return order === 'asc' ? comparison : -comparison
|
|
||||||
})
|
|
||||||
|
|
||||||
const sorted = idValuePairs.map(p => p.id)
|
const sorted = idValuePairs.map(p => p.id)
|
||||||
return topK !== undefined ? sorted.slice(0, topK) : sorted
|
return topK !== undefined ? sorted.slice(0, topK) : sorted
|
||||||
|
|
@ -2309,11 +2341,51 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
*
|
*
|
||||||
* @public (called from brainy.ts for sorted queries)
|
* @public (called from brainy.ts for sorted queries)
|
||||||
*/
|
*/
|
||||||
|
/**
|
||||||
|
* The cross-engine ordering contract in one comparator (sealed 2026-08-03):
|
||||||
|
* missing/null values sort LAST in BOTH directions — the direction flip
|
||||||
|
* never moves them to the front — and ties break by id ascending, so an
|
||||||
|
* ordered read is deterministic and identical on both engines. Numbers
|
||||||
|
* compare numerically; everything else by code-point (UTF-8 byte) order,
|
||||||
|
* matching the native column store exactly.
|
||||||
|
*/
|
||||||
|
private compareAddressedValues(
|
||||||
|
aVal: any,
|
||||||
|
bVal: any,
|
||||||
|
aId: string,
|
||||||
|
bId: string,
|
||||||
|
order: 'asc' | 'desc'
|
||||||
|
): number {
|
||||||
|
const aNull = aVal == null
|
||||||
|
const bNull = bVal == null
|
||||||
|
if (aNull || bNull) {
|
||||||
|
if (aNull && bNull) return aId < bId ? -1 : aId > bId ? 1 : 0
|
||||||
|
return aNull ? 1 : -1
|
||||||
|
}
|
||||||
|
let comparison = 0
|
||||||
|
if (aVal !== bVal) {
|
||||||
|
if (typeof aVal === 'number' && typeof bVal === 'number') {
|
||||||
|
comparison = aVal < bVal ? -1 : 1
|
||||||
|
} else {
|
||||||
|
comparison = compareCodePoints(String(aVal), String(bVal))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (comparison === 0) return aId < bId ? -1 : aId > bId ? 1 : 0
|
||||||
|
return order === 'asc' ? comparison : -comparison
|
||||||
|
}
|
||||||
|
|
||||||
async getFieldValueForEntity(entityId: string, field: string): Promise<any> {
|
async getFieldValueForEntity(entityId: string, field: string): Promise<any> {
|
||||||
// Path 1: Bucketed fields need the actual value from storage.
|
// `field` arrives as a FROZEN INDEX KEY (bare = user metadata;
|
||||||
|
// 'system.<field>' = engine scalar). Storage fallbacks read the matching
|
||||||
|
// side of the record — a system key reads the record scalar, a bare key
|
||||||
|
// reads the user's metadata bag; the two can never shadow each other.
|
||||||
|
const systemInner = field.startsWith('system.') ? field.slice('system.'.length) : null
|
||||||
|
|
||||||
|
// Path 1: Bucketed fields need the actual (un-bucketed) value from storage.
|
||||||
if (BUCKETED_INDEX_FIELDS.has(field)) {
|
if (BUCKETED_INDEX_FIELDS.has(field)) {
|
||||||
const noun = await this.storage.getNoun(entityId)
|
const noun = await this.storage.getNoun(entityId)
|
||||||
return noun ? resolveEntityField(noun, field) : undefined
|
if (!noun) return undefined
|
||||||
|
return (noun as unknown as Record<string, unknown>)[systemInner as string]
|
||||||
}
|
}
|
||||||
|
|
||||||
// Path 3 precondition: entity must be in the id mapper for bitmap lookup.
|
// Path 3 precondition: entity must be in the id mapper for bitmap lookup.
|
||||||
|
|
@ -2330,7 +2402,11 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
// yet indexed. resolveEntityField handles the shape contract.
|
// yet indexed. resolveEntityField handles the shape contract.
|
||||||
if (!sparseIndex) {
|
if (!sparseIndex) {
|
||||||
const noun = await this.storage.getNoun(entityId)
|
const noun = await this.storage.getNoun(entityId)
|
||||||
return noun ? resolveEntityField(noun, field) : undefined
|
if (!noun) return undefined
|
||||||
|
if (systemInner !== null) {
|
||||||
|
return (noun as unknown as Record<string, unknown>)[systemInner]
|
||||||
|
}
|
||||||
|
return (noun as { metadata?: Record<string, unknown> }).metadata?.[field]
|
||||||
}
|
}
|
||||||
|
|
||||||
// Path 3: Search sparse index chunks for this entity's value.
|
// Path 3: Search sparse index chunks for this entity's value.
|
||||||
|
|
@ -2757,6 +2833,17 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
// VFS Statistics Methods (uses existing Roaring bitmap infrastructure)
|
// VFS Statistics Methods (uses existing Roaring bitmap infrastructure)
|
||||||
// ============================================================================
|
// ============================================================================
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Read the type column's bitmap for one type value — frozen key first
|
||||||
|
* ('system.type', epoch 3), legacy 'noun' as the pre-rebuild fallback.
|
||||||
|
*/
|
||||||
|
private async getTypeBitmap(type: string): Promise<RoaringBitmap32 | null> {
|
||||||
|
return (
|
||||||
|
(await this.getBitmapFromChunks('system.type', type)) ??
|
||||||
|
(await this.getBitmapFromChunks('noun', type))
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Get VFS entity count for a specific type using Roaring bitmap intersection
|
* Get VFS entity count for a specific type using Roaring bitmap intersection
|
||||||
* Uses hardware-accelerated SIMD operations (AVX2/SSE4.2)
|
* Uses hardware-accelerated SIMD operations (AVX2/SSE4.2)
|
||||||
|
|
@ -2765,7 +2852,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
*/
|
*/
|
||||||
async getVFSEntityCountByType(type: string): Promise<number> {
|
async getVFSEntityCountByType(type: string): Promise<number> {
|
||||||
const vfsBitmap = await this.getBitmapFromChunks('isVFSEntity', true)
|
const vfsBitmap = await this.getBitmapFromChunks('isVFSEntity', true)
|
||||||
const typeBitmap = await this.getBitmapFromChunks('noun', type)
|
const typeBitmap = await this.getTypeBitmap(type)
|
||||||
|
|
||||||
if (!vfsBitmap || !typeBitmap) return 0
|
if (!vfsBitmap || !typeBitmap) return 0
|
||||||
|
|
||||||
|
|
@ -2788,7 +2875,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
|
|
||||||
// Iterate through all known types and compute VFS count via intersection
|
// Iterate through all known types and compute VFS count via intersection
|
||||||
for (const type of this.totalEntitiesByType.keys()) {
|
for (const type of this.totalEntitiesByType.keys()) {
|
||||||
const typeBitmap = await this.getBitmapFromChunks('noun', type)
|
const typeBitmap = await this.getTypeBitmap(type)
|
||||||
if (typeBitmap) {
|
if (typeBitmap) {
|
||||||
const intersection = RoaringBitmap32.and(vfsBitmap, typeBitmap)
|
const intersection = RoaringBitmap32.and(vfsBitmap, typeBitmap)
|
||||||
if (intersection.size > 0) {
|
if (intersection.size > 0) {
|
||||||
|
|
@ -3382,18 +3469,21 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
* Tracks which fields commonly appear with which entity types
|
* Tracks which fields commonly appear with which entity types
|
||||||
*/
|
*/
|
||||||
private updateTypeFieldAffinity(entityId: string, field: string, value: any, operation: 'add' | 'remove', metadata?: any): void {
|
private updateTypeFieldAffinity(entityId: string, field: string, value: any, operation: 'add' | 'remove', metadata?: any): void {
|
||||||
// Only track affinity for non-system fields (but allow 'noun' for type detection)
|
// Only track affinity for user fields (plus the type column itself,
|
||||||
if (this.config.excludeFields.includes(field) && field !== 'noun') return
|
// which drives detection). Engine columns carry the literal 'system.'
|
||||||
|
// prefix under the frozen key format.
|
||||||
|
if (field.startsWith('system.') && field !== 'system.type') return
|
||||||
|
|
||||||
// For the 'noun' field, the value IS the entity type
|
// For the type column ('system.type'), the value IS the entity type
|
||||||
let entityType: string | null = null
|
let entityType: string | null = null
|
||||||
|
|
||||||
if (field === 'noun') {
|
if (field === 'system.type') {
|
||||||
// This is the type definition itself
|
// This is the type definition itself
|
||||||
entityType = this.normalizeValue(value, field) // Pass field for bucketing!
|
entityType = this.normalizeValue(value, field) // Pass field for bucketing!
|
||||||
} else if (metadata && metadata.noun) {
|
} else if (metadata && (metadata.noun ?? metadata.type)) {
|
||||||
// Extract entity type from metadata
|
// Extract entity type from the source shape: stored records carry it
|
||||||
entityType = this.normalizeValue(metadata.noun, 'noun')
|
// under 'noun', entity-for-indexing views under 'type'.
|
||||||
|
entityType = this.normalizeValue(metadata.noun ?? metadata.type, 'system.type')
|
||||||
} else {
|
} else {
|
||||||
// No type information available, skip affinity tracking
|
// No type information available, skip affinity tracking
|
||||||
return
|
return
|
||||||
|
|
@ -3416,8 +3506,9 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
const currentCount = typeFields.get(field) || 0
|
const currentCount = typeFields.get(field) || 0
|
||||||
typeFields.set(field, currentCount + 1)
|
typeFields.set(field, currentCount + 1)
|
||||||
|
|
||||||
// Update total entities of this type (only count once per entity)
|
// Update total entities of this type (only count once per entity —
|
||||||
if (field === 'noun') {
|
// the type column appears exactly once per entity)
|
||||||
|
if (field === 'system.type') {
|
||||||
const newCount = this.totalEntitiesByType.get(entityType)! + 1
|
const newCount = this.totalEntitiesByType.get(entityType)! + 1
|
||||||
this.totalEntitiesByType.set(entityType, newCount)
|
this.totalEntitiesByType.set(entityType, newCount)
|
||||||
|
|
||||||
|
|
@ -3440,7 +3531,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
}
|
}
|
||||||
|
|
||||||
// Update total entities of this type
|
// Update total entities of this type
|
||||||
if (field === 'noun') {
|
if (field === 'system.type') {
|
||||||
const total = this.totalEntitiesByType.get(entityType)!
|
const total = this.totalEntitiesByType.get(entityType)!
|
||||||
if (total > 1) {
|
if (total > 1) {
|
||||||
const newCount = total - 1
|
const newCount = total - 1
|
||||||
|
|
|
||||||
|
|
@ -17,6 +17,7 @@ import { findCallerLocation } from './callerLocation.js'
|
||||||
// fallback branches that no supported runtime can reach.
|
// fallback branches that no supported runtime can reach.
|
||||||
import * as os from 'node:os'
|
import * as os from 'node:os'
|
||||||
import * as fs from 'node:fs'
|
import * as fs from 'node:fs'
|
||||||
|
import { parseFieldAddress, UnsupportedFindOptionError } from '../db/fieldAddressing.js'
|
||||||
|
|
||||||
const getSystemMemory = (): number => {
|
const getSystemMemory = (): number => {
|
||||||
if (os) {
|
if (os) {
|
||||||
|
|
@ -466,9 +467,31 @@ export function validateFindParams(params: FindParams): void {
|
||||||
throw new Error('cannot specify both query and vector - they are mutually exclusive')
|
throw new Error('cannot specify both query and vector - they are mutually exclusive')
|
||||||
}
|
}
|
||||||
|
|
||||||
// Universal truth: can't use both cursor and offset pagination
|
// ACCEPTED-AND-IGNORED DIED AS A CLASS (sealed 2026-08-03): options the
|
||||||
if (params.cursor !== undefined && params.offset !== undefined) {
|
// engine does not implement REFUSE with a typed error instead of silently
|
||||||
throw new Error('cannot use both cursor and offset pagination simultaneously')
|
// doing nothing — a production consumer discovered a no-op by measurement
|
||||||
|
// once; never again.
|
||||||
|
if (params.cursor !== undefined) {
|
||||||
|
throw new UnsupportedFindOptionError('cursor')
|
||||||
|
}
|
||||||
|
if ((params as Record<string, unknown>).includeRelations !== undefined) {
|
||||||
|
throw new UnsupportedFindOptionError('includeRelations')
|
||||||
|
}
|
||||||
|
if ((params as Record<string, unknown>).writeOnly !== undefined) {
|
||||||
|
throw new UnsupportedFindOptionError('writeOnly')
|
||||||
|
}
|
||||||
|
|
||||||
|
// THE ONE ADDRESSING LAW: the orderBy address must PARSE (bare/metadata. =
|
||||||
|
// user field, system.<field> = the ruled map, anything else refuses typed
|
||||||
|
// with the valid map in the message) and order must be a real direction.
|
||||||
|
if (params.orderBy !== undefined) {
|
||||||
|
if (typeof params.orderBy !== 'string') {
|
||||||
|
throw new Error('orderBy must be a string field address')
|
||||||
|
}
|
||||||
|
parseFieldAddress(params.orderBy, 'entity') // throws InvalidFieldAddressError on a bad address
|
||||||
|
}
|
||||||
|
if (params.order !== undefined && params.order !== 'asc' && params.order !== 'desc') {
|
||||||
|
throw new Error(`order must be 'asc' or 'desc', got '${String(params.order)}'`)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Auto-limit query length based on memory
|
// Auto-limit query length based on memory
|
||||||
|
|
@ -495,7 +518,28 @@ export function validateFindParams(params: FindParams): void {
|
||||||
/**
|
/**
|
||||||
* Validate add parameters
|
* Validate add parameters
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The namespace cannot be forged: a USER metadata key literally spelled
|
||||||
|
* 'system.<anything>' would collide with the engine's explicit address
|
||||||
|
* namespace at read time — refuse it at the write door, loudly, with the
|
||||||
|
* fix in the message (sealed 2026-08-03).
|
||||||
|
*/
|
||||||
|
function rejectForgedSystemKeys(metadata: Record<string, unknown> | undefined, site: string): void {
|
||||||
|
if (!metadata) return
|
||||||
|
for (const key of Object.keys(metadata)) {
|
||||||
|
if (key.startsWith('system.')) {
|
||||||
|
throw new Error(
|
||||||
|
`${site}: metadata key '${key}' is not allowed — the 'system.' prefix is the ` +
|
||||||
|
`engine's explicit address namespace and cannot be used as a user field name. ` +
|
||||||
|
`Rename the field (e.g. '${key.slice('system.'.length)}').`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
export function validateAddParams(params: AddParams): void {
|
export function validateAddParams(params: AddParams): void {
|
||||||
|
rejectForgedSystemKeys(params.metadata as Record<string, unknown> | undefined, 'add()')
|
||||||
// Universal truth: must have data or vector
|
// Universal truth: must have data or vector
|
||||||
if (!params.data && !params.vector) {
|
if (!params.data && !params.vector) {
|
||||||
throw new Error(
|
throw new Error(
|
||||||
|
|
@ -536,6 +580,7 @@ export function validateAddParams(params: AddParams): void {
|
||||||
* Validate update parameters
|
* Validate update parameters
|
||||||
*/
|
*/
|
||||||
export function validateUpdateParams(params: UpdateParams): void {
|
export function validateUpdateParams(params: UpdateParams): void {
|
||||||
|
rejectForgedSystemKeys(params.metadata as Record<string, unknown> | undefined, 'update()')
|
||||||
// Universal truth: must have an ID
|
// Universal truth: must have an ID
|
||||||
if (!params.id) {
|
if (!params.id) {
|
||||||
throw new Error('id is required for update')
|
throw new Error('id is required for update')
|
||||||
|
|
@ -573,6 +618,7 @@ export function validateUpdateParams(params: UpdateParams): void {
|
||||||
* Validate relate parameters
|
* Validate relate parameters
|
||||||
*/
|
*/
|
||||||
export function validateRelateParams(params: RelateParams): void {
|
export function validateRelateParams(params: RelateParams): void {
|
||||||
|
rejectForgedSystemKeys(params.metadata as Record<string, unknown> | undefined, 'relate()')
|
||||||
// 8.0 verb-id contract (L.7): verb ids are UUIDs, generated by brainy.
|
// 8.0 verb-id contract (L.7): verb ids are UUIDs, generated by brainy.
|
||||||
// RelateParams has no `id` field — an untyped caller passing one would
|
// RelateParams has no `id` field — an untyped caller passing one would
|
||||||
// previously have it silently ignored (a generated UUID was used instead).
|
// previously have it silently ignored (a generated UUID was used instead).
|
||||||
|
|
@ -621,6 +667,7 @@ export function validateRelateParams(params: RelateParams): void {
|
||||||
* accepts type/subtype/weight/confidence/data/metadata changes.
|
* accepts type/subtype/weight/confidence/data/metadata changes.
|
||||||
*/
|
*/
|
||||||
export function validateUpdateRelationParams(params: UpdateRelationParams): void {
|
export function validateUpdateRelationParams(params: UpdateRelationParams): void {
|
||||||
|
rejectForgedSystemKeys(params.metadata as Record<string, unknown> | undefined, 'updateRelation()')
|
||||||
if (!params.id) {
|
if (!params.id) {
|
||||||
throw new Error('id is required for updateRelation')
|
throw new Error('id is required for updateRelation')
|
||||||
}
|
}
|
||||||
|
|
|
||||||
307
tests/conformance/collider-fidelity.test.ts
Normal file
307
tests/conformance/collider-fidelity.test.ts
Normal file
|
|
@ -0,0 +1,307 @@
|
||||||
|
/**
|
||||||
|
* @module tests/conformance/collider-fidelity
|
||||||
|
* @description THE REOPEN-COLLIDER CONFORMANCE CASE (required cross-engine
|
||||||
|
* before any RC counts as gates-green — ruled 2026-08-03). The
|
||||||
|
* field-addressing law's fidelity half: user metadata may carry ANY name —
|
||||||
|
* including every engine spelling (`confidence`, `type`, `id`, `createdAt`,
|
||||||
|
* …) and every plumbing name (`level`, `data`, `vector`, `_rev`) — and the
|
||||||
|
* value survives, verbatim and reachable, across the FULL lifecycle: live
|
||||||
|
* reads, where/orderBy, flush, close+reopen, a forced epoch rebuild, and
|
||||||
|
* time travel. The engine scalars stay separately reachable at `system.*`
|
||||||
|
* the whole way. No halfway states.
|
||||||
|
*
|
||||||
|
* Self-arming like the namespace-law suite: skips loudly until the arming
|
||||||
|
* exports are present, so the suite can sit on a branch ahead of the build.
|
||||||
|
*/
|
||||||
|
import { describe, it, expect, beforeAll, afterAll } from 'vitest'
|
||||||
|
import { mkdtempSync, rmSync } from 'node:fs'
|
||||||
|
import { tmpdir } from 'node:os'
|
||||||
|
import { join } from 'node:path'
|
||||||
|
import * as brainyExports from '../../src/index.js'
|
||||||
|
import { Brainy, NounType, VerbType } from '../../src/index.js'
|
||||||
|
import {
|
||||||
|
BRAIN_FORMAT_PATH,
|
||||||
|
EXPECTED_INDEX_EPOCH
|
||||||
|
} from '../../src/storage/brainFormat.js'
|
||||||
|
|
||||||
|
const ARMED = 'UnresolvableFieldError' in brainyExports
|
||||||
|
const suite = ARMED ? describe : describe.skip
|
||||||
|
if (!ARMED) {
|
||||||
|
// eslint-disable-next-line no-console
|
||||||
|
console.warn(
|
||||||
|
'[collider-fidelity] SKIPPING: package root does not export the ' +
|
||||||
|
'field-addressing law surface yet (UnresolvableFieldError absent).'
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Every entity system scalar name written as a USER metadata field, with
|
||||||
|
* unmistakable user values, plus the plumbing names and naturals. */
|
||||||
|
const COLLIDER_BAG = {
|
||||||
|
// the ten entity system scalars, as user fields
|
||||||
|
id: 'user-id',
|
||||||
|
type: 'user-type',
|
||||||
|
subtype: 'user-subtype',
|
||||||
|
createdAt: 'user-createdAt',
|
||||||
|
updatedAt: 'user-updatedAt',
|
||||||
|
confidence: 'user-confidence',
|
||||||
|
weight: 'user-weight',
|
||||||
|
visibility: 'user-visibility',
|
||||||
|
service: 'user-service',
|
||||||
|
createdBy: 'user-createdBy',
|
||||||
|
// plumbing names, as user fields
|
||||||
|
level: 7,
|
||||||
|
data: 'user-data',
|
||||||
|
vector: 'user-vector',
|
||||||
|
_rev: 'user-rev',
|
||||||
|
// naturals previously silently un-indexed by name
|
||||||
|
content: 'user-content',
|
||||||
|
// a plain control field
|
||||||
|
plain: 'control'
|
||||||
|
} as const
|
||||||
|
|
||||||
|
|
||||||
|
suite('collider fidelity — the reopen-collider case (both suites, ruled)', () => {
|
||||||
|
let dir: string
|
||||||
|
let brain: Brainy
|
||||||
|
let colliderId: string
|
||||||
|
|
||||||
|
const open = async (): Promise<Brainy> => {
|
||||||
|
const b = new Brainy({
|
||||||
|
storage: { type: 'filesystem', path: dir },
|
||||||
|
requireSubtype: false
|
||||||
|
})
|
||||||
|
await b.init()
|
||||||
|
return b
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The full read battery — run at every lifecycle boundary. */
|
||||||
|
const verifyColliderTruth = async (label: string): Promise<void> => {
|
||||||
|
// 1. get(): the bag comes back verbatim; engine scalars stay engine.
|
||||||
|
const entity = await brain.get(colliderId)
|
||||||
|
expect(entity, `${label}: entity readable`).toBeTruthy()
|
||||||
|
for (const [k, v] of Object.entries(COLLIDER_BAG)) {
|
||||||
|
expect(
|
||||||
|
(entity!.metadata as Record<string, unknown>)[k],
|
||||||
|
`${label}: bag.${k} verbatim`
|
||||||
|
).toEqual(v)
|
||||||
|
}
|
||||||
|
expect(entity!.type, `${label}: engine type intact`).toBe(NounType.Document)
|
||||||
|
expect(entity!.confidence, `${label}: engine confidence intact`).toBe(0.25)
|
||||||
|
|
||||||
|
// 2. where on collider names (bare = the user's field, always).
|
||||||
|
for (const [k, v] of [
|
||||||
|
['confidence', 'user-confidence'],
|
||||||
|
['type', 'user-type'],
|
||||||
|
['id', 'user-id'],
|
||||||
|
['content', 'user-content'],
|
||||||
|
['data', 'user-data'],
|
||||||
|
['level', 7]
|
||||||
|
] as const) {
|
||||||
|
const rows = await brain.find({ where: { [k]: v }, limit: 10 })
|
||||||
|
expect(
|
||||||
|
rows.map((r) => r.id),
|
||||||
|
`${label}: where {${k}} finds the collider row`
|
||||||
|
).toContain(colliderId)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3. system.* keeps reading the ENGINE values.
|
||||||
|
const byEngine = await brain.find({
|
||||||
|
where: { 'system.confidence': 0.25 },
|
||||||
|
limit: 10
|
||||||
|
})
|
||||||
|
expect(
|
||||||
|
byEngine.map((r) => r.id),
|
||||||
|
`${label}: system.confidence reads the engine scalar`
|
||||||
|
).toContain(colliderId)
|
||||||
|
const byUserSpelledSystem = await brain.find({
|
||||||
|
where: { 'system.confidence': 'user-confidence' },
|
||||||
|
limit: 10
|
||||||
|
})
|
||||||
|
expect(
|
||||||
|
byUserSpelledSystem.map((r) => r.id),
|
||||||
|
`${label}: the user's value is NOT reachable via system.*`
|
||||||
|
).not.toContain(colliderId)
|
||||||
|
|
||||||
|
// 4. orderBy a collider name orders by the USER values.
|
||||||
|
const ordered = await brain.find({
|
||||||
|
type: NounType.Document,
|
||||||
|
orderBy: 'level',
|
||||||
|
order: 'desc',
|
||||||
|
limit: 10
|
||||||
|
})
|
||||||
|
expect(ordered.length, `${label}: ordered read complete`).toBe(3)
|
||||||
|
expect(
|
||||||
|
(ordered[0].metadata as Record<string, unknown>).plain,
|
||||||
|
`${label}: user level orders desc (7 first)`
|
||||||
|
).toBe('control')
|
||||||
|
}
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
dir = mkdtempSync(join(tmpdir(), 'brainy-collider-'))
|
||||||
|
brain = await open()
|
||||||
|
|
||||||
|
colliderId = await brain.add({
|
||||||
|
data: 'the collider probe document',
|
||||||
|
type: NounType.Document,
|
||||||
|
confidence: 0.25,
|
||||||
|
metadata: { ...COLLIDER_BAG }
|
||||||
|
})
|
||||||
|
// two ordering companions with smaller user `level`s
|
||||||
|
await brain.add({
|
||||||
|
data: 'ordering companion low',
|
||||||
|
type: NounType.Document,
|
||||||
|
metadata: { level: 3, plain: 'low' }
|
||||||
|
})
|
||||||
|
await brain.add({
|
||||||
|
data: 'ordering companion mid',
|
||||||
|
type: NounType.Document,
|
||||||
|
metadata: { level: 5, plain: 'mid' }
|
||||||
|
})
|
||||||
|
}, 120000)
|
||||||
|
|
||||||
|
afterAll(async () => {
|
||||||
|
await brain.close().catch(() => {})
|
||||||
|
rmSync(dir, { recursive: true, force: true })
|
||||||
|
})
|
||||||
|
|
||||||
|
it('LIVE: colliders are the user’s, verbatim and fully queryable', async () => {
|
||||||
|
await verifyColliderTruth('live')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('REOPEN: the restart boundary loses nothing', async () => {
|
||||||
|
await brain.flush()
|
||||||
|
await brain.close()
|
||||||
|
brain = await open()
|
||||||
|
await verifyColliderTruth('reopen')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('REBUILD: a forced epoch rebuild re-indexes the colliders from canonical', async () => {
|
||||||
|
await brain.close()
|
||||||
|
// Simulate epoch drift: a missing marker forces the full derived-index
|
||||||
|
// rebuild at open — the exact path every pre-law brain takes once.
|
||||||
|
rmSync(join(dir, BRAIN_FORMAT_PATH), { force: true })
|
||||||
|
brain = await open()
|
||||||
|
await verifyColliderTruth('rebuild')
|
||||||
|
// And the rebuild re-stamps the current epoch.
|
||||||
|
const marker = await (
|
||||||
|
brain as unknown as {
|
||||||
|
storage: { readRawObject(p: string): Promise<{ indexEpoch?: number } | null> }
|
||||||
|
}
|
||||||
|
).storage.readRawObject(BRAIN_FORMAT_PATH)
|
||||||
|
expect(marker?.indexEpoch).toBe(EXPECTED_INDEX_EPOCH)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('TIME TRAVEL: asOf reads historical collider values faithfully', async () => {
|
||||||
|
const gen = brain.generation()
|
||||||
|
await brain.update({ id: colliderId, metadata: { confidence: 'user-confidence-v2' } })
|
||||||
|
const now = await brain.get(colliderId)
|
||||||
|
expect((now!.metadata as Record<string, unknown>).confidence).toBe('user-confidence-v2')
|
||||||
|
|
||||||
|
const past = await brain.asOf(gen)
|
||||||
|
try {
|
||||||
|
const then = await past.get(colliderId)
|
||||||
|
expect(
|
||||||
|
(then!.metadata as Record<string, unknown>).confidence,
|
||||||
|
'asOf reads the pre-update USER value'
|
||||||
|
).toBe('user-confidence')
|
||||||
|
} finally {
|
||||||
|
await past.release()
|
||||||
|
}
|
||||||
|
// engine scalar untouched throughout
|
||||||
|
expect(now!.confidence).toBe(0.25)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('RELATION MIRROR: edge collider bags survive write → read → reopen', async () => {
|
||||||
|
const a = await brain.add({ data: 'edge endpoint a', type: NounType.Person, metadata: { plain: 'a' } })
|
||||||
|
const b = await brain.add({ data: 'edge endpoint b', type: NounType.Person, metadata: { plain: 'b' } })
|
||||||
|
const edgeBag = {
|
||||||
|
verb: 'user-verb',
|
||||||
|
confidence: 'user-edge-confidence',
|
||||||
|
weight: 'user-edge-weight',
|
||||||
|
subtype: 'user-edge-subtype',
|
||||||
|
createdAt: 'user-edge-createdAt',
|
||||||
|
service: 'user-edge-service'
|
||||||
|
}
|
||||||
|
const relId = await brain.relate({
|
||||||
|
from: a,
|
||||||
|
to: b,
|
||||||
|
type: VerbType.RelatedTo,
|
||||||
|
confidence: 0.5,
|
||||||
|
metadata: { ...edgeBag }
|
||||||
|
})
|
||||||
|
|
||||||
|
const check = async (label: string): Promise<void> => {
|
||||||
|
const rels = await brain.related({ from: a, type: VerbType.RelatedTo })
|
||||||
|
const rel = rels.find((r) => r.id === relId)
|
||||||
|
expect(rel, `${label}: relation readable`).toBeTruthy()
|
||||||
|
for (const [k, v] of Object.entries(edgeBag)) {
|
||||||
|
expect(
|
||||||
|
(rel!.metadata as Record<string, unknown>)[k],
|
||||||
|
`${label}: edge bag.${k} verbatim`
|
||||||
|
).toEqual(v)
|
||||||
|
}
|
||||||
|
expect(rel!.confidence, `${label}: engine edge confidence intact`).toBe(0.5)
|
||||||
|
expect(rel!.type, `${label}: engine verb intact`).toBe(VerbType.RelatedTo)
|
||||||
|
}
|
||||||
|
|
||||||
|
await check('live')
|
||||||
|
await brain.flush()
|
||||||
|
await brain.close()
|
||||||
|
brain = await open()
|
||||||
|
await check('reopen')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('FORGERY: user metadata keys spelled system.* refuse at every write door', async () => {
|
||||||
|
await expect(
|
||||||
|
brain.add({ data: 'forged', type: NounType.Document, metadata: { 'system.confidence': 1 } })
|
||||||
|
).rejects.toThrow(/system\./)
|
||||||
|
await expect(
|
||||||
|
brain.update({ id: colliderId, metadata: { 'system.type': 'x' } })
|
||||||
|
).rejects.toThrow(/system\./)
|
||||||
|
const a = await brain.add({ data: 'forgery endpoint a', type: NounType.Person, metadata: {} })
|
||||||
|
const b = await brain.add({ data: 'forgery endpoint b', type: NounType.Person, metadata: {} })
|
||||||
|
await expect(
|
||||||
|
brain.relate({ from: a, to: b, type: VerbType.RelatedTo, metadata: { 'system.verb': 'x' } })
|
||||||
|
).rejects.toThrow(/system\./)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('CONFIG: the dead reservedFieldPolicy option refuses loudly, never ignored', () => {
|
||||||
|
expect(
|
||||||
|
() => new Brainy({ storage: { type: 'memory' }, reservedFieldPolicy: 'throw' } as never)
|
||||||
|
).toThrow(/field-addressing law/)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('LEGACY: a pre-law flat record still reads with engine fields top-level', async () => {
|
||||||
|
const storage = (
|
||||||
|
brain as unknown as {
|
||||||
|
storage: {
|
||||||
|
saveNoun(n: unknown): Promise<void>
|
||||||
|
saveNounMetadata(id: string, m: Record<string, unknown>): Promise<void>
|
||||||
|
}
|
||||||
|
}
|
||||||
|
).storage
|
||||||
|
const legacyId = '00000000-0000-4000-8000-00000000f1a7'
|
||||||
|
await storage.saveNoun({ id: legacyId, vector: new Array(384).fill(0.01), connections: new Map(), level: 0 })
|
||||||
|
// Legacy FLAT shape: engine + user keys mixed at one level, NO _fmt stamp.
|
||||||
|
// Sound to split by name — the pre-law door refused user colliders.
|
||||||
|
await storage.saveNounMetadata(legacyId, {
|
||||||
|
noun: NounType.Document,
|
||||||
|
confidence: 0.75,
|
||||||
|
createdAt: 1700000000000,
|
||||||
|
updatedAt: 1700000000000,
|
||||||
|
_rev: 1,
|
||||||
|
legacyField: 'legacy-value'
|
||||||
|
})
|
||||||
|
const entity = await brain.get(legacyId)
|
||||||
|
expect(entity).toBeTruthy()
|
||||||
|
expect(entity!.confidence, 'legacy flat confidence = engine').toBe(0.75)
|
||||||
|
expect(
|
||||||
|
(entity!.metadata as Record<string, unknown>).legacyField,
|
||||||
|
'legacy custom field = user bag'
|
||||||
|
).toBe('legacy-value')
|
||||||
|
expect(
|
||||||
|
(entity!.metadata as Record<string, unknown>).confidence,
|
||||||
|
'legacy flat engine key never leaks into the bag'
|
||||||
|
).toBeUndefined()
|
||||||
|
})
|
||||||
|
})
|
||||||
486
tests/conformance/namespace-law.test.ts
Normal file
486
tests/conformance/namespace-law.test.ts
Normal file
|
|
@ -0,0 +1,486 @@
|
||||||
|
/**
|
||||||
|
* @module tests/conformance/namespace-law
|
||||||
|
* @description Conformance suite for the ruled field-addressing contract
|
||||||
|
* announced in RELEASES.md ("Coming next... one field-addressing law — bare
|
||||||
|
* names = user metadata, `system.<field>` for engine fields, typed refusals
|
||||||
|
* for unresolvable names"). This suite is the drift-proof shared by this
|
||||||
|
* engine and its native accelerator: both must satisfy every test here
|
||||||
|
* bit-for-bit, because they implement the SAME contract independently.
|
||||||
|
*
|
||||||
|
* The rule, in full:
|
||||||
|
* 1. A bare field name in `where` / `orderBy` / `groupBy` / aggregation
|
||||||
|
* `source.where` ALWAYS means the caller's own `metadata` field. No
|
||||||
|
* priority resolution, no engine fallback — ever.
|
||||||
|
* 2. `system.<field>` reaches an engine scalar, and ONLY an engine scalar,
|
||||||
|
* and ONLY when spelled explicitly. The addressable entity map is exactly
|
||||||
|
* ten names: id, type, subtype, createdAt, updatedAt, confidence, weight,
|
||||||
|
* visibility, service, createdBy. The relationship map is system.verb,
|
||||||
|
* system.sourceId, system.targetId, plus the eight scalars shared with
|
||||||
|
* entities.
|
||||||
|
* 3. Some names are invisible plumbing and are never addressable in either
|
||||||
|
* spelling: vector, connections, level, data, _rev. `system.level`,
|
||||||
|
* `system.vector`, and `system.data` all refuse — they are not in the
|
||||||
|
* system map. Bare `level` is a perfectly ordinary user field.
|
||||||
|
* 4. `metadata.<field>` is the explicit-user-scope spelling: identical
|
||||||
|
* semantics to the bare spelling, valid everywhere the bare spelling is.
|
||||||
|
* 5. Anything that resolves to neither a user field nor a system scalar is a
|
||||||
|
* typed refusal naming both candidates (`UnresolvableFieldError`).
|
||||||
|
* Unimplemented `find()` options (`cursor`, `includeRelations`,
|
||||||
|
* `writeOnly`) refuse with `UnsupportedFindOptionError` instead of being
|
||||||
|
* silently accepted and ignored.
|
||||||
|
* 6. Ordering is identical on both engines: rows missing/null on the
|
||||||
|
* `orderBy` field sort LAST in BOTH directions and are never dropped;
|
||||||
|
* ties break by id ascending.
|
||||||
|
*
|
||||||
|
* The motivating incident (told generically — see CLAUDE.md naming rule): an
|
||||||
|
* internal report from a production deployment showed a user metadata field
|
||||||
|
* literally named `level` silently shadowed by the engine's internal HNSW
|
||||||
|
* node layer, breaking sort order with zero errors raised. This contract
|
||||||
|
* makes that class of bug impossible, and testable forever.
|
||||||
|
*
|
||||||
|
* SELF-SKIP: the resolver this suite pins is being built in a parallel
|
||||||
|
* session and has not landed on every branch yet. Rather than going red on
|
||||||
|
* a branch that simply hasn't caught up, the suite detects whether the
|
||||||
|
* contract is live by the one thing any conformant implementation must
|
||||||
|
* export — `UnresolvableFieldError` from the package root — and skips
|
||||||
|
* loudly (never silently) until it does. This is the house pattern: a
|
||||||
|
* sibling engine's gate once went red because a test armed before its
|
||||||
|
* feature existed.
|
||||||
|
*/
|
||||||
|
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||||
|
import { Brainy } from '../../src/brainy.js'
|
||||||
|
import { NounType } from '../../src/types/graphTypes.js'
|
||||||
|
import * as brainyExports from '../../src/index.js'
|
||||||
|
|
||||||
|
const stubEmbedding = async (text: string): Promise<number[]> => {
|
||||||
|
const hash = text.split('').reduce((acc, char) => acc + char.charCodeAt(0), 0)
|
||||||
|
return new Array(384).fill(0).map((_, i) => Math.sin(hash + i))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Detected purely by the exported error-class NAME — never by reaching into
|
||||||
|
// implementation internals. Both engines building this contract must export
|
||||||
|
// it from the package root, so this is a legitimate, implementation-agnostic
|
||||||
|
// readiness probe.
|
||||||
|
const lawActive = 'UnresolvableFieldError' in brainyExports
|
||||||
|
const UnresolvableFieldError = (brainyExports as Record<string, unknown>).UnresolvableFieldError as new (
|
||||||
|
...args: any[]
|
||||||
|
) => Error
|
||||||
|
const UnsupportedFindOptionError = (brainyExports as Record<string, unknown>)
|
||||||
|
.UnsupportedFindOptionError as new (...args: any[]) => Error
|
||||||
|
|
||||||
|
// Always runs, regardless of lawActive — the loud signal that the rest of
|
||||||
|
// this file was skipped, and why.
|
||||||
|
it('namespace law armed?', () => {
|
||||||
|
if (!lawActive) {
|
||||||
|
console.warn(
|
||||||
|
'[conformance] namespace-law suite SKIPPED — UnresolvableFieldError not exported yet; arms when the resolver lands'
|
||||||
|
)
|
||||||
|
}
|
||||||
|
expect(true).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Awaits `promise`, asserting it rejects with an instance of `ErrorClass`
|
||||||
|
* whose `.message` contains every string in `mustContain`. Fails loudly if
|
||||||
|
* the promise resolves instead of rejecting.
|
||||||
|
*/
|
||||||
|
async function expectRefusal(
|
||||||
|
promise: Promise<unknown>,
|
||||||
|
ErrorClass: new (...args: any[]) => Error,
|
||||||
|
...mustContain: string[]
|
||||||
|
): Promise<void> {
|
||||||
|
let threw = false
|
||||||
|
try {
|
||||||
|
await promise
|
||||||
|
} catch (err) {
|
||||||
|
threw = true
|
||||||
|
expect(err).toBeInstanceOf(ErrorClass)
|
||||||
|
for (const fragment of mustContain) {
|
||||||
|
expect((err as Error).message).toContain(fragment)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
expect(threw).toBe(true)
|
||||||
|
}
|
||||||
|
|
||||||
|
describe.skipIf(!lawActive)('namespace law — bare/system/metadata field addressing', () => {
|
||||||
|
let brain: Brainy
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'memory' as const },
|
||||||
|
embeddingFunction: stubEmbedding
|
||||||
|
})
|
||||||
|
await brain.init()
|
||||||
|
})
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
/** The star case from the motivating incident: metadata.level 3/9/6. */
|
||||||
|
async function addLevelRows(): Promise<string[]> {
|
||||||
|
const ids: string[] = []
|
||||||
|
for (const level of [3, 9, 6]) {
|
||||||
|
ids.push(
|
||||||
|
await brain.add({
|
||||||
|
data: `probe level ${level}`,
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-level',
|
||||||
|
metadata: { name: `p-${level}`, level }
|
||||||
|
})
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return ids
|
||||||
|
}
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
// Rule 1 — bare field name = the user's metadata field, always.
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
|
||||||
|
it("bare orderBy 'level' reads user metadata, desc and asc (the star case)", async () => {
|
||||||
|
await addLevelRows()
|
||||||
|
|
||||||
|
const desc = await brain.find({
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-level',
|
||||||
|
orderBy: 'level',
|
||||||
|
order: 'desc',
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
expect(desc.map((r: any) => r.metadata?.level)).toEqual([9, 6, 3])
|
||||||
|
|
||||||
|
const asc = await brain.find({
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-level',
|
||||||
|
orderBy: 'level',
|
||||||
|
order: 'asc',
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
expect(asc.map((r: any) => r.metadata?.level)).toEqual([3, 6, 9])
|
||||||
|
})
|
||||||
|
|
||||||
|
it("bare where { level: N } matches the user's field", async () => {
|
||||||
|
const ids = await addLevelRows()
|
||||||
|
const hit = await brain.find({ type: NounType.Person, subtype: 'ns-law-level', where: { level: 9 } })
|
||||||
|
expect(hit).toHaveLength(1)
|
||||||
|
expect(hit[0].id).toBe(ids[1])
|
||||||
|
expect(hit[0].metadata?.level).toBe(9)
|
||||||
|
})
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
// Rule 4 — metadata.<field> is the explicit-user-scope spelling,
|
||||||
|
// identical semantics to bare, valid on every path including orderBy.
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
|
||||||
|
it("'metadata.level' resolves identically to bare 'level'", async () => {
|
||||||
|
await addLevelRows()
|
||||||
|
const desc = await brain.find({
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-level',
|
||||||
|
orderBy: 'metadata.level',
|
||||||
|
order: 'desc',
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
expect(desc.map((r: any) => r.metadata?.level)).toEqual([9, 6, 3])
|
||||||
|
})
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
// Rule 2 — system.<field> reaches an engine scalar explicitly.
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
|
||||||
|
it('system.createdAt sorts by entity age', async () => {
|
||||||
|
const ids: string[] = []
|
||||||
|
for (const name of ['first', 'second', 'third']) {
|
||||||
|
ids.push(
|
||||||
|
await brain.add({
|
||||||
|
data: `aged ${name}`,
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-aged',
|
||||||
|
metadata: { name }
|
||||||
|
})
|
||||||
|
)
|
||||||
|
// Guarantee distinct createdAt timestamps between adds.
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 5))
|
||||||
|
}
|
||||||
|
|
||||||
|
const asc = await brain.find({
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-aged',
|
||||||
|
orderBy: 'system.createdAt',
|
||||||
|
order: 'asc',
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
expect(asc.map((r: any) => r.id)).toEqual(ids)
|
||||||
|
|
||||||
|
const desc = await brain.find({
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-aged',
|
||||||
|
orderBy: 'system.createdAt',
|
||||||
|
order: 'desc',
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
expect(desc.map((r: any) => r.id)).toEqual([...ids].reverse())
|
||||||
|
})
|
||||||
|
|
||||||
|
it('where on system.confidence filters by the engine scalar', async () => {
|
||||||
|
const highId = await brain.add({
|
||||||
|
data: 'high confidence row',
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-confidence',
|
||||||
|
confidence: 0.95,
|
||||||
|
metadata: { name: 'hi' }
|
||||||
|
})
|
||||||
|
await brain.add({
|
||||||
|
data: 'low confidence row',
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-confidence',
|
||||||
|
confidence: 0.4,
|
||||||
|
metadata: { name: 'lo' }
|
||||||
|
})
|
||||||
|
|
||||||
|
const hit = await brain.find({
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-confidence',
|
||||||
|
where: { 'system.confidence': 0.95 }
|
||||||
|
})
|
||||||
|
expect(hit).toHaveLength(1)
|
||||||
|
expect(hit[0].id).toBe(highId)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('groupBy on system.subtype groups by the engine scalar, not user metadata', async () => {
|
||||||
|
await brain.add({ data: 'i1', type: NounType.Document, subtype: 'invoice' })
|
||||||
|
await brain.add({ data: 'i2', type: NounType.Document, subtype: 'invoice' })
|
||||||
|
await brain.add({ data: 'r1', type: NounType.Document, subtype: 'receipt' })
|
||||||
|
|
||||||
|
brain.defineAggregate({
|
||||||
|
name: 'ns_law_by_subtype_system',
|
||||||
|
source: { type: NounType.Document },
|
||||||
|
groupBy: ['system.subtype'],
|
||||||
|
metrics: { count: { op: 'count' } }
|
||||||
|
})
|
||||||
|
|
||||||
|
const groups = await brain.queryAggregate('ns_law_by_subtype_system')
|
||||||
|
const invoiceGroup = groups.find((g) => Object.values(g.groupKey).includes('invoice'))
|
||||||
|
const receiptGroup = groups.find((g) => Object.values(g.groupKey).includes('receipt'))
|
||||||
|
expect(invoiceGroup?.metrics.count).toBe(2)
|
||||||
|
expect(receiptGroup?.metrics.count).toBe(1)
|
||||||
|
})
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
// Rule 1 (groupBy face) — bare groupBy dimensions read user metadata,
|
||||||
|
// never the engine's own notion of the same-sounding name.
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
|
||||||
|
it('groupBy on a bare user metadata field groups by that field', async () => {
|
||||||
|
await brain.add({
|
||||||
|
data: 'd1',
|
||||||
|
type: NounType.Document,
|
||||||
|
subtype: 'ns-law-group-bare',
|
||||||
|
metadata: { team: 'alpha' }
|
||||||
|
})
|
||||||
|
await brain.add({
|
||||||
|
data: 'd2',
|
||||||
|
type: NounType.Document,
|
||||||
|
subtype: 'ns-law-group-bare',
|
||||||
|
metadata: { team: 'alpha' }
|
||||||
|
})
|
||||||
|
await brain.add({
|
||||||
|
data: 'd3',
|
||||||
|
type: NounType.Document,
|
||||||
|
subtype: 'ns-law-group-bare',
|
||||||
|
metadata: { team: 'beta' }
|
||||||
|
})
|
||||||
|
|
||||||
|
brain.defineAggregate({
|
||||||
|
name: 'ns_law_by_team_bare',
|
||||||
|
// system.subtype — bare 'subtype' would address user metadata under the
|
||||||
|
// law (the exact migration every fleet consumer's aggregates make).
|
||||||
|
source: { type: NounType.Document, where: { 'system.subtype': 'ns-law-group-bare' } },
|
||||||
|
groupBy: ['team'],
|
||||||
|
metrics: { count: { op: 'count' } }
|
||||||
|
})
|
||||||
|
|
||||||
|
const groups = await brain.queryAggregate('ns_law_by_team_bare')
|
||||||
|
const alphaGroup = groups.find((g) => Object.values(g.groupKey).includes('alpha'))
|
||||||
|
const betaGroup = groups.find((g) => Object.values(g.groupKey).includes('beta'))
|
||||||
|
expect(alphaGroup?.metrics.count).toBe(2)
|
||||||
|
expect(betaGroup?.metrics.count).toBe(1)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('where on a bare user metadata field filters normally (score, not a system name)', async () => {
|
||||||
|
await brain.add({
|
||||||
|
data: 'high score',
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-score',
|
||||||
|
metadata: { score: 42 }
|
||||||
|
})
|
||||||
|
await brain.add({
|
||||||
|
data: 'low score',
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-score',
|
||||||
|
metadata: { score: 7 }
|
||||||
|
})
|
||||||
|
|
||||||
|
const hit = await brain.find({ type: NounType.Person, subtype: 'ns-law-score', where: { score: 42 } })
|
||||||
|
expect(hit).toHaveLength(1)
|
||||||
|
expect(hit[0].metadata?.score).toBe(42)
|
||||||
|
})
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
// Rule 5 — typed refusals, naming both candidates.
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
|
||||||
|
it("bare orderBy 'createdAt' refuses when no such metadata field exists — names both candidates", async () => {
|
||||||
|
await brain.add({
|
||||||
|
data: 'no metadata.createdAt here',
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-refuse-createdAt',
|
||||||
|
metadata: { name: 'x' }
|
||||||
|
})
|
||||||
|
|
||||||
|
await expectRefusal(
|
||||||
|
brain.find({
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-refuse-createdAt',
|
||||||
|
orderBy: 'createdAt',
|
||||||
|
limit: 10
|
||||||
|
}),
|
||||||
|
UnresolvableFieldError,
|
||||||
|
'system.createdAt',
|
||||||
|
'metadata.createdAt'
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
// Rule 3 — invisible plumbing refuses in either spelling; system.<name>
|
||||||
|
// for a name that isn't in the ten-scalar map is unresolvable.
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
|
||||||
|
it('system.level refuses — level is invisible plumbing, never a system scalar', async () => {
|
||||||
|
await brain.add({
|
||||||
|
data: 'has a level metadata field',
|
||||||
|
type: NounType.Person,
|
||||||
|
metadata: { level: 5 }
|
||||||
|
})
|
||||||
|
await expectRefusal(brain.find({ orderBy: 'system.level', limit: 10 }), UnresolvableFieldError)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('system.vector refuses — vector is invisible plumbing, never a system scalar', async () => {
|
||||||
|
await brain.add({ data: 'row', type: NounType.Person, metadata: { name: 'x' } })
|
||||||
|
await expectRefusal(brain.find({ orderBy: 'system.vector', limit: 10 }), UnresolvableFieldError)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('system.data refuses — data is a payload container, never a system scalar', async () => {
|
||||||
|
await brain.add({ data: 'row', type: NounType.Person, metadata: { name: 'x' } })
|
||||||
|
await expectRefusal(brain.find({ orderBy: 'system.data', limit: 10 }), UnresolvableFieldError)
|
||||||
|
})
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
// Rule 6 — the ordering contract.
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
|
||||||
|
async function addOrderingProbeRows(): Promise<{ ranked: string[]; missing: string }> {
|
||||||
|
const low = await brain.add({
|
||||||
|
data: 'low score',
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-ordering',
|
||||||
|
metadata: { score: 5 }
|
||||||
|
})
|
||||||
|
const high = await brain.add({
|
||||||
|
data: 'high score',
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-ordering',
|
||||||
|
metadata: { score: 9 }
|
||||||
|
})
|
||||||
|
const missing = await brain.add({
|
||||||
|
data: 'no score field at all',
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-ordering',
|
||||||
|
metadata: { name: 'no-score' }
|
||||||
|
})
|
||||||
|
return { ranked: [low, high], missing }
|
||||||
|
}
|
||||||
|
|
||||||
|
it('a row missing the orderBy field sorts LAST in desc — and is never dropped', async () => {
|
||||||
|
const { ranked, missing } = await addOrderingProbeRows()
|
||||||
|
const desc = await brain.find({
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-ordering',
|
||||||
|
orderBy: 'score',
|
||||||
|
order: 'desc',
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
expect(desc).toHaveLength(3)
|
||||||
|
expect(desc.map((r: any) => r.id)).toEqual([ranked[1], ranked[0], missing])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a row missing the orderBy field sorts LAST in asc too — and is never dropped', async () => {
|
||||||
|
const { ranked, missing } = await addOrderingProbeRows()
|
||||||
|
const asc = await brain.find({
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-ordering',
|
||||||
|
orderBy: 'score',
|
||||||
|
order: 'asc',
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
expect(asc).toHaveLength(3)
|
||||||
|
expect(asc.map((r: any) => r.id)).toEqual([ranked[0], ranked[1], missing])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('ties on the orderBy field break by id ascending, in BOTH directions', async () => {
|
||||||
|
const tiedIds: string[] = []
|
||||||
|
for (let i = 0; i < 4; i++) {
|
||||||
|
tiedIds.push(
|
||||||
|
await brain.add({
|
||||||
|
data: `tied ${i}`,
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-ties',
|
||||||
|
metadata: { score: 5 }
|
||||||
|
})
|
||||||
|
)
|
||||||
|
}
|
||||||
|
const expectedOrder = [...tiedIds].sort()
|
||||||
|
|
||||||
|
const asc = await brain.find({
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-ties',
|
||||||
|
orderBy: 'score',
|
||||||
|
order: 'asc',
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
expect(asc.map((r: any) => r.id)).toEqual(expectedOrder)
|
||||||
|
|
||||||
|
const desc = await brain.find({
|
||||||
|
type: NounType.Person,
|
||||||
|
subtype: 'ns-law-ties',
|
||||||
|
orderBy: 'score',
|
||||||
|
order: 'desc',
|
||||||
|
limit: 100
|
||||||
|
})
|
||||||
|
// Same tie-break ordering regardless of the primary direction — the
|
||||||
|
// contract states one universal rule ("id ascending"), not "reverse of
|
||||||
|
// the primary order".
|
||||||
|
expect(desc.map((r: any) => r.id)).toEqual(expectedOrder)
|
||||||
|
})
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
// Rule 5 (options face) — unimplemented find() options refuse loudly
|
||||||
|
// instead of being accepted and silently ignored.
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
|
||||||
|
it('find({ cursor }) refuses with UnsupportedFindOptionError', async () => {
|
||||||
|
await brain.add({ data: 'row', type: NounType.Person, metadata: { name: 'x' } })
|
||||||
|
await expectRefusal(brain.find({ cursor: 'anything', limit: 10 }), UnsupportedFindOptionError)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('find({ includeRelations }) refuses with UnsupportedFindOptionError', async () => {
|
||||||
|
await brain.add({ data: 'row', type: NounType.Person, metadata: { name: 'x' } })
|
||||||
|
await expectRefusal(brain.find({ includeRelations: true, limit: 10 }), UnsupportedFindOptionError)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('find({ writeOnly }) refuses with UnsupportedFindOptionError', async () => {
|
||||||
|
await brain.add({ data: 'row', type: NounType.Person, metadata: { name: 'x' } })
|
||||||
|
await expectRefusal(brain.find({ writeOnly: true, limit: 10 }), UnsupportedFindOptionError)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
@ -164,19 +164,19 @@ describe('BR-ADV-FEATURES-BUN regression', () => {
|
||||||
await b.close()
|
await b.close()
|
||||||
})
|
})
|
||||||
|
|
||||||
it('groupBy "noun" resolves to the entity type, not null', async () => {
|
it('groupBy "system.type" resolves to the entity type, not null (the legacy "noun" alias is dead)', async () => {
|
||||||
const b: any = new Brainy({ requireSubtype: false, storage: { type: 'memory' } })
|
const b: any = new Brainy({ requireSubtype: false, storage: { type: 'memory' } })
|
||||||
await b.init()
|
await b.init()
|
||||||
await b.add({ data: 'p', type: NounType.Person })
|
await b.add({ data: 'p', type: NounType.Person })
|
||||||
b.defineAggregate({
|
b.defineAggregate({
|
||||||
name: 'byNoun',
|
name: 'byNoun',
|
||||||
source: { type: NounType.Person },
|
source: { type: NounType.Person },
|
||||||
groupBy: ['noun'],
|
groupBy: ['system.type'],
|
||||||
metrics: { count: { op: 'count' } }
|
metrics: { count: { op: 'count' } }
|
||||||
})
|
})
|
||||||
const rows: any[] = await b.find({ aggregate: 'byNoun' })
|
const rows: any[] = await b.find({ aggregate: 'byNoun' })
|
||||||
expect(rows.length).toBe(1)
|
expect(rows.length).toBe(1)
|
||||||
expect(rows[0].groupKey.noun).toBe(NounType.Person)
|
expect(rows[0].groupKey['system.type']).toBe(NounType.Person)
|
||||||
await b.close()
|
await b.close()
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
|
||||||
|
|
@ -42,8 +42,11 @@ describe('aggregation + query field-resolution law', () => {
|
||||||
it('reserved-field groupBy decrements on delete (the drift bug)', async () => {
|
it('reserved-field groupBy decrements on delete (the drift bug)', async () => {
|
||||||
brain.defineAggregate({
|
brain.defineAggregate({
|
||||||
name: 'by_subtype',
|
name: 'by_subtype',
|
||||||
|
// system.subtype — subtype is an add() param (an engine scalar), never
|
||||||
|
// a user metadata field; bare 'subtype' now addresses the user's own
|
||||||
|
// metadata bag under the sealed field-addressing law.
|
||||||
source: { type: NounType.Document },
|
source: { type: NounType.Document },
|
||||||
groupBy: ['subtype'],
|
groupBy: ['system.subtype'],
|
||||||
metrics: { count: { op: 'count' } }
|
metrics: { count: { op: 'count' } }
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|
@ -60,7 +63,7 @@ describe('aggregation + query field-resolution law', () => {
|
||||||
}
|
}
|
||||||
let groups = await brain.queryAggregate('by_subtype')
|
let groups = await brain.queryAggregate('by_subtype')
|
||||||
expect(groups).toHaveLength(1)
|
expect(groups).toHaveLength(1)
|
||||||
expect(groups[0].groupKey).toEqual({ subtype: 'note' })
|
expect(groups[0].groupKey).toEqual({ 'system.subtype': 'note' })
|
||||||
expect(groups[0].metrics.count).toBe(5)
|
expect(groups[0].metrics.count).toBe(5)
|
||||||
|
|
||||||
await brain.remove(ids[0])
|
await brain.remove(ids[0])
|
||||||
|
|
@ -76,7 +79,7 @@ describe('aggregation + query field-resolution law', () => {
|
||||||
brain.defineAggregate({
|
brain.defineAggregate({
|
||||||
name: 'by_subtype',
|
name: 'by_subtype',
|
||||||
source: { type: NounType.Document },
|
source: { type: NounType.Document },
|
||||||
groupBy: ['subtype'],
|
groupBy: ['system.subtype'],
|
||||||
metrics: { count: { op: 'count' } }
|
metrics: { count: { op: 'count' } }
|
||||||
})
|
})
|
||||||
const id = await brain.add({
|
const id = await brain.add({
|
||||||
|
|
@ -88,7 +91,7 @@ describe('aggregation + query field-resolution law', () => {
|
||||||
|
|
||||||
const groups = await brain.queryAggregate('by_subtype')
|
const groups = await brain.queryAggregate('by_subtype')
|
||||||
const byKey = Object.fromEntries(
|
const byKey = Object.fromEntries(
|
||||||
groups.map((g) => [String(g.groupKey.subtype), g.metrics.count])
|
groups.map((g) => [String(g.groupKey['system.subtype']), g.metrics.count])
|
||||||
)
|
)
|
||||||
expect(byKey['published']).toBe(1)
|
expect(byKey['published']).toBe(1)
|
||||||
// The old group must be gone or zero — never still counting the entity.
|
// The old group must be gone or zero — never still counting the entity.
|
||||||
|
|
@ -98,7 +101,7 @@ describe('aggregation + query field-resolution law', () => {
|
||||||
it('source.where on a reserved field filters instead of matching nothing', async () => {
|
it('source.where on a reserved field filters instead of matching nothing', async () => {
|
||||||
brain.defineAggregate({
|
brain.defineAggregate({
|
||||||
name: 'notes_only',
|
name: 'notes_only',
|
||||||
source: { type: NounType.Document, where: { subtype: 'note' } },
|
source: { type: NounType.Document, where: { 'system.subtype': 'note' } },
|
||||||
groupBy: ['team'],
|
groupBy: ['team'],
|
||||||
metrics: { count: { op: 'count' } }
|
metrics: { count: { op: 'count' } }
|
||||||
})
|
})
|
||||||
|
|
|
||||||
|
|
@ -331,8 +331,10 @@ describe('Comprehensive All-APIs Test', () => {
|
||||||
it('should handle metadata queries efficiently', async () => {
|
it('should handle metadata queries efficiently', async () => {
|
||||||
const start = Date.now()
|
const start = Date.now()
|
||||||
|
|
||||||
|
// system.type — the legacy where.type→noun alias is dead; bare 'type'
|
||||||
|
// in where now addresses the user's own metadata field.
|
||||||
const results = await brain.find({
|
const results = await brain.find({
|
||||||
where: { type: NounType.Document },
|
where: { 'system.type': NounType.Document },
|
||||||
limit: 100
|
limit: 100
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@ import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||||
import * as fs from 'node:fs'
|
import * as fs from 'node:fs'
|
||||||
import * as os from 'node:os'
|
import * as os from 'node:os'
|
||||||
import * as path from 'node:path'
|
import * as path from 'node:path'
|
||||||
import { Brainy, ProtectedArtifactError, type CommitFact } from '../../src/index.js'
|
import { Brainy, ProtectedArtifactError, splitNounMetadataRecord, type CommitFact } from '../../src/index.js'
|
||||||
|
|
||||||
async function allFacts(brain: any): Promise<CommitFact[]> {
|
async function allFacts(brain: any): Promise<CommitFact[]> {
|
||||||
const scan = brain.scanFacts()
|
const scan = brain.scanFacts()
|
||||||
|
|
@ -65,7 +65,13 @@ describe('fact log dual-write (memory adapter)', () => {
|
||||||
const updateFact = facts[facts.length - 1]
|
const updateFact = facts[facts.length - 1]
|
||||||
const op = updateFact.ops.find((o) => o.id === id)!
|
const op = updateFact.ops.find((o) => o.id === id)!
|
||||||
expect(op.record).not.toBeNull()
|
expect(op.record).not.toBeNull()
|
||||||
expect((op.record!.metadata as any).v).toBe('new')
|
// The fact log is byte-faithful: op.record.metadata is the RAW stored
|
||||||
|
// record (v2 nested-bag since the field-addressing law) — read the user
|
||||||
|
// field through the shape-aware split, like every other reader.
|
||||||
|
const { custom } = splitNounMetadataRecord(
|
||||||
|
op.record!.metadata as Record<string, unknown>
|
||||||
|
)
|
||||||
|
expect(custom.v).toBe('new')
|
||||||
})
|
})
|
||||||
|
|
||||||
it('a transact commits ONE fact carrying all its ops, with meta', async () => {
|
it('a transact commits ONE fact carrying all its ops, with meta', async () => {
|
||||||
|
|
|
||||||
|
|
@ -2,8 +2,8 @@
|
||||||
* @module tests/integration/lens-consistency
|
* @module tests/integration/lens-consistency
|
||||||
* @description The three metadata "lenses" over one corpus must agree with
|
* @description The three metadata "lenses" over one corpus must agree with
|
||||||
* canonical ground truth id-for-id, warm AND after a cold reopen:
|
* canonical ground truth id-for-id, warm AND after a cold reopen:
|
||||||
* - combined: find({ type: T, where: { subtype: S } })
|
* - combined: find({ type: T, where: { 'system.subtype': S } })
|
||||||
* - subtype-only: find({ where: { subtype: S } })
|
* - subtype-only: find({ where: { 'system.subtype': S } })
|
||||||
* - type-only: find({ type: T })
|
* - type-only: find({ type: T })
|
||||||
* Ported from the fresh-brain probe that closed the type+subtype lens-drop
|
* Ported from the fresh-brain probe that closed the type+subtype lens-drop
|
||||||
* investigation (a restored pre-8.2.2 torn capture had entities visible to the
|
* investigation (a restored pre-8.2.2 torn capture had entities visible to the
|
||||||
|
|
@ -63,8 +63,11 @@ async function assertAllLenses(brain: any): Promise<void> {
|
||||||
const subtypes = [...new Set(CORPUS.map((c) => c.subtype))]
|
const subtypes = [...new Set(CORPUS.map((c) => c.subtype))]
|
||||||
|
|
||||||
for (const { type, subtype } of CORPUS) {
|
for (const { type, subtype } of CORPUS) {
|
||||||
const combined = idSet(await brain.find({ type, where: { subtype }, limit: 1000 }))
|
// system.subtype — subtype is an add()/update() param (an engine scalar),
|
||||||
const subtypeOnly = idSet(await brain.find({ where: { subtype }, limit: 1000 }))
|
// never a user metadata field; bare 'subtype' now addresses the user's
|
||||||
|
// own metadata bag under the sealed field-addressing law.
|
||||||
|
const combined = idSet(await brain.find({ type, where: { 'system.subtype': subtype }, limit: 1000 }))
|
||||||
|
const subtypeOnly = idSet(await brain.find({ where: { 'system.subtype': subtype }, limit: 1000 }))
|
||||||
const truthPair = await groundTruth(brain, { type, subtype })
|
const truthPair = await groundTruth(brain, { type, subtype })
|
||||||
const truthSubtype = await groundTruth(brain, { subtype })
|
const truthSubtype = await groundTruth(brain, { subtype })
|
||||||
|
|
||||||
|
|
@ -82,7 +85,7 @@ async function assertAllLenses(brain: any): Promise<void> {
|
||||||
// Count cross-check against the corpus definition itself.
|
// Count cross-check against the corpus definition itself.
|
||||||
for (const subtype of subtypes) {
|
for (const subtype of subtypes) {
|
||||||
const expected = CORPUS.filter((c) => c.subtype === subtype).reduce((s, c) => s + c.count, 0)
|
const expected = CORPUS.filter((c) => c.subtype === subtype).reduce((s, c) => s + c.count, 0)
|
||||||
const got = (await brain.find({ where: { subtype }, limit: 1000 })).length
|
const got = (await brain.find({ where: { 'system.subtype': subtype }, limit: 1000 })).length
|
||||||
expect(got).toBe(expected)
|
expect(got).toBe(expected)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -123,16 +126,16 @@ describe('lens consistency — combined vs subtype-only vs canonical ground trut
|
||||||
|
|
||||||
it('after an update() flips type AND subtype, every lens tracks the move exactly', async () => {
|
it('after an update() flips type AND subtype, every lens tracks the move exactly', async () => {
|
||||||
// The historical cross-bucket-staleness path: change (concept, action) -> (task, review).
|
// The historical cross-bucket-staleness path: change (concept, action) -> (task, review).
|
||||||
const victims = await brain.find({ type: 'concept', where: { subtype: 'action' }, limit: 1 })
|
const victims = await brain.find({ type: 'concept', where: { 'system.subtype': 'action' }, limit: 1 })
|
||||||
expect(victims.length).toBe(1)
|
expect(victims.length).toBe(1)
|
||||||
const id = victims[0].id
|
const id = victims[0].id
|
||||||
await brain.update({ id, type: 'task', subtype: 'review' })
|
await brain.update({ id, type: 'task', subtype: 'review' })
|
||||||
|
|
||||||
const oldCombined = idSet(await brain.find({ type: 'concept', where: { subtype: 'action' }, limit: 1000 }))
|
const oldCombined = idSet(await brain.find({ type: 'concept', where: { 'system.subtype': 'action' }, limit: 1000 }))
|
||||||
expect(oldCombined.has(id)).toBe(false) // unposted from the old buckets
|
expect(oldCombined.has(id)).toBe(false) // unposted from the old buckets
|
||||||
const newCombined = idSet(await brain.find({ type: 'task', where: { subtype: 'review' }, limit: 1000 }))
|
const newCombined = idSet(await brain.find({ type: 'task', where: { 'system.subtype': 'review' }, limit: 1000 }))
|
||||||
expect(newCombined.has(id)).toBe(true) // posted to the new buckets
|
expect(newCombined.has(id)).toBe(true) // posted to the new buckets
|
||||||
const subtypeOnly = idSet(await brain.find({ where: { subtype: 'review' }, limit: 1000 }))
|
const subtypeOnly = idSet(await brain.find({ where: { 'system.subtype': 'review' }, limit: 1000 }))
|
||||||
expect(subtypeOnly.has(id)).toBe(true)
|
expect(subtypeOnly.has(id)).toBe(true)
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
|
||||||
|
|
@ -141,7 +141,54 @@ describe('level field shadow — user metadata named level is a real field', ()
|
||||||
expect(Array.isArray(after?.vector) && after!.vector!.length).toBe(384)
|
expect(Array.isArray(after?.vector) && after!.vector!.length).toBe(384)
|
||||||
})
|
})
|
||||||
|
|
||||||
it('this build runs index epoch 2 (the paired level-indexability rebuild)', () => {
|
it('this build runs index epoch 3 (the namespace-law key split rebuild)', () => {
|
||||||
expect(EXPECTED_INDEX_EPOCH).toBe(2)
|
expect(EXPECTED_INDEX_EPOCH).toBe(3)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('noun-record writes never stamp over stored graph state', () => {
|
||||||
|
let brain: Brainy
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'memory' as const },
|
||||||
|
embeddingFunction: stubEmbedding
|
||||||
|
})
|
||||||
|
await brain.init()
|
||||||
|
})
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a data-changing update preserves LEGACY inline connections in the record', async () => {
|
||||||
|
// Codec-era records carry an EMPTY connections field by design (the
|
||||||
|
// adjacency lives in a separate compressed blob) — the clobber window
|
||||||
|
// exists only for legacy pre-codec records whose adjacency is inline.
|
||||||
|
// Simulate one: write the record with inline connections directly.
|
||||||
|
const id = await brain.add({
|
||||||
|
data: 'legacy-shaped node',
|
||||||
|
type: NounType.Concept,
|
||||||
|
metadata: { n: 1 }
|
||||||
|
})
|
||||||
|
const storage = (brain as any).storage
|
||||||
|
const rec = await storage.getNoun(id)
|
||||||
|
const legacy = {
|
||||||
|
...rec,
|
||||||
|
connections: new Map([[0, new Set(['00000000-0000-4000-8000-00000000aaaa'])]]),
|
||||||
|
level: 1
|
||||||
|
}
|
||||||
|
await storage.saveNoun(legacy)
|
||||||
|
const before = await storage.getNoun(id)
|
||||||
|
expect(before.connections.size).toBeGreaterThan(0)
|
||||||
|
|
||||||
|
// A data-changing update stages SaveNounOperation with placeholder
|
||||||
|
// adjacency — the legacy inline connections must survive the write.
|
||||||
|
await brain.update({ id, data: 'completely re-embedded text' })
|
||||||
|
|
||||||
|
const after = await storage.getNoun(id)
|
||||||
|
expect(after.connections.size).toBeGreaterThan(0)
|
||||||
|
expect(after.level).toBe(1)
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
|
||||||
|
|
@ -20,6 +20,16 @@ import { MigrationRunner, MIGRATIONS } from '../../src/migration/index.js'
|
||||||
import type { Migration } from '../../src/migration/index.js'
|
import type { Migration } from '../../src/migration/index.js'
|
||||||
import { NounType, VerbType } from '../../src/types/graphTypes.js'
|
import { NounType, VerbType } from '../../src/types/graphTypes.js'
|
||||||
|
|
||||||
|
// THE VIEW CONTRACT (field-addressing law): transforms receive engine fields
|
||||||
|
// top-level and the USER's bag nested under `metadata` — user-field changes
|
||||||
|
// go inside the bag. These two helpers keep the one-liner migrations tidy.
|
||||||
|
const bagOf = (m: Record<string, unknown>): Record<string, unknown> =>
|
||||||
|
m.metadata as Record<string, unknown>
|
||||||
|
const withBag = (
|
||||||
|
m: Record<string, unknown>,
|
||||||
|
patch: Record<string, unknown>
|
||||||
|
): Record<string, unknown> => ({ ...m, metadata: { ...bagOf(m), ...patch } })
|
||||||
|
|
||||||
// Helper to temporarily inject migrations into the MIGRATIONS array
|
// Helper to temporarily inject migrations into the MIGRATIONS array
|
||||||
function withMigrations(migrations: Migration[], fn: () => Promise<void>): Promise<void> {
|
function withMigrations(migrations: Migration[], fn: () => Promise<void>): Promise<void> {
|
||||||
const original = MIGRATIONS.splice(0, MIGRATIONS.length)
|
const original = MIGRATIONS.splice(0, MIGRATIONS.length)
|
||||||
|
|
@ -78,9 +88,11 @@ describe('Migration System', () => {
|
||||||
description: 'Add version field to entities with status',
|
description: 'Add version field to entities with status',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => {
|
transform: (m) => {
|
||||||
// Only transform entities that have our specific 'status' field
|
// Only transform entities that have our specific 'status' USER field
|
||||||
if ('status' in m && !('version' in m)) {
|
// (user fields live in the nested bag — the view contract).
|
||||||
return { ...m, version: 1 }
|
const bag = m.metadata as Record<string, unknown>
|
||||||
|
if ('status' in bag && !('version' in bag)) {
|
||||||
|
return { ...m, metadata: { ...bag, version: 1 } }
|
||||||
}
|
}
|
||||||
return null
|
return null
|
||||||
}
|
}
|
||||||
|
|
@ -94,7 +106,8 @@ describe('Migration System', () => {
|
||||||
// All 3 entities have 'status' metadata
|
// All 3 entities have 'status' metadata
|
||||||
expect(p.affectedEntities).toBeGreaterThanOrEqual(3)
|
expect(p.affectedEntities).toBeGreaterThanOrEqual(3)
|
||||||
expect(p.sampleChanges.length).toBeGreaterThan(0)
|
expect(p.sampleChanges.length).toBeGreaterThan(0)
|
||||||
expect(p.sampleChanges[0].after.version).toBe(1)
|
// Samples carry the VIEW shape: user fields inside `.metadata`.
|
||||||
|
expect(p.sampleChanges[0].after.metadata.version).toBe(1)
|
||||||
|
|
||||||
// Verify no data was modified (dry-run)
|
// Verify no data was modified (dry-run)
|
||||||
const entity = await brain.get(id1)
|
const entity = await brain.get(id1)
|
||||||
|
|
@ -111,9 +124,10 @@ describe('Migration System', () => {
|
||||||
description: 'Rename state to status',
|
description: 'Rename state to status',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => {
|
transform: (m) => {
|
||||||
if ('state' in m) {
|
const bag = m.metadata as Record<string, unknown>
|
||||||
const { state, ...rest } = m
|
if ('state' in bag) {
|
||||||
return { ...rest, status: state }
|
const { state, ...rest } = bag
|
||||||
|
return { ...m, metadata: { ...rest, status: state } }
|
||||||
}
|
}
|
||||||
return null
|
return null
|
||||||
}
|
}
|
||||||
|
|
@ -124,11 +138,12 @@ describe('Migration System', () => {
|
||||||
const p = preview as any
|
const p = preview as any
|
||||||
expect(p.sampleChanges.length).toBeGreaterThanOrEqual(1)
|
expect(p.sampleChanges.length).toBeGreaterThanOrEqual(1)
|
||||||
|
|
||||||
// Find the sample for our entity (it has the 'state' field)
|
// Find the sample for our entity (it has the 'state' USER field —
|
||||||
const sample = p.sampleChanges.find((s: any) => s.before.state === 'draft')
|
// samples carry the VIEW shape, user fields inside `.metadata`)
|
||||||
|
const sample = p.sampleChanges.find((s: any) => s.before.metadata.state === 'draft')
|
||||||
expect(sample).toBeDefined()
|
expect(sample).toBeDefined()
|
||||||
expect(sample.after.status).toBe('draft')
|
expect(sample.after.metadata.status).toBe('draft')
|
||||||
expect(sample.after.state).toBeUndefined()
|
expect(sample.after.metadata.state).toBeUndefined()
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
@ -149,8 +164,8 @@ describe('Migration System', () => {
|
||||||
description: 'Add migrated flag to entities with priority',
|
description: 'Add migrated flag to entities with priority',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => {
|
transform: (m) => {
|
||||||
if ('priority' in m && !('migrated' in m)) {
|
if ('priority' in bagOf(m) && !('migrated' in bagOf(m))) {
|
||||||
return { ...m, migrated: true }
|
return withBag(m, { migrated: true })
|
||||||
}
|
}
|
||||||
return null
|
return null
|
||||||
}
|
}
|
||||||
|
|
@ -179,8 +194,8 @@ describe('Migration System', () => {
|
||||||
description: 'Uppercase status field only when present',
|
description: 'Uppercase status field only when present',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => {
|
transform: (m) => {
|
||||||
if (typeof m.status === 'string') {
|
if (typeof bagOf(m).status === 'string') {
|
||||||
return { ...m, status: (m.status as string).toUpperCase() }
|
return withBag(m, { status: (bagOf(m).status as string).toUpperCase() })
|
||||||
}
|
}
|
||||||
return null
|
return null
|
||||||
}
|
}
|
||||||
|
|
@ -203,7 +218,7 @@ describe('Migration System', () => {
|
||||||
version: '1.0.0',
|
version: '1.0.0',
|
||||||
description: 'Double count',
|
description: 'Double count',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => typeof m.count === 'number' ? { ...m, count: (m.count as number) * 2 } : null
|
transform: (m) => typeof bagOf(m).count === 'number' ? withBag(m, { count: (bagOf(m).count as number) * 2 }) : null
|
||||||
}
|
}
|
||||||
|
|
||||||
const migration2: Migration = {
|
const migration2: Migration = {
|
||||||
|
|
@ -211,7 +226,7 @@ describe('Migration System', () => {
|
||||||
version: '1.1.0',
|
version: '1.1.0',
|
||||||
description: 'Add 10 to count',
|
description: 'Add 10 to count',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => typeof m.count === 'number' ? { ...m, count: (m.count as number) + 10 } : null
|
transform: (m) => typeof bagOf(m).count === 'number' ? withBag(m, { count: (bagOf(m).count as number) + 10 }) : null
|
||||||
}
|
}
|
||||||
|
|
||||||
await withMigrations([migration1, migration2], async () => {
|
await withMigrations([migration1, migration2], async () => {
|
||||||
|
|
@ -229,7 +244,7 @@ describe('Migration System', () => {
|
||||||
version: '1.0.0',
|
version: '1.0.0',
|
||||||
description: 'Increment v',
|
description: 'Increment v',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => typeof m.v === 'number' ? { ...m, v: (m.v as number) + 1 } : null
|
transform: (m) => typeof bagOf(m).v === 'number' ? withBag(m, { v: (bagOf(m).v as number) + 1 }) : null
|
||||||
}
|
}
|
||||||
|
|
||||||
await withMigrations([migration], async () => {
|
await withMigrations([migration], async () => {
|
||||||
|
|
@ -266,7 +281,7 @@ describe('Migration System', () => {
|
||||||
version: '2.0.0',
|
version: '2.0.0',
|
||||||
description: 'Add y field to entities with x',
|
description: 'Add y field to entities with x',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => 'x' in m && !('y' in m) ? { ...m, y: 2 } : null
|
transform: (m) => 'x' in bagOf(m) && !('y' in bagOf(m)) ? withBag(m, { y: 2 }) : null
|
||||||
}
|
}
|
||||||
|
|
||||||
await withMigrations([migration], async () => {
|
await withMigrations([migration], async () => {
|
||||||
|
|
@ -290,8 +305,8 @@ describe('Migration System', () => {
|
||||||
description: 'Replace original with migrated',
|
description: 'Replace original with migrated',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => {
|
transform: (m) => {
|
||||||
if (m.original === true) {
|
if (bagOf(m).original === true) {
|
||||||
return { ...m, original: false, migrated: true }
|
return withBag(m, { original: false, migrated: true })
|
||||||
}
|
}
|
||||||
return null
|
return null
|
||||||
}
|
}
|
||||||
|
|
@ -323,7 +338,7 @@ describe('Migration System', () => {
|
||||||
version: '4.0.0',
|
version: '4.0.0',
|
||||||
description: 'Add field',
|
description: 'Add field',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => 'q' in m && !('r' in m) ? { ...m, r: 2 } : null
|
transform: (m) => 'q' in bagOf(m) && !('r' in bagOf(m)) ? withBag(m, { r: 2 }) : null
|
||||||
}
|
}
|
||||||
|
|
||||||
await withMigrations([migration], async () => {
|
await withMigrations([migration], async () => {
|
||||||
|
|
@ -384,7 +399,7 @@ describe('Migration System', () => {
|
||||||
version: '1.0.0',
|
version: '1.0.0',
|
||||||
description: 'Auto migrate test',
|
description: 'Auto migrate test',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => 'legacy' in m ? { ...m, legacy: false, upgraded: true } : null
|
transform: (m) => 'legacy' in bagOf(m) ? withBag(m, { legacy: false, upgraded: true }) : null
|
||||||
}
|
}
|
||||||
|
|
||||||
await withMigrations([migration], async () => {
|
await withMigrations([migration], async () => {
|
||||||
|
|
@ -410,7 +425,7 @@ describe('Migration System', () => {
|
||||||
version: '1.0.0',
|
version: '1.0.0',
|
||||||
description: 'Add y to entities with x',
|
description: 'Add y to entities with x',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => 'x' in m ? { ...m, y: true } : null
|
transform: (m) => 'x' in bagOf(m) ? withBag(m, { y: true }) : null
|
||||||
}
|
}
|
||||||
|
|
||||||
const progressCalls: any[] = []
|
const progressCalls: any[] = []
|
||||||
|
|
@ -444,7 +459,7 @@ describe('Migration System', () => {
|
||||||
version: '1.0.0',
|
version: '1.0.0',
|
||||||
description: 'Increment v on entities that have it',
|
description: 'Increment v on entities that have it',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => typeof m.v === 'number' ? { ...m, v: (m.v as number) + 1 } : null
|
transform: (m) => typeof bagOf(m).v === 'number' ? withBag(m, { v: (bagOf(m).v as number) + 1 }) : null
|
||||||
}
|
}
|
||||||
|
|
||||||
await withMigrations([migration], async () => {
|
await withMigrations([migration], async () => {
|
||||||
|
|
@ -477,9 +492,10 @@ describe('Migration System', () => {
|
||||||
description: 'Rename strength to intensity',
|
description: 'Rename strength to intensity',
|
||||||
applies: 'verbs',
|
applies: 'verbs',
|
||||||
transform: (m) => {
|
transform: (m) => {
|
||||||
if ('strength' in m) {
|
const bag = bagOf(m)
|
||||||
const { strength, ...rest } = m
|
if ('strength' in bag) {
|
||||||
return { ...rest, intensity: strength }
|
const { strength, ...rest } = bag
|
||||||
|
return { ...m, metadata: { ...rest, intensity: strength } }
|
||||||
}
|
}
|
||||||
return null
|
return null
|
||||||
}
|
}
|
||||||
|
|
@ -507,7 +523,7 @@ describe('Migration System', () => {
|
||||||
version: '1.0.0',
|
version: '1.0.0',
|
||||||
description: 'Update tag from old to new',
|
description: 'Update tag from old to new',
|
||||||
applies: 'both',
|
applies: 'both',
|
||||||
transform: (m) => m.tag === 'old' ? { ...m, tag: 'new' } : null
|
transform: (m) => bagOf(m).tag === 'old' ? withBag(m, { tag: 'new' }) : null
|
||||||
}
|
}
|
||||||
|
|
||||||
await withMigrations([migration], async () => {
|
await withMigrations([migration], async () => {
|
||||||
|
|
@ -577,11 +593,11 @@ describe('Migration System', () => {
|
||||||
description: 'Transform that throws on non-number values',
|
description: 'Transform that throws on non-number values',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => {
|
transform: (m) => {
|
||||||
if ('value' in m) {
|
if ('value' in bagOf(m)) {
|
||||||
if (typeof m.value !== 'number') {
|
if (typeof bagOf(m).value !== 'number') {
|
||||||
throw new Error('value must be a number')
|
throw new Error('value must be a number')
|
||||||
}
|
}
|
||||||
return { ...m, value: (m.value as number) * 10 }
|
return withBag(m, { value: (bagOf(m).value as number) * 10 })
|
||||||
}
|
}
|
||||||
return null
|
return null
|
||||||
}
|
}
|
||||||
|
|
@ -615,7 +631,7 @@ describe('Migration System', () => {
|
||||||
description: 'Always throws',
|
description: 'Always throws',
|
||||||
applies: 'nouns',
|
applies: 'nouns',
|
||||||
transform: (m) => {
|
transform: (m) => {
|
||||||
if ('boom' in m) {
|
if ('boom' in bagOf(m)) {
|
||||||
throw new Error('deliberate failure')
|
throw new Error('deliberate failure')
|
||||||
}
|
}
|
||||||
return null
|
return null
|
||||||
|
|
|
||||||
|
|
@ -56,7 +56,7 @@ describe('find({ orderBy }) sort bug regression', () => {
|
||||||
|
|
||||||
const results = await brain.find({
|
const results = await brain.find({
|
||||||
type: NounType.Concept,
|
type: NounType.Concept,
|
||||||
orderBy: 'createdAt',
|
orderBy: 'system.createdAt',
|
||||||
order: 'desc',
|
order: 'desc',
|
||||||
limit: 1
|
limit: 1
|
||||||
})
|
})
|
||||||
|
|
@ -76,7 +76,7 @@ describe('find({ orderBy }) sort bug regression', () => {
|
||||||
|
|
||||||
const results = await brain.find({
|
const results = await brain.find({
|
||||||
type: NounType.Concept,
|
type: NounType.Concept,
|
||||||
orderBy: 'createdAt',
|
orderBy: 'system.createdAt',
|
||||||
order: 'asc',
|
order: 'asc',
|
||||||
limit: 1
|
limit: 1
|
||||||
})
|
})
|
||||||
|
|
@ -94,7 +94,7 @@ describe('find({ orderBy }) sort bug regression', () => {
|
||||||
|
|
||||||
const results = await brain.find({
|
const results = await brain.find({
|
||||||
type: NounType.Concept,
|
type: NounType.Concept,
|
||||||
orderBy: 'createdAt',
|
orderBy: 'system.createdAt',
|
||||||
order: 'desc'
|
order: 'desc'
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|
@ -115,7 +115,7 @@ describe('find({ orderBy }) sort bug regression', () => {
|
||||||
|
|
||||||
const results = await brain.find({
|
const results = await brain.find({
|
||||||
type: NounType.Concept,
|
type: NounType.Concept,
|
||||||
orderBy: 'updatedAt',
|
orderBy: 'system.updatedAt',
|
||||||
order: 'desc',
|
order: 'desc',
|
||||||
limit: 1
|
limit: 1
|
||||||
})
|
})
|
||||||
|
|
@ -136,7 +136,7 @@ describe('find({ orderBy }) sort bug regression', () => {
|
||||||
const id3 = await brain.add({ data: 'third', type: NounType.Concept })
|
const id3 = await brain.add({ data: 'third', type: NounType.Concept })
|
||||||
|
|
||||||
const results = await brain.find({
|
const results = await brain.find({
|
||||||
orderBy: 'createdAt',
|
orderBy: 'system.createdAt',
|
||||||
order: 'desc',
|
order: 'desc',
|
||||||
limit: 2
|
limit: 2
|
||||||
})
|
})
|
||||||
|
|
|
||||||
|
|
@ -244,7 +244,10 @@ describe('Metadata index cleanup after remove / removeMany', () => {
|
||||||
const noConfidenceId = await addEntity({ type: 'thing' })
|
const noConfidenceId = await addEntity({ type: 'thing' })
|
||||||
const withConfidenceId = await addEntity({ type: 'thing', confidence: 0.9 })
|
const withConfidenceId = await addEntity({ type: 'thing', confidence: 0.9 })
|
||||||
|
|
||||||
const results = await brain.find({ where: { confidence: { exists: true } } })
|
// system.confidence — confidence is an engine scalar (an add() param),
|
||||||
|
// never a metadata field; bare 'confidence' now addresses the user's
|
||||||
|
// own metadata bag under the sealed field-addressing law.
|
||||||
|
const results = await brain.find({ where: { 'system.confidence': { exists: true } } })
|
||||||
const ids = results.map(r => r.id)
|
const ids = results.map(r => r.id)
|
||||||
|
|
||||||
expect(ids).toContain(withConfidenceId)
|
expect(ids).toContain(withConfidenceId)
|
||||||
|
|
@ -255,7 +258,8 @@ describe('Metadata index cleanup after remove / removeMany', () => {
|
||||||
const noWeightId = await addEntity({ type: 'thing' })
|
const noWeightId = await addEntity({ type: 'thing' })
|
||||||
const withWeightId = await addEntity({ type: 'thing', weight: 0.5 })
|
const withWeightId = await addEntity({ type: 'thing', weight: 0.5 })
|
||||||
|
|
||||||
const results = await brain.find({ where: { weight: { exists: true } } })
|
// system.weight — same reasoning as system.confidence above.
|
||||||
|
const results = await brain.find({ where: { 'system.weight': { exists: true } } })
|
||||||
const ids = results.map(r => r.id)
|
const ids = results.map(r => r.id)
|
||||||
|
|
||||||
expect(ids).toContain(withWeightId)
|
expect(ids).toContain(withWeightId)
|
||||||
|
|
@ -269,11 +273,12 @@ describe('Metadata index cleanup after remove / removeMany', () => {
|
||||||
const id = await addEntity({ type: 'thing' })
|
const id = await addEntity({ type: 'thing' })
|
||||||
await brain.remove(id)
|
await brain.remove(id)
|
||||||
|
|
||||||
// Entity must not appear in any confidence query
|
// Entity must not appear in any confidence query. system.confidence —
|
||||||
const existsTrue = await brain.find({ where: { confidence: { exists: true } } })
|
// same addressing as the two tests above.
|
||||||
|
const existsTrue = await brain.find({ where: { 'system.confidence': { exists: true } } })
|
||||||
expect(existsTrue.map(r => r.id)).not.toContain(id)
|
expect(existsTrue.map(r => r.id)).not.toContain(id)
|
||||||
|
|
||||||
const existsFalse = await brain.find({ where: { confidence: { exists: false } } })
|
const existsFalse = await brain.find({ where: { 'system.confidence': { exists: false } } })
|
||||||
expect(existsFalse.map(r => r.id)).not.toContain(id)
|
expect(existsFalse.map(r => r.id)).not.toContain(id)
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
|
||||||
|
|
@ -42,7 +42,8 @@ describe('find({ where, orderBy }) bounds the sort to the page (CTX-BR-FIND-ORDE
|
||||||
return real(f, ob, o, topK)
|
return real(f, ob, o, topK)
|
||||||
}
|
}
|
||||||
|
|
||||||
const results = await brain.find({ where: { bucket: 'x' }, orderBy: 'createdAt', order: 'desc', limit: 5 })
|
// system.createdAt — entity age, not a user metadata field named 'createdAt'.
|
||||||
|
const results = await brain.find({ where: { bucket: 'x' }, orderBy: 'system.createdAt', order: 'desc', limit: 5 })
|
||||||
|
|
||||||
expect(results).toHaveLength(5)
|
expect(results).toHaveLength(5)
|
||||||
// Page-bounded: ~ limit (5) + a small hidden-tier over-fetch — NOT all 50 matches.
|
// Page-bounded: ~ limit (5) + a small hidden-tier over-fetch — NOT all 50 matches.
|
||||||
|
|
|
||||||
|
|
@ -245,9 +245,10 @@ describe('rc.8 no-freeze migration deference (isMigrating / stampBrainFormat / b
|
||||||
it('the brain-format marker module exports the compiled epoch + data-format constants', () => {
|
it('the brain-format marker module exports the compiled epoch + data-format constants', () => {
|
||||||
// cor imports these from '@soulcraft/brainy/brain-format' (Hook 3) so both
|
// cor imports these from '@soulcraft/brainy/brain-format' (Hook 3) so both
|
||||||
// sides share ONE source of truth — no duplicated constant to drift.
|
// sides share ONE source of truth — no duplicated constant to drift.
|
||||||
// Epoch 2: user metadata named `level` became indexable (the reserved-name
|
// Epoch 3: the namespace-law key split (bare user keys · literal
|
||||||
// shadow fix, 2026-08-03) — pre-fix brains rebuild derived indexes at open.
|
// 'system.<field>' scalars, 2026-08-03) — every brain rebuilds onto the
|
||||||
expect(EXPECTED_INDEX_EPOCH).toBe(2)
|
// frozen keys at first open. (Epoch 2 same day: `level` indexability.)
|
||||||
|
expect(EXPECTED_INDEX_EPOCH).toBe(3)
|
||||||
expect(CURRENT_DATA_FORMAT).toBe('8.0')
|
expect(CURRENT_DATA_FORMAT).toBe('8.0')
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
|
||||||
|
|
@ -1,251 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/unit/brainy/reserved-field-policy
|
|
||||||
* @description The 8.0 `reservedFieldPolicy` matrix — what happens when an
|
|
||||||
* untyped (JavaScript) caller smuggles a Brainy-reserved field INSIDE the
|
|
||||||
* `metadata` bag of a write call, past the compile-time guard.
|
|
||||||
*
|
|
||||||
* 8.0 is a clean break with no silent failures. The decided contract:
|
|
||||||
* - `'throw'` (DEFAULT): a reserved key in the bag throws a clear Error naming
|
|
||||||
* the offending key(s) and the correct write path. No remap, no data loss.
|
|
||||||
* - `'warn'`: legacy remap PLUS a one-shot (per method+field, per process)
|
|
||||||
* warning for EVERY reserved key found.
|
|
||||||
* - `'remap'`: the pre-8.0 silent remap, no warning.
|
|
||||||
*
|
|
||||||
* The deep correctness of the remap itself (top-level precedence, system-managed
|
|
||||||
* drops, transact()/with() mirrors, read-side splitting) lives in
|
|
||||||
* tests/unit/brainy/update-reserved-metadata-remap.test.ts (which now runs under
|
|
||||||
* `reservedFieldPolicy: 'remap'`). This file pins the POLICY SELECTION and the
|
|
||||||
* throw/warn behaviors.
|
|
||||||
*
|
|
||||||
* Compile-time callers can't write these shapes at all (see
|
|
||||||
* tests/unit/types/reserved-metadata-keys.test-d.ts); the `as object` widenings
|
|
||||||
* below simulate untyped callers.
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
|
|
||||||
import { Brainy } from '../../../src/index.js'
|
|
||||||
import { NounType, VerbType } from '../../../src/types/graphTypes.js'
|
|
||||||
import { createTestConfig } from '../../helpers/test-factory.js'
|
|
||||||
import { prodLog } from '../../../src/utils/logger.js'
|
|
||||||
|
|
||||||
describe('reservedFieldPolicy', () => {
|
|
||||||
describe("default policy is 'throw'", () => {
|
|
||||||
let brain: Brainy
|
|
||||||
|
|
||||||
beforeEach(async () => {
|
|
||||||
// No reservedFieldPolicy override → resolves to 'throw'.
|
|
||||||
brain = new Brainy(createTestConfig())
|
|
||||||
await brain.init()
|
|
||||||
})
|
|
||||||
|
|
||||||
afterEach(async () => {
|
|
||||||
await brain.close()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('add() throws naming the offending key and the correct write path', async () => {
|
|
||||||
await expect(
|
|
||||||
brain.add({
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'x',
|
|
||||||
metadata: { confidence: 0.8 } as object
|
|
||||||
})
|
|
||||||
).rejects.toThrow(/metadata\.confidence is a reserved field/)
|
|
||||||
|
|
||||||
// The error names the right param and the reserved list for discoverability.
|
|
||||||
await expect(
|
|
||||||
brain.add({
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'x',
|
|
||||||
metadata: { confidence: 0.8 } as object
|
|
||||||
})
|
|
||||||
).rejects.toThrow(/'confidence' param.*RESERVED_ENTITY_FIELDS/s)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('add() lists EVERY offending key when several are present', async () => {
|
|
||||||
const err = await brain
|
|
||||||
.add({
|
|
||||||
type: NounType.Person,
|
|
||||||
data: 'multi',
|
|
||||||
metadata: { confidence: 0.5, weight: 0.6, subtype: 'employee' } as object
|
|
||||||
})
|
|
||||||
.catch((e) => e as Error)
|
|
||||||
expect(err).toBeInstanceOf(Error)
|
|
||||||
expect(err.message).toMatch(/confidence/)
|
|
||||||
expect(err.message).toMatch(/weight/)
|
|
||||||
expect(err.message).toMatch(/subtype/)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('update() throws on a reserved key in the patch', async () => {
|
|
||||||
const id = await brain.add({ type: NounType.Concept, subtype: 'general', data: 'y' })
|
|
||||||
await expect(
|
|
||||||
brain.update({ id, metadata: { confidence: 0.3 } as object })
|
|
||||||
).rejects.toThrow(/metadata\.confidence is a reserved field/)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('relate() throws on a reserved key in the bag', async () => {
|
|
||||||
const a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'A' })
|
|
||||||
const b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'B' })
|
|
||||||
await expect(
|
|
||||||
brain.relate({
|
|
||||||
from: a,
|
|
||||||
to: b,
|
|
||||||
type: VerbType.RelatedTo,
|
|
||||||
subtype: 'colleague',
|
|
||||||
metadata: { confidence: 0.4 } as object
|
|
||||||
})
|
|
||||||
).rejects.toThrow(/metadata\.confidence is a reserved field.*RESERVED_RELATION_FIELDS/s)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('updateRelation() throws on a reserved key in the patch', async () => {
|
|
||||||
const a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'A' })
|
|
||||||
const b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'B' })
|
|
||||||
const relId = await brain.relate({
|
|
||||||
from: a,
|
|
||||||
to: b,
|
|
||||||
type: VerbType.ReportsTo,
|
|
||||||
subtype: 'direct'
|
|
||||||
})
|
|
||||||
await expect(
|
|
||||||
brain.updateRelation({ id: relId, metadata: { weight: 0.2 } as object })
|
|
||||||
).rejects.toThrow(/metadata\.weight is a reserved field/)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('transact() add op throws on a reserved key in the bag', async () => {
|
|
||||||
await expect(
|
|
||||||
brain.transact([
|
|
||||||
{
|
|
||||||
op: 'add',
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'tx',
|
|
||||||
metadata: { confidence: 0.7 } as object
|
|
||||||
}
|
|
||||||
])
|
|
||||||
).rejects.toThrow(/metadata\.confidence is a reserved field/)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('a custom (non-reserved) key in the bag does NOT throw', async () => {
|
|
||||||
const id = await brain.add({
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'ok',
|
|
||||||
metadata: { status: 'draft', rating: 4 }
|
|
||||||
})
|
|
||||||
const entity = await brain.get(id)
|
|
||||||
expect(entity?.metadata).toEqual({ status: 'draft', rating: 4 })
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe("'remap' policy remaps silently (no warning)", () => {
|
|
||||||
let brain: Brainy
|
|
||||||
let warnSpy: ReturnType<typeof vi.spyOn>
|
|
||||||
|
|
||||||
beforeEach(async () => {
|
|
||||||
warnSpy = vi.spyOn(prodLog, 'warn').mockImplementation(() => {})
|
|
||||||
brain = new Brainy(createTestConfig({ reservedFieldPolicy: 'remap' }))
|
|
||||||
await brain.init()
|
|
||||||
})
|
|
||||||
|
|
||||||
afterEach(async () => {
|
|
||||||
await brain.close()
|
|
||||||
warnSpy.mockRestore()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('lifts user-mutable reserved fields to top-level without warning', async () => {
|
|
||||||
const id = await brain.add({
|
|
||||||
type: NounType.Person,
|
|
||||||
data: 'remap lift',
|
|
||||||
metadata: { confidence: 0.8, weight: 0.6, subtype: 'employee', dept: 'eng' } as object
|
|
||||||
})
|
|
||||||
const entity = await brain.get(id)
|
|
||||||
expect(entity?.confidence).toBe(0.8)
|
|
||||||
expect(entity?.weight).toBe(0.6)
|
|
||||||
expect(entity?.subtype).toBe('employee')
|
|
||||||
expect(entity?.metadata).toEqual({ dept: 'eng' })
|
|
||||||
// 'remap' is silent about reserved fields (unrelated storage logs may fire,
|
|
||||||
// so assert specifically that no reserved-field warning was emitted).
|
|
||||||
const reservedWarned = warnSpy.mock.calls.some((c) =>
|
|
||||||
String(c[0]).includes('reserved field')
|
|
||||||
)
|
|
||||||
expect(reservedWarned).toBe(false)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('preserves _originalId on natural-key ids through the remap path', async () => {
|
|
||||||
// A speculative view applies the same normalization and maps a natural-key
|
|
||||||
// id to a stable UUID, preserving the caller's original string.
|
|
||||||
const base = await brain.now()
|
|
||||||
const speculative = await base.with([
|
|
||||||
{
|
|
||||||
op: 'add',
|
|
||||||
id: 'remap-spec-entity',
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'spec',
|
|
||||||
metadata: { confidence: 0.65, custom: 'spec' } as object
|
|
||||||
}
|
|
||||||
])
|
|
||||||
const entity = await speculative.get('remap-spec-entity')
|
|
||||||
expect(entity?.confidence).toBe(0.65)
|
|
||||||
expect(entity?.metadata).toEqual({ custom: 'spec', _originalId: 'remap-spec-entity' })
|
|
||||||
await speculative.release()
|
|
||||||
await base.release()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe("'warn' policy remaps AND warns once per key", () => {
|
|
||||||
let brain: Brainy
|
|
||||||
let warnSpy: ReturnType<typeof vi.spyOn>
|
|
||||||
|
|
||||||
beforeEach(async () => {
|
|
||||||
warnSpy = vi.spyOn(prodLog, 'warn').mockImplementation(() => {})
|
|
||||||
brain = new Brainy(createTestConfig({ reservedFieldPolicy: 'warn' }))
|
|
||||||
await brain.init()
|
|
||||||
})
|
|
||||||
|
|
||||||
afterEach(async () => {
|
|
||||||
await brain.close()
|
|
||||||
warnSpy.mockRestore()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('remaps the value (same as remap) and emits a warning naming the field', async () => {
|
|
||||||
// Use a method+field combo unique to this test so the per-process one-shot
|
|
||||||
// registry has not already consumed it.
|
|
||||||
const id = await brain.add({
|
|
||||||
type: NounType.Person,
|
|
||||||
data: 'warn lift',
|
|
||||||
// weight is user-mutable → remapped; this is the only 'warn'-policy
|
|
||||||
// add({ weight }) in the suite, so the one-shot warning fires here.
|
|
||||||
metadata: { weight: 0.42, dept: 'eng' } as object
|
|
||||||
})
|
|
||||||
const entity = await brain.get(id)
|
|
||||||
// Value is honored (remap still happens under 'warn').
|
|
||||||
expect(entity?.weight).toBe(0.42)
|
|
||||||
expect(entity?.metadata).toEqual({ dept: 'eng' })
|
|
||||||
// And a warning was emitted naming the reserved field.
|
|
||||||
expect(warnSpy).toHaveBeenCalled()
|
|
||||||
const warned = warnSpy.mock.calls.some((c) =>
|
|
||||||
String(c[0]).includes("'weight'")
|
|
||||||
)
|
|
||||||
expect(warned).toBe(true)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('warns for system-managed keys too (closes the historical gap)', async () => {
|
|
||||||
// Pre-8.0 only system-managed fields warned; 'warn' warns for every key.
|
|
||||||
// 'createdBy' (system-managed on update) is unique to this test.
|
|
||||||
const id = await brain.add({ type: NounType.Concept, subtype: 'general', data: 'sys' })
|
|
||||||
warnSpy.mockClear()
|
|
||||||
await brain.update({ id, metadata: { createdBy: 'nope', keep: 'me' } as object })
|
|
||||||
const entity = await brain.get(id)
|
|
||||||
// System-managed key dropped; custom field merged.
|
|
||||||
expect((entity?.metadata as Record<string, unknown>)?.createdBy).toBeUndefined()
|
|
||||||
expect((entity?.metadata as Record<string, unknown>)?.keep).toBe('me')
|
|
||||||
// A warning was emitted for the dropped system-managed key.
|
|
||||||
const warned = warnSpy.mock.calls.some((c) =>
|
|
||||||
String(c[0]).includes("'createdBy'")
|
|
||||||
)
|
|
||||||
expect(warned).toBe(true)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
@ -1,403 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/unit/brainy/update-reserved-metadata-remap
|
|
||||||
* @description Regression tests for the reserved-field metadata-bag trap,
|
|
||||||
* ported from the 7.x fix and extended to the full 8.0 contract.
|
|
||||||
*
|
|
||||||
* History: `add({metadata: {confidence}})` lifted reserved fields to their
|
|
||||||
* canonical top-level location, but `update({metadata: {confidence}})`
|
|
||||||
* silently dropped the same shape — the patch value survived the merge and
|
|
||||||
* was then clobbered by the preserve-existing spread. A production
|
|
||||||
* consumer's confidence-evolution writes no-oped for weeks before being
|
|
||||||
* caught by reading values back.
|
|
||||||
*
|
|
||||||
* These tests pin the LEGACY REMAP behavior, which in 8.0 is opt-in via
|
|
||||||
* `reservedFieldPolicy: 'remap'` (the default is `'throw'` — see the policy
|
|
||||||
* matrix in tests/unit/brainy/reserved-field-policy.test.ts). The brain in
|
|
||||||
* every test below is constructed with `reservedFieldPolicy: 'remap'` so these
|
|
||||||
* deep correctness assertions about the remap path stay exercised.
|
|
||||||
*
|
|
||||||
* Remap contract under test (every write path, entities AND relationships):
|
|
||||||
* - user-mutable reserved fields (`confidence`, `weight`, `subtype` — plus
|
|
||||||
* `service`/`createdBy` at add()/relate() time) remap from the metadata
|
|
||||||
* bag to their dedicated top-level param, with top-level winning when both
|
|
||||||
* are present;
|
|
||||||
* - system-managed reserved fields (`createdAt`, `_rev`, `noun`/`verb`,
|
|
||||||
* `data`, …) are dropped from the bag;
|
|
||||||
* - the same normalization applies to `transact()` operations and `with()`
|
|
||||||
* speculative views;
|
|
||||||
* - reads NEVER echo a reserved field inside `metadata`.
|
|
||||||
*
|
|
||||||
* TypeScript callers can't write these shapes at all (compile-time guard on
|
|
||||||
* the metadata param types — see tests/unit/types/reserved-metadata-keys.test-d.ts);
|
|
||||||
* these tests simulate untyped (JavaScript) callers, hence the `as object`
|
|
||||||
* widenings on the metadata literals.
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
|
||||||
import { Brainy } from '../../../src/index.js'
|
|
||||||
import { NounType, VerbType } from '../../../src/types/graphTypes.js'
|
|
||||||
import { createTestConfig } from '../../helpers/test-factory.js'
|
|
||||||
|
|
||||||
describe('reserved-field metadata remap (8.0 legacy remap path)', () => {
|
|
||||||
let brain: Brainy
|
|
||||||
|
|
||||||
beforeEach(async () => {
|
|
||||||
// The remap path is opt-in in 8.0 (default policy is 'throw').
|
|
||||||
brain = new Brainy(createTestConfig({ reservedFieldPolicy: 'remap' }))
|
|
||||||
await brain.init()
|
|
||||||
})
|
|
||||||
|
|
||||||
afterEach(async () => {
|
|
||||||
await brain.close()
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('update() — the ported 7.x regression', () => {
|
|
||||||
it('remaps metadata.confidence to the top-level field (the production repro)', async () => {
|
|
||||||
const id = await brain.add({
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'x',
|
|
||||||
metadata: { confidence: 0.8 } as object
|
|
||||||
})
|
|
||||||
|
|
||||||
// Top-level write works (always did)
|
|
||||||
await brain.update({ id, confidence: 0.42 })
|
|
||||||
let entity = await brain.get(id)
|
|
||||||
expect(entity?.confidence).toBe(0.42)
|
|
||||||
|
|
||||||
// Metadata-patch write — silently dropped pre-fix, remapped now
|
|
||||||
await brain.update({ id, metadata: { confidence: 0.33 } as object })
|
|
||||||
entity = await brain.get(id)
|
|
||||||
expect(entity?.confidence).toBe(0.33)
|
|
||||||
// The reserved key must not linger inside the metadata bag
|
|
||||||
expect((entity?.metadata as Record<string, unknown>)?.confidence).toBeUndefined()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('remaps metadata.weight and metadata.subtype the same way', async () => {
|
|
||||||
const id = await brain.add({
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'y',
|
|
||||||
metadata: {}
|
|
||||||
})
|
|
||||||
|
|
||||||
await brain.update({ id, metadata: { weight: 0.7, subtype: 'specialized' } as object })
|
|
||||||
const entity = await brain.get(id)
|
|
||||||
expect(entity?.weight).toBe(0.7)
|
|
||||||
expect(entity?.subtype).toBe('specialized')
|
|
||||||
expect((entity?.metadata as Record<string, unknown>)?.weight).toBeUndefined()
|
|
||||||
expect((entity?.metadata as Record<string, unknown>)?.subtype).toBeUndefined()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('top-level param wins when both top-level and metadata-patch carry the field', async () => {
|
|
||||||
const id = await brain.add({
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'z',
|
|
||||||
metadata: { confidence: 0.5 } as object
|
|
||||||
})
|
|
||||||
|
|
||||||
await brain.update({ id, confidence: 0.9, metadata: { confidence: 0.1 } as object })
|
|
||||||
const entity = await brain.get(id)
|
|
||||||
expect(entity?.confidence).toBe(0.9)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('drops system-managed fields from patches without corrupting the entity', async () => {
|
|
||||||
const id = await brain.add({
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'w',
|
|
||||||
metadata: { keep: 'me' }
|
|
||||||
})
|
|
||||||
const before = await brain.get(id)
|
|
||||||
|
|
||||||
await brain.update({
|
|
||||||
id,
|
|
||||||
metadata: { createdAt: 1, _rev: 999, noun: 'organization', other: 'applied' } as object
|
|
||||||
})
|
|
||||||
const after = await brain.get(id)
|
|
||||||
|
|
||||||
expect(after?.createdAt).toBe(before?.createdAt) // immutable
|
|
||||||
expect(after?.type).toBe('concept') // noun patch ignored
|
|
||||||
expect(after?._rev).toBe((before?._rev ?? 1) + 1) // _rev patch ignored; normal bump applied
|
|
||||||
expect((after?.metadata as Record<string, unknown>)?.other).toBe('applied') // custom fields still merge
|
|
||||||
expect((after?.metadata as Record<string, unknown>)?.keep).toBe('me')
|
|
||||||
expect((after?.metadata as Record<string, unknown>)?._rev).toBeUndefined()
|
|
||||||
expect((after?.metadata as Record<string, unknown>)?.createdAt).toBeUndefined()
|
|
||||||
expect((after?.metadata as Record<string, unknown>)?.noun).toBeUndefined()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('custom (non-reserved) metadata patches are unaffected by the remap', async () => {
|
|
||||||
const id = await brain.add({
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'v',
|
|
||||||
metadata: { status: 'draft' }
|
|
||||||
})
|
|
||||||
|
|
||||||
await brain.update({ id, metadata: { status: 'reviewed', rating: 4.5 } })
|
|
||||||
const entity = await brain.get(id)
|
|
||||||
expect((entity?.metadata as Record<string, unknown>)?.status).toBe('reviewed')
|
|
||||||
expect((entity?.metadata as Record<string, unknown>)?.rating).toBe(4.5)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('add() — explicit lift, identical contract', () => {
|
|
||||||
it('lifts confidence/weight/subtype out of the bag to top level', async () => {
|
|
||||||
const id = await brain.add({
|
|
||||||
type: NounType.Person,
|
|
||||||
data: 'lift check',
|
|
||||||
metadata: { confidence: 0.8, weight: 0.6, subtype: 'employee', dept: 'eng' } as object
|
|
||||||
})
|
|
||||||
|
|
||||||
const entity = await brain.get(id)
|
|
||||||
expect(entity?.confidence).toBe(0.8)
|
|
||||||
expect(entity?.weight).toBe(0.6)
|
|
||||||
expect(entity?.subtype).toBe('employee')
|
|
||||||
expect(entity?.metadata).toEqual({ dept: 'eng' })
|
|
||||||
})
|
|
||||||
|
|
||||||
it('lifts service (settable at add time) and lets the top-level param win', async () => {
|
|
||||||
const lifted = await brain.add({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'service lift',
|
|
||||||
metadata: { service: 'orders' } as object
|
|
||||||
})
|
|
||||||
expect((await brain.get(lifted))?.service).toBe('orders')
|
|
||||||
|
|
||||||
const topLevelWins = await brain.add({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'service precedence',
|
|
||||||
service: 'billing',
|
|
||||||
metadata: { service: 'orders' } as object
|
|
||||||
})
|
|
||||||
const entity = await brain.get(topLevelWins)
|
|
||||||
expect(entity?.service).toBe('billing')
|
|
||||||
expect((entity?.metadata as Record<string, unknown>)?.service).toBeUndefined()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('a remapped subtype satisfies subtype enforcement like a top-level one', async () => {
|
|
||||||
brain.requireSubtype(NounType.Document)
|
|
||||||
|
|
||||||
// Top-level missing, but the bag carries it — must not throw.
|
|
||||||
const id = await brain.add({
|
|
||||||
type: NounType.Document,
|
|
||||||
data: 'enforcement via remap',
|
|
||||||
metadata: { subtype: 'invoice' } as object
|
|
||||||
})
|
|
||||||
expect((await brain.get(id))?.subtype).toBe('invoice')
|
|
||||||
|
|
||||||
// Neither place carries it — must throw.
|
|
||||||
await expect(
|
|
||||||
brain.add({ type: NounType.Document, data: 'no subtype anywhere' })
|
|
||||||
).rejects.toThrow(/subtype/)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('transact() — same remap on add and update ops', () => {
|
|
||||||
it('normalizes reserved fields in transact add + update ops', async () => {
|
|
||||||
const db1 = await brain.transact([
|
|
||||||
{
|
|
||||||
op: 'add',
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'tx',
|
|
||||||
metadata: { confidence: 0.7, custom: 'a' } as object
|
|
||||||
}
|
|
||||||
])
|
|
||||||
const id = db1.receipt!.ids[0]
|
|
||||||
|
|
||||||
let entity = await brain.get(id)
|
|
||||||
expect(entity?.confidence).toBe(0.7)
|
|
||||||
expect(entity?.metadata).toEqual({ custom: 'a' })
|
|
||||||
|
|
||||||
await brain.transact([
|
|
||||||
{ op: 'update', id, metadata: { confidence: 0.25, custom: 'b' } as object }
|
|
||||||
])
|
|
||||||
entity = await brain.get(id)
|
|
||||||
expect(entity?.confidence).toBe(0.25)
|
|
||||||
expect(entity?.metadata).toEqual({ custom: 'b' })
|
|
||||||
expect((entity?.metadata as Record<string, unknown>)?.confidence).toBeUndefined()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('historical asOf() reads surface reserved fields ONLY top-level', async () => {
|
|
||||||
const db1 = await brain.transact([
|
|
||||||
{
|
|
||||||
op: 'add',
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'historical',
|
|
||||||
metadata: { confidence: 0.9, custom: 'past' } as object
|
|
||||||
}
|
|
||||||
])
|
|
||||||
const id = db1.receipt!.ids[0]
|
|
||||||
|
|
||||||
// Move the world forward so generation db1 is historical.
|
|
||||||
await brain.transact([{ op: 'update', id, confidence: 0.1, metadata: { custom: 'now' } }])
|
|
||||||
|
|
||||||
const past = await brain.asOf(db1.generation)
|
|
||||||
const historical = await past.get(id)
|
|
||||||
expect(historical?.confidence).toBe(0.9)
|
|
||||||
expect(historical?.metadata).toEqual({ custom: 'past' })
|
|
||||||
await past.release()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('with() speculative views apply the same normalization', async () => {
|
|
||||||
const base = await brain.now()
|
|
||||||
const speculative = await base.with([
|
|
||||||
{
|
|
||||||
op: 'add',
|
|
||||||
id: 'spec-entity',
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'spec',
|
|
||||||
metadata: { confidence: 0.65, custom: 'spec' } as object
|
|
||||||
}
|
|
||||||
])
|
|
||||||
|
|
||||||
const entity = await speculative.get('spec-entity')
|
|
||||||
expect(entity?.confidence).toBe(0.65)
|
|
||||||
// 8.0 id normalization: a natural-key id is mapped to a stable UUID and
|
|
||||||
// the caller's original string is preserved under _originalId — surfaced
|
|
||||||
// here exactly as the durable transact()/add() paths do.
|
|
||||||
expect(entity?.metadata).toEqual({ custom: 'spec', _originalId: 'spec-entity' })
|
|
||||||
await speculative.release()
|
|
||||||
await base.release()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('read paths never echo reserved fields inside metadata', () => {
|
|
||||||
it('find() (storage pagination path) returns custom-only metadata with reserved fields top-level', async () => {
|
|
||||||
const id = await brain.add({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'pagination echo check',
|
|
||||||
confidence: 0.8,
|
|
||||||
weight: 0.6,
|
|
||||||
metadata: { dept: 'eng' }
|
|
||||||
})
|
|
||||||
|
|
||||||
// No query/filter → served by the direct storage pagination path
|
|
||||||
// (getNounsWithPagination), which historically echoed the full flat
|
|
||||||
// record (noun/subtype/createdAt/… inside metadata).
|
|
||||||
const results = await brain.find({ limit: 50 })
|
|
||||||
const result = results.find((r) => r.id === id)
|
|
||||||
expect(result).toBeDefined()
|
|
||||||
expect(result?.entity.metadata).toEqual({ dept: 'eng' })
|
|
||||||
expect(result?.entity.type).toBe(NounType.Person)
|
|
||||||
expect(result?.entity.subtype).toBe('employee')
|
|
||||||
expect(result?.entity.confidence).toBe(0.8)
|
|
||||||
expect(result?.entity.weight).toBe(0.6)
|
|
||||||
expect(typeof result?.entity.createdAt).toBe('number')
|
|
||||||
expect(result?.entity._rev).toBe(1)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('related() by target surfaces reserved fields top-level, custom-only metadata', async () => {
|
|
||||||
const a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'src' })
|
|
||||||
const b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'tgt' })
|
|
||||||
const relId = await brain.relate({
|
|
||||||
from: a,
|
|
||||||
to: b,
|
|
||||||
type: VerbType.ReportsTo,
|
|
||||||
subtype: 'direct',
|
|
||||||
confidence: 0.9,
|
|
||||||
weight: 0.5,
|
|
||||||
service: 'orders',
|
|
||||||
metadata: { note: 'target path' }
|
|
||||||
})
|
|
||||||
|
|
||||||
const relations = await brain.related({ to: b })
|
|
||||||
const rel = relations.find((r) => r.id === relId)
|
|
||||||
expect(rel).toBeDefined()
|
|
||||||
expect(rel?.metadata).toEqual({ note: 'target path' })
|
|
||||||
expect(rel?.subtype).toBe('direct')
|
|
||||||
expect(rel?.confidence).toBe(0.9)
|
|
||||||
expect(rel?.weight).toBe(0.5)
|
|
||||||
expect(rel?.service).toBe('orders')
|
|
||||||
expect(typeof rel?.createdAt).toBe('number')
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('relationships — relate() / updateRelation() mirror', () => {
|
|
||||||
let a: string
|
|
||||||
let b: string
|
|
||||||
|
|
||||||
beforeEach(async () => {
|
|
||||||
a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'A' })
|
|
||||||
b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'B' })
|
|
||||||
})
|
|
||||||
|
|
||||||
it('relate() persists the top-level confidence and service params', async () => {
|
|
||||||
const relId = await brain.relate({
|
|
||||||
from: a,
|
|
||||||
to: b,
|
|
||||||
type: VerbType.ReportsTo,
|
|
||||||
subtype: 'direct',
|
|
||||||
confidence: 0.77,
|
|
||||||
service: 'orders'
|
|
||||||
})
|
|
||||||
|
|
||||||
const relations = await brain.related({ from: a })
|
|
||||||
const rel = relations.find((r) => r.id === relId)
|
|
||||||
expect(rel?.confidence).toBe(0.77)
|
|
||||||
expect(rel?.service).toBe('orders')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('relate() remaps reserved fields out of the metadata bag', async () => {
|
|
||||||
const relId = await brain.relate({
|
|
||||||
from: a,
|
|
||||||
to: b,
|
|
||||||
type: VerbType.RelatedTo,
|
|
||||||
subtype: 'colleague',
|
|
||||||
metadata: { confidence: 0.4, weight: 0.3, role: 'peer' } as object
|
|
||||||
})
|
|
||||||
|
|
||||||
const relations = await brain.related({ from: a })
|
|
||||||
const rel = relations.find((r) => r.id === relId)
|
|
||||||
expect(rel?.confidence).toBe(0.4)
|
|
||||||
expect(rel?.weight).toBe(0.3)
|
|
||||||
expect(rel?.metadata).toEqual({ role: 'peer' })
|
|
||||||
})
|
|
||||||
|
|
||||||
it('relation.metadata never echoes the verb type key', async () => {
|
|
||||||
const relId = await brain.relate({
|
|
||||||
from: a,
|
|
||||||
to: b,
|
|
||||||
type: VerbType.RelatedTo,
|
|
||||||
subtype: 'colleague',
|
|
||||||
metadata: { note: 'no echo' }
|
|
||||||
})
|
|
||||||
|
|
||||||
const relations = await brain.related({ from: a })
|
|
||||||
const rel = relations.find((r) => r.id === relId)
|
|
||||||
expect(rel?.type).toBe(VerbType.RelatedTo)
|
|
||||||
expect((rel?.metadata as Record<string, unknown>)?.verb).toBeUndefined()
|
|
||||||
expect(rel?.metadata).toEqual({ note: 'no echo' })
|
|
||||||
})
|
|
||||||
|
|
||||||
it('updateRelation() remaps the user-mutable trio and preserves service', async () => {
|
|
||||||
const relId = await brain.relate({
|
|
||||||
from: a,
|
|
||||||
to: b,
|
|
||||||
type: VerbType.ReportsTo,
|
|
||||||
subtype: 'direct',
|
|
||||||
service: 'orders',
|
|
||||||
metadata: { keep: 'me' }
|
|
||||||
})
|
|
||||||
|
|
||||||
await brain.updateRelation({
|
|
||||||
id: relId,
|
|
||||||
metadata: { confidence: 0.55, subtype: 'dotted-line', extra: 'applied' } as object
|
|
||||||
})
|
|
||||||
|
|
||||||
const relations = await brain.related({ from: a })
|
|
||||||
const rel = relations.find((r) => r.id === relId)
|
|
||||||
expect(rel?.confidence).toBe(0.55)
|
|
||||||
expect(rel?.subtype).toBe('dotted-line')
|
|
||||||
expect(rel?.service).toBe('orders') // fixed at relate() time, never erased by updates
|
|
||||||
expect(rel?.metadata).toEqual({ keep: 'me', extra: 'applied' })
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
@ -198,60 +198,47 @@ describe('visibility (8.0 reserved field)', () => {
|
||||||
expect(entity?.visibility).toBeUndefined()
|
expect(entity?.visibility).toBeUndefined()
|
||||||
})
|
})
|
||||||
|
|
||||||
it('an untyped caller passing visibility inside metadata is normalized under reservedFieldPolicy:"remap" (lifted to top-level)', async () => {
|
it('metadata.visibility is the USER’s field (field-addressing law) — stored verbatim, never lifted to the engine tier', async () => {
|
||||||
// Simulate a JavaScript caller smuggling the reserved key past the compile-time guard.
|
const id = await brain.add({
|
||||||
// The legacy remap behavior is now opt-in (8.0 default is 'throw').
|
type: NounType.Concept,
|
||||||
const remapBrain = new Brainy(createTestConfig({ reservedFieldPolicy: 'remap' }))
|
data: 'y',
|
||||||
await remapBrain.init()
|
metadata: { visibility: 'internal', tag: 't' } as object
|
||||||
try {
|
})
|
||||||
const id = await remapBrain.add({
|
const entity = await brain.get(id)
|
||||||
type: NounType.Concept,
|
// The user's field lives in the bag, verbatim…
|
||||||
data: 'y',
|
expect((entity?.metadata as Record<string, unknown>)?.visibility).toBe('internal')
|
||||||
metadata: { visibility: 'internal', tag: 't' } as object
|
expect((entity?.metadata as Record<string, unknown>)?.tag).toBe('t')
|
||||||
})
|
// …and the ENGINE tier is untouched: absent === public, so the entity
|
||||||
const entity = await remapBrain.get(id)
|
// stays visible on default reads (the engine tier is set only via the
|
||||||
// Lifted to the top-level field…
|
// dedicated visibility param and reads at system.visibility).
|
||||||
expect(entity?.visibility).toBe('internal')
|
expect(entity?.visibility).toBeUndefined()
|
||||||
// …and stripped from the metadata bag.
|
const visible = await brain.find({ type: NounType.Concept, limit: 20 })
|
||||||
expect((entity?.metadata as Record<string, unknown>)?.visibility).toBeUndefined()
|
expect(visible.map((r) => r.id)).toContain(id)
|
||||||
expect((entity?.metadata as Record<string, unknown>)?.tag).toBe('t')
|
|
||||||
// It is excluded from the default count, exactly like a top-level internal write.
|
|
||||||
expect(await remapBrain.getNounCount()).toBe(0)
|
|
||||||
} finally {
|
|
||||||
await remapBrain.close()
|
|
||||||
}
|
|
||||||
})
|
})
|
||||||
|
|
||||||
it('a "system" value smuggled through metadata is dropped under reservedFieldPolicy:"remap", not honored', async () => {
|
it('a user field valued "system" cannot smuggle the Brainy-only tier — it is just user data', async () => {
|
||||||
// 'system' is Brainy-only; an untyped caller must not be able to set it.
|
const id = await brain.add({
|
||||||
const remapBrain = new Brainy(createTestConfig({ reservedFieldPolicy: 'remap' }))
|
type: NounType.Concept,
|
||||||
await remapBrain.init()
|
data: 'z',
|
||||||
try {
|
metadata: { visibility: 'system' } as object
|
||||||
const id = await remapBrain.add({
|
})
|
||||||
type: NounType.Concept,
|
const entity = await brain.get(id)
|
||||||
data: 'z',
|
// Engine tier unaffected → entity stays public (counted, visible);
|
||||||
metadata: { visibility: 'system' } as object
|
// the string 'system' is ordinary user data in the bag.
|
||||||
})
|
expect(entity?.visibility).toBeUndefined()
|
||||||
const entity = await remapBrain.get(id)
|
expect((entity?.metadata as Record<string, unknown>)?.visibility).toBe('system')
|
||||||
// The smuggled 'system' was dropped → entity stays public (counted, visible).
|
const found = await brain.find({ type: NounType.Concept, limit: 10 })
|
||||||
expect(entity?.visibility).toBeUndefined()
|
expect(found.map((r) => r.id)).toContain(id)
|
||||||
expect(await remapBrain.getNounCount()).toBe(1)
|
|
||||||
const found = await remapBrain.find({ type: NounType.Concept, limit: 10 })
|
|
||||||
expect(found.map((r) => r.id)).toContain(id)
|
|
||||||
} finally {
|
|
||||||
await remapBrain.close()
|
|
||||||
}
|
|
||||||
})
|
})
|
||||||
|
|
||||||
it('an untyped caller passing visibility inside metadata throws under the default policy', async () => {
|
it('a forged system.visibility key in metadata refuses loudly at the write door', async () => {
|
||||||
// 8.0 default: no silent remap — a reserved key in the bag is a loud error.
|
|
||||||
await expect(
|
await expect(
|
||||||
brain.add({
|
brain.add({
|
||||||
type: NounType.Concept,
|
type: NounType.Concept,
|
||||||
data: 'throws',
|
data: 'throws',
|
||||||
metadata: { visibility: 'internal', tag: 't' } as object
|
metadata: { 'system.visibility': 'internal' } as object
|
||||||
})
|
})
|
||||||
).rejects.toThrow(/visibility.*reserved field/)
|
).rejects.toThrow(/system\./)
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
|
||||||
145
tests/unit/db/fieldAddressing.test.ts
Normal file
145
tests/unit/db/fieldAddressing.test.ts
Normal file
|
|
@ -0,0 +1,145 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/db/fieldAddressing
|
||||||
|
* @description Unit pins for the one field-addressing law (ruled 2026-08-03).
|
||||||
|
* These pin the PURE half of the law — parsing, the ruled maps, plumbing
|
||||||
|
* invisibility, refusal text — including the RELATION map, which cannot be
|
||||||
|
* pinned through the public query API today (related() carries no
|
||||||
|
* field-addressing options): the verb mirror is contract-tested here at the
|
||||||
|
* module level so the two engines cannot drift on it.
|
||||||
|
*/
|
||||||
|
import { describe, it, expect } from 'vitest'
|
||||||
|
import {
|
||||||
|
SYSTEM_ENTITY_SCALARS,
|
||||||
|
SYSTEM_RELATION_SCALARS,
|
||||||
|
PLUMBING_FIELDS,
|
||||||
|
parseFieldAddress,
|
||||||
|
buildUnresolvableMessage,
|
||||||
|
InvalidFieldAddressError
|
||||||
|
} from '../../../src/db/fieldAddressing.js'
|
||||||
|
|
||||||
|
describe('field-addressing law — pure module pins', () => {
|
||||||
|
it('the entity system map is EXACTLY the ruled ten scalars', () => {
|
||||||
|
expect([...SYSTEM_ENTITY_SCALARS].sort()).toEqual(
|
||||||
|
[
|
||||||
|
'confidence',
|
||||||
|
'createdAt',
|
||||||
|
'createdBy',
|
||||||
|
'id',
|
||||||
|
'service',
|
||||||
|
'subtype',
|
||||||
|
'type',
|
||||||
|
'updatedAt',
|
||||||
|
'visibility',
|
||||||
|
'weight'
|
||||||
|
].sort()
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('the relation system map is the ruled verb mirror', () => {
|
||||||
|
expect([...SYSTEM_RELATION_SCALARS].sort()).toEqual(
|
||||||
|
[
|
||||||
|
'verb',
|
||||||
|
'sourceId',
|
||||||
|
'targetId',
|
||||||
|
'confidence',
|
||||||
|
'createdAt',
|
||||||
|
'createdBy',
|
||||||
|
'service',
|
||||||
|
'subtype',
|
||||||
|
'updatedAt',
|
||||||
|
'visibility',
|
||||||
|
'weight'
|
||||||
|
].sort()
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('plumbing is exactly the ruled five, and none of it leaks into a system map', () => {
|
||||||
|
expect([...PLUMBING_FIELDS].sort()).toEqual(
|
||||||
|
['_rev', 'connections', 'data', 'level', 'vector'].sort()
|
||||||
|
)
|
||||||
|
for (const field of PLUMBING_FIELDS) {
|
||||||
|
expect(SYSTEM_ENTITY_SCALARS.has(field)).toBe(false)
|
||||||
|
expect(SYSTEM_RELATION_SCALARS.has(field)).toBe(false)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('bare names address user metadata — even when the name matches a system scalar', () => {
|
||||||
|
expect(parseFieldAddress('level', 'entity')).toEqual({
|
||||||
|
scope: 'metadata',
|
||||||
|
field: 'level',
|
||||||
|
raw: 'level'
|
||||||
|
})
|
||||||
|
expect(parseFieldAddress('confidence', 'entity').scope).toBe('metadata')
|
||||||
|
expect(parseFieldAddress('createdAt', 'entity').scope).toBe('metadata')
|
||||||
|
expect(parseFieldAddress('verb', 'relation').scope).toBe('metadata')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('metadata.-prefix is the explicit spelling of the bare form', () => {
|
||||||
|
expect(parseFieldAddress('metadata.level', 'entity')).toEqual({
|
||||||
|
scope: 'metadata',
|
||||||
|
field: 'level',
|
||||||
|
raw: 'metadata.level'
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
it('system.-prefix reaches exactly the map — entity and relation', () => {
|
||||||
|
for (const field of SYSTEM_ENTITY_SCALARS) {
|
||||||
|
expect(parseFieldAddress(`system.${field}`, 'entity')).toEqual({
|
||||||
|
scope: 'system',
|
||||||
|
field,
|
||||||
|
raw: `system.${field}`
|
||||||
|
})
|
||||||
|
}
|
||||||
|
for (const field of SYSTEM_RELATION_SCALARS) {
|
||||||
|
expect(parseFieldAddress(`system.${field}`, 'relation').scope).toBe('system')
|
||||||
|
}
|
||||||
|
// The structural relation members are NOT entity scalars.
|
||||||
|
expect(() => parseFieldAddress('system.verb', 'entity')).toThrow(InvalidFieldAddressError)
|
||||||
|
expect(() => parseFieldAddress('system.sourceId', 'entity')).toThrow(InvalidFieldAddressError)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('plumbing refuses in the system spelling, on both record kinds', () => {
|
||||||
|
for (const field of PLUMBING_FIELDS) {
|
||||||
|
expect(() => parseFieldAddress(`system.${field}`, 'entity')).toThrow(
|
||||||
|
InvalidFieldAddressError
|
||||||
|
)
|
||||||
|
expect(() => parseFieldAddress(`system.${field}`, 'relation')).toThrow(
|
||||||
|
InvalidFieldAddressError
|
||||||
|
)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('refusal text carries the whole valid map — the fix lives in the message', () => {
|
||||||
|
try {
|
||||||
|
parseFieldAddress('system.level', 'entity')
|
||||||
|
expect.unreachable('should have thrown')
|
||||||
|
} catch (e) {
|
||||||
|
const msg = (e as Error).message
|
||||||
|
for (const field of SYSTEM_ENTITY_SCALARS) {
|
||||||
|
expect(msg).toContain(`system.${field}`)
|
||||||
|
}
|
||||||
|
expect(msg).toContain('plumbing')
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('malformed addresses refuse: empty name, bare metadata. prefix', () => {
|
||||||
|
expect(() => parseFieldAddress('', 'entity')).toThrow(InvalidFieldAddressError)
|
||||||
|
expect(() => parseFieldAddress('metadata.', 'entity')).toThrow(InvalidFieldAddressError)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('the did-you-mean names BOTH candidates for a system-colliding bare name', () => {
|
||||||
|
const msg = buildUnresolvableMessage('createdAt', 'entity')
|
||||||
|
expect(msg).toContain('system.createdAt')
|
||||||
|
expect(msg).toContain('metadata.createdAt')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a non-colliding unknown bare name names both spellings — system.<f> explicitly as NOT valid', () => {
|
||||||
|
// Cross-engine pin (cor's suite greps for both spellings in every
|
||||||
|
// refusal): the metadata candidate is the fix; the system spelling is
|
||||||
|
// named but HONESTLY marked invalid, never offered as a candidate.
|
||||||
|
const msg = buildUnresolvableMessage('scoore', 'entity')
|
||||||
|
expect(msg).toContain('metadata.scoore')
|
||||||
|
expect(msg).toContain('system.scoore')
|
||||||
|
expect(msg).toContain('NOT valid')
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
@ -32,7 +32,7 @@ function entity(overrides: Partial<Entity> = {}): Entity {
|
||||||
}
|
}
|
||||||
|
|
||||||
describe('db/whereMatcher — resolveEntityField', () => {
|
describe('db/whereMatcher — resolveEntityField', () => {
|
||||||
it('resolves standard top-level fields', () => {
|
it('system.<field> resolves the entity scalar; bare/metadata. reads the metadata bag only (sealed 2026-08-03)', () => {
|
||||||
const e = entity({
|
const e = entity({
|
||||||
subtype: 'invoice',
|
subtype: 'invoice',
|
||||||
service: 'billing',
|
service: 'billing',
|
||||||
|
|
@ -41,17 +41,32 @@ describe('db/whereMatcher — resolveEntityField', () => {
|
||||||
_rev: 3,
|
_rev: 3,
|
||||||
data: 'payload'
|
data: 'payload'
|
||||||
})
|
})
|
||||||
expect(resolveEntityField(e, 'id')).toBe('e-1')
|
|
||||||
expect(resolveEntityField(e, 'type')).toBe(NounType.Document)
|
// system.<field> is the ONLY spelling that reaches an entity scalar.
|
||||||
expect(resolveEntityField(e, 'noun')).toBe(NounType.Document) // alias
|
expect(resolveEntityField(e, 'system.id')).toBe('e-1')
|
||||||
expect(resolveEntityField(e, 'subtype')).toBe('invoice')
|
expect(resolveEntityField(e, 'system.type')).toBe(NounType.Document)
|
||||||
expect(resolveEntityField(e, 'service')).toBe('billing')
|
expect(resolveEntityField(e, 'system.subtype')).toBe('invoice')
|
||||||
expect(resolveEntityField(e, 'confidence')).toBe(0.9)
|
expect(resolveEntityField(e, 'system.service')).toBe('billing')
|
||||||
expect(resolveEntityField(e, 'weight')).toBe(0.5)
|
expect(resolveEntityField(e, 'system.confidence')).toBe(0.9)
|
||||||
expect(resolveEntityField(e, '_rev')).toBe(3)
|
expect(resolveEntityField(e, 'system.weight')).toBe(0.5)
|
||||||
expect(resolveEntityField(e, 'createdAt')).toBe(1000)
|
expect(resolveEntityField(e, 'system.createdAt')).toBe(1000)
|
||||||
expect(resolveEntityField(e, 'updatedAt')).toBe(2000)
|
expect(resolveEntityField(e, 'system.updatedAt')).toBe(2000)
|
||||||
expect(resolveEntityField(e, 'data')).toBe('payload')
|
|
||||||
|
// Plumbing (_rev, data) is invisible even via system. — not in the
|
||||||
|
// ten-scalar map, so this internal resolver reads it as absent (the typed
|
||||||
|
// refusal for these lives one layer up, at the query-surface parser).
|
||||||
|
expect(resolveEntityField(e, 'system._rev')).toBeUndefined()
|
||||||
|
expect(resolveEntityField(e, 'system.data')).toBeUndefined()
|
||||||
|
|
||||||
|
// Bare names are ALWAYS the user's metadata field — even when they share
|
||||||
|
// a spelling with an engine scalar, or with the now-dead 'noun' alias.
|
||||||
|
// This entity's metadata bag is empty, so every bare name below reads
|
||||||
|
// absent rather than silently falling back to the entity scalar.
|
||||||
|
expect(resolveEntityField(e, 'id')).toBeUndefined()
|
||||||
|
expect(resolveEntityField(e, 'type')).toBeUndefined()
|
||||||
|
expect(resolveEntityField(e, 'noun')).toBeUndefined() // legacy alias is dead
|
||||||
|
expect(resolveEntityField(e, 'subtype')).toBeUndefined()
|
||||||
|
expect(resolveEntityField(e, 'createdAt')).toBeUndefined()
|
||||||
})
|
})
|
||||||
|
|
||||||
it('resolves custom fields from the metadata bag', () => {
|
it('resolves custom fields from the metadata bag', () => {
|
||||||
|
|
|
||||||
|
|
@ -30,6 +30,9 @@ function allTestFiles(dir: string, out: string[] = []): string[] {
|
||||||
* conscious decision — a NEW orphan not listed here fails the guard below.
|
* conscious decision — a NEW orphan not listed here fails the guard below.
|
||||||
*/
|
*/
|
||||||
const MANUAL_ONLY = new Set<string>([
|
const MANUAL_ONLY = new Set<string>([
|
||||||
|
// Conformance suites run as an explicit gate stage (both engines run them
|
||||||
|
// by direct invocation), never swept into the unit/integration configs.
|
||||||
|
'tests/conformance/collider-fidelity.test.ts',
|
||||||
'tests/api/performance-benchmarks.test.ts',
|
'tests/api/performance-benchmarks.test.ts',
|
||||||
'tests/critical-neural-validation.test.ts',
|
'tests/critical-neural-validation.test.ts',
|
||||||
'tests/critical-performance-benchmark.test.ts',
|
'tests/critical-performance-benchmark.test.ts',
|
||||||
|
|
@ -38,7 +41,15 @@ const MANUAL_ONLY = new Set<string>([
|
||||||
'tests/package-size-limit.test.ts',
|
'tests/package-size-limit.test.ts',
|
||||||
'tests/performance/graph-scale-performance.test.ts',
|
'tests/performance/graph-scale-performance.test.ts',
|
||||||
'tests/performance/triple-intelligence-scale.test.ts',
|
'tests/performance/triple-intelligence-scale.test.ts',
|
||||||
'tests/performance/typeAware.bench.test.ts'
|
'tests/performance/typeAware.bench.test.ts',
|
||||||
|
// Cross-engine field-addressing conformance suite: pinned bit-for-bit against
|
||||||
|
// the native accelerator's implementation of the SAME contract, and invoked
|
||||||
|
// directly (`npx vitest run tests/conformance/namespace-law.test.ts`), never
|
||||||
|
// swept into the unit/integration gates — a run against a branch where the
|
||||||
|
// resolver hasn't landed yet must SKIP loudly (see the file's own SELF-SKIP
|
||||||
|
// doc), not silently pass/fail as a side effect of which gate happened to
|
||||||
|
// pick it up.
|
||||||
|
'tests/conformance/namespace-law.test.ts'
|
||||||
])
|
])
|
||||||
|
|
||||||
function inGate(rel: string): boolean {
|
function inGate(rel: string): boolean {
|
||||||
|
|
|
||||||
127
tests/unit/types/nestedBagRecord.test.ts
Normal file
127
tests/unit/types/nestedBagRecord.test.ts
Normal file
|
|
@ -0,0 +1,127 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/types/nestedBagRecord
|
||||||
|
* @description Unit pins for the v2 (nested-bag) stored-record layer — the
|
||||||
|
* storage half of the field-addressing law. The write door accepts ANY user
|
||||||
|
* metadata name; what makes that lossless on disk is the record shape:
|
||||||
|
* engine fields top-level, the user bag NESTED verbatim, discriminated by
|
||||||
|
* the engine-written format stamp (never by names — names are the user's).
|
||||||
|
* These pins hold the builders, the discriminator, and the shape-aware
|
||||||
|
* split that every read path (live, batch, historical) routes through.
|
||||||
|
*/
|
||||||
|
import { describe, it, expect } from 'vitest'
|
||||||
|
import {
|
||||||
|
buildNounMetadataRecord,
|
||||||
|
buildVerbMetadataRecord,
|
||||||
|
splitNounMetadataRecord,
|
||||||
|
splitVerbMetadataRecord,
|
||||||
|
isNestedBagRecord,
|
||||||
|
METADATA_RECORD_FORMAT_KEY,
|
||||||
|
NESTED_BAG_FORMAT
|
||||||
|
} from '../../../src/types/reservedFields.js'
|
||||||
|
|
||||||
|
const COLLIDER_BAG = {
|
||||||
|
confidence: 'user-confidence',
|
||||||
|
weight: 'user-weight',
|
||||||
|
subtype: 'user-subtype',
|
||||||
|
createdAt: 'user-createdAt',
|
||||||
|
service: 'user-service',
|
||||||
|
data: 'user-data',
|
||||||
|
noun: 'user-noun',
|
||||||
|
_rev: 'user-rev',
|
||||||
|
level: 7,
|
||||||
|
plain: 'control'
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('v2 nested-bag stored records — build / discriminate / split', () => {
|
||||||
|
it('build → split round-trips a fully colliding user bag VERBATIM', () => {
|
||||||
|
const record = buildNounMetadataRecord(
|
||||||
|
{ noun: 'document', confidence: 0.25, createdAt: 111, updatedAt: 222, _rev: 1 },
|
||||||
|
{ ...COLLIDER_BAG }
|
||||||
|
)
|
||||||
|
expect(isNestedBagRecord(record)).toBe(true)
|
||||||
|
expect(record[METADATA_RECORD_FORMAT_KEY]).toBe(NESTED_BAG_FORMAT)
|
||||||
|
|
||||||
|
const { reserved, custom } = splitNounMetadataRecord(record)
|
||||||
|
// The engine half is exactly what the engine wrote…
|
||||||
|
expect(reserved.noun).toBe('document')
|
||||||
|
expect(reserved.confidence).toBe(0.25)
|
||||||
|
expect(reserved._rev).toBe(1)
|
||||||
|
// …and the user bag comes back byte-for-byte, colliders included.
|
||||||
|
expect(custom).toEqual(COLLIDER_BAG)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('the verb mirror round-trips an edge collider bag verbatim', () => {
|
||||||
|
const record = buildVerbMetadataRecord(
|
||||||
|
{ verb: 'relatedTo', weight: 1.0, confidence: 0.5, createdAt: 333 },
|
||||||
|
{ verb: 'user-verb', confidence: 'user-c', tag: 't' }
|
||||||
|
)
|
||||||
|
expect(isNestedBagRecord(record)).toBe(true)
|
||||||
|
const { reserved, custom } = splitVerbMetadataRecord(record)
|
||||||
|
expect(reserved.verb).toBe('relatedTo')
|
||||||
|
expect(reserved.confidence).toBe(0.5)
|
||||||
|
expect(custom).toEqual({ verb: 'user-verb', confidence: 'user-c', tag: 't' })
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a LEGACY flat record (no stamp) splits BY NAME — sound because the pre-law door refused colliders', () => {
|
||||||
|
const legacy = {
|
||||||
|
noun: 'document',
|
||||||
|
confidence: 0.75,
|
||||||
|
createdAt: 111,
|
||||||
|
_rev: 2,
|
||||||
|
legacyField: 'legacy-value'
|
||||||
|
}
|
||||||
|
expect(isNestedBagRecord(legacy)).toBe(false)
|
||||||
|
const { reserved, custom } = splitNounMetadataRecord(legacy)
|
||||||
|
expect(reserved.confidence).toBe(0.75)
|
||||||
|
expect(reserved._rev).toBe(2)
|
||||||
|
expect(custom).toEqual({ legacyField: 'legacy-value' })
|
||||||
|
})
|
||||||
|
|
||||||
|
it('the stamp is the discriminator, never the name: a legacy user OBJECT field named `metadata` does not fake a v2 record', () => {
|
||||||
|
// Pre-law, 'metadata' was never a reserved name — a flat record could
|
||||||
|
// legally carry a user object field spelled exactly 'metadata'. Without
|
||||||
|
// the engine-written stamp it must split as legacy, with that object
|
||||||
|
// preserved as an ordinary user field.
|
||||||
|
const legacyWithMetadataField = {
|
||||||
|
noun: 'document',
|
||||||
|
confidence: 0.5,
|
||||||
|
metadata: { nested: 'user-object' }
|
||||||
|
}
|
||||||
|
expect(isNestedBagRecord(legacyWithMetadataField)).toBe(false)
|
||||||
|
const { reserved, custom } = splitNounMetadataRecord(legacyWithMetadataField)
|
||||||
|
expect(reserved.confidence).toBe(0.5)
|
||||||
|
expect(custom).toEqual({ metadata: { nested: 'user-object' } })
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a malformed stamp (right key, wrong value / non-object bag) never discriminates as v2', () => {
|
||||||
|
expect(
|
||||||
|
isNestedBagRecord({ [METADATA_RECORD_FORMAT_KEY]: 999, metadata: {} })
|
||||||
|
).toBe(false)
|
||||||
|
expect(
|
||||||
|
isNestedBagRecord({ [METADATA_RECORD_FORMAT_KEY]: NESTED_BAG_FORMAT, metadata: 'not-a-bag' })
|
||||||
|
).toBe(false)
|
||||||
|
expect(
|
||||||
|
isNestedBagRecord({ [METADATA_RECORD_FORMAT_KEY]: NESTED_BAG_FORMAT, metadata: [1, 2] })
|
||||||
|
).toBe(false)
|
||||||
|
expect(isNestedBagRecord(null)).toBe(false)
|
||||||
|
expect(isNestedBagRecord(undefined)).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('the v2 split never surfaces the stamp or the bag container as fields', () => {
|
||||||
|
const record = buildNounMetadataRecord({ noun: 'document', _rev: 1 }, { a: 1 })
|
||||||
|
const { reserved, custom } = splitNounMetadataRecord(record)
|
||||||
|
expect(METADATA_RECORD_FORMAT_KEY in reserved).toBe(false)
|
||||||
|
expect(METADATA_RECORD_FORMAT_KEY in custom).toBe(false)
|
||||||
|
expect('metadata' in reserved).toBe(false)
|
||||||
|
expect(custom).toEqual({ a: 1 })
|
||||||
|
})
|
||||||
|
|
||||||
|
it('builders copy the bag (no aliasing): later caller mutation cannot reach the record', () => {
|
||||||
|
const bag: Record<string, unknown> = { a: 1 }
|
||||||
|
const record = buildNounMetadataRecord({ noun: 'document' }, bag)
|
||||||
|
bag.a = 999
|
||||||
|
bag.b = 'sneaky'
|
||||||
|
expect((record.metadata as Record<string, unknown>).a).toBe(1)
|
||||||
|
expect('b' in (record.metadata as Record<string, unknown>)).toBe(false)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
@ -1,265 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/unit/types/reserved-metadata-keys.test-d
|
|
||||||
* @description Compile-time tests for the reserved-field contract (layer 1 of
|
|
||||||
* three — see src/types/reservedFields.ts): a literal reserved key inside any
|
|
||||||
* `metadata` param is a TypeScript error, while the generic `T` ergonomics
|
|
||||||
* stay intact (typed bags, untyped brains, index-signature shapes, and the
|
|
||||||
* documented exemption for consumers who explicitly declare a reserved key in
|
|
||||||
* their own metadata type).
|
|
||||||
*
|
|
||||||
* Runs under vitest typecheck mode (`test.typecheck` in
|
|
||||||
* tests/configs/vitest.unit.config.ts) — these assertions are validated by
|
|
||||||
* `tsc`, never executed. The runtime half of the contract (the write-path
|
|
||||||
* remap for untyped callers) is pinned by
|
|
||||||
* tests/unit/brainy/update-reserved-metadata-remap.test.ts.
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { describe, it, assertType } from 'vitest'
|
|
||||||
import type {
|
|
||||||
AddParams,
|
|
||||||
UpdateParams,
|
|
||||||
RelateParams,
|
|
||||||
UpdateRelationParams,
|
|
||||||
TxOperation
|
|
||||||
} from '../../../src/index.js'
|
|
||||||
import { NounType, VerbType } from '../../../src/types/graphTypes.js'
|
|
||||||
|
|
||||||
describe('reserved entity keys in metadata are compile errors', () => {
|
|
||||||
it('AddParams (untyped brain) rejects every reserved key but stays open for custom fields', () => {
|
|
||||||
// Custom fields of any shape remain legal — exactly the pre-8.0 latitude.
|
|
||||||
assertType<AddParams>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
metadata: { dept: 'eng', level: 3, tags: ['a', 'b'], nested: { ok: true } }
|
|
||||||
})
|
|
||||||
|
|
||||||
assertType<AddParams>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
// @ts-expect-error — 'noun' is reserved (the entity type travels via the top-level 'type' param)
|
|
||||||
metadata: { noun: 'organization' }
|
|
||||||
})
|
|
||||||
assertType<AddParams>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
// @ts-expect-error — 'subtype' is reserved (use the top-level 'subtype' param)
|
|
||||||
metadata: { subtype: 'contractor' }
|
|
||||||
})
|
|
||||||
assertType<AddParams>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
// @ts-expect-error — 'createdAt' is reserved (system-managed)
|
|
||||||
metadata: { createdAt: Date.now() }
|
|
||||||
})
|
|
||||||
assertType<AddParams>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
// @ts-expect-error — 'updatedAt' is reserved (system-managed)
|
|
||||||
metadata: { updatedAt: Date.now() }
|
|
||||||
})
|
|
||||||
assertType<AddParams>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
// @ts-expect-error — 'confidence' is reserved (use the top-level 'confidence' param)
|
|
||||||
metadata: { confidence: 0.8 }
|
|
||||||
})
|
|
||||||
assertType<AddParams>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
// @ts-expect-error — 'weight' is reserved (use the top-level 'weight' param)
|
|
||||||
metadata: { weight: 0.5 }
|
|
||||||
})
|
|
||||||
assertType<AddParams>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
// @ts-expect-error — 'service' is reserved (use the top-level 'service' param)
|
|
||||||
metadata: { service: 'orders' }
|
|
||||||
})
|
|
||||||
assertType<AddParams>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
// @ts-expect-error — 'data' is reserved (use the top-level 'data' param)
|
|
||||||
metadata: { data: 'content' }
|
|
||||||
})
|
|
||||||
assertType<AddParams>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
// @ts-expect-error — 'createdBy' is reserved (use the top-level 'createdBy' param)
|
|
||||||
metadata: { createdBy: { augmentation: 'importer', version: '1.0' } }
|
|
||||||
})
|
|
||||||
assertType<AddParams>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
// @ts-expect-error — '_rev' is reserved (system-managed revision counter)
|
|
||||||
metadata: { _rev: 7 }
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('AddParams<T> (typed brain) rejects reserved keys alongside the declared shape', () => {
|
|
||||||
interface EmployeeMeta {
|
|
||||||
dept: string
|
|
||||||
level: number
|
|
||||||
}
|
|
||||||
|
|
||||||
assertType<AddParams<EmployeeMeta>>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
metadata: { dept: 'eng', level: 3 }
|
|
||||||
})
|
|
||||||
|
|
||||||
assertType<AddParams<EmployeeMeta>>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
// @ts-expect-error — 'confidence' is reserved even when T declares other fields
|
|
||||||
metadata: { dept: 'eng', level: 3, confidence: 0.8 }
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('documented exemptions: T-declared reserved keys and index-signature shapes stay assignable', () => {
|
|
||||||
// A consumer who *explicitly* types a reserved key into their metadata
|
|
||||||
// shape keeps a working (if unwise) type — the guard exempts keyof T.
|
|
||||||
interface LegacyMeta {
|
|
||||||
confidence: number
|
|
||||||
note: string
|
|
||||||
}
|
|
||||||
assertType<AddParams<LegacyMeta>>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
metadata: { confidence: 0.8, note: 'declared by the consumer type' }
|
|
||||||
})
|
|
||||||
|
|
||||||
// Index-signature metadata types (keyof T = string) remain fully open.
|
|
||||||
assertType<AddParams<Record<string, unknown>>>({
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
data: 'x',
|
|
||||||
metadata: { anything: 'goes', confidence: 0.8 }
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('UpdateParams patch rejects reserved keys but accepts partial custom patches', () => {
|
|
||||||
interface EmployeeMeta {
|
|
||||||
dept: string
|
|
||||||
level: number
|
|
||||||
}
|
|
||||||
|
|
||||||
// Partial patch of the declared shape is legal.
|
|
||||||
assertType<UpdateParams<EmployeeMeta>>({ id: 'e1', metadata: { dept: 'sales' } })
|
|
||||||
// Untyped patch with custom fields is legal.
|
|
||||||
assertType<UpdateParams>({ id: 'e1', metadata: { status: 'reviewed', rating: 4.5 } })
|
|
||||||
|
|
||||||
// @ts-expect-error — 'confidence' is reserved (use the top-level 'confidence' param)
|
|
||||||
assertType<UpdateParams>({ id: 'e1', metadata: { confidence: 0.33 } })
|
|
||||||
// @ts-expect-error — 'subtype' is reserved (use the top-level 'subtype' param)
|
|
||||||
assertType<UpdateParams>({ id: 'e1', metadata: { subtype: 'specialized' } })
|
|
||||||
// @ts-expect-error — '_rev' is reserved (pass 'ifRev' for optimistic concurrency)
|
|
||||||
assertType<UpdateParams>({ id: 'e1', metadata: { _rev: 3 } })
|
|
||||||
// @ts-expect-error — 'confidence' is reserved even when T declares other fields
|
|
||||||
assertType<UpdateParams<EmployeeMeta>>({ id: 'e1', metadata: { confidence: 0.1 } })
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('reserved relationship keys in metadata are compile errors', () => {
|
|
||||||
it('RelateParams rejects reserved keys but stays open for custom edge fields', () => {
|
|
||||||
assertType<RelateParams>({
|
|
||||||
from: 'a',
|
|
||||||
to: 'b',
|
|
||||||
type: VerbType.ReportsTo,
|
|
||||||
subtype: 'direct',
|
|
||||||
metadata: { role: 'peer', since: 2024 }
|
|
||||||
})
|
|
||||||
|
|
||||||
assertType<RelateParams>({
|
|
||||||
from: 'a',
|
|
||||||
to: 'b',
|
|
||||||
type: VerbType.ReportsTo,
|
|
||||||
subtype: 'direct',
|
|
||||||
// @ts-expect-error — 'verb' is reserved (the relationship type travels via the top-level 'type' param)
|
|
||||||
metadata: { verb: 'relatedTo' }
|
|
||||||
})
|
|
||||||
assertType<RelateParams>({
|
|
||||||
from: 'a',
|
|
||||||
to: 'b',
|
|
||||||
type: VerbType.ReportsTo,
|
|
||||||
subtype: 'direct',
|
|
||||||
// @ts-expect-error — 'confidence' is reserved (use the top-level 'confidence' param)
|
|
||||||
metadata: { confidence: 0.9 }
|
|
||||||
})
|
|
||||||
assertType<RelateParams>({
|
|
||||||
from: 'a',
|
|
||||||
to: 'b',
|
|
||||||
type: VerbType.ReportsTo,
|
|
||||||
subtype: 'direct',
|
|
||||||
// @ts-expect-error — 'weight' is reserved (use the top-level 'weight' param)
|
|
||||||
metadata: { weight: 0.4 }
|
|
||||||
})
|
|
||||||
assertType<RelateParams>({
|
|
||||||
from: 'a',
|
|
||||||
to: 'b',
|
|
||||||
type: VerbType.ReportsTo,
|
|
||||||
subtype: 'direct',
|
|
||||||
// @ts-expect-error — 'service' is reserved (use the top-level 'service' param)
|
|
||||||
metadata: { service: 'orders' }
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
it('UpdateRelationParams patch rejects reserved keys', () => {
|
|
||||||
assertType<UpdateRelationParams>({ id: 'r1', metadata: { note: 'fine' } })
|
|
||||||
|
|
||||||
// @ts-expect-error — 'confidence' is reserved (use the top-level 'confidence' param)
|
|
||||||
assertType<UpdateRelationParams>({ id: 'r1', metadata: { confidence: 0.5 } })
|
|
||||||
// @ts-expect-error — 'subtype' is reserved (use the top-level 'subtype' param)
|
|
||||||
assertType<UpdateRelationParams>({ id: 'r1', metadata: { subtype: 'dotted-line' } })
|
|
||||||
// @ts-expect-error — 'createdAt' is reserved (system-managed)
|
|
||||||
assertType<UpdateRelationParams>({ id: 'r1', metadata: { createdAt: 1 } })
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('transact() operations inherit the same guard', () => {
|
|
||||||
it('TxOperation add/update/relate metadata rejects reserved keys', () => {
|
|
||||||
assertType<TxOperation>({
|
|
||||||
op: 'add',
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'tx',
|
|
||||||
metadata: { custom: 'a' }
|
|
||||||
})
|
|
||||||
assertType<TxOperation>({
|
|
||||||
op: 'add',
|
|
||||||
type: NounType.Concept,
|
|
||||||
subtype: 'general',
|
|
||||||
data: 'tx',
|
|
||||||
// @ts-expect-error — 'confidence' is reserved on transact add ops too
|
|
||||||
metadata: { confidence: 0.7 }
|
|
||||||
})
|
|
||||||
assertType<TxOperation>({
|
|
||||||
op: 'update',
|
|
||||||
id: 'e1',
|
|
||||||
// @ts-expect-error — 'weight' is reserved on transact update ops too
|
|
||||||
metadata: { weight: 0.2 }
|
|
||||||
})
|
|
||||||
assertType<TxOperation>({
|
|
||||||
op: 'relate',
|
|
||||||
from: 'a',
|
|
||||||
to: 'b',
|
|
||||||
type: VerbType.RelatedTo,
|
|
||||||
subtype: 'colleague',
|
|
||||||
// @ts-expect-error — 'verb' is reserved on transact relate ops too
|
|
||||||
metadata: { verb: 'contains' }
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
@ -56,11 +56,15 @@ describe('Zero-Config Parameter Validation', () => {
|
||||||
})).toThrow('cannot specify both query and vector')
|
})).toThrow('cannot specify both query and vector')
|
||||||
})
|
})
|
||||||
|
|
||||||
it('should reject both cursor and offset', () => {
|
it('should refuse cursor outright — even paired with offset — as an unimplemented option', () => {
|
||||||
|
// cursor is now a typed, unconditional refusal (UnsupportedFindOptionError):
|
||||||
|
// it used to be accepted-and-ignored, only conflicting when offset was also
|
||||||
|
// given. Accepted-and-ignored died as a class — cursor refuses on its own,
|
||||||
|
// so pairing it with offset refuses too, but with the SAME message.
|
||||||
expect(() => validateFindParams({
|
expect(() => validateFindParams({
|
||||||
cursor: 'abc123',
|
cursor: 'abc123',
|
||||||
offset: 10
|
offset: 10
|
||||||
})).toThrow('cannot use both cursor and offset pagination')
|
})).toThrow("find() option 'cursor' is not implemented")
|
||||||
})
|
})
|
||||||
|
|
||||||
it('should validate vector dimensions', () => {
|
it('should validate vector dimensions', () => {
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue