soulcraftlabs/open-brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 343 Packages Wiki Activity Actions 8

feat(namespace): NO SPECIAL NAMES + storage fidelity — the ruled completion of the field-addressing law
All checks were successful
CI / Node 22 (push) Successful in 12m9s
Details
CI / Node 24 (push) Successful in 12m4s
Details
CI / Bun (latest) (push) Successful in 12m52s
Details

Browse source 
The write side of the law, ruled 2026-08-03: data is either in main space
where developers can use anything, or it is in system.*.

- The reserved-name write door DIES: add/update/relate/updateRelation
  metadata bags accept EVERY name (confidence, type, id, data, level,
  content, ...) as ordinary user fields — indexed, filterable, sortable,
  aggregatable, identical to any other field. The remap/enforce/warn
  machinery, the reservedFieldPolicy config (now a typed init refusal),
  and the compile-time metadata key bans are all removed. The one write
  refusal left: keys spelled 'system.*' (namespace forgery), now enforced
  on all four write doors.
- STORED RECORDS GO NESTED (v2): engine fields top-level, the user bag
  nested verbatim under 'metadata', sealed by a format stamp — by-name
  storage discrimination is unsound once colliders are admitted. Legacy
  flat records stay readable forever through the shape-aware splitters
  (sound for them: the old door refused colliders). Time travel rides the
  same split (generation store snapshots whole records).
- Name-based index exclusions DIE: user frame indexes every name; the
  excludeFields/indexedFields knobs and their silent-[] holes are gone;
  bulk-payload protection is value-shape only, uniform across names.
- Consumer-sweep findings fixed in the same wave: per-type counts read
  the frozen 'system.type' column (addToIndex sort, affinity tracking,
  cold-count rehydration, VFS type bitmaps — legacy 'noun' fallback for
  pre-rebuild reads); resolveHiddenIds addresses 'system.visibility'
  (bare 'visibility' was a silent no-op under the law — VFS/system
  entities leaked into default reads).
- Fidelity fallout fixed in the owning layers: readEntityFieldAddress
  reads the bag first (colliders were absent-shadowed by its own guard)
  and never serves system addresses from the bag; blob history refs read
  the bag shape-aware; migration transforms now receive ONE normalized
  view (engine fields + nested bag) regardless of stored era, and stray
  flat-habit keys refuse with the fix in the message.
- THE REOPEN-COLLIDER CONFORMANCE CASE (required before any RC counts as
  gates-green): all ten collider names + plumbing names written as user
  fields, verified verbatim + queryable across live reads, flush+reopen,
  a forced epoch rebuild, and asOf time travel; relation mirror; forgery
  refusals; legacy flat-record compat. 8/8 green.

Gates: unit 1901/1901 (exit 0) · integration 758 (exit 0) · conformance
27/27 (exit 0) · consumer test sweep migrated (10 files).
This commit is contained in:
David Snelling 2026-08-03 16:59:13 -07:00
parent 48a6130a50
commit 24bf6cdbc5
32 changed files with 1355 additions and 1905 deletions
Download patch file Download diff file Expand all files Collapse all files

697
src/brainy.ts
View file

@ -148,7 +148,9 @@ import {
import { NounType, VerbType, TypeUtils } from './types/graphTypes.js'
import {
splitNounMetadataRecord,
splitVerbMetadataRecord
splitVerbMetadataRecord,
buildNounMetadataRecord,
buildVerbMetadataRecord
} from './types/reservedFields.js'
import { BrainyInterface } from './types/brainyInterface.js'
import type { IntegrationHub, IntegrationHubConfig } from './integrations/core/IntegrationHub.js'
@ -746,6 +748,21 @@ export class Brainy<T = any> implements BrainyInterface<T> {
private lazyRebuildPromise: Promise<void> | null = null
constructor(config?: BrainyConfig) {
// The reserved-field write policy died with the field-addressing law:
// every metadata name is the user's now (engine scalars write via their
// dedicated params and read at `system.*`), so there is nothing left for
// the policy to govern. A config still passing it refuses loudly rather
// than being silently ignored.
if (config && 'reservedFieldPolicy' in (config as Record<string, unknown>)) {
throw new Error(
`reservedFieldPolicy was removed by the field-addressing law: metadata field ` +
`names are never reserved anymore — every name in the metadata bag is the ` +
`user's and works like any other field. Set engine scalars via their ` +
`dedicated params (confidence, weight, subtype, …) and query them as ` +
`system.<field>. Remove the reservedFieldPolicy option.`
)
}
// Normalize configuration with defaults
this.config = this.normalizeConfig(config)
@ -2018,12 +2035,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Zero-config validation (static import for performance)
validateAddParams(params)
// Reserved fields arriving via the metadata bag (untyped callers — the
// compile-time guard stops TypeScript callers) are normalized to their
// canonical top-level location BEFORE any enforcement runs, so a
// remapped subtype participates in subtype-pairing enforcement and the
// indexed metadata bag carries only custom fields.
params = this.remapReservedAddMetadata(params)
// Tracked-field vocabulary enforcement (Layer 2). Walks both bags so a
// tracked field declared at top level (e.g. 'subtype') and one declared in
@ -2095,28 +2106,33 @@ export class Brainy<T = any> implements BrainyInterface<T> {
)
}
// Prepare metadata for storage
// data is stored opaquely in the 'data' field - NOT spread into top-level metadata.
// Only metadata fields are queryable via find({ where }).
const storageMetadata = {
...params.metadata,
// Preserve the caller's original (non-UUID) id when normalized, so reads
// can surface it. A real UUID passes through with no _originalId.
...(originalId !== undefined && { [ORIGINAL_ID_KEY]: originalId }),
data: params.data,
noun: params.type,
...(params.subtype !== undefined && { subtype: params.subtype }),
// visibility: stored only when not 'public' (absent === public, keeps records lean)
...(params.visibility !== undefined &&
params.visibility !== 'public' && { visibility: params.visibility }),
service: params.service,
createdAt: Date.now(),
updatedAt: Date.now(),
_rev: 1,
...(params.confidence !== undefined && { confidence: params.confidence }),
...(params.weight !== undefined && { weight: params.weight }),
...(params.createdBy && { createdBy: params.createdBy })
}
// Prepare metadata for storage: a v2 nested-bag record — engine fields
// top-level, the user's bag nested VERBATIM (any name, including engine
// spellings like `confidence` or `type`, is the user's and survives
// faithfully; the field-addressing law).
const storageMetadata = buildNounMetadataRecord(
{
data: params.data,
noun: params.type,
...(params.subtype !== undefined && { subtype: params.subtype }),
// visibility: stored only when not 'public' (absent === public, keeps records lean)
...(params.visibility !== undefined &&
params.visibility !== 'public' && { visibility: params.visibility }),
service: params.service,
createdAt: Date.now(),
updatedAt: Date.now(),
_rev: 1,
...(params.confidence !== undefined && { confidence: params.confidence }),
...(params.weight !== undefined && { weight: params.weight }),
...(params.createdBy && { createdBy: params.createdBy })
},
{
...params.metadata,
// Preserve the caller's original (non-UUID) id when normalized, so reads
// can surface it. A real UUID passes through with no _originalId.
...(originalId !== undefined && { [ORIGINAL_ID_KEY]: originalId })
}
)
// Build entity structure for indexing (NEW - with top-level fields)
// Optional fields must use conditional spreading to match storageMetadata exactly.
@ -2627,320 +2643,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
return entity
}
/** One-shot registry for reserved-field warnings (per process, per method+field). */
private static warnedReservedFields = new Set<string>()
/**
* @description Resolve the human-readable "correct write path" guidance for a
* reserved field on a given write method. Single source of truth shared by the
* `'throw'` (Error message) and `'warn'` (one-shot warning) paths so the two
* never drift. The trio `confidence` / `weight` / `subtype` and the
* add()/relate()-time fields `service` / `createdBy` / `visibility` map to a
* dedicated param; everything else is system-managed.
* @param method - The public write method the bag arrived through.
* @param field - The reserved field name found in the metadata bag.
* @returns Guidance naming the correct way to set the field.
*/
private reservedWritePath(
method: 'add' | 'update' | 'relate' | 'updateRelation',
field: string
): string {
const typeParam = "the top-level 'type' param"
switch (field) {
case 'noun':
case 'verb':
return typeParam
case 'data':
return "the top-level 'data' param"
case 'confidence':
return "the 'confidence' param"
case 'weight':
return "the 'weight' param"
case 'subtype':
return "the 'subtype' param"
case 'visibility':
return "the 'visibility' param ('public' | 'internal')"
case 'service':
return method === 'add'
? "the 'service' param of add()"
: method === 'relate'
? "the 'service' param of relate()"
: 'nothing — service is fixed at create time'
case 'createdBy':
return method === 'add'
? "the 'createdBy' param of add()"
: 'nothing — createdBy is system-managed'
case 'createdAt':
return 'nothing — creation time is set automatically'
case 'updatedAt':
return 'nothing — set automatically on every write'
case '_rev':
return method === 'update'
? "the 'ifRev' param for optimistic concurrency"
: 'nothing — revisions are system-managed'
default:
return 'a dedicated top-level param'
}
}
/**
* @description Enforce {@link BrainyConfig.reservedFieldPolicy} for reserved
* fields found inside a metadata bag. Called by every write-path remap once
* the bag has been split and at least one reserved key is present.
*
* - `'throw'` (default): throw a clear Error naming every offending key and
* its correct write path. The caller never reaches the remap.
* - `'warn'`: emit a ONE-SHOT (per method+field, per process) warning for
* EVERY reserved key found — both the user-mutable fields that are about to
* be remapped and the system-managed fields that are about to be dropped —
* then fall through to the legacy remap.
* - `'remap'`: silent legacy remap, no warning.
*
* @param method - The public write method the bag arrived through.
* @param reserved - The reserved half of the split metadata bag (non-empty).
* @param reservedListName - `'RESERVED_ENTITY_FIELDS'` or
* `'RESERVED_RELATION_FIELDS'` — named in the thrown Error for discoverability.
* @returns `true` when the caller should proceed with the legacy remap
* (`'warn'` / `'remap'`); `'throw'` never returns (it throws first).
* @throws {Error} When the policy is `'throw'` and any reserved key is present.
*/
private enforceReservedPolicy(
method: 'add' | 'update' | 'relate' | 'updateRelation',
reserved: Partial<Record<string, unknown>>,
reservedListName: 'RESERVED_ENTITY_FIELDS' | 'RESERVED_RELATION_FIELDS'
): boolean {
const policy = this.config.reservedFieldPolicy ?? 'throw'
const keys = Object.keys(reserved)
if (keys.length === 0) return true
if (policy === 'throw') {
const detail = keys
.map((k) => {
const path = this.reservedWritePath(method, k)
// System-managed fields resolve to a "nothing — …" sentinel; phrase
// those as "is system-managed" rather than "pass it as the nothing".
return path.startsWith('nothing')
? `metadata.${k} is a reserved field (${path.replace(/^nothing\s*—\s*/, '')}) and cannot be set through ${method}()`
: `metadata.${k} is a reserved field — pass it as ${path} to ${method}()`
})
.join('; ')
throw new Error(
`${detail} (reserved: see ${reservedListName}). ` +
`Set reservedFieldPolicy:'remap' to opt into legacy remapping, ` +
`or reservedFieldPolicy:'warn' to remap with a warning.`
)
}
if (policy === 'warn') {
// One-shot warning for EVERY reserved key (today only system-managed ones
// warn — this closes that gap so user-mutable remaps are visible too).
for (const k of keys) {
this.warnReservedRemapped(method, k, this.reservedWritePath(method, k))
}
}
// 'warn' and 'remap' both fall through to the legacy remap.
return true
}
/**
* @description One-shot (per method+field, per process) warning that a
* reserved field arrived inside a metadata bag under the `'warn'` policy. The
* wording is neutral on "remapped vs dropped" — `reservedWritePath()` already
* tells the caller where the value goes (a dedicated param, or "nothing").
* @param method - The public write method the bag arrived through.
* @param field - The reserved field name found in the bag.
* @param rightPath - Guidance naming the correct write path.
*/
private warnReservedRemapped(method: string, field: string, rightPath: string): void {
const key = `${method}:${field}`
if (Brainy.warnedReservedFields.has(key)) return
Brainy.warnedReservedFields.add(key)
// System-managed fields resolve to a "nothing — …" sentinel; phrase the
// guidance so it reads cleanly in both the remapped and dropped cases.
const guidance = rightPath.startsWith('nothing')
? `it is ${rightPath.replace(/^nothing\s*—\s*/, '')} and was dropped`
: `set it via ${rightPath} instead`
prodLog.warn(
`[brainy] ${method}(): '${field}' is a reserved field and was found inside the ` +
`metadata bag — ${guidance}. (Legacy remap applied because ` +
`reservedFieldPolicy is 'warn'. This warning is shown once per field per process.)`
)
}
/**
* @description Normalize an `add()` params object with respect to
* Brainy-reserved fields arriving inside `metadata` (untyped callers only —
* the compile-time guard on `AddParams.metadata` stops TypeScript callers).
* Governed by {@link BrainyConfig.reservedFieldPolicy} (default `'throw'`):
* `'throw'` rejects the write naming the offending key(s); `'warn'`/`'remap'`
* fall through to the legacy remap, where fields with a dedicated `add()`
* param (`confidence`, `weight`, `subtype`, `visibility`, `service`,
* `createdBy`) are remapped to that param unless the caller also passed it
* explicitly (top-level wins) and system-managed fields (`noun`, `data`,
* `createdAt`, `updatedAt`, `_rev`) are dropped. A remapped `subtype` flows
* through subtype-pairing enforcement exactly like a top-level one.
* @param params - The caller's add params (not mutated).
* @returns Params with reserved fields normalized out of `metadata`.
* @throws {Error} When `reservedFieldPolicy` is `'throw'` and the bag carries a reserved key.
*/
private remapReservedAddMetadata(params: AddParams<T>): AddParams<T> {
const bag = params.metadata as Record<string, unknown> | undefined
if (!bag || typeof bag !== 'object') return params
const { reserved, custom } = splitNounMetadataRecord(bag)
if (Object.keys(reserved).length === 0) return params
// Policy gate: 'throw' (default) throws here; 'warn' warns once per key then
// remaps; 'remap' silently remaps. (Throw never returns.)
this.enforceReservedPolicy('add', reserved, 'RESERVED_ENTITY_FIELDS')
const createdBy = reserved.createdBy as { augmentation?: unknown; version?: unknown } | undefined
const createdByValid =
typeof createdBy === 'object' &&
createdBy !== null &&
typeof createdBy.augmentation === 'string' &&
typeof createdBy.version === 'string'
return {
...params,
metadata: custom as AddParams<T>['metadata'],
...(params.confidence === undefined &&
typeof reserved.confidence === 'number' && { confidence: reserved.confidence }),
...(params.weight === undefined &&
typeof reserved.weight === 'number' && { weight: reserved.weight }),
...(params.subtype === undefined &&
typeof reserved.subtype === 'string' && { subtype: reserved.subtype }),
...(params.visibility === undefined &&
(reserved.visibility === 'public' || reserved.visibility === 'internal') && {
visibility: reserved.visibility as 'public' | 'internal'
}),
...(params.service === undefined &&
typeof reserved.service === 'string' && { service: reserved.service }),
...(params.createdBy === undefined &&
createdByValid && { createdBy: createdBy as { augmentation: string; version: string } })
}
}
/**
* @description Normalize an `update()` params object with respect to
* Brainy-reserved fields arriving inside the metadata patch — the `update()`
* mirror of {@link remapReservedAddMetadata}, closing the historical trap
* where `add({metadata:{confidence}})` lifted the field but
* `update({metadata:{confidence}})` silently dropped it (the patch value
* survived the merge and was then clobbered by the preserve-existing
* spread; a production consumer's confidence-evolution writes no-oped until
* read back). Governed by {@link BrainyConfig.reservedFieldPolicy} (default
* `'throw'`): `'throw'` rejects the write; `'warn'`/`'remap'` remap
* user-mutable fields (`confidence`, `weight`, `subtype`) to their dedicated
* param unless the caller also passed it (top-level wins) and drop everything
* else (`noun`, `data`, `createdAt`, `updatedAt`, `service`, `createdBy`,
* `_rev`) as system-managed or fixed at `add()` time.
* @param params - The caller's update params (not mutated).
* @returns Params with reserved fields normalized out of `metadata`.
* @throws {Error} When `reservedFieldPolicy` is `'throw'` and the bag carries a reserved key.
*/
private remapReservedUpdateMetadata(params: UpdateParams<T>): UpdateParams<T> {
const bag = params.metadata as Record<string, unknown> | undefined
if (!bag || typeof bag !== 'object') return params
const { reserved, custom } = splitNounMetadataRecord(bag)
if (Object.keys(reserved).length === 0) return params
// Policy gate: 'throw' (default) throws; 'warn' warns once per key then
// remaps; 'remap' silently remaps.
this.enforceReservedPolicy('update', reserved, 'RESERVED_ENTITY_FIELDS')
return {
...params,
metadata: custom as UpdateParams<T>['metadata'],
...(params.confidence === undefined &&
typeof reserved.confidence === 'number' && { confidence: reserved.confidence }),
...(params.weight === undefined &&
typeof reserved.weight === 'number' && { weight: reserved.weight }),
...(params.subtype === undefined &&
typeof reserved.subtype === 'string' && { subtype: reserved.subtype })
}
}
/**
* @description Normalize a `relate()` params object with respect to
* Brainy-reserved fields arriving inside `metadata` — the relationship
* mirror of {@link remapReservedAddMetadata}. Governed by
* {@link BrainyConfig.reservedFieldPolicy} (default `'throw'`): `'throw'`
* rejects the write; `'warn'`/`'remap'` remap fields with a dedicated
* `relate()` param (`confidence`, `weight`, `subtype`, `visibility`,
* `service`) to that param (top-level wins) and drop system-managed fields
* (`verb`, `data`, `createdAt`, `updatedAt`, `createdBy`, `_rev`).
* @param params - The caller's relate params (not mutated).
* @returns Params with reserved fields normalized out of `metadata`.
* @throws {Error} When `reservedFieldPolicy` is `'throw'` and the bag carries a reserved key.
*/
private remapReservedRelateMetadata(params: RelateParams<T>): RelateParams<T> {
const bag = params.metadata as Record<string, unknown> | undefined
if (!bag || typeof bag !== 'object') return params
const { reserved, custom } = splitVerbMetadataRecord(bag)
if (Object.keys(reserved).length === 0) return params
// Policy gate: 'throw' (default) throws; 'warn' warns once per key then
// remaps; 'remap' silently remaps.
this.enforceReservedPolicy('relate', reserved, 'RESERVED_RELATION_FIELDS')
return {
...params,
metadata: custom as RelateParams<T>['metadata'],
...(params.confidence === undefined &&
typeof reserved.confidence === 'number' && { confidence: reserved.confidence }),
...(params.weight === undefined &&
typeof reserved.weight === 'number' && { weight: reserved.weight }),
...(params.subtype === undefined &&
typeof reserved.subtype === 'string' && { subtype: reserved.subtype }),
...(params.visibility === undefined &&
(reserved.visibility === 'public' || reserved.visibility === 'internal') && {
visibility: reserved.visibility as 'public' | 'internal'
}),
...(params.service === undefined &&
typeof reserved.service === 'string' && { service: reserved.service })
}
}
/**
* @description Normalize an `updateRelation()` params object with respect
* to Brainy-reserved fields arriving inside the metadata patch — the
* relationship mirror of {@link remapReservedUpdateMetadata}. Governed by
* {@link BrainyConfig.reservedFieldPolicy} (default `'throw'`): `'throw'`
* rejects the write; `'warn'`/`'remap'` remap user-mutable fields
* (`confidence`, `weight`, `subtype`, `visibility`) to their dedicated param
* (top-level wins) and drop everything else.
* @param params - The caller's update-relation params (not mutated).
* @returns Params with reserved fields normalized out of `metadata`.
* @throws {Error} When `reservedFieldPolicy` is `'throw'` and the bag carries a reserved key.
*/
private remapReservedUpdateRelationMetadata(
params: UpdateRelationParams<T>
): UpdateRelationParams<T> {
const bag = params.metadata as Record<string, unknown> | undefined
if (!bag || typeof bag !== 'object') return params
const { reserved, custom } = splitVerbMetadataRecord(bag)
if (Object.keys(reserved).length === 0) return params
// Policy gate: 'throw' (default) throws; 'warn' warns once per key then
// remaps; 'remap' silently remaps.
this.enforceReservedPolicy('updateRelation', reserved, 'RESERVED_RELATION_FIELDS')
return {
...params,
metadata: custom as UpdateRelationParams<T>['metadata'],
...(params.confidence === undefined &&
typeof reserved.confidence === 'number' && { confidence: reserved.confidence }),
...(params.weight === undefined &&
typeof reserved.weight === 'number' && { weight: reserved.weight }),
...(params.subtype === undefined &&
typeof reserved.subtype === 'string' && { subtype: reserved.subtype }),
...(params.visibility === undefined &&
(reserved.visibility === 'public' || reserved.visibility === 'internal') && {
visibility: reserved.visibility as 'public' | 'internal'
})
}
}
/**
* Update an existing entity
@ -3006,12 +2708,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Reserved fields arriving via the metadata patch are remapped to their
// canonical top-level location, mirroring add()'s lift. Without this the
// patch value survived the merge but was then clobbered by the
// preserve-existing spreads below — a silent no-op consumers could only
// detect by reading values back. User-mutable fields (confidence,
// weight, subtype) remap unless the same field was also passed top-level
// (top-level wins); system-managed fields are dropped with a one-shot
// warning naming the right path.
params = this.remapReservedUpdateMetadata(params)
// Tracked-field vocabulary enforcement (Layer 2). Same as add() — the
// metadata bag carries fields registered via trackField(), and subtype is
@ -3078,31 +2774,33 @@ export class Brainy<T = any> implements BrainyInterface<T> {
? { ...existing.metadata, ...params.metadata }
: params.metadata || existing.metadata
// Prepare updated metadata object
// data is stored opaquely in the 'data' field - NOT spread into top-level metadata.
const updatedMetadata = {
...newMetadata,
data: params.data !== undefined ? params.data : existing.data,
noun: params.type || existing.type,
service: existing.service,
createdAt: existing.createdAt,
updatedAt: Date.now(),
_rev: currentRev + 1,
// Update confidence and weight if provided, otherwise preserve existing
...(params.confidence !== undefined && { confidence: params.confidence }),
...(params.weight !== undefined && { weight: params.weight }),
...(params.confidence === undefined && existing.confidence !== undefined && { confidence: existing.confidence }),
...(params.weight === undefined && existing.weight !== undefined && { weight: existing.weight }),
// Update subtype if provided, otherwise preserve existing
...(params.subtype !== undefined && { subtype: params.subtype }),
...(params.subtype === undefined && existing.subtype !== undefined && { subtype: existing.subtype }),
// Visibility: take the new value if provided, else preserve existing. Stored only
// when the effective value is not 'public' (absent === public, keeps records lean).
// A change to 'public' therefore drops the field entirely.
...(((params.visibility ?? existing.visibility) ?? 'public') !== 'public' && {
visibility: params.visibility ?? existing.visibility
})
}
// Prepare the updated v2 nested-bag record: engine fields top-level,
// the merged user bag nested verbatim (collider names stay the user's).
const updatedMetadata = buildNounMetadataRecord(
{
data: params.data !== undefined ? params.data : existing.data,
noun: params.type || existing.type,
service: existing.service,
createdAt: existing.createdAt,
updatedAt: Date.now(),
_rev: currentRev + 1,
// Update confidence and weight if provided, otherwise preserve existing
...(params.confidence !== undefined && { confidence: params.confidence }),
...(params.weight !== undefined && { weight: params.weight }),
...(params.confidence === undefined && existing.confidence !== undefined && { confidence: existing.confidence }),
...(params.weight === undefined && existing.weight !== undefined && { weight: existing.weight }),
// Update subtype if provided, otherwise preserve existing
...(params.subtype !== undefined && { subtype: params.subtype }),
...(params.subtype === undefined && existing.subtype !== undefined && { subtype: existing.subtype }),
// Visibility: take the new value if provided, else preserve existing. Stored only
// when the effective value is not 'public' (absent === public, keeps records lean).
// A change to 'public' therefore drops the field entirely.
...(((params.visibility ?? existing.visibility) ?? 'public') !== 'public' && {
visibility: params.visibility ?? existing.visibility
})
},
newMetadata as Record<string, unknown>
)
// Build entity structure for metadata index (with top-level fields).
// No `level`: engine plumbing never enters the indexing view (it
@ -4043,9 +3741,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// engine-minted UUID — relation ids are never caller-supplied here.)
params = { ...params, from: resolveEntityId(params.from), to: resolveEntityId(params.to) }
// Reserved fields arriving via the metadata bag are normalized to their
// canonical top-level params before enforcement — mirror of add()'s lift.
params = this.remapReservedRelateMetadata(params)
// Subtype pairing enforcement (Layer 3 — 7.30.0). Per-type rules registered
// via brain.requireSubtype() compose with the brain-wide strict-mode flag.
@ -4097,25 +3792,28 @@ export class Brainy<T = any> implements BrainyInterface<T> {
(v, i) => (v + toEntity.vector[i]) / 2
)
// Prepare verb metadata
// User metadata spread FIRST, then system fields ALWAYS win (prevents collision)
// Prepare verb metadata: a v2 nested-bag record — engine fields
// top-level, the user's edge bag nested verbatim (any name is the
// user's; the field-addressing law).
// One timestamp for both createdAt and updatedAt so a never-updated edge reports a
// stable updatedAt (=== createdAt) instead of a fresh Date.now() fabricated per read.
const relateTs = Date.now()
const verbMetadata = {
...(params.metadata || {}),
verb: params.type,
...(params.subtype !== undefined && { subtype: params.subtype }),
// visibility: stored only when not 'public' (absent === public, keeps records lean)
...(params.visibility !== undefined &&
params.visibility !== 'public' && { visibility: params.visibility }),
weight: params.weight ?? 1.0,
...(params.confidence !== undefined && { confidence: params.confidence }),
...(params.service !== undefined && { service: params.service }),
createdAt: relateTs,
updatedAt: relateTs,
...(params.data !== undefined && { data: params.data })
}
const verbMetadata = buildVerbMetadataRecord(
{
verb: params.type,
...(params.subtype !== undefined && { subtype: params.subtype }),
// visibility: stored only when not 'public' (absent === public, keeps records lean)
...(params.visibility !== undefined &&
params.visibility !== 'public' && { visibility: params.visibility }),
weight: params.weight ?? 1.0,
...(params.confidence !== undefined && { confidence: params.confidence }),
...(params.service !== undefined && { service: params.service }),
createdAt: relateTs,
updatedAt: relateTs,
...(params.data !== undefined && { data: params.data })
},
(params.metadata as Record<string, unknown>) || {}
)
// Save to storage (vector and metadata separately)
const verb: GraphVerb = {
@ -4347,9 +4045,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
validateUpdateRelationParams(params)
// Reserved fields arriving via the metadata patch are remapped to their
// canonical top-level params — mirror of update()'s normalization.
params = this.remapReservedUpdateRelationMetadata(params)
const existing = await this.storage.getVerb(params.id)
if (!existing) {
@ -4378,32 +4073,36 @@ export class Brainy<T = any> implements BrainyInterface<T> {
? { ...(existingRec.metadata || {}), ...(params.metadata || {}) }
: params.metadata || existingRec.metadata
// Build updated stored metadata. System fields ALWAYS win — same shape as relate().
const updatedMetadata = {
...newMetadata,
verb: newVerbType,
...(params.subtype !== undefined
? { subtype: params.subtype }
: existingRec.subtype !== undefined && { subtype: existingRec.subtype }),
// Visibility: new value if provided, else preserve existing; stored only when the
// effective value is not 'public' (a change to 'public' drops the field).
...(((params.visibility ?? existingRec.visibility) ?? 'public') !== 'public' && {
visibility: params.visibility ?? existingRec.visibility
}),
weight: params.weight ?? existingRec.weight ?? 1.0,
...(params.confidence !== undefined
? { confidence: params.confidence }
: existingRec.confidence !== undefined && { confidence: existingRec.confidence }),
// service/createdBy are fixed at relate() time — always carried forward
// (omitting them here silently erased them on every updateRelation()).
...(existingRec.service !== undefined && { service: existingRec.service }),
...(existingRec.createdBy !== undefined && { createdBy: existingRec.createdBy }),
createdAt: existingRec.createdAt,
updatedAt: Date.now(),
...(params.data !== undefined
? { data: params.data }
: existingRec.data !== undefined && { data: existingRec.data })
}
// Build the updated stored record: v2 nested-bag — engine fields
// top-level, the merged user bag nested verbatim (mirror of update()).
const updatedWeight = params.weight ?? existingRec.weight ?? 1.0
const updatedData =
params.data !== undefined ? params.data : existingRec.data
const updatedMetadata = buildVerbMetadataRecord(
{
verb: newVerbType,
...(params.subtype !== undefined
? { subtype: params.subtype }
: existingRec.subtype !== undefined && { subtype: existingRec.subtype }),
// Visibility: new value if provided, else preserve existing; stored only when the
// effective value is not 'public' (a change to 'public' drops the field).
...(((params.visibility ?? existingRec.visibility) ?? 'public') !== 'public' && {
visibility: params.visibility ?? existingRec.visibility
}),
weight: updatedWeight,
...(params.confidence !== undefined
? { confidence: params.confidence }
: existingRec.confidence !== undefined && { confidence: existingRec.confidence }),
// service/createdBy are fixed at relate() time — always carried forward
// (omitting them here silently erased them on every updateRelation()).
...(existingRec.service !== undefined && { service: existingRec.service }),
...(existingRec.createdBy !== undefined && { createdBy: existingRec.createdBy }),
createdAt: existingRec.createdAt,
updatedAt: Date.now(),
...(updatedData !== undefined && { data: updatedData })
},
newMetadata as Record<string, unknown>
)
// Build the verb view used by the graph index — top-level fields mirror relate()'s.
const verbForIndex: GraphVerb = {
@ -4419,9 +4118,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
...(((params.visibility ?? existingRec.visibility) ?? 'public') !== 'public' && {
visibility: params.visibility ?? existingRec.visibility
}),
weight: updatedMetadata.weight,
weight: updatedWeight,
metadata: newMetadata,
data: updatedMetadata.data,
data: updatedData,
createdAt: existingRec.createdAt
}
@ -6027,8 +5726,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
): Promise<Set<string>> {
const excluded = this.excludedVisibilityTiers(params)
if (!excluded) return new Set()
// 'system.visibility' — the engine scalar's frozen address. A bare
// 'visibility' key would address the USER's metadata bag under the
// field-addressing law and silently hide nothing (VFS/system entities
// would leak into every default read).
const ids = await this.metadataIndex.getIdsForFilter({
visibility: excluded.length === 1 ? excluded[0] : { oneOf: excluded }
'system.visibility': excluded.length === 1 ? excluded[0] : { oneOf: excluded }
})
return new Set(ids)
}
@ -9282,10 +8985,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
): Promise<string> {
const { op: _discriminator, ...rawParams } = op
validateAddParams(rawParams as AddParams<T>)
// Same reserved-field normalization as add() — the metadata bag is
// cleaned BEFORE enforcement so a remapped subtype participates in
// subtype-pairing enforcement and only custom fields reach the index.
const params = this.remapReservedAddMetadata(rawParams as AddParams<T>)
const params = rawParams as AddParams<T>
this.enforceTrackedFieldValues(params.metadata as Record<string, unknown> | undefined, 'metadata')
this.enforceTrackedFieldValues({ subtype: params.subtype } as Record<string, unknown>, 'top-level')
this.enforceSubtypeOnAdd('add', params.type, params.subtype, params.metadata)
@ -9360,25 +9060,31 @@ export class Brainy<T = any> implements BrainyInterface<T> {
plan.createdNouns.add(id)
const now = Date.now()
const storageMetadata = {
...params.metadata,
// Preserve the caller's original (non-UUID) id when normalized — mirror
// of add(). A real UUID passes through with no _originalId.
...(originalId !== undefined && { [ORIGINAL_ID_KEY]: originalId }),
data: params.data,
noun: params.type,
...(params.subtype !== undefined && { subtype: params.subtype }),
// visibility: stored only when not 'public' (absent === public, keeps records lean)
...(params.visibility !== undefined &&
params.visibility !== 'public' && { visibility: params.visibility }),
service: params.service,
createdAt: now,
updatedAt: now,
_rev: 1,
...(params.confidence !== undefined && { confidence: params.confidence }),
...(params.weight !== undefined && { weight: params.weight }),
...(params.createdBy && { createdBy: params.createdBy })
}
// v2 nested-bag record — mirror of add(): engine fields top-level, the
// user's bag nested verbatim (collider names stay the user's).
const storageMetadata = buildNounMetadataRecord(
{
data: params.data,
noun: params.type,
...(params.subtype !== undefined && { subtype: params.subtype }),
// visibility: stored only when not 'public' (absent === public, keeps records lean)
...(params.visibility !== undefined &&
params.visibility !== 'public' && { visibility: params.visibility }),
service: params.service,
createdAt: now,
updatedAt: now,
_rev: 1,
...(params.confidence !== undefined && { confidence: params.confidence }),
...(params.weight !== undefined && { weight: params.weight }),
...(params.createdBy && { createdBy: params.createdBy })
},
{
...params.metadata,
// Preserve the caller's original (non-UUID) id when normalized — mirror
// of add(). A real UUID passes through with no _originalId.
...(originalId !== undefined && { [ORIGINAL_ID_KEY]: originalId })
}
)
const entityForIndexing = {
id,
vector,
@ -9441,10 +9147,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
): Promise<string> {
const { op: _discriminator, ...rawParams } = op
validateUpdateParams(rawParams as UpdateParams<T>)
// Same reserved-field normalization as update() — user-mutable fields
// remap to their dedicated param (top-level wins), system-managed fields
// drop with a one-shot warning.
const params = this.remapReservedUpdateMetadata(rawParams as UpdateParams<T>)
const params = rawParams as UpdateParams<T>
// Id normalization (8.0) — mirror of update(): a natural key resolves to the
// canonical UUID add() stored. A real UUID passes through.
params.id = resolveEntityId(params.id)
@ -9496,29 +9199,33 @@ export class Brainy<T = any> implements BrainyInterface<T> {
? { ...existing.metadata, ...params.metadata }
: params.metadata || existing.metadata
const now = Date.now()
const updatedMetadata = {
...newMetadata,
data: params.data !== undefined ? params.data : existing.data,
noun: params.type || existing.type,
service: existing.service,
createdAt: existing.createdAt,
updatedAt: now,
_rev: currentRev + 1,
...(params.confidence !== undefined && { confidence: params.confidence }),
...(params.weight !== undefined && { weight: params.weight }),
...(params.confidence === undefined &&
existing.confidence !== undefined && { confidence: existing.confidence }),
...(params.weight === undefined &&
existing.weight !== undefined && { weight: existing.weight }),
...(params.subtype !== undefined && { subtype: params.subtype }),
...(params.subtype === undefined &&
existing.subtype !== undefined && { subtype: existing.subtype }),
// Visibility: new value if provided, else preserve existing; stored only when the
// effective value is not 'public' (a change to 'public' drops the field).
...(((params.visibility ?? existing.visibility) ?? 'public') !== 'public' && {
visibility: params.visibility ?? existing.visibility
})
}
// v2 nested-bag record — mirror of update(): engine fields top-level,
// the merged user bag nested verbatim.
const updatedMetadata = buildNounMetadataRecord(
{
data: params.data !== undefined ? params.data : existing.data,
noun: params.type || existing.type,
service: existing.service,
createdAt: existing.createdAt,
updatedAt: now,
_rev: currentRev + 1,
...(params.confidence !== undefined && { confidence: params.confidence }),
...(params.weight !== undefined && { weight: params.weight }),
...(params.confidence === undefined &&
existing.confidence !== undefined && { confidence: existing.confidence }),
...(params.weight === undefined &&
existing.weight !== undefined && { weight: existing.weight }),
...(params.subtype !== undefined && { subtype: params.subtype }),
...(params.subtype === undefined &&
existing.subtype !== undefined && { subtype: existing.subtype }),
// Visibility: new value if provided, else preserve existing; stored only when the
// effective value is not 'public' (a change to 'public' drops the field).
...(((params.visibility ?? existing.visibility) ?? 'public') !== 'public' && {
visibility: params.visibility ?? existing.visibility
})
},
newMetadata as Record<string, unknown>
)
// Register for the authoritative under-mutex CAS re-verify + rev re-stamp
// (see PlannedTransact.casUpdates). The staged UpdateNounMetadataOperation
@ -9739,8 +9446,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
): Promise<string> {
const { op: _discriminator, ...rawParams } = op
validateRelateParams(rawParams as RelateParams<T>)
// Same reserved-field normalization as relate().
const params = this.remapReservedRelateMetadata(rawParams as RelateParams<T>)
const params = rawParams as RelateParams<T>
// Id normalization (8.0) — mirror of relate(): resolve BOTH endpoints to the
// canonical UUID add() stored, so a relate op may reference either side by
// natural key. Real UUIDs pass through. (Relationship ids are engine-minted.)
@ -9790,19 +9496,23 @@ export class Brainy<T = any> implements BrainyInterface<T> {
const id = uuidv4()
const relationVector = fromEntity.vector.map((v, i) => (v + toEntity.vector[i]) / 2)
const now = Date.now()
const verbMetadata = {
...(params.metadata || {}),
verb: params.type,
...(params.subtype !== undefined && { subtype: params.subtype }),
// visibility: stored only when not 'public' (absent === public, keeps records lean)
...(params.visibility !== undefined &&
params.visibility !== 'public' && { visibility: params.visibility }),
weight: params.weight ?? 1.0,
...(params.confidence !== undefined && { confidence: params.confidence }),
...(params.service !== undefined && { service: params.service }),
createdAt: now,
...(params.data !== undefined && { data: params.data })
}
// v2 nested-bag record — mirror of relate(): engine fields top-level,
// the user's edge bag nested verbatim.
const verbMetadata = buildVerbMetadataRecord(
{
verb: params.type,
...(params.subtype !== undefined && { subtype: params.subtype }),
// visibility: stored only when not 'public' (absent === public, keeps records lean)
...(params.visibility !== undefined &&
params.visibility !== 'public' && { visibility: params.visibility }),
weight: params.weight ?? 1.0,
...(params.confidence !== undefined && { confidence: params.confidence }),
...(params.service !== undefined && { service: params.service }),
createdAt: now,
...(params.data !== undefined && { data: params.data })
},
(params.metadata as Record<string, unknown>) || {}
)
const verb: GraphVerb = {
id,
vector: relationVector,
@ -15115,12 +14825,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
requireSubtype: config?.requireSubtype ?? true,
// Multi-process safety
mode: config?.mode ?? 'writer',
force: config?.force ?? false,
// Reserved-field-in-metadata-bag policy (8.0 — no silent failures).
// Default 'throw': an untyped caller that smuggles a reserved key past
// the compile guard gets a loud Error naming the correct write path.
// 'warn' = remap + one-shot warning per key; 'remap' = legacy silent remap.
reservedFieldPolicy: config?.reservedFieldPolicy ?? 'throw'
force: config?.force ?? false
}
}

67
src/db/db.ts
View file

@ -59,10 +59,6 @@ import type {
import type { StorageAdapter } from '../coreTypes.js'
import { exportGraph } from './portableGraph.js'
import type { ExportSelector, ExportOptions, PortableGraph } from './portableGraph.js'
import {
splitNounMetadataRecord,
splitVerbMetadataRecord
} from '../types/reservedFields.js'
import { v4 as uuidv4 } from '../universal/uuid.js'
import { coerceNewEntityId, resolveEntityId, ORIGINAL_ID_KEY } from '../utils/idNormalization.js'
import { EntityNotFoundError } from '../errors/notFound.js'
@ -705,23 +701,15 @@ export class Db<T = any> {
for (const op of ops) {
switch (op.op) {
case 'add': {
// Reserved-field normalization — mirror of the brain.transact()
// write path: user-settable fields lift to their dedicated field
// (top-level wins), system-managed fields drop, and the entity's
// metadata bag carries ONLY custom fields. Speculative views skip
// the one-shot warnings — committing the same ops through
// `brain.transact()` warns on the real write path.
const { reserved, custom } = splitNounMetadataRecord(
op.metadata as Record<string, unknown> | undefined
)
const confidence =
op.confidence ?? (typeof reserved.confidence === 'number' ? reserved.confidence : undefined)
const weight =
op.weight ?? (typeof reserved.weight === 'number' ? reserved.weight : undefined)
const subtype =
op.subtype ?? (typeof reserved.subtype === 'string' ? reserved.subtype : undefined)
const service =
op.service ?? (typeof reserved.service === 'string' ? reserved.service : undefined)
// Field-addressing law: the metadata bag is the user's, VERBATIM —
// no reserved-name lift, no drops. Engine scalars come ONLY from
// their dedicated op fields; a bag field named `confidence` is an
// ordinary user field, exactly as on the committed write path.
const custom = { ...(op.metadata as Record<string, unknown> | undefined) }
const confidence = op.confidence
const weight = op.weight
const subtype = op.subtype
const service = op.service
// Id normalization (8.0) — mirror of the committed transact() add
// path: a natural key coerces to a STABLE UUID (v5), preserving the
@ -759,16 +747,12 @@ export class Db<T = any> {
`with(): entity ${updateId} not found at generation ${this.gen}`
)
}
// Same reserved-field normalization as the committed update path.
const { reserved, custom } = splitNounMetadataRecord(
op.metadata as Record<string, unknown> | undefined
)
const confidence =
op.confidence ?? (typeof reserved.confidence === 'number' ? reserved.confidence : undefined)
const weight =
op.weight ?? (typeof reserved.weight === 'number' ? reserved.weight : undefined)
const subtype =
op.subtype ?? (typeof reserved.subtype === 'string' ? reserved.subtype : undefined)
// Field-addressing law — mirror of the add case: the patch bag is
// the user's verbatim; engine scalars only from dedicated op fields.
const custom = { ...(op.metadata as Record<string, unknown> | undefined) }
const confidence = op.confidence
const weight = op.weight
const subtype = op.subtype
const mergedMetadata =
op.merge !== false
? ({ ...(base.metadata as object), ...custom } as T)
@ -830,19 +814,14 @@ export class Db<T = any> {
}
if (duplicate) break
// Reserved-field normalization — relationship mirror of the add
// op above (and of the committed relate() path).
const { reserved, custom } = splitVerbMetadataRecord(
op.metadata as Record<string, unknown> | undefined
)
const confidence =
op.confidence ?? (typeof reserved.confidence === 'number' ? reserved.confidence : undefined)
const weight =
op.weight ?? (typeof reserved.weight === 'number' ? reserved.weight : undefined)
const subtype =
op.subtype ?? (typeof reserved.subtype === 'string' ? reserved.subtype : undefined)
const service =
op.service ?? (typeof reserved.service === 'string' ? reserved.service : undefined)
// Field-addressing law — relationship mirror of the add case: the
// edge bag is the user's verbatim; engine scalars only from
// dedicated op fields.
const custom = { ...(op.metadata as Record<string, unknown> | undefined) }
const confidence = op.confidence
const weight = op.weight
const subtype = op.subtype
const service = op.service
const id = uuidv4()
overlay.verbs.set(id, {

28
src/db/fieldAddressing.ts
View file

@ -174,23 +174,26 @@ export function readEntityFieldAddress(
: null
if (address.scope === 'system') {
// Entity views carry system scalars top-level; raw storage shapes carry
// them inside the stored metadata record (where `type` is spelled `noun`).
// Read top-level first, then the record — never the user's namespace.
// System scalars live at the record's top level, NEVER in the user's
// bag — a user field named `confidence` must be unreachable from
// system.confidence (and vice versa). Entity views carry the scalars
// top-level directly; record-derived views spell the type `noun`.
const top = rec[address.field]
if (top !== undefined) return top
if (bag) {
if (address.field === 'type') return bag.type ?? bag.noun
return bag[address.field]
}
if (address.field === 'type') return rec.noun
return undefined
}
// User scope. The write-path remap guarantees the user can never OWN a
// field named like a system scalar (those lift top-level at write), so a
// bare system name reads as ABSENT — reading the stored record's reserved
// key here would re-create the shadow this module exists to kill. Same for
// plumbing and the legacy 'noun' spelling.
// User scope: the bag IS the user's namespace, authoritative — EVERY name
// reads from it, engine spellings included (`bag.confidence` is the user's
// confidence field under the field-addressing law).
if (bag) return bag[address.field]
// No bag at all: a LEGACY flat record (pre-nested-bag storage). Its keys
// matching system/plumbing names are the ENGINE's — the pre-law write door
// refused user colliders — so a bare system name reads as ABSENT rather
// than resurrecting the shadow this module exists to kill. Same for the
// legacy 'noun' spelling.
if (
SYSTEM_ENTITY_SCALARS.has(address.field) ||
PLUMBING_FIELDS.has(address.field) ||
@ -198,7 +201,6 @@ export function readEntityFieldAddress(
) {
return undefined
}
if (bag) return bag[address.field]
return rec[address.field]
}

62
src/import/ImportCoordinator.ts
View file

@ -22,7 +22,6 @@ import { SmartYAMLImporter } from '../importers/SmartYAMLImporter.js'
import { SmartDOCXImporter } from '../importers/SmartDOCXImporter.js'
import { VFSStructureGenerator } from '../importers/VFSStructureGenerator.js'
import { NounType, VerbType } from '../types/graphTypes.js'
import { splitNounMetadataRecord, splitVerbMetadataRecord } from '../types/reservedFields.js'
import { v4 as uuidv4 } from '../universal/uuid.js'
import * as fs from 'fs'
import * as path from 'path'
@ -871,35 +870,18 @@ export class ImportCoordinator {
}
/**
* Strip Brainy-reserved entity keys out of an extractor-supplied metadata bag.
*
* Extractors (and consumer `customMetadata`) can carry reserved keys
* (`confidence`, `subtype`, `weight`, …) inside `metadata`. Brainy 8.0's
* default `reservedFieldPolicy` is `'throw'`, so spreading such a bag into
* `add({ metadata })` would reject the whole import. The import pipeline owns
* the correct write path: user-mutable reserved values are passed as dedicated
* `AddParams` params (see the call sites), so here we simply drop the reserved
* half of the bag and keep only the custom fields that belong in `metadata`.
*
* Normalize an extractor/consumer metadata bag for spreading — the
* field-addressing law: the bag is the user's, VERBATIM. No name is
* reserved anymore ('confidence', 'subtype', 'type', … in a source bag
* import as ordinary user fields); the old reserved-key strip was data
* loss under the law and is gone. A forged 'system.'-prefixed key still
* refuses loudly at the write door (`rejectForgedSystemKeys`).
* @param bag - The extractor/consumer metadata bag (may be undefined).
* @returns The custom-only metadata (reserved keys removed).
* @returns The bag itself, or `{}` for non-object inputs.
*/
private stripReservedFromBag(bag: Record<string, any> | undefined | null): Record<string, any> {
private bagVerbatim(bag: Record<string, any> | undefined | null): Record<string, any> {
if (!bag || typeof bag !== 'object') return {}
return splitNounMetadataRecord(bag).custom
}
/**
* Relationship mirror of {@link stripReservedFromBag} — strips reserved verb
* keys (`verb`, `confidence`, `weight`, `subtype`, …) out of an edge metadata
* bag so it carries only custom fields. Reserved values that have a dedicated
* `RelateParams` param are passed there by the call site instead.
* @param bag - The extractor/consumer edge metadata bag (may be undefined).
* @returns The custom-only edge metadata (reserved keys removed).
*/
private stripReservedFromRelationBag(bag: Record<string, any> | undefined | null): Record<string, any> {
if (!bag || typeof bag !== 'object') return {}
return splitVerbMetadataRecord(bag).custom
return bag
}
/**
@ -1017,7 +999,7 @@ export class ImportCoordinator {
importedAt: trackingContext.importedAt,
importFormat: trackingContext.importFormat,
importSource: trackingContext.importSource,
...this.stripReservedFromBag(trackingContext.customMetadata)
...this.bagVerbatim(trackingContext.customMetadata)
})
}
})
@ -1045,13 +1027,11 @@ export class ImportCoordinator {
data: entity.description || entity.name,
type: entity.type,
subtype: entity.subtype ?? options.defaultSubtype ?? 'imported',
// `confidence` is a reserved field — pass it as the dedicated param,
// never inside the metadata bag (8.0 reservedFieldPolicy defaults to 'throw').
// Engine confidence rides its dedicated param; the bag below is
// the user's verbatim (no name is reserved — field-addressing law).
confidence: entity.confidence,
metadata: {
// Extractor/consumer bags may smuggle reserved keys — strip them so
// the bag carries only custom fields.
...this.stripReservedFromBag(entity.metadata),
...this.bagVerbatim(entity.metadata),
name: entity.name,
vfsPath: vfsFile?.path,
importedFrom: 'import-coordinator',
@ -1064,7 +1044,7 @@ export class ImportCoordinator {
importSource: trackingContext.importSource,
sourceRow: row.rowNumber,
sourceSheet: row.sheet,
...this.stripReservedFromBag(trackingContext.customMetadata)
...this.bagVerbatim(trackingContext.customMetadata)
})
}
}
@ -1145,7 +1125,7 @@ export class ImportCoordinator {
importIds: [trackingContext.importId],
projectId: trackingContext.projectId,
importFormat: trackingContext.importFormat,
...this.stripReservedFromRelationBag(trackingContext.customMetadata)
...this.bagVerbatim(trackingContext.customMetadata)
})
}
}
@ -1180,7 +1160,7 @@ export class ImportCoordinator {
confidence: entity.confidence,
metadata: {
// Strip any reserved keys an extractor smuggled into the bag.
...this.stripReservedFromBag(entity.metadata),
...this.bagVerbatim(entity.metadata),
name: entity.name,
vfsPath: vfsFile?.path,
importedFrom: 'import-coordinator',
@ -1194,7 +1174,7 @@ export class ImportCoordinator {
importSource: trackingContext.importSource,
sourceRow: row.rowNumber,
sourceSheet: row.sheet,
...this.stripReservedFromBag(trackingContext.customMetadata)
...this.bagVerbatim(trackingContext.customMetadata)
})
}
})
@ -1234,7 +1214,7 @@ export class ImportCoordinator {
importIds: [trackingContext.importId],
projectId: trackingContext.projectId,
importFormat: trackingContext.importFormat,
...this.stripReservedFromRelationBag(trackingContext.customMetadata)
...this.bagVerbatim(trackingContext.customMetadata)
})
}
})
@ -1289,7 +1269,7 @@ export class ImportCoordinator {
projectId: trackingContext.projectId,
importedAt: trackingContext.importedAt,
importFormat: trackingContext.importFormat,
...this.stripReservedFromBag(trackingContext.customMetadata)
...this.bagVerbatim(trackingContext.customMetadata)
})
}
})
@ -1319,7 +1299,7 @@ export class ImportCoordinator {
projectId: trackingContext.projectId,
importedAt: trackingContext.importedAt,
importFormat: trackingContext.importFormat,
...this.stripReservedFromRelationBag(trackingContext.customMetadata)
...this.bagVerbatim(trackingContext.customMetadata)
})
}
})
@ -1422,7 +1402,7 @@ export class ImportCoordinator {
...(typeof (rel as any).confidence === 'number' && { confidence: (rel as any).confidence }),
...(typeof (rel as any).weight === 'number' && { weight: (rel as any).weight }),
metadata: {
...this.stripReservedFromRelationBag(rel.metadata),
...this.bagVerbatim(rel.metadata),
relationshipType: 'semantic', // Distinguish from VFS/provenance
inferredType: verbType !== rel.type, // Track if type was enhanced
originalType: rel.type

7
src/index.ts
View file

@ -89,7 +89,12 @@ export {
RESERVED_ENTITY_FIELDS,
RESERVED_RELATION_FIELDS,
splitNounMetadataRecord,
splitVerbMetadataRecord
splitVerbMetadataRecord,
buildNounMetadataRecord,
buildVerbMetadataRecord,
isNestedBagRecord,
METADATA_RECORD_FORMAT_KEY,
NESTED_BAG_FORMAT
} from './types/reservedFields.js'
export type {
ReservedEntityField,

95
src/migration/MigrationRunner.ts
View file

@ -9,6 +9,67 @@ import type { BaseStorage } from '../storage/baseStorage.js'
import type { NounMetadata, VerbMetadata } from '../coreTypes.js'
import type { Migration, MigrationState, MigrationPreview, MigrationResult, MigrateOptions, MigrationError } from './types.js'
import { MIGRATIONS } from './migrations.js'
import {
splitNounMetadataRecord,
splitVerbMetadataRecord,
buildNounMetadataRecord,
buildVerbMetadataRecord,
RESERVED_ENTITY_FIELDS,
RESERVED_RELATION_FIELDS
} from '../types/reservedFields.js'
const RESERVED_NOUN_SET: ReadonlySet<string> = new Set(RESERVED_ENTITY_FIELDS)
const RESERVED_VERB_SET: ReadonlySet<string> = new Set(RESERVED_RELATION_FIELDS)
/**
* Normalize a stored record (either era: legacy flat OR v2 nested-bag) into
* THE transform view — the one shape every migration transform receives:
* engine fields top-level, the user's metadata bag nested under `metadata`.
* Transforms never see the storage era; a migration written today works on
* a brain of any age.
*/
function toTransformView(
record: Record<string, unknown>,
kind: 'noun' | 'verb'
): Record<string, unknown> {
const { reserved, custom } =
kind === 'noun' ? splitNounMetadataRecord(record) : splitVerbMetadataRecord(record)
return { ...reserved, metadata: { ...custom } }
}
/**
* Convert a transform's returned view back into a stamped v2 stored record.
* LOUD CONTRACT: user fields belong inside `.metadata` — a stray top-level
* key that is not an engine field is a migration bug under the
* field-addressing law (pre-law transforms wrote user fields flat), and it
* refuses with the fix in the message rather than silently dropping or
* silently storing it as an engine key.
*/
function fromTransformView(
view: Record<string, unknown>,
kind: 'noun' | 'verb'
): Record<string, unknown> {
const reservedSet = kind === 'noun' ? RESERVED_NOUN_SET : RESERVED_VERB_SET
const engine: Record<string, unknown> = {}
for (const [key, value] of Object.entries(view)) {
if (key === 'metadata') continue
if (!reservedSet.has(key)) {
throw new Error(
`migration transform returned a top-level key '${key}' that is not an ` +
`engine field — under the field-addressing law user fields live inside ` +
`.metadata (return { ...view, metadata: { ...view.metadata, ${key}: … } }).`
)
}
engine[key] = value
}
const bag =
view.metadata && typeof view.metadata === 'object' && !Array.isArray(view.metadata)
? (view.metadata as Record<string, unknown>)
: {}
return kind === 'noun'
? buildNounMetadataRecord(engine, bag)
: buildVerbMetadataRecord(engine, bag)
}
const MIGRATION_STATE_KEY = '__migration_state__'
const PREVIEW_SAMPLE_SIZE = 5
@ -125,14 +186,16 @@ export class MigrationRunner {
const entityMeta = metadataBatch.get(entity.id)
if (!entityMeta) continue
const metadata = entityMeta as Record<string, unknown>
const result = this.applyTransforms(metadata, nounMigrations)
// Transforms see THE view (engine fields + nested user bag),
// never the raw storage era.
const view = toTransformView(entityMeta as Record<string, unknown>, 'noun')
const result = this.applyTransforms(view, nounMigrations)
if (result !== null) {
affectedEntities++
if (sampleChanges.length < PREVIEW_SAMPLE_SIZE) {
sampleChanges.push({
id: entity.id,
before: { ...metadata },
before: view,
after: result
})
}
@ -157,14 +220,14 @@ export class MigrationRunner {
const verbMeta = await this.storage.getVerbMetadata(verb.id)
if (!verbMeta) continue
const metadata = verbMeta as Record<string, unknown>
const result = this.applyTransforms(metadata, verbMigrations)
const view = toTransformView(verbMeta as Record<string, unknown>, 'verb')
const result = this.applyTransforms(view, verbMigrations)
if (result !== null) {
affectedEntities++
if (sampleChanges.length < PREVIEW_SAMPLE_SIZE) {
sampleChanges.push({
id: verb.id,
before: { ...metadata },
before: view,
after: result
})
}
@ -289,9 +352,16 @@ export class MigrationRunner {
if (!entityMeta) continue
try {
const transformed = migration.transform(entityMeta as Record<string, unknown>)
const transformed = migration.transform(
toTransformView(entityMeta as Record<string, unknown>, 'noun')
)
if (transformed !== null) {
await this.storage.saveNounMetadata(entity.id, transformed as NounMetadata)
// Re-stamp as a v2 record (also upgrades legacy records touched
// by a migration onto the nested-bag shape).
await this.storage.saveNounMetadata(
entity.id,
fromTransformView(transformed, 'noun') as NounMetadata
)
modified++
}
} catch (err) {
@ -357,9 +427,14 @@ export class MigrationRunner {
if (!metadata) continue
try {
const transformed = migration.transform(metadata as Record<string, unknown>)
const transformed = migration.transform(
toTransformView(metadata as Record<string, unknown>, 'verb')
)
if (transformed !== null) {
await this.storage.saveVerbMetadata(verb.id, transformed as VerbMetadata)
await this.storage.saveVerbMetadata(
verb.id,
fromTransformView(transformed, 'verb') as VerbMetadata
)
modified++
}
} catch (err) {

14
src/migration/types.ts
View file

@ -14,7 +14,19 @@ export interface Migration {
description: string
/** Which entity types this migration applies to */
applies: 'nouns' | 'verbs' | 'both'
/** Return transformed metadata, or null if no change needed */
/**
* Return the transformed record view, or null if no change needed.
*
* THE VIEW CONTRACT (field-addressing law): the transform receives ONE
* normalized shape regardless of how old the stored record is — engine
* fields top-level (`noun`/`verb`, `subtype`, `confidence`, `weight`,
* timestamps, `_rev`, …) and the USER's metadata bag nested under
* `metadata` (where every name is the user's, engine spellings included).
* Return the same shape: user-field changes go inside `.metadata`; a
* stray non-engine top-level key in the returned object refuses loudly
* (it is the pre-law flat habit, and silently guessing its namespace
* would corrupt data).
*/
transform: (metadata: Record<string, unknown>) => Record<string, unknown> | null
}

19
src/neural/neuralImport.ts
View file

@ -7,7 +7,6 @@
import { Brainy } from '../brainy.js'
import { NounType, VerbType } from '../types/graphTypes.js'
import { splitNounMetadataRecord, splitVerbMetadataRecord } from '../types/reservedFields.js'
import * as fs from '../universal/fs.js'
import * as path from '../universal/path.js'
// @ts-ignore
@ -803,12 +802,14 @@ export class NeuralImport {
data: this.extractMainText(entity.originalData),
type: entity.nounType as NounType,
subtype: entity.subtype ?? options.defaultSubtype ?? 'extracted',
// `confidence` is a reserved field — dedicated param, not metadata
// (8.0 reservedFieldPolicy defaults to 'throw').
// Engine confidence rides its dedicated param; the source object
// imports as the user's bag VERBATIM — no name is reserved
// (field-addressing law).
confidence: entity.confidence,
metadata: {
// Strip any reserved keys the source data smuggled into the bag.
...splitNounMetadataRecord(entity.originalData).custom,
...(typeof entity.originalData === 'object' && entity.originalData !== null
? entity.originalData
: {}),
id: entity.suggestedId
}
})
@ -822,11 +823,13 @@ export class NeuralImport {
type: relationship.verbType as VerbType,
subtype: relationship.subtype ?? options.defaultSubtype ?? 'extracted',
weight: relationship.weight,
confidence: relationship.confidence, // reserved field — dedicated param, not metadata
confidence: relationship.confidence, // engine confidence — dedicated param
metadata: {
context: relationship.context,
// Strip any reserved keys smuggled into the edge metadata bag.
...splitVerbMetadataRecord(relationship.metadata).custom
// The edge bag imports verbatim — no name is reserved.
...(typeof relationship.metadata === 'object' && relationship.metadata !== null
? relationship.metadata
: {})
}
})
}

13
src/storage/baseStorage.ts
View file

@ -36,7 +36,8 @@ import { BrainyError, ProtectedArtifactError, DerivedArtifactMissingError } from
import { MetadataWriteBuffer } from '../utils/metadataWriteBuffer.js'
import {
splitNounMetadataRecord,
splitVerbMetadataRecord
splitVerbMetadataRecord,
isNestedBagRecord
} from '../types/reservedFields.js'
/**
@ -1013,8 +1014,14 @@ export abstract class BaseStorage extends BaseStorageAdapter {
const hashes: string[] = []
for (const record of records) {
if (record.kind !== 'noun') continue
const storage = (record.metadata as { storage?: { type?: string; hash?: unknown } } | null)
?.storage
// The VFS blob pointer (`storage: {type:'blob', hash}`) is a USER-bag
// field: in a v2 nested-bag record it lives inside `metadata`, in a
// legacy flat record it sits at the top level — read shape-aware.
const raw = record.metadata as Record<string, unknown> | null
const bag = isNestedBagRecord(raw)
? (raw!.metadata as Record<string, unknown>)
: raw
const storage = (bag as { storage?: { type?: string; hash?: unknown } } | null)?.storage
if (storage?.type === 'blob' && typeof storage.hash === 'string') {
hashes.push(storage.hash)
}

73
src/types/brainy.types.ts
View file

@ -320,15 +320,18 @@ export interface AddParams<T = any> {
*/
visibility?: 'public' | 'internal'
/**
* Structured queryable fields — indexed by MetadataIndex, used in `where` filters.
* Structured queryable fields — indexed by MetadataIndex, used in `where`
* filters, `orderBy`, and aggregation.
*
* Reserved entity fields (`RESERVED_ENTITY_FIELDS` — `noun`, `subtype`, `visibility`,
* `createdAt`, `updatedAt`, `confidence`, `weight`, `service`, `data`, `createdBy`,
* `_rev`) may NOT appear here — they have dedicated top-level params and the type makes
* a literal reserved key a compile error. Untyped (JavaScript) callers that pass one
* anyway are normalized at write time: user-settable fields remap to their top-level
* param (top-level wins when both are supplied), system-managed fields are dropped with
* a one-shot warning.
* THE FIELD-ADDRESSING LAW: every name here is YOURS. There are no
* reserved metadata names — `confidence`, `type`, `id`, `level`, `data`,
* `content`, … are ordinary user fields that index, filter, sort, and
* aggregate like any other, and survive faithfully across restarts and
* rebuilds. Engine scalars are set only via their dedicated params
* (`confidence`, `weight`, `subtype`, …) and are queried explicitly as
* `system.<field>` (`where: { 'system.confidence': … }`). The ONE illegal
* spelling is a key starting `'system.'` — the engine's explicit address
* namespace cannot be forged; such a write refuses with a typed error.
*/
metadata?: EntityMetadataInput<T>
/** Custom entity ID. When omitted, a time-ordered UUID v7 is generated; a supplied natural-key string is normalized to a stable UUID v5. */
@ -386,12 +389,11 @@ export interface UpdateParams<T = any> {
*/
visibility?: EntityVisibility
/**
* Metadata fields to merge (or replace when `merge: false`). Reserved entity
* fields (`RESERVED_ENTITY_FIELDS`) may NOT appear here — `confidence` /
* `weight` / `subtype` / `visibility` have dedicated params on this call, and the rest
* are system-managed. A literal reserved key is a compile error; untyped callers
* are normalized at write time (remap user-settable, drop system-managed
* with a one-shot warning).
* Metadata fields to merge (or replace when `merge: false`). Every name is
* the user's (the field-addressing law) — a patch field named `confidence`
* updates YOUR field of that name, never the engine scalar (use the
* dedicated `confidence` param for that). Keys spelled `'system.…'` refuse
* with a typed error (namespace forgery).
*/
metadata?: EntityMetadataPatch<T>
merge?: boolean // Merge or replace metadata (default: true)
@ -444,11 +446,11 @@ export interface RelateParams<T = any> {
/** Content for the relationship (optional — overrides auto-computed vector) */
data?: any
/**
* Structured queryable fields on the edge. Reserved relationship fields
* (`RESERVED_RELATION_FIELDS` — `verb`, `subtype`, `visibility`, `createdAt`,
* `updatedAt`, `confidence`, `weight`, `service`, `data`, `createdBy`, `_rev`) may NOT
* appear here — they have dedicated params. A literal reserved key is a
* compile error; untyped callers are normalized at write time.
* Structured queryable fields on the edge. Every name is the user's (the
* field-addressing law) — `verb`, `confidence`, `weight`, … in this bag are
* ordinary user fields; engine scalars ride their dedicated params and are
* addressed as `system.<field>`. Keys spelled `'system.…'` refuse with a
* typed error (namespace forgery).
*/
metadata?: RelationMetadataInput<T>
/** Create reverse edge too (default: false) */
@ -478,10 +480,9 @@ export interface UpdateRelationParams<T = any> {
confidence?: number // New confidence (0-1)
data?: any // New content
/**
* Metadata fields to merge (or replace when `merge: false`). Reserved
* relationship fields (`RESERVED_RELATION_FIELDS`) may NOT appear here —
* a literal reserved key is a compile error; untyped callers are
* normalized at write time.
* Metadata fields to merge (or replace when `merge: false`). Every name is
* the user's (the field-addressing law); engine scalars ride their
* dedicated params. Keys spelled `'system.…'` refuse with a typed error.
*/
metadata?: RelationMetadataPatch<T>
merge?: boolean // Merge or replace metadata
@ -2027,32 +2028,6 @@ export interface BrainyConfig {
*/
force?: boolean
/**
* How write paths react when an untyped (JavaScript) caller smuggles a
* Brainy-reserved field (`RESERVED_ENTITY_FIELDS` / `RESERVED_RELATION_FIELDS`
* — `confidence`, `weight`, `subtype`, `visibility`, `service`, `createdBy`,
* `noun`/`verb`, `data`, `createdAt`, `updatedAt`, `_rev`) **inside the
* `metadata` bag** of `add()` / `update()` / `relate()` / `updateRelation()`
* (and their `transact()` / `with()` mirrors). TypeScript callers can't write
* these shapes at all — the compile-time guard on the metadata param types
* (`NoReservedEntityKeys` / `NoReservedRelationKeys`) rejects a literal
* reserved key — so this policy only governs untyped callers that slip one
* past the compiler.
*
* - `'throw'` (**default, 8.0**): a reserved key in the bag throws a clear
* `Error` naming the offending key(s) and the correct write path. No silent
* remap, no data loss, no surprise. This is the 8.0 "no silent failures"
* contract.
* - `'warn'`: legacy remapping with a loud, one-shot (per key, per process)
* warning for EVERY reserved key found — user-mutable fields are remapped to
* their dedicated top-level param (top-level wins when both are supplied),
* system-managed fields are dropped. Use while migrating untyped call sites.
* - `'remap'`: the pre-8.0 silent remapping, no warning. Last-resort
* compatibility hatch for code that intentionally relies on the bag path.
*
* @default 'throw'
*/
reservedFieldPolicy?: 'throw' | 'warn' | 'remap'
}
// ============= Neural API Types =============

264
src/types/reservedFields.ts
View file

@ -1,35 +1,54 @@
/**
* @module types/reservedFields
* @description The canonical reserved-field contract — ONE place that defines
* which keys belong to Brainy (top-level entity/relationship fields) and may
* therefore never live inside a `metadata` bag.
* @description The stored-record layout contract — ONE place that defines
* which keys of a persisted metadata record belong to the ENGINE (top-level
* entity/relationship fields) and how the USER's metadata bag is kept apart
* from them, faithfully, across flush / reopen / rebuild / time travel.
*
* Three layers enforce the contract, all driven by the constants below:
* THE FIELD-ADDRESSING LAW (ruled 2026-08-03, VENUE-BRAINY-ORDERBY-NOOP):
* data is either in main space — where developers can use ANY name, and it
* all works with every database function — or it is in `system.*`. There are
* NO reserved user-facing metadata names anymore: `confidence`, `type`,
* `level`, `data`, `id`, `content` … inside a metadata bag are ordinary user
* fields. The only refused write is a user metadata key literally starting
* with `'system.'` (namespace forgery — see `rejectForgedSystemKeys`).
*
* 1. **Compile time** — `AddParams.metadata`, `UpdateParams.metadata`,
* `RelateParams.metadata` and `UpdateRelationParams.metadata` are typed so
* a literal reserved key is a TypeScript error (see
* {@link EntityMetadataInput} / {@link RelationMetadataInput}).
* 2. **Write time** — for untyped (JavaScript) callers that smuggle a
* reserved key past the compiler anyway, every write path normalizes the
* bag: user-mutable fields are remapped to their dedicated top-level
* param (top-level wins when both are supplied) and system-managed fields
* are dropped with a one-shot warning naming the correct write path.
* 3. **Read time** — every read path splits the stored flat record through
* {@link splitNounMetadataRecord} / {@link splitVerbMetadataRecord}, so a
* reserved field is surfaced ONLY at top level and `entity.metadata` /
* `relation.metadata` contain ONLY custom fields, always — live reads,
* batch reads, and historical (`asOf`) reads alike.
* That law makes name-based storage discrimination unsound for NEW records
* (a user field named `confidence` may now legally sit beside the engine's
* confidence scalar), so persisted metadata records carry the user bag
* NESTED, shape-discriminated by a format stamp:
*
* Documented for consumers in `docs/concepts/consistency-model.md`
* ("Reserved fields").
* - **v2 (nested-bag)** — `{ …engine fields…, [METADATA_RECORD_FORMAT_KEY]:
* NESTED_BAG_FORMAT, metadata: { …user bag, verbatim… } }`. Built ONLY by
* {@link buildNounMetadataRecord} / {@link buildVerbMetadataRecord}; the
* engine half and the user bag can never collide because they never share
* a level.
* - **legacy (flat)** — engine fields and user fields mixed at one level,
* discriminated BY NAME through the RESERVED_* lists. Sound for legacy
* records precisely because the pre-law write door REFUSED user metadata
* carrying those names — a flat key matching a reserved name IS the
* engine's value in any record the old door admitted.
*
* {@link splitNounMetadataRecord} / {@link splitVerbMetadataRecord} read
* BOTH shapes (stamp first, name split as the legacy fallback) and are the
* single read-side choke point for live, batch, AND historical (`asOf`)
* reads — the generation store snapshots whole records, so time travel
* rides the same split.
*
* The RESERVED_* lists therefore no longer describe a user-facing ban — they
* describe the ENGINE HALF of the stored record layout (and drive the legacy
* split). The write-door remap machinery and the compile-time metadata key
* bans that used to enforce the old contract are gone.
*/
/**
* @description Entity (noun) field names reserved by Brainy. These keys are
* stored in the flat per-entity metadata record alongside custom fields, but
* they belong to Brainy: every read path extracts them to top-level
* `Entity` fields, and no write path accepts them inside `metadata`.
* @description Entity (noun) field names owned by the ENGINE in a stored
* metadata record. In v2 (nested-bag) records these are the legal TOP-LEVEL
* keys beside the nested `metadata` bag; in legacy flat records they drive
* the by-name split. They are NOT a user-facing ban list: since the
* field-addressing law, a user metadata field may carry any of these names
* and remains the user's — it lives inside the nested bag, never at the
* record's top level.
*
* | Key | Canonical write path |
* |-----|----------------------|
@ -119,68 +138,54 @@ export type ReservedRelationField = (typeof RESERVED_RELATION_FIELDS)[number]
type IsAny<T> = 0 extends 1 & T ? true : false
/**
* @description Compile-time tripwire: marks every reserved entity key as
* `never` so an object literal carrying one fails to type-check. Keys that
* `T` itself declares (including via an index signature, where
* `keyof T = string`) are exempted — a consumer who *explicitly* types a
* reserved key into their metadata shape keeps a working (if unwise) type,
* and index-signature metadata types remain assignable.
* @deprecated The compile-time reserved-key ban died with the
* field-addressing law: every name is legal user metadata now. Kept as an
* empty (no-op) guard so external type references keep compiling; it bans
* nothing.
*/
export type NoReservedEntityKeys<T> = {
readonly [K in ReservedEntityField as K extends keyof T ? never : K]?: never
}
export type NoReservedEntityKeys<T> = unknown
/**
* @description Relationship mirror of {@link NoReservedEntityKeys}.
* @deprecated Relationship mirror of {@link NoReservedEntityKeys} — no-op
* for the same reason.
*/
export type NoReservedRelationKeys<T> = {
readonly [K in ReservedRelationField as K extends keyof T ? never : K]?: never
}
/**
* @description The metadata bag shape for untyped brains (`T = any`): an
* open index signature (any custom key, any value — exactly the pre-8.0
* latitude) intersected with the reserved-key guard, whose declared
* `?: never` properties take precedence over the index signature so a
* literal reserved key is still a compile error.
*/
type OpenBag<Guard> = { [key: string]: any } & Guard
export type NoReservedRelationKeys<T> = unknown
/**
* @description The type of `AddParams.metadata`: the consumer's metadata
* shape `T` with reserved entity keys forbidden at compile time. For untyped
* brains (`T = any`) the bag stays open ({@link OpenBag}), so arbitrary
* custom fields remain legal while literal reserved keys still error.
* shape `T`, open. Under the field-addressing law EVERY key is a legal user
* field (engine scalars are written only via their dedicated params and read
* at `system.*`), so no name is banned at compile time. The one illegal
* spelling — a key starting `'system.'` — cannot be expressed as a mapped
* type ban and is refused at runtime (`rejectForgedSystemKeys`).
*/
export type EntityMetadataInput<T> = IsAny<T> extends true
? OpenBag<NoReservedEntityKeys<object>>
: T & NoReservedEntityKeys<T>
? { [key: string]: any }
: T
/**
* @description The type of `UpdateParams.metadata`: a partial patch of the
* consumer's metadata shape with reserved entity keys forbidden at compile
* time. Same `T = any` handling as {@link EntityMetadataInput}.
* consumer's metadata shape. Same openness as {@link EntityMetadataInput}.
*/
export type EntityMetadataPatch<T> = IsAny<T> extends true
? OpenBag<NoReservedEntityKeys<object>>
: Partial<T> & NoReservedEntityKeys<T>
? { [key: string]: any }
: Partial<T>
/**
* @description The type of `RelateParams.metadata`: the consumer's edge
* metadata shape with reserved relationship keys forbidden at compile time.
* metadata shape, open — the relation mirror of {@link EntityMetadataInput}.
*/
export type RelationMetadataInput<T> = IsAny<T> extends true
? OpenBag<NoReservedRelationKeys<object>>
: T & NoReservedRelationKeys<T>
? { [key: string]: any }
: T
/**
* @description The type of `UpdateRelationParams.metadata`: a partial patch
* of the consumer's edge metadata shape with reserved relationship keys
* forbidden at compile time.
* of the consumer's edge metadata shape, open.
*/
export type RelationMetadataPatch<T> = IsAny<T> extends true
? OpenBag<NoReservedRelationKeys<object>>
: Partial<T> & NoReservedRelationKeys<T>
? { [key: string]: any }
: Partial<T>
/**
* @description Result of splitting a stored flat metadata record into its
@ -196,6 +201,103 @@ export interface SplitMetadataRecord<F extends string> {
const RESERVED_ENTITY_SET: ReadonlySet<string> = new Set(RESERVED_ENTITY_FIELDS)
const RESERVED_RELATION_SET: ReadonlySet<string> = new Set(RESERVED_RELATION_FIELDS)
/**
* @description The format-stamp key of a persisted metadata record. Its
* presence with the exact value {@link NESTED_BAG_FORMAT} marks a v2
* (nested-bag) record; its absence marks a legacy flat record. The stamp is
* what makes the shape check collision-proof against legacy user data: a
* pre-law record COULD carry a user field named `metadata` (the name was
* never reserved), but it cannot also carry this engine-written stamp.
*/
export const METADATA_RECORD_FORMAT_KEY = '_fmt'
/**
* @description The nested-bag record format stamp (v2, the field-addressing
* law's storage shape, 2026-08-03): engine fields at top level, the user's
* metadata bag NESTED verbatim under `metadata`. Cross-engine: the native
* provider discriminates record shapes by the same stamp.
*/
export const NESTED_BAG_FORMAT = 2
/**
* @description `true` when a persisted record carries the v2 nested-bag
* stamp (and a structurally valid nested bag).
*/
export function isNestedBagRecord(
record: Record<string, unknown> | null | undefined
): boolean {
return (
record !== null &&
record !== undefined &&
typeof record === 'object' &&
record[METADATA_RECORD_FORMAT_KEY] === NESTED_BAG_FORMAT &&
typeof record.metadata === 'object' &&
record.metadata !== null &&
!Array.isArray(record.metadata)
)
}
/**
* @description Build a v2 (nested-bag) entity metadata record — THE only
* sanctioned way to construct a persisted noun metadata record. The engine
* half goes top-level; the user bag nests verbatim under `metadata`; the
* format stamp seals the shape. Because the two halves never share a level,
* a user field named `confidence` (or any other engine spelling) survives
* flush / reopen / rebuild / time travel exactly as written.
* @param engineFields - The engine-owned half (keys from
* {@link RESERVED_ENTITY_FIELDS} — `noun`, timestamps, `_rev`, …).
* @param userBag - The consumer's metadata bag, stored verbatim.
* @returns The stamped v2 record.
*/
export function buildNounMetadataRecord(
engineFields: Partial<Record<ReservedEntityField, unknown>>,
userBag: Record<string, unknown> | undefined
): Record<string, unknown> {
return {
...engineFields,
[METADATA_RECORD_FORMAT_KEY]: NESTED_BAG_FORMAT,
metadata: { ...(userBag ?? {}) }
}
}
/**
* @description Build a v2 (nested-bag) relationship metadata record — the
* verb mirror of {@link buildNounMetadataRecord}.
* @param engineFields - The engine-owned half (keys from
* {@link RESERVED_RELATION_FIELDS} — `verb`, `weight`, timestamps, …).
* @param userBag - The consumer's edge metadata bag, stored verbatim.
* @returns The stamped v2 record.
*/
export function buildVerbMetadataRecord(
engineFields: Partial<Record<ReservedRelationField, unknown>>,
userBag: Record<string, unknown> | undefined
): Record<string, unknown> {
return {
...engineFields,
[METADATA_RECORD_FORMAT_KEY]: NESTED_BAG_FORMAT,
metadata: { ...(userBag ?? {}) }
}
}
/**
* @description Shape-first split of a v2 record: the engine half is the top
* level filtered through the reserved list (belt — the builders only ever
* write reserved names there), the user bag is `record.metadata` verbatim.
*/
function splitNestedRecord<F extends string>(
record: Record<string, unknown>,
reservedSet: ReadonlySet<string>
): SplitMetadataRecord<F> {
const reserved: Record<string, unknown> = {}
for (const [key, value] of Object.entries(record)) {
if (reservedSet.has(key)) reserved[key] = value
}
return {
reserved: reserved as Partial<Record<F, unknown>>,
custom: { ...(record.metadata as Record<string, unknown>) }
}
}
/**
* @description Shared splitter — partitions a record's keys against a
* reserved-name set. `null`/`undefined` records split to two empty objects.
@ -222,33 +324,45 @@ function splitRecord<F extends string>(
}
/**
* @description Split a stored entity (noun) flat metadata record into
* reserved fields and custom metadata — THE canonical read-side split. Every
* entity read path (live `get()`, batch reads, paginated listings, and
* historical `asOf()` materialization) goes through this function, so the
* reserved list can never drift between read paths.
* @param record - The stored flat metadata record.
* @returns `reserved` (Brainy-owned fields) and `custom` (the consumer's metadata bag).
* @description Split a stored entity (noun) metadata record into engine
* fields and the user's metadata bag — THE canonical read-side split, shape
* aware. v2 (nested-bag) records split by SHAPE: engine half top-level, bag
* = `record.metadata` verbatim (user collider names survive faithfully).
* Legacy flat records split BY NAME through the reserved list — sound for
* them because the pre-law write door refused user metadata carrying those
* names. Every entity read path (live `get()`, batch reads, paginated
* listings, and historical `asOf()` materialization — the generation store
* snapshots whole records) goes through this function, so the two shapes
* can never drift between read paths.
* @param record - The stored metadata record (either shape).
* @returns `reserved` (engine-owned fields) and `custom` (the consumer's metadata bag).
* @example
* const { reserved, custom } = splitNounMetadataRecord(stored)
* // reserved.noun → entity.type, reserved.confidence → entity.confidence, …
* // custom → entity.metadata (custom fields only, always)
* // custom → entity.metadata (the user's fields only, always — ANY names)
*/
export function splitNounMetadataRecord(
record: Record<string, unknown> | null | undefined
): SplitMetadataRecord<ReservedEntityField> {
if (isNestedBagRecord(record)) {
return splitNestedRecord(record as Record<string, unknown>, RESERVED_ENTITY_SET)
}
return splitRecord(record, RESERVED_ENTITY_SET)
}
/**
* @description Split a stored relationship (verb) flat metadata record into
* reserved fields and custom metadata — the verb mirror of
* {@link splitNounMetadataRecord}, used by every relationship read path.
* @param record - The stored flat metadata record.
* @returns `reserved` (Brainy-owned fields) and `custom` (the consumer's metadata bag).
* @description Split a stored relationship (verb) metadata record into
* engine fields and the user's edge metadata bag — the verb mirror of
* {@link splitNounMetadataRecord}, shape aware, used by every relationship
* read path.
* @param record - The stored metadata record (either shape).
* @returns `reserved` (engine-owned fields) and `custom` (the consumer's metadata bag).
*/
export function splitVerbMetadataRecord(
record: Record<string, unknown> | null | undefined
): SplitMetadataRecord<ReservedRelationField> {
if (isNestedBagRecord(record)) {
return splitNestedRecord(record as Record<string, unknown>, RESERVED_RELATION_SET)
}
return splitRecord(record, RESERVED_RELATION_SET)
}

223
src/utils/metadataIndex.ts
View file

@ -73,8 +73,11 @@ export interface MetadataIndexConfig {
maxIndexSize?: number // Max number of entries per field value (default: 10000)
rebuildThreshold?: number // Rebuild if index is this % stale (default: 0.1)
autoOptimize?: boolean // Auto-cleanup unused entries (default: true)
indexedFields?: string[] // Only index these fields (default: all)
excludeFields?: string[] // Never index these fields
// NOTE: the name-based indexedFields/excludeFields knobs died with the
// field-addressing law ("no special names"): EVERY user field indexes,
// whatever its name. Bulk-payload protection is value-SHAPE based and
// uniform across all names (large arrays never become posting scalars;
// long values index hashed) — shape is not a name carve-out.
}
export interface MetadataIndexOptions {
@ -185,31 +188,12 @@ export class MetadataIndexManager implements MetadataIndexProvider {
this.config = {
maxIndexSize: config.maxIndexSize ?? 10000,
rebuildThreshold: config.rebuildThreshold ?? 0.1,
autoOptimize: config.autoOptimize ?? true,
indexedFields: config.indexedFields ?? [],
excludeFields: config.excludeFields ?? [
// ONLY exclude truly un-indexable fields (binary data, large content)
// Timestamps are NOW indexed with automatic bucketing (prevents pollution)
// Vectors and embeddings (binary data, already have HNSW indexes)
'embedding',
'vector',
'embeddings',
'vectors',
// Large content fields (too large for metadata indexing)
'content',
'data',
'originalData',
'_data',
// Primary keys (use direct lookups instead)
'id'
// NOTE: 'accessed', 'modified', 'createdAt', etc. are NO LONGER excluded!
// They are now indexed with automatic 1-minute bucketing to prevent file pollution
// This enables range queries like: modified > yesterday
]
autoOptimize: config.autoOptimize ?? true
// No name-based exclude/allow lists — the field-addressing law: every
// user field indexes, whatever its name ('content', 'data', 'id',
// 'vector', … included). Bulk payloads are kept out by uniform value-
// SHAPE rules in extractIndexableFields (arrays >10 never become
// posting scalars; >100-char values index hashed), never by name.
}
// Initialize metadata cache with similar config to search cache
@ -301,7 +285,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
}
// Warm the cache with common fields (lazy loading optimization)
// This loads the 'noun' sparse index which is needed for type counts
// This loads the type column ('system.type') needed for type counts
await this.warmCache()
// Load type counts AFTER warmCache (sparse index is now cached)
@ -350,8 +334,9 @@ export class MetadataIndexManager implements MetadataIndexProvider {
* Target: >80% cache hit rate for typical workloads
*/
async warmCache(): Promise<void> {
// Common fields used in most queries
const commonFields = ['noun', 'type', 'service', 'createdAt']
// Common columns used in most queries — the frozen system keys, plus
// legacy spellings for a pre-epoch-3 brain read before its rebuild runs.
const commonFields = ['system.type', 'system.service', 'system.createdAt', 'noun']
prodLog.debug(`🔥 Warming metadata cache with common fields: ${commonFields.join(', ')}`)
@ -537,9 +522,11 @@ export class MetadataIndexManager implements MetadataIndexProvider {
}
/**
* Lazy load entity counts from the 'noun' field sparse index (O(n) where n = number of types)
* Lazy load entity counts from the type column (O(n) where n = number of
* types). The frozen key is 'system.type' (epoch 3); the legacy 'noun'
* column is read as a fallback for a pre-epoch-3 brain observed before its
* rebuild has run (e.g. a reader-mode open against an old writer).
* FIX: Previously read from stats.nounCount which was SERVICE-keyed, not TYPE-keyed
* Now computes counts from the sparse index which has the correct type information
*/
private async lazyLoadCounts(): Promise<void> {
try {
@ -549,23 +536,31 @@ export class MetadataIndexManager implements MetadataIndexProvider {
this.entityCountsByTypeFixed.fill(0)
this.verbCountsByTypeFixed.fill(0)
// PRIMARY (8.0+): rehydrate per-type counts from the column store's 'noun'
// field — the authoritative on-disk source after a cold reopen.
// PRIMARY (8.0+): rehydrate per-type counts from the column store's
// type column — the authoritative on-disk source after a cold reopen.
// Frozen key first ('system.type', epoch 3), legacy 'noun' as the
// pre-rebuild fallback.
//
// The chunked sparse-index WRITE path was removed in 7.20.0 (commit
// 11be039): new workspaces persist the 'noun' field ONLY to the column
// store, never to a `__sparse_index__noun` blob. So the legacy sparse
// path below finds nothing and leaves every count at 0 — which is exactly
// why counts.byType/byTypeEnum/topTypes/allNounTypeCounts all read empty
// 11be039): new workspaces persist the type column ONLY to the column
// store, never to a sparse-index blob. So the legacy sparse path below
// finds nothing and leaves every count at 0 — which is exactly why
// counts.byType/byTypeEnum/topTypes/allNounTypeCounts all read empty
// after close()+reopen while find()/getNounCount() (different sources)
// stay correct. The column store's per-value cardinality matches the warm
// `updateTypeFieldAffinity` counts EXACTLY because both are driven from the
// same `addToIndex` field set, in lockstep, with no visibility gate on
// either — so this rehydration reproduces the warm values precisely.
if (this.columnStore && this.columnStore.getIndexedFields().includes('noun')) {
const nounValues = await this.columnStore.getFilterValues('noun')
const indexedCols = this.columnStore ? this.columnStore.getIndexedFields() : []
const typeCol = indexedCols.includes('system.type')
? 'system.type'
: indexedCols.includes('noun')
? 'noun'
: null
if (this.columnStore && typeCol) {
const nounValues = await this.columnStore.getFilterValues(typeCol)
for (const value of nounValues) {
const bitmap = await this.columnStore.filter('noun', value)
const bitmap = await this.columnStore.filter(typeCol, value)
if (bitmap.size > 0) {
// Use the stored value directly as the key (the legacy sparse path
// did the same): it is already the normalized type string that
@ -580,16 +575,17 @@ export class MetadataIndexManager implements MetadataIndexProvider {
}
// LEGACY FALLBACK (pre-7.20.0 workspaces still on the chunked sparse index).
const nounSparseIndex = await this.loadSparseIndex('noun')
const sparseCol = (await this.loadSparseIndex('system.type')) ? 'system.type' : 'noun'
const nounSparseIndex = await this.loadSparseIndex(sparseCol)
if (!nounSparseIndex) {
// No column-store 'noun' field and no sparse index yet — counts will be
// No column-store type column and no sparse index yet — counts will be
// populated as entities are added.
return
}
// Iterate through all chunks and sum up bitmap sizes by type
for (const chunkId of nounSparseIndex.getAllChunkIds()) {
const chunk = await this.chunkManager.loadChunk('noun', chunkId)
const chunk = await this.chunkManager.loadChunk(sparseCol, chunkId)
if (chunk) {
for (const [type, bitmap] of chunk.entries) {
const currentCount = this.totalEntitiesByType.get(type) || 0
@ -1179,66 +1175,46 @@ export class MetadataIndexManager implements MetadataIndexProvider {
return `__HASH_${Math.abs(hash).toString(36)}`
}
/**
* Check if field should be indexed
*/
private shouldIndexField(field: string): boolean {
if (this.config.excludeFields.includes(field)) return false
if (this.config.indexedFields.length > 0) {
return this.config.indexedFields.includes(field)
}
return true
}
/**
* Extract indexable field-value pairs from entity or metadata
*
* Now handles BOTH entity structure (with top-level fields) AND plain metadata
* - Extracts from top-level fields (confidence, weight, timestamps, type, service, etc.)
* - Also extracts from nested metadata field (custom user fields)
* - Skips HNSW-specific fields (vector, connections, level, id)
* - Maps 'type' → 'noun' for backward compatibility with existing indexes
*
* BUG FIX: Exclude vector embeddings and large arrays from indexing
* BUG FIX: Also exclude purely numeric field names (array indices)
* - Vector fields (384+ dimensions) were creating 825K chunk files for 1,144 entities
* - Arrays converted to objects with numeric keys were still being indexed
* Handles BOTH entity structure (with top-level fields) AND record shapes
* - Record-frame system scalars index under literal 'system.<field>' keys
* - The user's metadata bag indexes under bare keys — EVERY name (the
* field-addressing law: no special names; 'level', 'data', 'id',
* 'content', 'vector' in a bag are ordinary user fields)
* - Record-frame plumbing (vector, connections, level, data, _rev, id)
* never indexes — that is namespace routing, not a name carve-out
* - Value-SHAPE rules apply uniformly to all names: arrays >10 never
* become posting scalars; purely numeric key names (array indices)
* skip; >100-char values index hashed (normalizeValue)
*/
private extractIndexableFields(data: any): Array<{ field: string, value: any }> {
const fields: Array<{ field: string, value: any }> = []
// Fields that should NEVER be indexed: bulk structural payloads that would
// blow up the index (the 384-dim vector, embeddings, the adjacency list).
// These are also caught by the array-size guard below, but naming them is
// belt-and-suspenders. NOTE: `level` was previously here (an HNSW node's
// layer) but it never actually reaches this path — every caller passes a
// metadata bag or Entity record, neither of which carries the node's
// `level` — so its only effect was to silently drop a legitimate USER
// metadata field named `level` (log level, skill level, access level…),
// making `where: { level: … }` return nothing. Removed. (`id` stays: it is
// the reserved entity-identity field, resolved specially by find().)
const NEVER_INDEX = new Set(['vector', 'embedding', 'embeddings', 'connections', 'id'])
// RECORD-FRAME-ONLY plumbing guard: on an entity/stored-record frame
// these keys are the engine's structural payloads (the 384-dim vector,
// embeddings, the adjacency list, the identity field) and never index.
// This set is NEVER applied inside the user's metadata bag — under the
// field-addressing law every user name indexes; a real vector-sized
// value in a bag is kept out by the uniform array-size shape guard, not
// by its name.
const RECORD_PLUMBING = new Set(['vector', 'embedding', 'embeddings', 'connections', 'id'])
// THE FROZEN INDEX KEY FORMAT (cross-engine, sealed 2026-08-03; the native
// accelerator keys identically — epoch 3 rebuilds every brain onto it):
// user fields index under BARE keys exactly as the caller wrote them;
// the ten system scalars index under literal 'system.<field>' keys — the
// key IS the query address, so the two namespaces can never collide
// inside the index again. `origin` tracks which side of the record a key
// came from: 'record' = the entity/stored-record frame (system scalars,
// plumbing, and the metadata bag live here — the WRITE PATH's reserved-
// name remap guarantees a record-frame key matching a system name IS the
// system value); 'user' = inside the flattened metadata bag (everything
// is the user's, including natural names like `level` and `data`).
// Frame kinds: 'entity-record' = entityForIndexing shape (user fields
// nested under `metadata`; stray top-level keys are DROPPED, not guessed —
// epoch-3's rebuild-from-canonical normalizes historical shapes);
// 'flat-record' = the stored metadata-record shape (user fields FLAT
// beside the reserved ones — the write path's reserved-name remap
// guarantees a key matching a system name IS the system value, so
// non-system keys here are the user's and index bare); 'user' = inside
// the metadata bag (everything is the user's, including natural names
// like `level` and `data`).
// inside the index again.
// Frame kinds: 'entity-record' = entityForIndexing shape / v2 nested-bag
// stored record (user fields nested under `metadata`; stray top-level
// keys are DROPPED, not guessed); 'flat-record' = the LEGACY stored
// metadata-record shape (user fields flat beside the engine's — sound to
// split by name because the pre-law write door refused user metadata
// carrying engine names, so a flat key matching a system name IS the
// system value); 'user' = inside the metadata bag, where EVERY key is
// the user's and indexes bare — collider names included.
type Frame = 'entity-record' | 'flat-record' | 'user'
const extract = (obj: any, prefix = '', frame: Frame = 'entity-record'): void => {
for (const [key, value] of Object.entries(obj)) {
@ -1254,30 +1230,25 @@ export class MetadataIndexManager implements MetadataIndexProvider {
} else if (SYSTEM_ENTITY_SCALARS.has(key) && key !== 'id') {
fullKey = `system.${key}`
} else if (
key === 'data' || key === '_rev' || key === 'level' || NEVER_INDEX.has(key)
key === 'data' || key === '_rev' || key === 'level' || key === '_fmt' ||
RECORD_PLUMBING.has(key)
) {
continue // plumbing / identity / bulk payloads — never indexed from a record frame
continue // plumbing / identity / format stamp — never indexed from a record frame
} else if (frame === 'entity-record') {
continue // stray entity-frame key: dropped, not guessed
}
// flat-record fallthrough: a non-system, non-plumbing key IS a user
// field (flat beside the reserved ones) — indexes bare via fullKey.
} else if (!prefix && NEVER_INDEX.has(key)) {
// User frame: only the bulk-payload guards apply — natural names
// like `level` and `data` are real user fields here. (`id` as a
// user metadata field remains un-indexed this train — documented
// limitation; system.id resolves via the id mapper, never a column.)
continue
// field (flat beside the engine's, legacy shape) — indexes bare.
}
// User frame: NO name-based skips — every user field indexes, whatever
// its name (the field-addressing law). Only the uniform value-shape
// guards below apply.
// Skip purely numeric field names (array indices converted to object keys)
// Legitimate field names should never be purely numeric
// This catches vectors stored as objects: {0: 0.1, 1: 0.2, ...}
if (/^\d+$/.test(key)) continue
// Skip fields based on user configuration
if (!this.shouldIndexField(fullKey)) continue
// Skip large arrays (> 10 elements) - likely vectors or bulk data
if (Array.isArray(value) && value.length > 10) continue
@ -1510,10 +1481,11 @@ export class MetadataIndexManager implements MetadataIndexProvider {
prodLog.debug(`Entity ${id} has ${wordFields.length} indexed words (large document)`)
}
// Sort fields to process 'noun' field first for type-field affinity tracking
// Sort fields to process the type column first for type-field affinity
// tracking ('system.type' is the frozen key; 'noun' died at epoch 3).
fields.sort((a, b) => {
if (a.field === 'noun') return -1
if (b.field === 'noun') return 1
if (a.field === 'system.type') return -1
if (b.field === 'system.type') return 1
return 0
})
@ -2861,6 +2833,17 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// VFS Statistics Methods (uses existing Roaring bitmap infrastructure)
// ============================================================================
/**
* Read the type column's bitmap for one type value — frozen key first
* ('system.type', epoch 3), legacy 'noun' as the pre-rebuild fallback.
*/
private async getTypeBitmap(type: string): Promise<RoaringBitmap32 | null> {
return (
(await this.getBitmapFromChunks('system.type', type)) ??
(await this.getBitmapFromChunks('noun', type))
)
}
/**
* Get VFS entity count for a specific type using Roaring bitmap intersection
* Uses hardware-accelerated SIMD operations (AVX2/SSE4.2)
@ -2869,7 +2852,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
*/
async getVFSEntityCountByType(type: string): Promise<number> {
const vfsBitmap = await this.getBitmapFromChunks('isVFSEntity', true)
const typeBitmap = await this.getBitmapFromChunks('noun', type)
const typeBitmap = await this.getTypeBitmap(type)
if (!vfsBitmap || !typeBitmap) return 0
@ -2892,7 +2875,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// Iterate through all known types and compute VFS count via intersection
for (const type of this.totalEntitiesByType.keys()) {
const typeBitmap = await this.getBitmapFromChunks('noun', type)
const typeBitmap = await this.getTypeBitmap(type)
if (typeBitmap) {
const intersection = RoaringBitmap32.and(vfsBitmap, typeBitmap)
if (intersection.size > 0) {
@ -3486,18 +3469,21 @@ export class MetadataIndexManager implements MetadataIndexProvider {
* Tracks which fields commonly appear with which entity types
*/
private updateTypeFieldAffinity(entityId: string, field: string, value: any, operation: 'add' | 'remove', metadata?: any): void {
// Only track affinity for non-system fields (but allow 'noun' for type detection)
if (this.config.excludeFields.includes(field) && field !== 'noun') return
// Only track affinity for user fields (plus the type column itself,
// which drives detection). Engine columns carry the literal 'system.'
// prefix under the frozen key format.
if (field.startsWith('system.') && field !== 'system.type') return
// For the 'noun' field, the value IS the entity type
// For the type column ('system.type'), the value IS the entity type
let entityType: string | null = null
if (field === 'noun') {
if (field === 'system.type') {
// This is the type definition itself
entityType = this.normalizeValue(value, field) // Pass field for bucketing!
} else if (metadata && metadata.noun) {
// Extract entity type from metadata
entityType = this.normalizeValue(metadata.noun, 'noun')
} else if (metadata && (metadata.noun ?? metadata.type)) {
// Extract entity type from the source shape: stored records carry it
// under 'noun', entity-for-indexing views under 'type'.
entityType = this.normalizeValue(metadata.noun ?? metadata.type, 'system.type')
} else {
// No type information available, skip affinity tracking
return
@ -3520,8 +3506,9 @@ export class MetadataIndexManager implements MetadataIndexProvider {
const currentCount = typeFields.get(field) || 0
typeFields.set(field, currentCount + 1)
// Update total entities of this type (only count once per entity)
if (field === 'noun') {
// Update total entities of this type (only count once per entity —
// the type column appears exactly once per entity)
if (field === 'system.type') {
const newCount = this.totalEntitiesByType.get(entityType)! + 1
this.totalEntitiesByType.set(entityType, newCount)
@ -3544,7 +3531,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
}
// Update total entities of this type
if (field === 'noun') {
if (field === 'system.type') {
const total = this.totalEntitiesByType.get(entityType)!
if (total > 1) {
const newCount = total - 1

2
src/utils/paramValidation.ts
View file

@ -618,6 +618,7 @@ export function validateUpdateParams(params: UpdateParams): void {
* Validate relate parameters
*/
export function validateRelateParams(params: RelateParams): void {
rejectForgedSystemKeys(params.metadata as Record<string, unknown> | undefined, 'relate()')
// 8.0 verb-id contract (L.7): verb ids are UUIDs, generated by brainy.
// RelateParams has no `id` field — an untyped caller passing one would
// previously have it silently ignored (a generated UUID was used instead).
@ -666,6 +667,7 @@ export function validateRelateParams(params: RelateParams): void {
* accepts type/subtype/weight/confidence/data/metadata changes.
*/
export function validateUpdateRelationParams(params: UpdateRelationParams): void {
rejectForgedSystemKeys(params.metadata as Record<string, unknown> | undefined, 'updateRelation()')
if (!params.id) {
throw new Error('id is required for updateRelation')
}