feat(namespace): NO SPECIAL NAMES + storage fidelity — the ruled completion of the field-addressing law
All checks were successful
CI / Node 22 (push) Successful in 12m9s
CI / Node 24 (push) Successful in 12m4s
CI / Bun (latest) (push) Successful in 12m52s

The write side of the law, ruled 2026-08-03: data is either in main space
where developers can use anything, or it is in system.*.

- The reserved-name write door DIES: add/update/relate/updateRelation
  metadata bags accept EVERY name (confidence, type, id, data, level,
  content, ...) as ordinary user fields — indexed, filterable, sortable,
  aggregatable, identical to any other field. The remap/enforce/warn
  machinery, the reservedFieldPolicy config (now a typed init refusal),
  and the compile-time metadata key bans are all removed. The one write
  refusal left: keys spelled 'system.*' (namespace forgery), now enforced
  on all four write doors.
- STORED RECORDS GO NESTED (v2): engine fields top-level, the user bag
  nested verbatim under 'metadata', sealed by a format stamp — by-name
  storage discrimination is unsound once colliders are admitted. Legacy
  flat records stay readable forever through the shape-aware splitters
  (sound for them: the old door refused colliders). Time travel rides the
  same split (generation store snapshots whole records).
- Name-based index exclusions DIE: user frame indexes every name; the
  excludeFields/indexedFields knobs and their silent-[] holes are gone;
  bulk-payload protection is value-shape only, uniform across names.
- Consumer-sweep findings fixed in the same wave: per-type counts read
  the frozen 'system.type' column (addToIndex sort, affinity tracking,
  cold-count rehydration, VFS type bitmaps — legacy 'noun' fallback for
  pre-rebuild reads); resolveHiddenIds addresses 'system.visibility'
  (bare 'visibility' was a silent no-op under the law — VFS/system
  entities leaked into default reads).
- Fidelity fallout fixed in the owning layers: readEntityFieldAddress
  reads the bag first (colliders were absent-shadowed by its own guard)
  and never serves system addresses from the bag; blob history refs read
  the bag shape-aware; migration transforms now receive ONE normalized
  view (engine fields + nested bag) regardless of stored era, and stray
  flat-habit keys refuse with the fix in the message.
- THE REOPEN-COLLIDER CONFORMANCE CASE (required before any RC counts as
  gates-green): all ten collider names + plumbing names written as user
  fields, verified verbatim + queryable across live reads, flush+reopen,
  a forced epoch rebuild, and asOf time travel; relation mirror; forgery
  refusals; legacy flat-record compat. 8/8 green.

Gates: unit 1901/1901 (exit 0) · integration 758 (exit 0) · conformance
27/27 (exit 0) · consumer test sweep migrated (10 files).
This commit is contained in:
David Snelling 2026-08-03 16:59:13 -07:00
parent 48a6130a50
commit 24bf6cdbc5
32 changed files with 1355 additions and 1905 deletions

View file

@ -320,15 +320,18 @@ export interface AddParams<T = any> {
*/
visibility?: 'public' | 'internal'
/**
* Structured queryable fields indexed by MetadataIndex, used in `where` filters.
* Structured queryable fields indexed by MetadataIndex, used in `where`
* filters, `orderBy`, and aggregation.
*
* Reserved entity fields (`RESERVED_ENTITY_FIELDS` `noun`, `subtype`, `visibility`,
* `createdAt`, `updatedAt`, `confidence`, `weight`, `service`, `data`, `createdBy`,
* `_rev`) may NOT appear here they have dedicated top-level params and the type makes
* a literal reserved key a compile error. Untyped (JavaScript) callers that pass one
* anyway are normalized at write time: user-settable fields remap to their top-level
* param (top-level wins when both are supplied), system-managed fields are dropped with
* a one-shot warning.
* THE FIELD-ADDRESSING LAW: every name here is YOURS. There are no
* reserved metadata names `confidence`, `type`, `id`, `level`, `data`,
* `content`, are ordinary user fields that index, filter, sort, and
* aggregate like any other, and survive faithfully across restarts and
* rebuilds. Engine scalars are set only via their dedicated params
* (`confidence`, `weight`, `subtype`, ) and are queried explicitly as
* `system.<field>` (`where: { 'system.confidence': … }`). The ONE illegal
* spelling is a key starting `'system.'` the engine's explicit address
* namespace cannot be forged; such a write refuses with a typed error.
*/
metadata?: EntityMetadataInput<T>
/** Custom entity ID. When omitted, a time-ordered UUID v7 is generated; a supplied natural-key string is normalized to a stable UUID v5. */
@ -386,12 +389,11 @@ export interface UpdateParams<T = any> {
*/
visibility?: EntityVisibility
/**
* Metadata fields to merge (or replace when `merge: false`). Reserved entity
* fields (`RESERVED_ENTITY_FIELDS`) may NOT appear here `confidence` /
* `weight` / `subtype` / `visibility` have dedicated params on this call, and the rest
* are system-managed. A literal reserved key is a compile error; untyped callers
* are normalized at write time (remap user-settable, drop system-managed
* with a one-shot warning).
* Metadata fields to merge (or replace when `merge: false`). Every name is
* the user's (the field-addressing law) a patch field named `confidence`
* updates YOUR field of that name, never the engine scalar (use the
* dedicated `confidence` param for that). Keys spelled `'system.…'` refuse
* with a typed error (namespace forgery).
*/
metadata?: EntityMetadataPatch<T>
merge?: boolean // Merge or replace metadata (default: true)
@ -444,11 +446,11 @@ export interface RelateParams<T = any> {
/** Content for the relationship (optional — overrides auto-computed vector) */
data?: any
/**
* Structured queryable fields on the edge. Reserved relationship fields
* (`RESERVED_RELATION_FIELDS` `verb`, `subtype`, `visibility`, `createdAt`,
* `updatedAt`, `confidence`, `weight`, `service`, `data`, `createdBy`, `_rev`) may NOT
* appear here they have dedicated params. A literal reserved key is a
* compile error; untyped callers are normalized at write time.
* Structured queryable fields on the edge. Every name is the user's (the
* field-addressing law) `verb`, `confidence`, `weight`, in this bag are
* ordinary user fields; engine scalars ride their dedicated params and are
* addressed as `system.<field>`. Keys spelled `'system.…'` refuse with a
* typed error (namespace forgery).
*/
metadata?: RelationMetadataInput<T>
/** Create reverse edge too (default: false) */
@ -478,10 +480,9 @@ export interface UpdateRelationParams<T = any> {
confidence?: number // New confidence (0-1)
data?: any // New content
/**
* Metadata fields to merge (or replace when `merge: false`). Reserved
* relationship fields (`RESERVED_RELATION_FIELDS`) may NOT appear here
* a literal reserved key is a compile error; untyped callers are
* normalized at write time.
* Metadata fields to merge (or replace when `merge: false`). Every name is
* the user's (the field-addressing law); engine scalars ride their
* dedicated params. Keys spelled `'system.…'` refuse with a typed error.
*/
metadata?: RelationMetadataPatch<T>
merge?: boolean // Merge or replace metadata
@ -2027,32 +2028,6 @@ export interface BrainyConfig {
*/
force?: boolean
/**
* How write paths react when an untyped (JavaScript) caller smuggles a
* Brainy-reserved field (`RESERVED_ENTITY_FIELDS` / `RESERVED_RELATION_FIELDS`
* `confidence`, `weight`, `subtype`, `visibility`, `service`, `createdBy`,
* `noun`/`verb`, `data`, `createdAt`, `updatedAt`, `_rev`) **inside the
* `metadata` bag** of `add()` / `update()` / `relate()` / `updateRelation()`
* (and their `transact()` / `with()` mirrors). TypeScript callers can't write
* these shapes at all the compile-time guard on the metadata param types
* (`NoReservedEntityKeys` / `NoReservedRelationKeys`) rejects a literal
* reserved key so this policy only governs untyped callers that slip one
* past the compiler.
*
* - `'throw'` (**default, 8.0**): a reserved key in the bag throws a clear
* `Error` naming the offending key(s) and the correct write path. No silent
* remap, no data loss, no surprise. This is the 8.0 "no silent failures"
* contract.
* - `'warn'`: legacy remapping with a loud, one-shot (per key, per process)
* warning for EVERY reserved key found user-mutable fields are remapped to
* their dedicated top-level param (top-level wins when both are supplied),
* system-managed fields are dropped. Use while migrating untyped call sites.
* - `'remap'`: the pre-8.0 silent remapping, no warning. Last-resort
* compatibility hatch for code that intentionally relies on the bag path.
*
* @default 'throw'
*/
reservedFieldPolicy?: 'throw' | 'warn' | 'remap'
}
// ============= Neural API Types =============

View file

@ -1,35 +1,54 @@
/**
* @module types/reservedFields
* @description The canonical reserved-field contract ONE place that defines
* which keys belong to Brainy (top-level entity/relationship fields) and may
* therefore never live inside a `metadata` bag.
* @description The stored-record layout contract ONE place that defines
* which keys of a persisted metadata record belong to the ENGINE (top-level
* entity/relationship fields) and how the USER's metadata bag is kept apart
* from them, faithfully, across flush / reopen / rebuild / time travel.
*
* Three layers enforce the contract, all driven by the constants below:
* THE FIELD-ADDRESSING LAW (ruled 2026-08-03, VENUE-BRAINY-ORDERBY-NOOP):
* data is either in main space where developers can use ANY name, and it
* all works with every database function or it is in `system.*`. There are
* NO reserved user-facing metadata names anymore: `confidence`, `type`,
* `level`, `data`, `id`, `content` inside a metadata bag are ordinary user
* fields. The only refused write is a user metadata key literally starting
* with `'system.'` (namespace forgery see `rejectForgedSystemKeys`).
*
* 1. **Compile time** `AddParams.metadata`, `UpdateParams.metadata`,
* `RelateParams.metadata` and `UpdateRelationParams.metadata` are typed so
* a literal reserved key is a TypeScript error (see
* {@link EntityMetadataInput} / {@link RelationMetadataInput}).
* 2. **Write time** for untyped (JavaScript) callers that smuggle a
* reserved key past the compiler anyway, every write path normalizes the
* bag: user-mutable fields are remapped to their dedicated top-level
* param (top-level wins when both are supplied) and system-managed fields
* are dropped with a one-shot warning naming the correct write path.
* 3. **Read time** every read path splits the stored flat record through
* {@link splitNounMetadataRecord} / {@link splitVerbMetadataRecord}, so a
* reserved field is surfaced ONLY at top level and `entity.metadata` /
* `relation.metadata` contain ONLY custom fields, always live reads,
* batch reads, and historical (`asOf`) reads alike.
* That law makes name-based storage discrimination unsound for NEW records
* (a user field named `confidence` may now legally sit beside the engine's
* confidence scalar), so persisted metadata records carry the user bag
* NESTED, shape-discriminated by a format stamp:
*
* Documented for consumers in `docs/concepts/consistency-model.md`
* ("Reserved fields").
* - **v2 (nested-bag)** `{ …engine fields…, [METADATA_RECORD_FORMAT_KEY]:
* NESTED_BAG_FORMAT, metadata: { user bag, verbatim } }`. Built ONLY by
* {@link buildNounMetadataRecord} / {@link buildVerbMetadataRecord}; the
* engine half and the user bag can never collide because they never share
* a level.
* - **legacy (flat)** engine fields and user fields mixed at one level,
* discriminated BY NAME through the RESERVED_* lists. Sound for legacy
* records precisely because the pre-law write door REFUSED user metadata
* carrying those names a flat key matching a reserved name IS the
* engine's value in any record the old door admitted.
*
* {@link splitNounMetadataRecord} / {@link splitVerbMetadataRecord} read
* BOTH shapes (stamp first, name split as the legacy fallback) and are the
* single read-side choke point for live, batch, AND historical (`asOf`)
* reads the generation store snapshots whole records, so time travel
* rides the same split.
*
* The RESERVED_* lists therefore no longer describe a user-facing ban they
* describe the ENGINE HALF of the stored record layout (and drive the legacy
* split). The write-door remap machinery and the compile-time metadata key
* bans that used to enforce the old contract are gone.
*/
/**
* @description Entity (noun) field names reserved by Brainy. These keys are
* stored in the flat per-entity metadata record alongside custom fields, but
* they belong to Brainy: every read path extracts them to top-level
* `Entity` fields, and no write path accepts them inside `metadata`.
* @description Entity (noun) field names owned by the ENGINE in a stored
* metadata record. In v2 (nested-bag) records these are the legal TOP-LEVEL
* keys beside the nested `metadata` bag; in legacy flat records they drive
* the by-name split. They are NOT a user-facing ban list: since the
* field-addressing law, a user metadata field may carry any of these names
* and remains the user's it lives inside the nested bag, never at the
* record's top level.
*
* | Key | Canonical write path |
* |-----|----------------------|
@ -119,68 +138,54 @@ export type ReservedRelationField = (typeof RESERVED_RELATION_FIELDS)[number]
type IsAny<T> = 0 extends 1 & T ? true : false
/**
* @description Compile-time tripwire: marks every reserved entity key as
* `never` so an object literal carrying one fails to type-check. Keys that
* `T` itself declares (including via an index signature, where
* `keyof T = string`) are exempted a consumer who *explicitly* types a
* reserved key into their metadata shape keeps a working (if unwise) type,
* and index-signature metadata types remain assignable.
* @deprecated The compile-time reserved-key ban died with the
* field-addressing law: every name is legal user metadata now. Kept as an
* empty (no-op) guard so external type references keep compiling; it bans
* nothing.
*/
export type NoReservedEntityKeys<T> = {
readonly [K in ReservedEntityField as K extends keyof T ? never : K]?: never
}
export type NoReservedEntityKeys<T> = unknown
/**
* @description Relationship mirror of {@link NoReservedEntityKeys}.
* @deprecated Relationship mirror of {@link NoReservedEntityKeys} no-op
* for the same reason.
*/
export type NoReservedRelationKeys<T> = {
readonly [K in ReservedRelationField as K extends keyof T ? never : K]?: never
}
/**
* @description The metadata bag shape for untyped brains (`T = any`): an
* open index signature (any custom key, any value exactly the pre-8.0
* latitude) intersected with the reserved-key guard, whose declared
* `?: never` properties take precedence over the index signature so a
* literal reserved key is still a compile error.
*/
type OpenBag<Guard> = { [key: string]: any } & Guard
export type NoReservedRelationKeys<T> = unknown
/**
* @description The type of `AddParams.metadata`: the consumer's metadata
* shape `T` with reserved entity keys forbidden at compile time. For untyped
* brains (`T = any`) the bag stays open ({@link OpenBag}), so arbitrary
* custom fields remain legal while literal reserved keys still error.
* shape `T`, open. Under the field-addressing law EVERY key is a legal user
* field (engine scalars are written only via their dedicated params and read
* at `system.*`), so no name is banned at compile time. The one illegal
* spelling a key starting `'system.'` cannot be expressed as a mapped
* type ban and is refused at runtime (`rejectForgedSystemKeys`).
*/
export type EntityMetadataInput<T> = IsAny<T> extends true
? OpenBag<NoReservedEntityKeys<object>>
: T & NoReservedEntityKeys<T>
? { [key: string]: any }
: T
/**
* @description The type of `UpdateParams.metadata`: a partial patch of the
* consumer's metadata shape with reserved entity keys forbidden at compile
* time. Same `T = any` handling as {@link EntityMetadataInput}.
* consumer's metadata shape. Same openness as {@link EntityMetadataInput}.
*/
export type EntityMetadataPatch<T> = IsAny<T> extends true
? OpenBag<NoReservedEntityKeys<object>>
: Partial<T> & NoReservedEntityKeys<T>
? { [key: string]: any }
: Partial<T>
/**
* @description The type of `RelateParams.metadata`: the consumer's edge
* metadata shape with reserved relationship keys forbidden at compile time.
* metadata shape, open the relation mirror of {@link EntityMetadataInput}.
*/
export type RelationMetadataInput<T> = IsAny<T> extends true
? OpenBag<NoReservedRelationKeys<object>>
: T & NoReservedRelationKeys<T>
? { [key: string]: any }
: T
/**
* @description The type of `UpdateRelationParams.metadata`: a partial patch
* of the consumer's edge metadata shape with reserved relationship keys
* forbidden at compile time.
* of the consumer's edge metadata shape, open.
*/
export type RelationMetadataPatch<T> = IsAny<T> extends true
? OpenBag<NoReservedRelationKeys<object>>
: Partial<T> & NoReservedRelationKeys<T>
? { [key: string]: any }
: Partial<T>
/**
* @description Result of splitting a stored flat metadata record into its
@ -196,6 +201,103 @@ export interface SplitMetadataRecord<F extends string> {
const RESERVED_ENTITY_SET: ReadonlySet<string> = new Set(RESERVED_ENTITY_FIELDS)
const RESERVED_RELATION_SET: ReadonlySet<string> = new Set(RESERVED_RELATION_FIELDS)
/**
* @description The format-stamp key of a persisted metadata record. Its
* presence with the exact value {@link NESTED_BAG_FORMAT} marks a v2
* (nested-bag) record; its absence marks a legacy flat record. The stamp is
* what makes the shape check collision-proof against legacy user data: a
* pre-law record COULD carry a user field named `metadata` (the name was
* never reserved), but it cannot also carry this engine-written stamp.
*/
export const METADATA_RECORD_FORMAT_KEY = '_fmt'
/**
* @description The nested-bag record format stamp (v2, the field-addressing
* law's storage shape, 2026-08-03): engine fields at top level, the user's
* metadata bag NESTED verbatim under `metadata`. Cross-engine: the native
* provider discriminates record shapes by the same stamp.
*/
export const NESTED_BAG_FORMAT = 2
/**
* @description `true` when a persisted record carries the v2 nested-bag
* stamp (and a structurally valid nested bag).
*/
export function isNestedBagRecord(
record: Record<string, unknown> | null | undefined
): boolean {
return (
record !== null &&
record !== undefined &&
typeof record === 'object' &&
record[METADATA_RECORD_FORMAT_KEY] === NESTED_BAG_FORMAT &&
typeof record.metadata === 'object' &&
record.metadata !== null &&
!Array.isArray(record.metadata)
)
}
/**
* @description Build a v2 (nested-bag) entity metadata record THE only
* sanctioned way to construct a persisted noun metadata record. The engine
* half goes top-level; the user bag nests verbatim under `metadata`; the
* format stamp seals the shape. Because the two halves never share a level,
* a user field named `confidence` (or any other engine spelling) survives
* flush / reopen / rebuild / time travel exactly as written.
* @param engineFields - The engine-owned half (keys from
* {@link RESERVED_ENTITY_FIELDS} `noun`, timestamps, `_rev`, ).
* @param userBag - The consumer's metadata bag, stored verbatim.
* @returns The stamped v2 record.
*/
export function buildNounMetadataRecord(
engineFields: Partial<Record<ReservedEntityField, unknown>>,
userBag: Record<string, unknown> | undefined
): Record<string, unknown> {
return {
...engineFields,
[METADATA_RECORD_FORMAT_KEY]: NESTED_BAG_FORMAT,
metadata: { ...(userBag ?? {}) }
}
}
/**
* @description Build a v2 (nested-bag) relationship metadata record the
* verb mirror of {@link buildNounMetadataRecord}.
* @param engineFields - The engine-owned half (keys from
* {@link RESERVED_RELATION_FIELDS} `verb`, `weight`, timestamps, ).
* @param userBag - The consumer's edge metadata bag, stored verbatim.
* @returns The stamped v2 record.
*/
export function buildVerbMetadataRecord(
engineFields: Partial<Record<ReservedRelationField, unknown>>,
userBag: Record<string, unknown> | undefined
): Record<string, unknown> {
return {
...engineFields,
[METADATA_RECORD_FORMAT_KEY]: NESTED_BAG_FORMAT,
metadata: { ...(userBag ?? {}) }
}
}
/**
* @description Shape-first split of a v2 record: the engine half is the top
* level filtered through the reserved list (belt the builders only ever
* write reserved names there), the user bag is `record.metadata` verbatim.
*/
function splitNestedRecord<F extends string>(
record: Record<string, unknown>,
reservedSet: ReadonlySet<string>
): SplitMetadataRecord<F> {
const reserved: Record<string, unknown> = {}
for (const [key, value] of Object.entries(record)) {
if (reservedSet.has(key)) reserved[key] = value
}
return {
reserved: reserved as Partial<Record<F, unknown>>,
custom: { ...(record.metadata as Record<string, unknown>) }
}
}
/**
* @description Shared splitter partitions a record's keys against a
* reserved-name set. `null`/`undefined` records split to two empty objects.
@ -222,33 +324,45 @@ function splitRecord<F extends string>(
}
/**
* @description Split a stored entity (noun) flat metadata record into
* reserved fields and custom metadata THE canonical read-side split. Every
* entity read path (live `get()`, batch reads, paginated listings, and
* historical `asOf()` materialization) goes through this function, so the
* reserved list can never drift between read paths.
* @param record - The stored flat metadata record.
* @returns `reserved` (Brainy-owned fields) and `custom` (the consumer's metadata bag).
* @description Split a stored entity (noun) metadata record into engine
* fields and the user's metadata bag THE canonical read-side split, shape
* aware. v2 (nested-bag) records split by SHAPE: engine half top-level, bag
* = `record.metadata` verbatim (user collider names survive faithfully).
* Legacy flat records split BY NAME through the reserved list sound for
* them because the pre-law write door refused user metadata carrying those
* names. Every entity read path (live `get()`, batch reads, paginated
* listings, and historical `asOf()` materialization the generation store
* snapshots whole records) goes through this function, so the two shapes
* can never drift between read paths.
* @param record - The stored metadata record (either shape).
* @returns `reserved` (engine-owned fields) and `custom` (the consumer's metadata bag).
* @example
* const { reserved, custom } = splitNounMetadataRecord(stored)
* // reserved.noun → entity.type, reserved.confidence → entity.confidence, …
* // custom → entity.metadata (custom fields only, always)
* // custom → entity.metadata (the user's fields only, always — ANY names)
*/
export function splitNounMetadataRecord(
record: Record<string, unknown> | null | undefined
): SplitMetadataRecord<ReservedEntityField> {
if (isNestedBagRecord(record)) {
return splitNestedRecord(record as Record<string, unknown>, RESERVED_ENTITY_SET)
}
return splitRecord(record, RESERVED_ENTITY_SET)
}
/**
* @description Split a stored relationship (verb) flat metadata record into
* reserved fields and custom metadata the verb mirror of
* {@link splitNounMetadataRecord}, used by every relationship read path.
* @param record - The stored flat metadata record.
* @returns `reserved` (Brainy-owned fields) and `custom` (the consumer's metadata bag).
* @description Split a stored relationship (verb) metadata record into
* engine fields and the user's edge metadata bag the verb mirror of
* {@link splitNounMetadataRecord}, shape aware, used by every relationship
* read path.
* @param record - The stored metadata record (either shape).
* @returns `reserved` (engine-owned fields) and `custom` (the consumer's metadata bag).
*/
export function splitVerbMetadataRecord(
record: Record<string, unknown> | null | undefined
): SplitMetadataRecord<ReservedRelationField> {
if (isNestedBagRecord(record)) {
return splitNestedRecord(record as Record<string, unknown>, RESERVED_RELATION_SET)
}
return splitRecord(record, RESERVED_RELATION_SET)
}