David Snelling
24bf6cdbc5
The write side of the law, ruled 2026-08-03: data is either in main space where developers can use anything, or it is in system.*. - The reserved-name write door DIES: add/update/relate/updateRelation metadata bags accept EVERY name (confidence, type, id, data, level, content, ...) as ordinary user fields — indexed, filterable, sortable, aggregatable, identical to any other field. The remap/enforce/warn machinery, the reservedFieldPolicy config (now a typed init refusal), and the compile-time metadata key bans are all removed. The one write refusal left: keys spelled 'system.*' (namespace forgery), now enforced on all four write doors. - STORED RECORDS GO NESTED (v2): engine fields top-level, the user bag nested verbatim under 'metadata', sealed by a format stamp — by-name storage discrimination is unsound once colliders are admitted. Legacy flat records stay readable forever through the shape-aware splitters (sound for them: the old door refused colliders). Time travel rides the same split (generation store snapshots whole records). - Name-based index exclusions DIE: user frame indexes every name; the excludeFields/indexedFields knobs and their silent-[] holes are gone; bulk-payload protection is value-shape only, uniform across names. - Consumer-sweep findings fixed in the same wave: per-type counts read the frozen 'system.type' column (addToIndex sort, affinity tracking, cold-count rehydration, VFS type bitmaps — legacy 'noun' fallback for pre-rebuild reads); resolveHiddenIds addresses 'system.visibility' (bare 'visibility' was a silent no-op under the law — VFS/system entities leaked into default reads). - Fidelity fallout fixed in the owning layers: readEntityFieldAddress reads the bag first (colliders were absent-shadowed by its own guard) and never serves system addresses from the bag; blob history refs read the bag shape-aware; migration transforms now receive ONE normalized view (engine fields + nested bag) regardless of stored era, and stray flat-habit keys refuse with the fix in the message. - THE REOPEN-COLLIDER CONFORMANCE CASE (required before any RC counts as gates-green): all ten collider names + plumbing names written as user fields, verified verbatim + queryable across live reads, flush+reopen, a forced epoch rebuild, and asOf time travel; relation mirror; forgery refusals; legacy flat-record compat. 8/8 green. Gates: unit 1901/1901 (exit 0) · integration 758 (exit 0) · conformance 27/27 (exit 0) · consumer test sweep migrated (10 files).
368 lines
15 KiB
TypeScript
368 lines
15 KiB
TypeScript
/**
|
|
* @module types/reservedFields
|
|
* @description The stored-record layout contract — ONE place that defines
|
|
* which keys of a persisted metadata record belong to the ENGINE (top-level
|
|
* entity/relationship fields) and how the USER's metadata bag is kept apart
|
|
* from them, faithfully, across flush / reopen / rebuild / time travel.
|
|
*
|
|
* THE FIELD-ADDRESSING LAW (ruled 2026-08-03, VENUE-BRAINY-ORDERBY-NOOP):
|
|
* data is either in main space — where developers can use ANY name, and it
|
|
* all works with every database function — or it is in `system.*`. There are
|
|
* NO reserved user-facing metadata names anymore: `confidence`, `type`,
|
|
* `level`, `data`, `id`, `content` … inside a metadata bag are ordinary user
|
|
* fields. The only refused write is a user metadata key literally starting
|
|
* with `'system.'` (namespace forgery — see `rejectForgedSystemKeys`).
|
|
*
|
|
* That law makes name-based storage discrimination unsound for NEW records
|
|
* (a user field named `confidence` may now legally sit beside the engine's
|
|
* confidence scalar), so persisted metadata records carry the user bag
|
|
* NESTED, shape-discriminated by a format stamp:
|
|
*
|
|
* - **v2 (nested-bag)** — `{ …engine fields…, [METADATA_RECORD_FORMAT_KEY]:
|
|
* NESTED_BAG_FORMAT, metadata: { …user bag, verbatim… } }`. Built ONLY by
|
|
* {@link buildNounMetadataRecord} / {@link buildVerbMetadataRecord}; the
|
|
* engine half and the user bag can never collide because they never share
|
|
* a level.
|
|
* - **legacy (flat)** — engine fields and user fields mixed at one level,
|
|
* discriminated BY NAME through the RESERVED_* lists. Sound for legacy
|
|
* records precisely because the pre-law write door REFUSED user metadata
|
|
* carrying those names — a flat key matching a reserved name IS the
|
|
* engine's value in any record the old door admitted.
|
|
*
|
|
* {@link splitNounMetadataRecord} / {@link splitVerbMetadataRecord} read
|
|
* BOTH shapes (stamp first, name split as the legacy fallback) and are the
|
|
* single read-side choke point for live, batch, AND historical (`asOf`)
|
|
* reads — the generation store snapshots whole records, so time travel
|
|
* rides the same split.
|
|
*
|
|
* The RESERVED_* lists therefore no longer describe a user-facing ban — they
|
|
* describe the ENGINE HALF of the stored record layout (and drive the legacy
|
|
* split). The write-door remap machinery and the compile-time metadata key
|
|
* bans that used to enforce the old contract are gone.
|
|
*/
|
|
|
|
/**
|
|
* @description Entity (noun) field names owned by the ENGINE in a stored
|
|
* metadata record. In v2 (nested-bag) records these are the legal TOP-LEVEL
|
|
* keys beside the nested `metadata` bag; in legacy flat records they drive
|
|
* the by-name split. They are NOT a user-facing ban list: since the
|
|
* field-addressing law, a user metadata field may carry any of these names
|
|
* and remains the user's — it lives inside the nested bag, never at the
|
|
* record's top level.
|
|
*
|
|
* | Key | Canonical write path |
|
|
* |-----|----------------------|
|
|
* | `noun` | the `type` param of `add()` / `update()` (stored under the key `noun`) |
|
|
* | `subtype` | the `subtype` param |
|
|
* | `visibility` | the `visibility` param (`'public'` \| `'internal'`; `'system'` is Brainy-only) |
|
|
* | `createdAt` | system-managed — set once at `add()` time |
|
|
* | `updatedAt` | system-managed — set on every write |
|
|
* | `confidence` | the `confidence` param |
|
|
* | `weight` | the `weight` param |
|
|
* | `service` | the `service` param of `add()` (immutable afterwards) |
|
|
* | `data` | the `data` param |
|
|
* | `createdBy` | the `createdBy` param of `add()` (immutable afterwards) |
|
|
* | `_rev` | system-managed revision counter — pass `ifRev` to `update()` for CAS |
|
|
*
|
|
* @example
|
|
* import { RESERVED_ENTITY_FIELDS } from '@soulcraft/brainy'
|
|
* const isReserved = (key: string) =>
|
|
* (RESERVED_ENTITY_FIELDS as readonly string[]).includes(key)
|
|
*/
|
|
export const RESERVED_ENTITY_FIELDS = [
|
|
'noun',
|
|
'subtype',
|
|
'visibility',
|
|
'createdAt',
|
|
'updatedAt',
|
|
'confidence',
|
|
'weight',
|
|
'service',
|
|
'data',
|
|
'createdBy',
|
|
'_rev'
|
|
] as const
|
|
|
|
/**
|
|
* @description Union of the entity field names reserved by Brainy — the
|
|
* element type of {@link RESERVED_ENTITY_FIELDS}.
|
|
*/
|
|
export type ReservedEntityField = (typeof RESERVED_ENTITY_FIELDS)[number]
|
|
|
|
/**
|
|
* @description Relationship (verb) field names reserved by Brainy — the verb
|
|
* mirror of {@link RESERVED_ENTITY_FIELDS}. The stored flat record keys the
|
|
* relationship type under `verb` (the public `Relation` field is `type`);
|
|
* everything else matches the entity list.
|
|
*
|
|
* | Key | Canonical write path |
|
|
* |-----|----------------------|
|
|
* | `verb` | the `type` param of `relate()` / `updateRelation()` (stored under the key `verb`) |
|
|
* | `subtype` | the `subtype` param |
|
|
* | `visibility` | the `visibility` param (`'public'` \| `'internal'`; `'system'` is Brainy-only) |
|
|
* | `createdAt` | system-managed — set once at `relate()` time |
|
|
* | `updatedAt` | system-managed — set on every write |
|
|
* | `confidence` | the `confidence` param |
|
|
* | `weight` | the `weight` param |
|
|
* | `service` | the `service` param of `relate()` (immutable afterwards) |
|
|
* | `data` | the `data` param |
|
|
* | `createdBy` | system-managed |
|
|
* | `_rev` | system-managed |
|
|
*/
|
|
export const RESERVED_RELATION_FIELDS = [
|
|
'verb',
|
|
'subtype',
|
|
'visibility',
|
|
'createdAt',
|
|
'updatedAt',
|
|
'confidence',
|
|
'weight',
|
|
'service',
|
|
'data',
|
|
'createdBy',
|
|
'_rev'
|
|
] as const
|
|
|
|
/**
|
|
* @description Union of the relationship field names reserved by Brainy —
|
|
* the element type of {@link RESERVED_RELATION_FIELDS}.
|
|
*/
|
|
export type ReservedRelationField = (typeof RESERVED_RELATION_FIELDS)[number]
|
|
|
|
/**
|
|
* @description `true` when `T` is exactly `any` (the classic
|
|
* `0 extends 1 & T` probe — only `any` absorbs the impossible intersection).
|
|
* Used to keep the reserved-key guard active for untyped brains, where a
|
|
* plain `T & guard` intersection would collapse to `any` and check nothing.
|
|
*/
|
|
type IsAny<T> = 0 extends 1 & T ? true : false
|
|
|
|
/**
|
|
* @deprecated The compile-time reserved-key ban died with the
|
|
* field-addressing law: every name is legal user metadata now. Kept as an
|
|
* empty (no-op) guard so external type references keep compiling; it bans
|
|
* nothing.
|
|
*/
|
|
export type NoReservedEntityKeys<T> = unknown
|
|
|
|
/**
|
|
* @deprecated Relationship mirror of {@link NoReservedEntityKeys} — no-op
|
|
* for the same reason.
|
|
*/
|
|
export type NoReservedRelationKeys<T> = unknown
|
|
|
|
/**
|
|
* @description The type of `AddParams.metadata`: the consumer's metadata
|
|
* shape `T`, open. Under the field-addressing law EVERY key is a legal user
|
|
* field (engine scalars are written only via their dedicated params and read
|
|
* at `system.*`), so no name is banned at compile time. The one illegal
|
|
* spelling — a key starting `'system.'` — cannot be expressed as a mapped
|
|
* type ban and is refused at runtime (`rejectForgedSystemKeys`).
|
|
*/
|
|
export type EntityMetadataInput<T> = IsAny<T> extends true
|
|
? { [key: string]: any }
|
|
: T
|
|
|
|
/**
|
|
* @description The type of `UpdateParams.metadata`: a partial patch of the
|
|
* consumer's metadata shape. Same openness as {@link EntityMetadataInput}.
|
|
*/
|
|
export type EntityMetadataPatch<T> = IsAny<T> extends true
|
|
? { [key: string]: any }
|
|
: Partial<T>
|
|
|
|
/**
|
|
* @description The type of `RelateParams.metadata`: the consumer's edge
|
|
* metadata shape, open — the relation mirror of {@link EntityMetadataInput}.
|
|
*/
|
|
export type RelationMetadataInput<T> = IsAny<T> extends true
|
|
? { [key: string]: any }
|
|
: T
|
|
|
|
/**
|
|
* @description The type of `UpdateRelationParams.metadata`: a partial patch
|
|
* of the consumer's edge metadata shape, open.
|
|
*/
|
|
export type RelationMetadataPatch<T> = IsAny<T> extends true
|
|
? { [key: string]: any }
|
|
: Partial<T>
|
|
|
|
/**
|
|
* @description Result of splitting a stored flat metadata record into its
|
|
* reserved (Brainy-owned) and custom (consumer-owned) halves.
|
|
*/
|
|
export interface SplitMetadataRecord<F extends string> {
|
|
/** The reserved fields present in the record, keyed by reserved name. */
|
|
reserved: Partial<Record<F, unknown>>
|
|
/** Every other key — the consumer's custom metadata, and nothing else. */
|
|
custom: Record<string, unknown>
|
|
}
|
|
|
|
const RESERVED_ENTITY_SET: ReadonlySet<string> = new Set(RESERVED_ENTITY_FIELDS)
|
|
const RESERVED_RELATION_SET: ReadonlySet<string> = new Set(RESERVED_RELATION_FIELDS)
|
|
|
|
/**
|
|
* @description The format-stamp key of a persisted metadata record. Its
|
|
* presence with the exact value {@link NESTED_BAG_FORMAT} marks a v2
|
|
* (nested-bag) record; its absence marks a legacy flat record. The stamp is
|
|
* what makes the shape check collision-proof against legacy user data: a
|
|
* pre-law record COULD carry a user field named `metadata` (the name was
|
|
* never reserved), but it cannot also carry this engine-written stamp.
|
|
*/
|
|
export const METADATA_RECORD_FORMAT_KEY = '_fmt'
|
|
|
|
/**
|
|
* @description The nested-bag record format stamp (v2, the field-addressing
|
|
* law's storage shape, 2026-08-03): engine fields at top level, the user's
|
|
* metadata bag NESTED verbatim under `metadata`. Cross-engine: the native
|
|
* provider discriminates record shapes by the same stamp.
|
|
*/
|
|
export const NESTED_BAG_FORMAT = 2
|
|
|
|
/**
|
|
* @description `true` when a persisted record carries the v2 nested-bag
|
|
* stamp (and a structurally valid nested bag).
|
|
*/
|
|
export function isNestedBagRecord(
|
|
record: Record<string, unknown> | null | undefined
|
|
): boolean {
|
|
return (
|
|
record !== null &&
|
|
record !== undefined &&
|
|
typeof record === 'object' &&
|
|
record[METADATA_RECORD_FORMAT_KEY] === NESTED_BAG_FORMAT &&
|
|
typeof record.metadata === 'object' &&
|
|
record.metadata !== null &&
|
|
!Array.isArray(record.metadata)
|
|
)
|
|
}
|
|
|
|
/**
|
|
* @description Build a v2 (nested-bag) entity metadata record — THE only
|
|
* sanctioned way to construct a persisted noun metadata record. The engine
|
|
* half goes top-level; the user bag nests verbatim under `metadata`; the
|
|
* format stamp seals the shape. Because the two halves never share a level,
|
|
* a user field named `confidence` (or any other engine spelling) survives
|
|
* flush / reopen / rebuild / time travel exactly as written.
|
|
* @param engineFields - The engine-owned half (keys from
|
|
* {@link RESERVED_ENTITY_FIELDS} — `noun`, timestamps, `_rev`, …).
|
|
* @param userBag - The consumer's metadata bag, stored verbatim.
|
|
* @returns The stamped v2 record.
|
|
*/
|
|
export function buildNounMetadataRecord(
|
|
engineFields: Partial<Record<ReservedEntityField, unknown>>,
|
|
userBag: Record<string, unknown> | undefined
|
|
): Record<string, unknown> {
|
|
return {
|
|
...engineFields,
|
|
[METADATA_RECORD_FORMAT_KEY]: NESTED_BAG_FORMAT,
|
|
metadata: { ...(userBag ?? {}) }
|
|
}
|
|
}
|
|
|
|
/**
|
|
* @description Build a v2 (nested-bag) relationship metadata record — the
|
|
* verb mirror of {@link buildNounMetadataRecord}.
|
|
* @param engineFields - The engine-owned half (keys from
|
|
* {@link RESERVED_RELATION_FIELDS} — `verb`, `weight`, timestamps, …).
|
|
* @param userBag - The consumer's edge metadata bag, stored verbatim.
|
|
* @returns The stamped v2 record.
|
|
*/
|
|
export function buildVerbMetadataRecord(
|
|
engineFields: Partial<Record<ReservedRelationField, unknown>>,
|
|
userBag: Record<string, unknown> | undefined
|
|
): Record<string, unknown> {
|
|
return {
|
|
...engineFields,
|
|
[METADATA_RECORD_FORMAT_KEY]: NESTED_BAG_FORMAT,
|
|
metadata: { ...(userBag ?? {}) }
|
|
}
|
|
}
|
|
|
|
/**
|
|
* @description Shape-first split of a v2 record: the engine half is the top
|
|
* level filtered through the reserved list (belt — the builders only ever
|
|
* write reserved names there), the user bag is `record.metadata` verbatim.
|
|
*/
|
|
function splitNestedRecord<F extends string>(
|
|
record: Record<string, unknown>,
|
|
reservedSet: ReadonlySet<string>
|
|
): SplitMetadataRecord<F> {
|
|
const reserved: Record<string, unknown> = {}
|
|
for (const [key, value] of Object.entries(record)) {
|
|
if (reservedSet.has(key)) reserved[key] = value
|
|
}
|
|
return {
|
|
reserved: reserved as Partial<Record<F, unknown>>,
|
|
custom: { ...(record.metadata as Record<string, unknown>) }
|
|
}
|
|
}
|
|
|
|
/**
|
|
* @description Shared splitter — partitions a record's keys against a
|
|
* reserved-name set. `null`/`undefined` records split to two empty objects.
|
|
* @param record - The stored flat metadata record (reserved + custom keys mixed).
|
|
* @param reservedSet - The reserved-name set to partition against.
|
|
* @returns The `{ reserved, custom }` halves.
|
|
*/
|
|
function splitRecord<F extends string>(
|
|
record: Record<string, unknown> | null | undefined,
|
|
reservedSet: ReadonlySet<string>
|
|
): SplitMetadataRecord<F> {
|
|
const reserved: Record<string, unknown> = {}
|
|
const custom: Record<string, unknown> = {}
|
|
if (record && typeof record === 'object') {
|
|
for (const [key, value] of Object.entries(record)) {
|
|
if (reservedSet.has(key)) {
|
|
reserved[key] = value
|
|
} else {
|
|
custom[key] = value
|
|
}
|
|
}
|
|
}
|
|
return { reserved: reserved as Partial<Record<F, unknown>>, custom }
|
|
}
|
|
|
|
/**
|
|
* @description Split a stored entity (noun) metadata record into engine
|
|
* fields and the user's metadata bag — THE canonical read-side split, shape
|
|
* aware. v2 (nested-bag) records split by SHAPE: engine half top-level, bag
|
|
* = `record.metadata` verbatim (user collider names survive faithfully).
|
|
* Legacy flat records split BY NAME through the reserved list — sound for
|
|
* them because the pre-law write door refused user metadata carrying those
|
|
* names. Every entity read path (live `get()`, batch reads, paginated
|
|
* listings, and historical `asOf()` materialization — the generation store
|
|
* snapshots whole records) goes through this function, so the two shapes
|
|
* can never drift between read paths.
|
|
* @param record - The stored metadata record (either shape).
|
|
* @returns `reserved` (engine-owned fields) and `custom` (the consumer's metadata bag).
|
|
* @example
|
|
* const { reserved, custom } = splitNounMetadataRecord(stored)
|
|
* // reserved.noun → entity.type, reserved.confidence → entity.confidence, …
|
|
* // custom → entity.metadata (the user's fields only, always — ANY names)
|
|
*/
|
|
export function splitNounMetadataRecord(
|
|
record: Record<string, unknown> | null | undefined
|
|
): SplitMetadataRecord<ReservedEntityField> {
|
|
if (isNestedBagRecord(record)) {
|
|
return splitNestedRecord(record as Record<string, unknown>, RESERVED_ENTITY_SET)
|
|
}
|
|
return splitRecord(record, RESERVED_ENTITY_SET)
|
|
}
|
|
|
|
/**
|
|
* @description Split a stored relationship (verb) metadata record into
|
|
* engine fields and the user's edge metadata bag — the verb mirror of
|
|
* {@link splitNounMetadataRecord}, shape aware, used by every relationship
|
|
* read path.
|
|
* @param record - The stored metadata record (either shape).
|
|
* @returns `reserved` (engine-owned fields) and `custom` (the consumer's metadata bag).
|
|
*/
|
|
export function splitVerbMetadataRecord(
|
|
record: Record<string, unknown> | null | undefined
|
|
): SplitMetadataRecord<ReservedRelationField> {
|
|
if (isNestedBagRecord(record)) {
|
|
return splitNestedRecord(record as Record<string, unknown>, RESERVED_RELATION_SET)
|
|
}
|
|
return splitRecord(record, RESERVED_RELATION_SET)
|
|
}
|