diff --git a/RELEASES.md b/RELEASES.md index 59aa50cf..1bc72f57 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -198,6 +198,25 @@ this rename; only the API surface moved. through earlier pins and are not reported by `db.since()`. Writes you want to travel back through go through `transact()`. This is the documented contract, stated rather than papered over. +- **Reserved fields have one canonical location — enforced at every layer.** + The Brainy-owned field names (`RESERVED_ENTITY_FIELDS`: + `noun`/`subtype`/`createdAt`/`updatedAt`/`confidence`/`weight`/`service`/ + `data`/`createdBy`/`_rev`; verb mirror `RESERVED_RELATION_FIELDS` with + `verb` for the type key) are now (1) a **compile error** inside any + `metadata` param — `add`/`update`/`relate`/`updateRelation` and the + matching `transact()` ops; (2) **normalized at write time** for untyped + callers — user-settable fields remap to their dedicated top-level param + (top-level wins; `update({metadata:{confidence}})` no longer silently + no-ops, closing a 7.x trap), system-managed fields drop with a one-shot + warning naming the right path; (3) **split at read time** through one + canonical helper, so `entity.metadata` / `relation.metadata` contain ONLY + custom fields on every read path — `get`, `find`, `getRelations` (several + paginated/by-source/by-target paths previously echoed the full stored + record, including the `verb` type key, inside `metadata`), batch reads, + and historical `asOf()` reads. `getRelations()` results now also surface + `confidence`/`updatedAt` top-level, and `updateRelation()` no longer + erases a relationship's `service`/`createdBy`. See "Reserved fields" in + `docs/concepts/consistency-model.md`. - **Unchanged from 7.31:** per-entity `_rev`, `update({ ifRev })` (`RevisionConflictError`), and `add({ ifAbsent })` work exactly as before, and `ifRev` is also accepted on `transact()` update operations (a conflict diff --git a/docs/concepts/consistency-model.md b/docs/concepts/consistency-model.md index ad5fa1c6..2634cc9f 100644 --- a/docs/concepts/consistency-model.md +++ b/docs/concepts/consistency-model.md @@ -5,7 +5,7 @@ public: true category: concepts template: concept order: 4 -description: The exact guarantees behind Brainy's Db API — snapshot isolation, atomic transactions, two levels of compare-and-swap, time travel, retention, snapshots, and crash recovery. +description: The exact guarantees behind Brainy's Db API — snapshot isolation, atomic transactions, two levels of compare-and-swap, time travel, retention, snapshots, crash recovery, and the reserved-field contract. next: - guides/snapshots-and-time-travel - guides/optimistic-concurrency @@ -251,6 +251,55 @@ Two rules keep snapshots honest: self-contained **read-only** store with the full query surface, including vector search. +## Reserved fields + +Some field names belong to Brainy, not to your metadata. They live at **top +level** on every entity and relationship, have dedicated write paths, and may +never appear inside a `metadata` bag: + +| Entities (nouns) | Relationships (verbs) | Canonical write path | +|---|---|---| +| `noun` | `verb` | the `type` param of `add()` / `relate()` | +| `subtype` | `subtype` | the `subtype` param | +| `confidence` | `confidence` | the `confidence` param | +| `weight` | `weight` | the `weight` param | +| `service` | `service` | the `service` param (fixed at create time) | +| `data` | `data` | the `data` param | +| `createdBy` | `createdBy` | the `createdBy` param of `add()` (system-managed on verbs) | +| `createdAt`, `updatedAt`, `_rev` | `createdAt`, `updatedAt`, `_rev` | system-managed (`ifRev` for CAS) | + +The canonical machine-readable lists are exported as +`RESERVED_ENTITY_FIELDS` and `RESERVED_RELATION_FIELDS` (defined in +`src/types/reservedFields.ts`, the single source of truth). Three layers +enforce the contract: + +1. **Compile time** — every `metadata` param (`add`, `update`, `relate`, + `updateRelation`, and the matching `transact()` operations) rejects a + literal reserved key as a TypeScript error. +2. **Write time** — untyped (JavaScript) callers that pass one anyway are + normalized: user-settable fields (`confidence`, `weight`, `subtype`, and + `service`/`createdBy` at create time) are remapped to their dedicated + param — **top-level wins** when both are supplied — and system-managed + fields are dropped with a one-shot warning naming the correct write path. + `update({ metadata: { confidence: 0.9 } })` therefore behaves exactly + like `update({ confidence: 0.9 })`. +3. **Read time** — every read path (`get`, `find`, `search`, + `getRelations`, batch reads, and historical `asOf()` materialization) + surfaces reserved fields **only at top level**: `entity.metadata` and + `relation.metadata` contain only your custom fields, always. + +```typescript +const id = await brain.add({ + type: 'document', subtype: 'invoice', + data: 'Invoice #42', confidence: 0.95, // reserved → top-level params + metadata: { customer: 'acme', total: 129.5 } // custom fields only +}) + +const entity = await brain.get(id) +entity.confidence // 0.95 — top level +entity.metadata // { customer: 'acme', total: 129.5 } +``` + ## What is not guaranteed Stated plainly, so nothing surprises you in production: diff --git a/src/brainy.ts b/src/brainy.ts index 8710f79e..7cd45eb1 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -100,6 +100,10 @@ import { FillSubtypesResult } from './types/brainy.types.js' import { NounType, VerbType, TypeUtils } from './types/graphTypes.js' +import { + splitNounMetadataRecord, + splitVerbMetadataRecord +} from './types/reservedFields.js' import { BrainyInterface } from './types/brainyInterface.js' import type { IntegrationHub } from './integrations/core/IntegrationHub.js' import { MigrationRunner } from './migration/MigrationRunner.js' @@ -1163,6 +1167,13 @@ export class Brainy implements BrainyInterface { // Zero-config validation (static import for performance) validateAddParams(params) + // Reserved fields arriving via the metadata bag (untyped callers — the + // compile-time guard stops TypeScript callers) are normalized to their + // canonical top-level location BEFORE any enforcement runs, so a + // remapped subtype participates in subtype-pairing enforcement and the + // indexed metadata bag carries only custom fields. + params = this.remapReservedAddMetadata(params) + // Tracked-field vocabulary enforcement (Layer 2). Walks both bags so a // tracked field declared at top level (e.g. 'subtype') and one declared in // metadata (e.g. 'status') both validate. @@ -1574,34 +1585,291 @@ export class Brainy implements BrainyInterface { // Metadata-only entity (no vector loading) // This is 76-81% faster for operations that don't need semantic similarity - // Extract standard fields, rest are custom metadata - // Same destructuring as baseStorage.getNoun() to ensure consistency - const { noun, subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, _rev, ...customMetadata } = metadata + // Canonical reserved/custom split — same single source of truth as every + // other read path (see src/types/reservedFields.ts). + const { reserved, custom } = splitNounMetadataRecord(metadata) const entity: Entity = { id, vector: [], // Stub vector (empty array - vectors not loaded for metadata-only) - type: noun as NounType || NounType.Thing, - subtype, + type: (reserved.noun as NounType) || NounType.Thing, + subtype: reserved.subtype as string | undefined, // Standard fields from metadata - confidence, - weight, - createdAt: createdAt || Date.now(), - updatedAt: updatedAt || Date.now(), - service, - data, - createdBy, + confidence: reserved.confidence as number | undefined, + weight: reserved.weight as number | undefined, + createdAt: (reserved.createdAt as number) || Date.now(), + updatedAt: (reserved.updatedAt as number) || Date.now(), + service: reserved.service as string | undefined, + data: reserved.data, + createdBy: reserved.createdBy as Entity['createdBy'], // 7.31.0 — surface revision counter (defaults to 1 for pre-7.31.0 entities) - _rev: typeof _rev === 'number' ? _rev : 1, + _rev: typeof reserved._rev === 'number' ? reserved._rev : 1, // Custom user fields (standard fields removed, only custom remain) - metadata: customMetadata as T + metadata: custom as T } return entity } + /** One-shot registry for reserved-field drop warnings (per process, per method+field). */ + private static warnedReservedFields = new Set() + + /** + * @description Warn once per process (per method+field) that a reserved + * field arrived inside a metadata bag and was dropped, naming the correct + * write path. Only dropped (system-managed) fields warn — user-settable + * fields are remapped to their dedicated param and honored silently. + * @param method - The public write method the bag arrived through. + * @param field - The reserved field name that was dropped. + * @param rightPath - Human guidance naming the correct write path. + */ + private warnDroppedReservedField(method: string, field: string, rightPath: string): void { + const key = `${method}:${field}` + if (Brainy.warnedReservedFields.has(key)) return + Brainy.warnedReservedFields.add(key) + prodLog.warn( + `[brainy] ${method}(): '${field}' is a reserved field and cannot be set ` + + `through the metadata bag — use ${rightPath}. The value was ignored. ` + + `(This warning is shown once per field per process.)` + ) + } + + /** + * @description Emit the one-shot drop warnings for system-managed entity + * fields found in a metadata bag. Shared by the `add` and `update` remaps + * (live calls and their `transact()` mirrors). + * @param method - `'add'` or `'update'` (the contract is identical for the + * matching `transact()` operation). + * @param reserved - The reserved half of the split metadata bag. + */ + private warnDroppedReservedEntityFields( + method: 'add' | 'update', + reserved: Partial> + ): void { + if (reserved.noun !== undefined) { + this.warnDroppedReservedField(method, 'noun', "the top-level 'type' param") + } + if (reserved.data !== undefined) { + this.warnDroppedReservedField(method, 'data', "the top-level 'data' param") + } + if (reserved.createdAt !== undefined) { + this.warnDroppedReservedField( + method, + 'createdAt', + method === 'add' + ? 'nothing — creation time is set automatically' + : 'nothing — creation time is immutable' + ) + } + if (reserved.updatedAt !== undefined) { + this.warnDroppedReservedField(method, 'updatedAt', 'nothing — set automatically on every write') + } + if (reserved._rev !== undefined) { + this.warnDroppedReservedField( + method, + '_rev', + method === 'update' + ? "the 'ifRev' param for optimistic concurrency" + : 'nothing — revisions are system-managed' + ) + } + if (method === 'update') { + if (reserved.service !== undefined) { + this.warnDroppedReservedField(method, 'service', 'nothing — fixed at add() time') + } + if (reserved.createdBy !== undefined) { + this.warnDroppedReservedField(method, 'createdBy', 'nothing — fixed at add() time') + } + } + } + + /** + * @description Normalize an `add()` params object with respect to + * Brainy-reserved fields arriving inside `metadata` (untyped callers only — + * the compile-time guard on `AddParams.metadata` stops TypeScript callers). + * Fields with a dedicated `add()` param (`confidence`, `weight`, `subtype`, + * `service`, `createdBy`) are remapped to that param unless the caller also + * passed it explicitly (top-level wins); system-managed fields (`noun`, + * `data`, `createdAt`, `updatedAt`, `_rev`) are dropped with a one-shot + * warning naming the correct write path. The remapped `subtype` flows + * through subtype-pairing enforcement exactly like a top-level one. + * @param params - The caller's add params (not mutated). + * @returns Params with reserved fields normalized out of `metadata`. + */ + private remapReservedAddMetadata(params: AddParams): AddParams { + const bag = params.metadata as Record | undefined + if (!bag || typeof bag !== 'object') return params + const { reserved, custom } = splitNounMetadataRecord(bag) + if (Object.keys(reserved).length === 0) return params + + this.warnDroppedReservedEntityFields('add', reserved) + + const createdBy = reserved.createdBy as { augmentation?: unknown; version?: unknown } | undefined + const createdByValid = + typeof createdBy === 'object' && + createdBy !== null && + typeof createdBy.augmentation === 'string' && + typeof createdBy.version === 'string' + + return { + ...params, + metadata: custom as AddParams['metadata'], + ...(params.confidence === undefined && + typeof reserved.confidence === 'number' && { confidence: reserved.confidence }), + ...(params.weight === undefined && + typeof reserved.weight === 'number' && { weight: reserved.weight }), + ...(params.subtype === undefined && + typeof reserved.subtype === 'string' && { subtype: reserved.subtype }), + ...(params.service === undefined && + typeof reserved.service === 'string' && { service: reserved.service }), + ...(params.createdBy === undefined && + createdByValid && { createdBy: createdBy as { augmentation: string; version: string } }) + } + } + + /** + * @description Normalize an `update()` params object with respect to + * Brainy-reserved fields arriving inside the metadata patch — the `update()` + * mirror of {@link remapReservedAddMetadata}, closing the historical trap + * where `add({metadata:{confidence}})` lifted the field but + * `update({metadata:{confidence}})` silently dropped it (the patch value + * survived the merge and was then clobbered by the preserve-existing + * spread; a production consumer's confidence-evolution writes no-oped until + * read back). User-mutable fields (`confidence`, `weight`, `subtype`) + * remap to their dedicated param unless the caller also passed it + * (top-level wins); everything else (`noun`, `data`, `createdAt`, + * `updatedAt`, `service`, `createdBy`, `_rev`) is system-managed or fixed + * at `add()` time and is dropped with a one-shot warning. + * @param params - The caller's update params (not mutated). + * @returns Params with reserved fields normalized out of `metadata`. + */ + private remapReservedUpdateMetadata(params: UpdateParams): UpdateParams { + const bag = params.metadata as Record | undefined + if (!bag || typeof bag !== 'object') return params + const { reserved, custom } = splitNounMetadataRecord(bag) + if (Object.keys(reserved).length === 0) return params + + this.warnDroppedReservedEntityFields('update', reserved) + + return { + ...params, + metadata: custom as UpdateParams['metadata'], + ...(params.confidence === undefined && + typeof reserved.confidence === 'number' && { confidence: reserved.confidence }), + ...(params.weight === undefined && + typeof reserved.weight === 'number' && { weight: reserved.weight }), + ...(params.subtype === undefined && + typeof reserved.subtype === 'string' && { subtype: reserved.subtype }) + } + } + + /** + * @description Emit the one-shot drop warnings for system-managed + * relationship fields found in a metadata bag. Shared by the `relate` and + * `updateRelation` remaps (live calls and the `transact()` relate mirror). + * @param method - `'relate'` or `'updateRelation'`. + * @param reserved - The reserved half of the split metadata bag. + */ + private warnDroppedReservedRelationFields( + method: 'relate' | 'updateRelation', + reserved: Partial> + ): void { + if (reserved.verb !== undefined) { + this.warnDroppedReservedField(method, 'verb', "the top-level 'type' param") + } + if (reserved.data !== undefined) { + this.warnDroppedReservedField(method, 'data', "the top-level 'data' param") + } + if (reserved.createdAt !== undefined) { + this.warnDroppedReservedField( + method, + 'createdAt', + method === 'relate' + ? 'nothing — creation time is set automatically' + : 'nothing — creation time is immutable' + ) + } + if (reserved.updatedAt !== undefined) { + this.warnDroppedReservedField(method, 'updatedAt', 'nothing — set automatically on every write') + } + if (reserved.createdBy !== undefined) { + this.warnDroppedReservedField(method, 'createdBy', 'nothing — system-managed') + } + if (reserved._rev !== undefined) { + this.warnDroppedReservedField(method, '_rev', 'nothing — system-managed') + } + if (method === 'updateRelation' && reserved.service !== undefined) { + this.warnDroppedReservedField(method, 'service', 'nothing — fixed at relate() time') + } + } + + /** + * @description Normalize a `relate()` params object with respect to + * Brainy-reserved fields arriving inside `metadata` — the relationship + * mirror of {@link remapReservedAddMetadata}. Fields with a dedicated + * `relate()` param (`confidence`, `weight`, `subtype`, `service`) remap to + * that param (top-level wins); system-managed fields (`verb`, `data`, + * `createdAt`, `updatedAt`, `createdBy`, `_rev`) are dropped with a + * one-shot warning. + * @param params - The caller's relate params (not mutated). + * @returns Params with reserved fields normalized out of `metadata`. + */ + private remapReservedRelateMetadata(params: RelateParams): RelateParams { + const bag = params.metadata as Record | undefined + if (!bag || typeof bag !== 'object') return params + const { reserved, custom } = splitVerbMetadataRecord(bag) + if (Object.keys(reserved).length === 0) return params + + this.warnDroppedReservedRelationFields('relate', reserved) + + return { + ...params, + metadata: custom as RelateParams['metadata'], + ...(params.confidence === undefined && + typeof reserved.confidence === 'number' && { confidence: reserved.confidence }), + ...(params.weight === undefined && + typeof reserved.weight === 'number' && { weight: reserved.weight }), + ...(params.subtype === undefined && + typeof reserved.subtype === 'string' && { subtype: reserved.subtype }), + ...(params.service === undefined && + typeof reserved.service === 'string' && { service: reserved.service }) + } + } + + /** + * @description Normalize an `updateRelation()` params object with respect + * to Brainy-reserved fields arriving inside the metadata patch — the + * relationship mirror of {@link remapReservedUpdateMetadata}. User-mutable + * fields (`confidence`, `weight`, `subtype`) remap to their dedicated + * param (top-level wins); everything else is dropped with a one-shot + * warning. + * @param params - The caller's update-relation params (not mutated). + * @returns Params with reserved fields normalized out of `metadata`. + */ + private remapReservedUpdateRelationMetadata( + params: UpdateRelationParams + ): UpdateRelationParams { + const bag = params.metadata as Record | undefined + if (!bag || typeof bag !== 'object') return params + const { reserved, custom } = splitVerbMetadataRecord(bag) + if (Object.keys(reserved).length === 0) return params + + this.warnDroppedReservedRelationFields('updateRelation', reserved) + + return { + ...params, + metadata: custom as UpdateRelationParams['metadata'], + ...(params.confidence === undefined && + typeof reserved.confidence === 'number' && { confidence: reserved.confidence }), + ...(params.weight === undefined && + typeof reserved.weight === 'number' && { weight: reserved.weight }), + ...(params.subtype === undefined && + typeof reserved.subtype === 'string' && { subtype: reserved.subtype }) + } + } + /** * Update an existing entity * @@ -1659,6 +1927,16 @@ export class Brainy implements BrainyInterface { // Zero-config validation (static import for performance) validateUpdateParams(params) + // Reserved fields arriving via the metadata patch are remapped to their + // canonical top-level location, mirroring add()'s lift. Without this the + // patch value survived the merge but was then clobbered by the + // preserve-existing spreads below — a silent no-op consumers could only + // detect by reading values back. User-mutable fields (confidence, + // weight, subtype) remap unless the same field was also passed top-level + // (top-level wins); system-managed fields are dropped with a one-shot + // warning naming the right path. + params = this.remapReservedUpdateMetadata(params) + // Tracked-field vocabulary enforcement (Layer 2). Same as add() — the // metadata bag carries fields registered via trackField(), and subtype is // a tracked top-level candidate. @@ -1690,10 +1968,10 @@ export class Brainy implements BrainyInterface { } // ifRev (7.31.0) — optimistic concurrency. If caller supplied ifRev, the persisted - // _rev must match exactly. Pre-7.31.0 entities without _rev are treated as rev 1. - const currentRev = typeof (existing.metadata as any)?._rev === 'number' - ? (existing.metadata as any)._rev - : (typeof (existing as any)._rev === 'number' ? (existing as any)._rev : 1) + // _rev must match exactly. `_rev` is reserved: every read path surfaces it ONLY + // top-level (entities without one are read as rev 1), so that is the one place + // to look. + const currentRev = typeof existing._rev === 'number' ? existing._rev : 1 if (typeof params.ifRev === 'number' && params.ifRev !== currentRev) { throw new RevisionConflictError(params.id, params.ifRev, currentRev) } @@ -1900,13 +2178,15 @@ export class Brainy implements BrainyInterface { // Aggregation hook (outside transaction — derived data) if (this._aggregationIndex && metadata) { - // Reconstruct entity-like object from stored metadata - const { noun, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata + // Reconstruct entity-like object from stored metadata via the + // canonical reserved/custom split (the hand-rolled destructure here + // missed subtype/_rev, leaking them into the aggregation view). + const { reserved, custom } = splitNounMetadataRecord(metadata) const entityForAgg = { - type: noun, - service, - data, - metadata: customMetadata + type: reserved.noun, + service: reserved.service, + data: reserved.data, + metadata: custom } this._aggregationIndex.onEntityDeleted(id, entityForAgg) } @@ -2168,6 +2448,10 @@ export class Brainy implements BrainyInterface { // Zero-config validation (static import for performance) validateRelateParams(params) + // Reserved fields arriving via the metadata bag are normalized to their + // canonical top-level params before enforcement — mirror of add()'s lift. + params = this.remapReservedRelateMetadata(params) + // Subtype pairing enforcement (Layer 3 — 7.30.0). Per-type rules registered // via brain.requireSubtype() compose with the brain-wide strict-mode flag. // Metadata is passed so infrastructure edges (VFS containment) can bypass @@ -2225,8 +2509,10 @@ export class Brainy implements BrainyInterface { verb: params.type, ...(params.subtype !== undefined && { subtype: params.subtype }), weight: params.weight ?? 1.0, + ...(params.confidence !== undefined && { confidence: params.confidence }), + ...(params.service !== undefined && { service: params.service }), createdAt: Date.now(), - ...((params as any).data !== undefined && { data: (params as any).data }) + ...(params.data !== undefined && { data: params.data }) } // Save to storage (vector and metadata separately) @@ -2239,8 +2525,10 @@ export class Brainy implements BrainyInterface { type: params.type, ...(params.subtype !== undefined && { subtype: params.subtype }), weight: params.weight ?? 1.0, + ...(params.confidence !== undefined && { confidence: params.confidence }), + ...(params.service !== undefined && { service: params.service }), metadata: params.metadata, - data: (params as any).data, + data: params.data, createdAt: Date.now() } @@ -2392,6 +2680,10 @@ export class Brainy implements BrainyInterface { validateUpdateRelationParams(params) + // Reserved fields arriving via the metadata patch are remapped to their + // canonical top-level params — mirror of update()'s normalization. + params = this.remapReservedUpdateRelationMetadata(params) + const existing = await this.storage.getVerb(params.id) if (!existing) { throw new Error(`Relation ${params.id} not found`) @@ -2428,6 +2720,10 @@ export class Brainy implements BrainyInterface { ...(params.confidence !== undefined ? { confidence: params.confidence } : existingAny.confidence !== undefined && { confidence: existingAny.confidence }), + // service/createdBy are fixed at relate() time — always carried forward + // (omitting them here silently erased them on every updateRelation()). + ...(existingAny.service !== undefined && { service: existingAny.service }), + ...(existingAny.createdBy !== undefined && { createdBy: existingAny.createdBy }), createdAt: existingAny.createdAt, updatedAt: Date.now(), ...(params.data !== undefined @@ -5030,31 +5326,25 @@ export class Brainy implements BrainyInterface { ) } - const { - verb, - subtype, - createdAt, - updatedAt, - confidence, - weight, - service, - data, - ...customMetadata - } = record.metadata as Record + // Canonical reserved/custom split — same single source of truth as the + // live verb read paths (see src/types/reservedFields.ts). + const { reserved, custom } = splitVerbMetadataRecord( + record.metadata as Record + ) return { id, from: core.sourceId, to: core.targetId, - type: (verb ?? core.verb) as VerbType, - ...(subtype !== undefined && { subtype: subtype as string }), - weight: (weight as number) ?? 1.0, - data, - metadata: customMetadata as T, - service: service as string, - createdAt: typeof createdAt === 'number' ? createdAt : Date.now(), - ...(typeof updatedAt === 'number' && { updatedAt }), - ...(typeof confidence === 'number' && { confidence }) + type: (reserved.verb ?? core.verb) as VerbType, + ...(reserved.subtype !== undefined && { subtype: reserved.subtype as string }), + weight: (reserved.weight as number) ?? 1.0, + data: reserved.data, + metadata: custom as T, + service: reserved.service as string, + createdAt: typeof reserved.createdAt === 'number' ? reserved.createdAt : Date.now(), + ...(typeof reserved.updatedAt === 'number' && { updatedAt: reserved.updatedAt }), + ...(typeof reserved.confidence === 'number' && { confidence: reserved.confidence }) } } @@ -5318,8 +5608,12 @@ export class Brainy implements BrainyInterface { state: TxPlanState, plan: PlannedTransact ): Promise { - const { op: _discriminator, ...params } = op - validateAddParams(params as AddParams) + const { op: _discriminator, ...rawParams } = op + validateAddParams(rawParams as AddParams) + // Same reserved-field normalization as add() — the metadata bag is + // cleaned BEFORE enforcement so a remapped subtype participates in + // subtype-pairing enforcement and only custom fields reach the index. + const params = this.remapReservedAddMetadata(rawParams as AddParams) this.enforceTrackedFieldValues(params.metadata as Record | undefined, 'metadata') this.enforceTrackedFieldValues({ subtype: params.subtype } as Record, 'top-level') this.enforceSubtypeOnAdd('add', params.type, params.subtype, params.metadata) @@ -5404,8 +5698,12 @@ export class Brainy implements BrainyInterface { state: TxPlanState, plan: PlannedTransact ): Promise { - const { op: _discriminator, ...params } = op - validateUpdateParams(params as UpdateParams) + const { op: _discriminator, ...rawParams } = op + validateUpdateParams(rawParams as UpdateParams) + // Same reserved-field normalization as update() — user-mutable fields + // remap to their dedicated param (top-level wins), system-managed fields + // drop with a one-shot warning. + const params = this.remapReservedUpdateMetadata(rawParams as UpdateParams) this.enforceTrackedFieldValues(params.metadata as Record | undefined, 'metadata') if (params.subtype !== undefined) { this.enforceTrackedFieldValues({ subtype: params.subtype } as Record, 'top-level') @@ -5424,13 +5722,9 @@ export class Brainy implements BrainyInterface { } // ifRev CAS — identical resolution to update(); a conflict rejects the - // WHOLE batch before anything is staged or applied. - const currentRev = - typeof (existing.metadata as any)?._rev === 'number' - ? (existing.metadata as any)._rev - : typeof existing._rev === 'number' - ? existing._rev - : 1 + // WHOLE batch before anything is staged or applied. `_rev` is reserved: + // every read path surfaces it ONLY top-level. + const currentRev = typeof existing._rev === 'number' ? existing._rev : 1 if (typeof params.ifRev === 'number' && params.ifRev !== currentRev) { throw new RevisionConflictError(params.id, params.ifRev, currentRev) } @@ -5601,8 +5895,14 @@ export class Brainy implements BrainyInterface { plan.touchedNouns.push(id) if (metadata) { - const { noun: nounType, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata - const entityForAgg = { type: nounType, service, data, metadata: customMetadata } + // Canonical reserved/custom split — mirror of delete()'s aggregation hook. + const { reserved, custom } = splitNounMetadataRecord(metadata) + const entityForAgg = { + type: reserved.noun, + service: reserved.service, + data: reserved.data, + metadata: custom + } plan.postCommit.push(() => { if (this._aggregationIndex) { this._aggregationIndex.onEntityDeleted(id, entityForAgg) @@ -5626,8 +5926,10 @@ export class Brainy implements BrainyInterface { state: TxPlanState, plan: PlannedTransact ): Promise { - const { op: _discriminator, ...params } = op - validateRelateParams(params as RelateParams) + const { op: _discriminator, ...rawParams } = op + validateRelateParams(rawParams as RelateParams) + // Same reserved-field normalization as relate(). + const params = this.remapReservedRelateMetadata(rawParams as RelateParams) this.enforceSubtypeOnRelate('relate', params.type, params.subtype, params.metadata) const fromEntity = await this.planGetEntity(state, params.from) @@ -5677,6 +5979,8 @@ export class Brainy implements BrainyInterface { verb: params.type, ...(params.subtype !== undefined && { subtype: params.subtype }), weight: params.weight ?? 1.0, + ...(params.confidence !== undefined && { confidence: params.confidence }), + ...(params.service !== undefined && { service: params.service }), createdAt: now, ...(params.data !== undefined && { data: params.data }) } @@ -5689,6 +5993,8 @@ export class Brainy implements BrainyInterface { type: params.type, ...(params.subtype !== undefined && { subtype: params.subtype }), weight: params.weight ?? 1.0, + ...(params.confidence !== undefined && { confidence: params.confidence }), + ...(params.service !== undefined && { service: params.service }), metadata: params.metadata, data: params.data, createdAt: now @@ -8238,7 +8544,7 @@ export class Brainy implements BrainyInterface { if (!readBoth && from.kind === 'metadata') { delete nextMeta[from.field] } - update.metadata = nextMeta as T + update.metadata = nextMeta as UpdateParams['metadata'] update.merge = false } else { // data path — only meaningful when data is an object @@ -8256,7 +8562,7 @@ export class Brainy implements BrainyInterface { const base = (entity.metadata as unknown as Record) ?? {} const nextMeta: Record = { ...base } delete nextMeta[from.field] - update.metadata = nextMeta as T + update.metadata = nextMeta as UpdateParams['metadata'] update.merge = false } else if (from.kind === 'data' && to.kind !== 'data') { const base = (entity.data && typeof entity.data === 'object') ? { ...(entity.data as Record) } : null @@ -9560,7 +9866,12 @@ export class Brainy implements BrainyInterface { } /** - * Convert verbs to relations (read from top-level) + * Convert storage-shape verbs to public `Relation`s. The storage layer has + * already done the canonical reserved/custom split (every combine goes + * through `hydrateVerbWithMetadata` — see src/types/reservedFields.ts), so + * this is a pure top-level field mapping: `metadata` carries ONLY custom + * fields, and every reserved field (`subtype`, `weight`, `confidence`, + * timestamps, `service`, `data`) surfaces at top level. */ private verbsToRelations(verbs: GraphVerb[]): Relation[] { return verbs.map((v) => { @@ -9572,10 +9883,12 @@ export class Brainy implements BrainyInterface { type: (v.verb || v.type) as VerbType, ...(va.subtype !== undefined && { subtype: va.subtype as string }), weight: v.weight ?? 1.0, + ...(typeof va.confidence === 'number' && { confidence: va.confidence as number }), data: v.data, metadata: v.metadata, service: v.service as string, - createdAt: typeof v.createdAt === 'number' ? v.createdAt : Date.now() + createdAt: typeof v.createdAt === 'number' ? v.createdAt : Date.now(), + ...(typeof va.updatedAt === 'number' && { updatedAt: va.updatedAt as number }) } }) } diff --git a/src/cli/commands/import.ts b/src/cli/commands/import.ts index d4db46fe..7f6c2696 100644 --- a/src/cli/commands/import.ts +++ b/src/cli/commands/import.ts @@ -324,16 +324,16 @@ export const importCommands = { const extractedEntities = await brain.extract(text) - // Add extracted entities + // Add extracted entities (ExtractedEntity is fully typed — + // text/type/confidence come straight off the extraction result) for (const extracted of extractedEntities) { - const type = (extracted as any).type || NounType.Thing await brain.add({ - data: (extracted as any).content || (extracted as any).text, - type: type, + data: extracted.text, + type: extracted.type || NounType.Thing, + confidence: extracted.confidence, // reserved field — dedicated param, not metadata metadata: { extractedFrom: entity.id, - extractionMethod: 'nlp', - confidence: (extracted as any).confidence + extractionMethod: 'nlp' } }) entitiesExtracted++ diff --git a/src/db/db.ts b/src/db/db.ts index 4e94f63d..39df27a0 100644 --- a/src/db/db.ts +++ b/src/db/db.ts @@ -56,6 +56,10 @@ import type { Relation, Result } from '../types/brainy.types.js' +import { + splitNounMetadataRecord, + splitVerbMetadataRecord +} from '../types/reservedFields.js' import { v4 as uuidv4 } from '../universal/uuid.js' import { SpeculativeOverlayError } from './errors.js' import type { GenerationStore } from './generationStore.js' @@ -544,20 +548,38 @@ export class Db { for (const op of ops) { switch (op.op) { case 'add': { + // Reserved-field normalization — mirror of the brain.transact() + // write path: user-settable fields lift to their dedicated field + // (top-level wins), system-managed fields drop, and the entity's + // metadata bag carries ONLY custom fields. Speculative views skip + // the one-shot warnings — committing the same ops through + // `brain.transact()` warns on the real write path. + const { reserved, custom } = splitNounMetadataRecord( + op.metadata as Record | undefined + ) + const confidence = + op.confidence ?? (typeof reserved.confidence === 'number' ? reserved.confidence : undefined) + const weight = + op.weight ?? (typeof reserved.weight === 'number' ? reserved.weight : undefined) + const subtype = + op.subtype ?? (typeof reserved.subtype === 'string' ? reserved.subtype : undefined) + const service = + op.service ?? (typeof reserved.service === 'string' ? reserved.service : undefined) + const id = op.id ?? uuidv4() const now = Date.now() overlay.nouns.set(id, { id, vector: op.vector ?? [], type: op.type, - ...(op.subtype !== undefined && { subtype: op.subtype }), + ...(subtype !== undefined && { subtype }), data: op.data, - metadata: (op.metadata ?? {}) as T, - ...(op.service !== undefined && { service: op.service }), + metadata: custom as T, + ...(service !== undefined && { service }), createdAt: now, updatedAt: now, - ...(op.confidence !== undefined && { confidence: op.confidence }), - ...(op.weight !== undefined && { weight: op.weight }), + ...(confidence !== undefined && { confidence }), + ...(weight !== undefined && { weight }), _rev: 1 }) break @@ -567,18 +589,28 @@ export class Db { if (!base) { throw new Error(`with(): entity ${op.id} not found at generation ${this.gen}`) } + // Same reserved-field normalization as the committed update path. + const { reserved, custom } = splitNounMetadataRecord( + op.metadata as Record | undefined + ) + const confidence = + op.confidence ?? (typeof reserved.confidence === 'number' ? reserved.confidence : undefined) + const weight = + op.weight ?? (typeof reserved.weight === 'number' ? reserved.weight : undefined) + const subtype = + op.subtype ?? (typeof reserved.subtype === 'string' ? reserved.subtype : undefined) const mergedMetadata = op.merge !== false - ? ({ ...(base.metadata as object), ...(op.metadata as object) } as T) - : ((op.metadata ?? base.metadata) as T) + ? ({ ...(base.metadata as object), ...custom } as T) + : ((op.metadata !== undefined ? custom : base.metadata) as T) overlay.nouns.set(op.id, { ...base, ...(op.type !== undefined && { type: op.type }), - ...(op.subtype !== undefined && { subtype: op.subtype }), + ...(subtype !== undefined && { subtype }), ...(op.data !== undefined && { data: op.data }), ...(op.vector !== undefined && { vector: op.vector }), - ...(op.confidence !== undefined && { confidence: op.confidence }), - ...(op.weight !== undefined && { weight: op.weight }), + ...(confidence !== undefined && { confidence }), + ...(weight !== undefined && { weight }), metadata: mergedMetadata, updatedAt: Date.now(), _rev: (base._rev ?? 1) + 1 @@ -617,17 +649,32 @@ export class Db { } if (duplicate) break + // Reserved-field normalization — relationship mirror of the add + // op above (and of the committed relate() path). + const { reserved, custom } = splitVerbMetadataRecord( + op.metadata as Record | undefined + ) + const confidence = + op.confidence ?? (typeof reserved.confidence === 'number' ? reserved.confidence : undefined) + const weight = + op.weight ?? (typeof reserved.weight === 'number' ? reserved.weight : undefined) + const subtype = + op.subtype ?? (typeof reserved.subtype === 'string' ? reserved.subtype : undefined) + const service = + op.service ?? (typeof reserved.service === 'string' ? reserved.service : undefined) + const id = uuidv4() overlay.verbs.set(id, { id, from: op.from, to: op.to, type: op.type, - ...(op.subtype !== undefined && { subtype: op.subtype }), - weight: op.weight ?? 1.0, + ...(subtype !== undefined && { subtype }), + weight: weight ?? 1.0, + ...(confidence !== undefined && { confidence }), ...(op.data !== undefined && { data: op.data }), - metadata: op.metadata, - ...(op.service !== undefined && { service: op.service }), + metadata: custom as T, + ...(service !== undefined && { service }), createdAt: Date.now() }) if (op.bidirectional) { @@ -637,11 +684,12 @@ export class Db { from: op.to, to: op.from, type: op.type, - ...(op.subtype !== undefined && { subtype: op.subtype }), - weight: op.weight ?? 1.0, + ...(subtype !== undefined && { subtype }), + weight: weight ?? 1.0, + ...(confidence !== undefined && { confidence }), ...(op.data !== undefined && { data: op.data }), - metadata: op.metadata, - ...(op.service !== undefined && { service: op.service }), + metadata: custom as T, + ...(service !== undefined && { service }), createdAt: Date.now() }) } diff --git a/src/import/EntityDeduplicator.ts b/src/import/EntityDeduplicator.ts index 58f15d39..08247695 100644 --- a/src/import/EntityDeduplicator.ts +++ b/src/import/EntityDeduplicator.ts @@ -155,7 +155,15 @@ export class EntityDeduplicator { throw new Error(`Entity ${existingId} not found`) } - // Merge metadata + // Update confidence (weighted average) — `confidence` is a reserved + // top-level field, so it travels via the dedicated update() param, not + // the metadata bag. + const mergedConfidence = this.mergeConfidence( + existing.confidence ?? 0.5, + candidate.confidence + ) + + // Merge metadata (custom fields only — reserved fields are top-level) const mergedMetadata = { ...existing.metadata, // Track provenance @@ -168,11 +176,6 @@ export class EntityDeduplicator { ...(existing.metadata?.vfsPaths || [existing.metadata?.vfsPath]).filter(Boolean), candidate.metadata?.vfsPath ].filter(Boolean), - // Update confidence (weighted average) - confidence: this.mergeConfidence( - existing.metadata?.confidence || 0.5, - candidate.confidence - ), // Merge other metadata ...this.mergeMetadataFields(existing.metadata, candidate.metadata), // Track last update @@ -183,6 +186,7 @@ export class EntityDeduplicator { // Update entity await this.brain.update({ id: existingId, + confidence: mergedConfidence, metadata: mergedMetadata, merge: true }) @@ -191,7 +195,7 @@ export class EntityDeduplicator { mergedEntityId: existingId, wasMerged: true, mergedWith: existing.metadata?.name || existingId, - confidence: mergedMetadata.confidence, + confidence: mergedConfidence, provenance: mergedMetadata.imports } } catch (error) { @@ -218,17 +222,18 @@ export class EntityDeduplicator { // No duplicate found, create new entity. Preserve any subtype the candidate // carried (set by the extractor or upstream importer), else fall back to // `'imported'` so enforcement doesn't fire (added 7.30.1). + // `confidence` is a reserved top-level field (dedicated add() param); + // creation time is system-managed — neither belongs in the metadata bag. const entityId = await this.brain.add({ data: candidate.description || candidate.name, type: candidate.type, subtype: (candidate as any).subtype ?? 'imported', + confidence: candidate.confidence, metadata: { ...candidate.metadata, name: candidate.name, - confidence: candidate.confidence, imports: [importSource], vfsPaths: [candidate.metadata?.vfsPath].filter(Boolean), - createdAt: Date.now(), mergeCount: 0 } }) diff --git a/src/import/ImportCoordinator.ts b/src/import/ImportCoordinator.ts index f53dc120..8adee43c 100644 --- a/src/import/ImportCoordinator.ts +++ b/src/import/ImportCoordinator.ts @@ -1051,7 +1051,6 @@ export class ImportCoordinator { ...(trackingContext && { importIds: [trackingContext.importId], projectId: trackingContext.projectId, - createdAt: Date.now(), importFormat: trackingContext.importFormat, ...trackingContext.customMetadata }) @@ -1133,11 +1132,12 @@ export class ImportCoordinator { rowNumber: row.rowNumber, extractedAt: Date.now(), format: sourceInfo?.format, - // Import tracking metadata + // Import tracking metadata (`createdAt` is reserved — the + // relationship's own creation time is system-managed, and the + // import timestamp already travels as `extractedAt`) ...(trackingContext && { importIds: [trackingContext.importId], projectId: trackingContext.projectId, - createdAt: Date.now(), importFormat: trackingContext.importFormat, ...trackingContext.customMetadata }) diff --git a/src/importers/SmartImportOrchestrator.ts b/src/importers/SmartImportOrchestrator.ts index ac61bbb6..962476e4 100644 --- a/src/importers/SmartImportOrchestrator.ts +++ b/src/importers/SmartImportOrchestrator.ts @@ -194,10 +194,10 @@ export class SmartImportOrchestrator { data: extracted.entity.description, type: extracted.entity.type, subtype: (extracted.entity as any).subtype ?? options.defaultSubtype ?? 'imported', + confidence: extracted.entity.confidence, // reserved field — dedicated param, not metadata metadata: { ...extracted.entity.metadata, name: extracted.entity.name, - confidence: extracted.entity.confidence, importedFrom: 'smart-import' } }) @@ -602,7 +602,8 @@ export class SmartImportOrchestrator { data: extracted.entity.description, type: extracted.entity.type, subtype: (extracted.entity as any).subtype ?? options.defaultSubtype ?? 'imported', - metadata: { ...extracted.entity.metadata, name: extracted.entity.name, confidence: extracted.entity.confidence, importedFrom: 'smart-import' } + confidence: extracted.entity.confidence, // reserved field — dedicated param, not metadata + metadata: { ...extracted.entity.metadata, name: extracted.entity.name, importedFrom: 'smart-import' } }) result.entityIds.push(entityId) result.stats.entitiesCreated++ @@ -617,7 +618,7 @@ export class SmartImportOrchestrator { onProgress?.({ phase: 'creating', message: 'Preparing relationships...', processed: 0, total: result.extraction.rows.length, entities: result.entityIds.length, relationships: 0 }) // Collect all relationship parameters - const relationshipParams: Array<{from: string; to: string; type: VerbType; subtype?: string; metadata?: any}> = [] + const relationshipParams: Array<{from: string; to: string; type: VerbType; subtype?: string; confidence?: number; metadata?: any}> = [] for (const extracted of result.extraction.rows) { for (const rel of extracted.relationships) { @@ -635,7 +636,8 @@ export class SmartImportOrchestrator { result.entityIds.push(toEntityId) } // Relationship subtype precedence: extractor → caller default → `'imported'` (7.30.1). - relationshipParams.push({ from: extracted.entity.id, to: toEntityId, type: rel.type, subtype: (rel as any).subtype ?? options.defaultSubtype ?? 'imported', metadata: { confidence: rel.confidence, evidence: rel.evidence } }) + // `confidence` is a reserved top-level field — dedicated relate() param, not metadata + relationshipParams.push({ from: extracted.entity.id, to: toEntityId, type: rel.type, subtype: (rel as any).subtype ?? options.defaultSubtype ?? 'imported', confidence: rel.confidence, metadata: { evidence: rel.evidence } }) } catch (error: any) { result.errors.push(`Failed to prepare relationship: ${error.message}`) } diff --git a/src/index.ts b/src/index.ts index 58799f4a..8fded6e1 100644 --- a/src/index.ts +++ b/src/index.ts @@ -26,6 +26,7 @@ export type { AddParams, UpdateParams, RelateParams, + UpdateRelationParams, FindParams, SubtypeRegistry, FillSubtypeRule, @@ -44,6 +45,26 @@ export type { AggregationProvider } from './types/brainy.types.js' +// Reserved-field contract — the canonical list of Brainy-owned field names +// that may never appear inside a `metadata` bag (see docs/concepts/consistency-model.md) +export { + RESERVED_ENTITY_FIELDS, + RESERVED_RELATION_FIELDS, + splitNounMetadataRecord, + splitVerbMetadataRecord +} from './types/reservedFields.js' +export type { + ReservedEntityField, + ReservedRelationField, + EntityMetadataInput, + EntityMetadataPatch, + RelationMetadataInput, + RelationMetadataPatch, + NoReservedEntityKeys, + NoReservedRelationKeys, + SplitMetadataRecord +} from './types/reservedFields.js' + // Export Aggregation Engine export { AggregationIndex, AggregateMaterializer, bucketTimestamp, parseBucketRange } from './aggregation/index.js' diff --git a/src/neural/neuralImport.ts b/src/neural/neuralImport.ts index c353602d..4c1c7648 100644 --- a/src/neural/neuralImport.ts +++ b/src/neural/neuralImport.ts @@ -808,13 +808,12 @@ export class NeuralImport { type: relationship.verbType as VerbType, subtype: (relationship as any).subtype ?? options.defaultSubtype ?? 'extracted', weight: relationship.weight, + confidence: relationship.confidence, // reserved field — dedicated param, not metadata metadata: { - confidence: relationship.confidence, - context: relationship.context, - ...relationship.metadata - } + context: relationship.context, + ...relationship.metadata } - ) + }) } spinner.succeed(this.colors.success( diff --git a/src/storage/adapters/fileSystemStorage.ts b/src/storage/adapters/fileSystemStorage.ts index c4467769..af9589b4 100644 --- a/src/storage/adapters/fileSystemStorage.ts +++ b/src/storage/adapters/fileSystemStorage.ts @@ -4,11 +4,8 @@ */ import { - GraphVerb, HNSWNoun, HNSWVerb, - NounMetadata, - VerbMetadata, HNSWNounWithMetadata, HNSWVerbWithMetadata, StatisticsData, @@ -2866,26 +2863,22 @@ export class FileSystemStorage extends BaseStorage { connections = connectionsMap } - // Extract standard fields from metadata to top-level - const metadataObj = (metadata || {}) as VerbMetadata - const { createdAt, updatedAt, confidence, weight, service, data: dataField, createdBy, ...customMetadata } = metadataObj - - const verbWithMetadata: HNSWVerbWithMetadata = { - id: edge.id, - vector: edge.vector, - connections: connections || new Map(), - verb: edge.verb, - sourceId: edge.sourceId, - targetId: edge.targetId, - createdAt: (createdAt as number) || Date.now(), - updatedAt: (updatedAt as number) || Date.now(), - confidence: confidence as number | undefined, - weight: weight as number | undefined, - service: service as string | undefined, - data: dataField as Record | undefined, - createdBy, - metadata: customMetadata - } + // Canonical hydration (single source of truth: + // src/types/reservedFields.ts) — reserved fields top-level, ONLY + // custom fields in `metadata`. The previous hand-rolled + // destructure here missed `subtype` (so streamed verbs lost it) + // and `verb` (so the type key echoed inside custom metadata). + const verbWithMetadata: HNSWVerbWithMetadata = this.hydrateVerbWithMetadata( + { + id: edge.id, + vector: edge.vector, + connections: connections || new Map(), + verb: edge.verb, + sourceId: edge.sourceId, + targetId: edge.targetId + }, + metadata + ) // Apply filters if (options.filter) { diff --git a/src/storage/baseStorage.ts b/src/storage/baseStorage.ts index 4914735b..da66e34e 100644 --- a/src/storage/baseStorage.ts +++ b/src/storage/baseStorage.ts @@ -30,6 +30,32 @@ import { BlobStorage, type BlobStoreAdapter } from './blobStorage.js' import { unwrapBinaryData } from './binaryDataCodec.js' import { prodLog } from '../utils/logger.js' import { MetadataWriteBuffer } from '../utils/metadataWriteBuffer.js' +import { + splitNounMetadataRecord, + splitVerbMetadataRecord +} from '../types/reservedFields.js' + +/** + * Normalize a stored timestamp value to epoch milliseconds. Brainy 8.0 writes + * plain numbers; records written by pre-8.0 cloud adapters may carry the + * `{ seconds, nanoseconds }` object form. Anything else falls back to now — + * matching the long-standing `|| Date.now()` combine behavior. + * @param value - The raw `createdAt`/`updatedAt` value from a stored metadata record. + * @returns Epoch milliseconds. + */ +function normalizeStoredTimestamp(value: unknown): number { + if (typeof value === 'number' && value > 0) { + return value + } + if ( + value !== null && + typeof value === 'object' && + typeof (value as { seconds?: unknown }).seconds === 'number' + ) { + return (value as { seconds: number }).seconds * 1000 + } + return Date.now() +} /** * Storage key analysis result @@ -969,6 +995,74 @@ export abstract class BaseStorage extends BaseStorageAdapter { await this.saveNoun_internal(noun) } + /** + * Hydrate a deserialized noun (pure HNSW vector data) with its stored flat + * metadata record — THE canonical noun combine for every storage read path. + * The record is split through `splitNounMetadataRecord` (single source of + * truth: src/types/reservedFields.ts): reserved fields surface ONLY at + * top level and `metadata` carries ONLY the consumer's custom fields. + * Adding a combine site that bypasses this helper reintroduces the + * reserved-field echo bug — don't. + * + * @param noun - The deserialized HNSW noun (id/vector/connections/level). + * @param metadata - The stored flat metadata record (reserved + custom keys). + * @returns The combined noun with reserved fields top-level, custom fields in `metadata`. + */ + protected hydrateNounWithMetadata( + noun: HNSWNoun, + metadata: Record | null | undefined + ): HNSWNounWithMetadata { + const { reserved, custom } = splitNounMetadataRecord(metadata) + return { + ...noun, + // Standard fields at top-level + type: (reserved.noun as NounType) || NounType.Thing, + subtype: reserved.subtype as string | undefined, + createdAt: normalizeStoredTimestamp(reserved.createdAt), + updatedAt: normalizeStoredTimestamp(reserved.updatedAt), + confidence: reserved.confidence as number | undefined, + weight: reserved.weight as number | undefined, + service: reserved.service as string | undefined, + data: reserved.data as Record | undefined, + createdBy: reserved.createdBy as HNSWNounWithMetadata['createdBy'], + _rev: typeof reserved._rev === 'number' ? reserved._rev : 1, + // Only custom user fields remain in metadata + metadata: custom + } + } + + /** + * Hydrate a deserialized verb (structural core) with its stored flat + * metadata record — THE canonical verb combine, the relationship mirror of + * {@link hydrateNounWithMetadata}. Splitting through + * `splitVerbMetadataRecord` extracts `verb` too, so the type key never + * echoes inside `metadata`. + * + * @param verb - The deserialized HNSW verb (id/vector/connections/verb/sourceId/targetId). + * @param metadata - The stored flat metadata record (reserved + custom keys). + * @returns The combined verb with reserved fields top-level, custom fields in `metadata`. + */ + protected hydrateVerbWithMetadata( + verb: HNSWVerb, + metadata: Record | null | undefined + ): HNSWVerbWithMetadata { + const { reserved, custom } = splitVerbMetadataRecord(metadata) + return { + ...verb, + // Standard fields at top-level + subtype: reserved.subtype as string | undefined, + createdAt: normalizeStoredTimestamp(reserved.createdAt), + updatedAt: normalizeStoredTimestamp(reserved.updatedAt), + confidence: reserved.confidence as number | undefined, + weight: reserved.weight as number | undefined, + service: reserved.service as string | undefined, + data: reserved.data as Record | undefined, + createdBy: reserved.createdBy as HNSWVerbWithMetadata['createdBy'], + // Only custom user fields remain in metadata + metadata: custom + } + } + /** * Get a noun from storage (returns combined HNSWNounWithMetadata) * @param id Entity ID @@ -990,28 +1084,7 @@ export abstract class BaseStorage extends BaseStorageAdapter { return null } - // Combine into HNSWNounWithMetadata - Extract standard fields to top-level - const { noun, subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, _rev, ...customMetadata } = metadata - - return { - id: vector.id, - vector: vector.vector, - connections: vector.connections, - level: vector.level, - // Standard fields at top-level - type: (noun as NounType) || NounType.Thing, - subtype: subtype as string | undefined, - createdAt: (createdAt as number) || Date.now(), - updatedAt: (updatedAt as number) || Date.now(), - confidence: confidence as number | undefined, - weight: weight as number | undefined, - service: service as string | undefined, - data: data as Record | undefined, - createdBy, - _rev: typeof _rev === 'number' ? _rev : 1, - // Only custom user fields remain in metadata - metadata: customMetadata - } + return this.hydrateNounWithMetadata(vector, metadata) } /** @@ -1025,29 +1098,12 @@ export abstract class BaseStorage extends BaseStorageAdapter { // Internal method returns HNSWNoun[], need to combine with metadata const nouns = await this.getNounsByNounType_internal(nounType) - // Combine each noun with its metadata - Extract standard fields to top-level + // Combine each noun with its metadata via the canonical hydration helper const nounsWithMetadata: HNSWNounWithMetadata[] = [] for (const noun of nouns) { const metadata = await this.getNounMetadata(noun.id) if (metadata) { - const { noun: nounType, subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, _rev, ...customMetadata } = metadata - - nounsWithMetadata.push({ - ...noun, - // Standard fields at top-level - type: (nounType as NounType) || NounType.Thing, - subtype: subtype as string | undefined, - createdAt: (createdAt as number) || Date.now(), - updatedAt: (updatedAt as number) || Date.now(), - confidence: confidence as number | undefined, - weight: weight as number | undefined, - service: service as string | undefined, - data: data as Record | undefined, - createdBy, - _rev: typeof _rev === 'number' ? _rev : 1, - // Only custom user fields in metadata - metadata: customMetadata - }) + nounsWithMetadata.push(this.hydrateNounWithMetadata(noun, metadata)) } } @@ -1109,28 +1165,7 @@ export abstract class BaseStorage extends BaseStorageAdapter { return null } - // Combine into HNSWVerbWithMetadata - Extract standard fields to top-level - const { subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, _rev, ...customMetadata } = metadata - - return { - id: verb.id, - vector: verb.vector, - connections: verb.connections, - verb: verb.verb, - sourceId: verb.sourceId, - targetId: verb.targetId, - // Standard fields at top-level - subtype: subtype as string | undefined, - createdAt: (createdAt as number) || Date.now(), - updatedAt: (updatedAt as number) || Date.now(), - confidence: confidence as number | undefined, - weight: weight as number | undefined, - service: service as string | undefined, - data: data as Record | undefined, - createdBy, - // Only custom user fields remain in metadata - metadata: customMetadata - } + return this.hydrateVerbWithMetadata(verb, metadata) } /** @@ -1181,98 +1216,15 @@ export abstract class BaseStorage extends BaseStorageAdapter { const metadataData = metadataResults.get(metadataPath) if (vectorData && metadataData) { - // Deserialize verb + // Deserialize, then combine via the canonical hydration helper const verb = this.deserializeVerb(vectorData) - - // Extract standard fields to top-level - const { subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, _rev, ...customMetadata } = metadataData - - results.set(id, { - id: verb.id, - vector: verb.vector, - connections: verb.connections, - verb: verb.verb, - sourceId: verb.sourceId, - targetId: verb.targetId, - // Standard fields at top-level - subtype: subtype as string | undefined, - createdAt: (createdAt as number) || Date.now(), - updatedAt: (updatedAt as number) || Date.now(), - confidence: confidence as number | undefined, - weight: weight as number | undefined, - service: service as string | undefined, - data: data as Record | undefined, - createdBy, - // Only custom user fields remain in metadata - metadata: customMetadata - }) + results.set(id, this.hydrateVerbWithMetadata(verb, metadataData)) } } return results } - /** - * Convert an HNSW verb (raw vector store entry) into a `GraphVerb` shape by - * combining with the verb's metadata. Used by internal code paths that need - * the graph-layer shape rather than the storage-layer shape. - */ - protected async convertHNSWVerbToGraphVerb(hnswVerb: HNSWVerb): Promise { - try { - // Load metadata - const metadata = await this.getVerbMetadata(hnswVerb.id) - - // Create default timestamp in Firestore format - const defaultTimestamp = { - seconds: Math.floor(Date.now() / 1000), - nanoseconds: (Date.now() % 1000) * 1000000 - } - - // Create default createdBy if not present - const defaultCreatedBy = { - augmentation: 'unknown', - version: '1.0' - } - - // Convert flexible timestamp to Firestore format for GraphVerb - const normalizeTimestamp = (ts: any) => { - if (!ts) return defaultTimestamp - if (typeof ts === 'number') { - return { - seconds: Math.floor(ts / 1000), - nanoseconds: (ts % 1000) * 1000000 - } - } - return ts - } - - return { - id: hnswVerb.id, - vector: hnswVerb.vector, - - // CORE FIELDS from HNSWVerb - verb: hnswVerb.verb, - sourceId: hnswVerb.sourceId, - targetId: hnswVerb.targetId, - - // Alias for ergonomic access - type: hnswVerb.verb, - - // Optional fields from metadata file - weight: metadata?.weight || 1.0, - metadata: metadata as any || {}, - createdAt: normalizeTimestamp(metadata?.createdAt), - updatedAt: normalizeTimestamp(metadata?.updatedAt), - createdBy: metadata?.createdBy || defaultCreatedBy, - data: metadata?.data as Record | undefined, - embedding: hnswVerb.vector - } - } catch (error) { - prodLog.error(`Failed to convert HNSWVerb to GraphVerb for ${hnswVerb.id}:`, error) - return null - } - } - /** * Internal method for loading all verbs - used by performance optimizations * @internal - Do not use directly, use getVerbs() with pagination instead @@ -1571,27 +1523,10 @@ export abstract class BaseStorage extends BaseStorageAdapter { } } - // Combine noun + metadata. Subtype is surfaced to top-level here - // (mirrors what `getNoun()` already does) so callers — including - // the 7.30.1 `brain.audit()` diagnostic — see a consistent shape - // regardless of which getter path they reach. - collectedNouns.push({ - ...deserialized, - type: (metadata.noun || 'thing') as NounType, - subtype: (metadata as any).subtype as string | undefined, - confidence: metadata.confidence, - weight: metadata.weight, - createdAt: metadata.createdAt - ? (typeof metadata.createdAt === 'number' ? metadata.createdAt : metadata.createdAt.seconds * 1000) - : Date.now(), - updatedAt: metadata.updatedAt - ? (typeof metadata.updatedAt === 'number' ? metadata.updatedAt : metadata.updatedAt.seconds * 1000) - : Date.now(), - service: metadata.service, - data: metadata.data as Record | undefined, - createdBy: metadata.createdBy, - metadata: metadata || ({} as NounMetadata) - }) + // Combine noun + metadata via the canonical hydration helper — + // reserved fields top-level, ONLY custom fields in `metadata` + // (this site previously echoed the full flat record). + collectedNouns.push(this.hydrateNounWithMetadata(deserialized, metadata)) } } } catch (error) { @@ -1723,22 +1658,10 @@ export abstract class BaseStorage extends BaseStorageAdapter { } } - // Combine verb + metadata - collectedVerbs.push({ - ...verb, - subtype: (metadata as any)?.subtype as string | undefined, - weight: metadata?.weight, - confidence: metadata?.confidence, - createdAt: metadata?.createdAt - ? (typeof metadata.createdAt === 'number' ? metadata.createdAt : metadata.createdAt.seconds * 1000) - : Date.now(), - updatedAt: metadata?.updatedAt - ? (typeof metadata.updatedAt === 'number' ? metadata.updatedAt : metadata.updatedAt.seconds * 1000) - : Date.now(), - service: metadata?.service, - createdBy: metadata?.createdBy, - metadata: metadata || ({} as VerbMetadata) - }) + // Combine verb + metadata via the canonical hydration helper — + // reserved fields top-level, ONLY custom fields in `metadata` + // (this site previously echoed the full flat record). + collectedVerbs.push(this.hydrateVerbWithMetadata(verb, metadata)) } catch (error) { // Skip verbs that fail to load } @@ -2571,31 +2494,9 @@ export abstract class BaseStorage extends BaseStorageAdapter { const metadataData = metadataResults.get(metadataPath) if (vectorData && metadataData) { - // Deserialize noun + // Deserialize, then combine via the canonical hydration helper const noun = this.deserializeNoun(vectorData) - - // Extract standard fields to top-level - const { noun: nounType, subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, _rev, ...customMetadata } = metadataData - - results.set(id, { - id: noun.id, - vector: noun.vector, - connections: noun.connections, - level: noun.level, - // Standard fields at top-level - type: (nounType as NounType) || NounType.Thing, - subtype: subtype as string | undefined, - createdAt: (createdAt as number) || Date.now(), - updatedAt: (updatedAt as number) || Date.now(), - confidence: confidence as number | undefined, - weight: weight as number | undefined, - service: service as string | undefined, - data: data as Record | undefined, - createdBy, - _rev: typeof _rev === 'number' ? _rev : 1, - // Only custom user fields remain in metadata - metadata: customMetadata - }) + results.set(id, this.hydrateNounWithMetadata(noun, metadataData)) } } @@ -3734,24 +3635,11 @@ export abstract class BaseStorage extends BaseStorageAdapter { const metadata = metadataMap.get(metadataPath) if (rawVerb && metadata) { - // CRITICAL - Deserialize connections Map from JSON storage format + // CRITICAL - Deserialize connections Map from JSON storage format, + // then combine via the canonical hydration helper (reserved fields + // top-level, ONLY custom fields in `metadata`). const verb = this.deserializeVerb(rawVerb) - - results.push({ - ...verb, - subtype: (metadata as any).subtype as string | undefined, - weight: metadata.weight, - confidence: metadata.confidence, - createdAt: metadata.createdAt - ? (typeof metadata.createdAt === 'number' ? metadata.createdAt : metadata.createdAt.seconds * 1000) - : Date.now(), - updatedAt: metadata.updatedAt - ? (typeof metadata.updatedAt === 'number' ? metadata.updatedAt : metadata.updatedAt.seconds * 1000) - : Date.now(), - service: metadata.service, - createdBy: metadata.createdBy, - metadata: metadata || {} as VerbMetadata - }) + results.push(this.hydrateVerbWithMetadata(verb, metadata)) } } @@ -3791,22 +3679,9 @@ export abstract class BaseStorage extends BaseStorageAdapter { if (verb.sourceId === sourceId) { const metadataPath = getVerbMetadataPath(verb.id) const metadata = await this.readCanonicalObject(metadataPath) - - results.push({ - ...verb, - subtype: (metadata as any)?.subtype as string | undefined, - weight: metadata?.weight, - confidence: metadata?.confidence, - createdAt: metadata?.createdAt - ? (typeof metadata.createdAt === 'number' ? metadata.createdAt : metadata.createdAt.seconds * 1000) - : Date.now(), - updatedAt: metadata?.updatedAt - ? (typeof metadata.updatedAt === 'number' ? metadata.updatedAt : metadata.updatedAt.seconds * 1000) - : Date.now(), - service: metadata?.service, - createdBy: metadata?.createdBy, - metadata: metadata || {} as VerbMetadata - }) + // Canonical hydration — reserved fields top-level, ONLY custom + // fields in `metadata`. + results.push(this.hydrateVerbWithMetadata(verb, metadata)) } } catch (error) { // Skip verbs that fail to load @@ -3919,20 +3794,10 @@ export abstract class BaseStorage extends BaseStorageAdapter { const metadataPath = getVerbMetadataPath(verbId) const metadata = metadataMap.get(metadataPath) || {} - const hydratedVerb: HNSWVerbWithMetadata = { - ...verbData, - weight: metadata?.weight, - confidence: metadata?.confidence, - createdAt: metadata?.createdAt - ? (typeof metadata.createdAt === 'number' ? metadata.createdAt : metadata.createdAt.seconds * 1000) - : Date.now(), - updatedAt: metadata?.updatedAt - ? (typeof metadata.updatedAt === 'number' ? metadata.updatedAt : metadata.updatedAt.seconds * 1000) - : Date.now(), - service: metadata?.service, - createdBy: metadata?.createdBy, - metadata: metadata as VerbMetadata - } + // Canonical hydration — reserved fields top-level (including + // subtype/data, which this site previously dropped), ONLY custom + // fields in `metadata`. + const hydratedVerb = this.hydrateVerbWithMetadata(verbData, metadata) // Add to results for this sourceId const sourceVerbs = results.get(verbData.sourceId)! @@ -3976,21 +3841,9 @@ export abstract class BaseStorage extends BaseStorageAdapter { const metadata = await this.getVerbMetadata(verbId) if (verb && metadata) { - results.push({ - ...verb, - subtype: (metadata as any).subtype as string | undefined, - weight: metadata.weight, - confidence: metadata.confidence, - createdAt: metadata.createdAt - ? (typeof metadata.createdAt === 'number' ? metadata.createdAt : metadata.createdAt.seconds * 1000) - : Date.now(), - updatedAt: metadata.updatedAt - ? (typeof metadata.updatedAt === 'number' ? metadata.updatedAt : metadata.updatedAt.seconds * 1000) - : Date.now(), - service: metadata.service, - createdBy: metadata.createdBy, - metadata: metadata || {} as VerbMetadata - }) + // Canonical hydration — reserved fields top-level, ONLY custom + // fields in `metadata`. + results.push(this.hydrateVerbWithMetadata(verb, metadata)) } } @@ -4023,22 +3876,9 @@ export abstract class BaseStorage extends BaseStorageAdapter { if (verb.targetId === targetId) { const metadataPath = getVerbMetadataPath(verb.id) const metadata = await this.readCanonicalObject(metadataPath) - - results.push({ - ...verb, - subtype: (metadata as any)?.subtype as string | undefined, - weight: metadata?.weight, - confidence: metadata?.confidence, - createdAt: metadata?.createdAt - ? (typeof metadata.createdAt === 'number' ? metadata.createdAt : metadata.createdAt.seconds * 1000) - : Date.now(), - updatedAt: metadata?.updatedAt - ? (typeof metadata.updatedAt === 'number' ? metadata.updatedAt : metadata.updatedAt.seconds * 1000) - : Date.now(), - service: metadata?.service, - createdBy: metadata?.createdBy, - metadata: metadata || {} as VerbMetadata - }) + // Canonical hydration — reserved fields top-level, ONLY custom + // fields in `metadata`. + results.push(this.hydrateVerbWithMetadata(verb, metadata)) } } catch (error) { // Skip verbs that fail to load @@ -4079,32 +3919,15 @@ export abstract class BaseStorage extends BaseStorageAdapter { // Filter by verb type if (hnswVerb.verb !== verbType) continue - // Load metadata separately (optional) + // Load metadata separately (optional), then combine via the + // canonical hydration helper (defensive vector copy preserved) const metadata = await this.getVerbMetadata(hnswVerb.id) - - // Extract standard fields from metadata to top-level - const metadataObj = (metadata || {}) as VerbMetadata - const { subtype, createdAt, updatedAt, confidence, weight, service, data, createdBy, _rev, ...customMetadata } = metadataObj - - const verbWithMetadata: HNSWVerbWithMetadata = { - id: hnswVerb.id, - vector: [...hnswVerb.vector], - connections: hnswVerb.connections, // Already deserialized - verb: hnswVerb.verb, - sourceId: hnswVerb.sourceId, - targetId: hnswVerb.targetId, - subtype: subtype as string | undefined, - createdAt: (createdAt as number) || Date.now(), - updatedAt: (updatedAt as number) || Date.now(), - confidence: confidence as number | undefined, - weight: weight as number | undefined, - service: service as string | undefined, - data: data as Record | undefined, - createdBy, - metadata: customMetadata - } - - verbs.push(verbWithMetadata) + verbs.push( + this.hydrateVerbWithMetadata( + { ...hnswVerb, vector: [...hnswVerb.vector] }, + metadata + ) + ) } catch (error) { // Skip verbs that fail to load } diff --git a/src/types/brainy.types.ts b/src/types/brainy.types.ts index 0e958d01..184ff49b 100644 --- a/src/types/brainy.types.ts +++ b/src/types/brainy.types.ts @@ -6,6 +6,12 @@ import { StorageAdapter, Vector } from '../coreTypes.js' import { NounType, VerbType } from './graphTypes.js' +import type { + EntityMetadataInput, + EntityMetadataPatch, + RelationMetadataInput, + RelationMetadataPatch +} from './reservedFields.js' // ============= Core Types ============= @@ -283,8 +289,18 @@ export interface AddParams { * aggregation on the standard-field fast path. */ subtype?: string - /** Structured queryable fields — indexed by MetadataIndex, used in `where` filters */ - metadata?: T + /** + * Structured queryable fields — indexed by MetadataIndex, used in `where` filters. + * + * Reserved entity fields (`RESERVED_ENTITY_FIELDS` — `noun`, `subtype`, `createdAt`, + * `updatedAt`, `confidence`, `weight`, `service`, `data`, `createdBy`, `_rev`) may NOT + * appear here — they have dedicated top-level params and the type makes a literal + * reserved key a compile error. Untyped (JavaScript) callers that pass one anyway are + * normalized at write time: user-settable fields remap to their top-level param + * (top-level wins when both are supplied), system-managed fields are dropped with a + * one-shot warning. + */ + metadata?: EntityMetadataInput /** Custom entity ID (auto-generated UUID v4 if not provided) */ id?: string /** Pre-computed embedding vector (skips auto-embedding when provided) */ @@ -314,7 +330,15 @@ export interface UpdateParams { data?: any // New content to re-embed type?: NounType // Change type subtype?: string // Change subtype (set to '' or null-equivalent via dedicated unset is future work) - metadata?: Partial // Metadata to update + /** + * Metadata fields to merge (or replace when `merge: false`). Reserved entity + * fields (`RESERVED_ENTITY_FIELDS`) may NOT appear here — `confidence` / + * `weight` / `subtype` have dedicated params on this call, and the rest are + * system-managed. A literal reserved key is a compile error; untyped callers + * are normalized at write time (remap user-settable, drop system-managed + * with a one-shot warning). + */ + metadata?: EntityMetadataPatch merge?: boolean // Merge or replace metadata (default: true) vector?: Vector // New pre-computed vector confidence?: number // Update type classification confidence @@ -355,8 +379,14 @@ export interface RelateParams { weight?: number /** Content for the relationship (optional — overrides auto-computed vector) */ data?: any - /** Structured queryable fields on the edge */ - metadata?: T + /** + * Structured queryable fields on the edge. Reserved relationship fields + * (`RESERVED_RELATION_FIELDS` — `verb`, `subtype`, `createdAt`, `updatedAt`, + * `confidence`, `weight`, `service`, `data`, `createdBy`, `_rev`) may NOT + * appear here — they have dedicated params. A literal reserved key is a + * compile error; untyped callers are normalized at write time. + */ + metadata?: RelationMetadataInput /** Create reverse edge too (default: false) */ bidirectional?: boolean /** Multi-tenancy service identifier */ @@ -377,7 +407,13 @@ export interface UpdateRelationParams { weight?: number // New weight confidence?: number // New confidence (0-1) data?: any // New content - metadata?: Partial // Metadata to update + /** + * Metadata fields to merge (or replace when `merge: false`). Reserved + * relationship fields (`RESERVED_RELATION_FIELDS`) may NOT appear here — + * a literal reserved key is a compile error; untyped callers are + * normalized at write time. + */ + metadata?: RelationMetadataPatch merge?: boolean // Merge or replace metadata } diff --git a/src/types/reservedFields.ts b/src/types/reservedFields.ts new file mode 100644 index 00000000..a67b0a5c --- /dev/null +++ b/src/types/reservedFields.ts @@ -0,0 +1,250 @@ +/** + * @module types/reservedFields + * @description The canonical reserved-field contract — ONE place that defines + * which keys belong to Brainy (top-level entity/relationship fields) and may + * therefore never live inside a `metadata` bag. + * + * Three layers enforce the contract, all driven by the constants below: + * + * 1. **Compile time** — `AddParams.metadata`, `UpdateParams.metadata`, + * `RelateParams.metadata` and `UpdateRelationParams.metadata` are typed so + * a literal reserved key is a TypeScript error (see + * {@link EntityMetadataInput} / {@link RelationMetadataInput}). + * 2. **Write time** — for untyped (JavaScript) callers that smuggle a + * reserved key past the compiler anyway, every write path normalizes the + * bag: user-mutable fields are remapped to their dedicated top-level + * param (top-level wins when both are supplied) and system-managed fields + * are dropped with a one-shot warning naming the correct write path. + * 3. **Read time** — every read path splits the stored flat record through + * {@link splitNounMetadataRecord} / {@link splitVerbMetadataRecord}, so a + * reserved field is surfaced ONLY at top level and `entity.metadata` / + * `relation.metadata` contain ONLY custom fields, always — live reads, + * batch reads, and historical (`asOf`) reads alike. + * + * Documented for consumers in `docs/concepts/consistency-model.md` + * ("Reserved fields"). + */ + +/** + * @description Entity (noun) field names reserved by Brainy. These keys are + * stored in the flat per-entity metadata record alongside custom fields, but + * they belong to Brainy: every read path extracts them to top-level + * `Entity` fields, and no write path accepts them inside `metadata`. + * + * | Key | Canonical write path | + * |-----|----------------------| + * | `noun` | the `type` param of `add()` / `update()` (stored under the key `noun`) | + * | `subtype` | the `subtype` param | + * | `createdAt` | system-managed — set once at `add()` time | + * | `updatedAt` | system-managed — set on every write | + * | `confidence` | the `confidence` param | + * | `weight` | the `weight` param | + * | `service` | the `service` param of `add()` (immutable afterwards) | + * | `data` | the `data` param | + * | `createdBy` | the `createdBy` param of `add()` (immutable afterwards) | + * | `_rev` | system-managed revision counter — pass `ifRev` to `update()` for CAS | + * + * @example + * import { RESERVED_ENTITY_FIELDS } from '@soulcraft/brainy' + * const isReserved = (key: string) => + * (RESERVED_ENTITY_FIELDS as readonly string[]).includes(key) + */ +export const RESERVED_ENTITY_FIELDS = [ + 'noun', + 'subtype', + 'createdAt', + 'updatedAt', + 'confidence', + 'weight', + 'service', + 'data', + 'createdBy', + '_rev' +] as const + +/** + * @description Union of the entity field names reserved by Brainy — the + * element type of {@link RESERVED_ENTITY_FIELDS}. + */ +export type ReservedEntityField = (typeof RESERVED_ENTITY_FIELDS)[number] + +/** + * @description Relationship (verb) field names reserved by Brainy — the verb + * mirror of {@link RESERVED_ENTITY_FIELDS}. The stored flat record keys the + * relationship type under `verb` (the public `Relation` field is `type`); + * everything else matches the entity list. + * + * | Key | Canonical write path | + * |-----|----------------------| + * | `verb` | the `type` param of `relate()` / `updateRelation()` (stored under the key `verb`) | + * | `subtype` | the `subtype` param | + * | `createdAt` | system-managed — set once at `relate()` time | + * | `updatedAt` | system-managed — set on every write | + * | `confidence` | the `confidence` param | + * | `weight` | the `weight` param | + * | `service` | the `service` param of `relate()` (immutable afterwards) | + * | `data` | the `data` param | + * | `createdBy` | system-managed | + * | `_rev` | system-managed | + */ +export const RESERVED_RELATION_FIELDS = [ + 'verb', + 'subtype', + 'createdAt', + 'updatedAt', + 'confidence', + 'weight', + 'service', + 'data', + 'createdBy', + '_rev' +] as const + +/** + * @description Union of the relationship field names reserved by Brainy — + * the element type of {@link RESERVED_RELATION_FIELDS}. + */ +export type ReservedRelationField = (typeof RESERVED_RELATION_FIELDS)[number] + +/** + * @description `true` when `T` is exactly `any` (the classic + * `0 extends 1 & T` probe — only `any` absorbs the impossible intersection). + * Used to keep the reserved-key guard active for untyped brains, where a + * plain `T & guard` intersection would collapse to `any` and check nothing. + */ +type IsAny = 0 extends 1 & T ? true : false + +/** + * @description Compile-time tripwire: marks every reserved entity key as + * `never` so an object literal carrying one fails to type-check. Keys that + * `T` itself declares (including via an index signature, where + * `keyof T = string`) are exempted — a consumer who *explicitly* types a + * reserved key into their metadata shape keeps a working (if unwise) type, + * and index-signature metadata types remain assignable. + */ +export type NoReservedEntityKeys = { + readonly [K in ReservedEntityField as K extends keyof T ? never : K]?: never +} + +/** + * @description Relationship mirror of {@link NoReservedEntityKeys}. + */ +export type NoReservedRelationKeys = { + readonly [K in ReservedRelationField as K extends keyof T ? never : K]?: never +} + +/** + * @description The metadata bag shape for untyped brains (`T = any`): an + * open index signature (any custom key, any value — exactly the pre-8.0 + * latitude) intersected with the reserved-key guard, whose declared + * `?: never` properties take precedence over the index signature so a + * literal reserved key is still a compile error. + */ +type OpenBag = { [key: string]: any } & Guard + +/** + * @description The type of `AddParams.metadata`: the consumer's metadata + * shape `T` with reserved entity keys forbidden at compile time. For untyped + * brains (`T = any`) the bag stays open ({@link OpenBag}), so arbitrary + * custom fields remain legal while literal reserved keys still error. + */ +export type EntityMetadataInput = IsAny extends true + ? OpenBag> + : T & NoReservedEntityKeys + +/** + * @description The type of `UpdateParams.metadata`: a partial patch of the + * consumer's metadata shape with reserved entity keys forbidden at compile + * time. Same `T = any` handling as {@link EntityMetadataInput}. + */ +export type EntityMetadataPatch = IsAny extends true + ? OpenBag> + : Partial & NoReservedEntityKeys + +/** + * @description The type of `RelateParams.metadata`: the consumer's edge + * metadata shape with reserved relationship keys forbidden at compile time. + */ +export type RelationMetadataInput = IsAny extends true + ? OpenBag> + : T & NoReservedRelationKeys + +/** + * @description The type of `UpdateRelationParams.metadata`: a partial patch + * of the consumer's edge metadata shape with reserved relationship keys + * forbidden at compile time. + */ +export type RelationMetadataPatch = IsAny extends true + ? OpenBag> + : Partial & NoReservedRelationKeys + +/** + * @description Result of splitting a stored flat metadata record into its + * reserved (Brainy-owned) and custom (consumer-owned) halves. + */ +export interface SplitMetadataRecord { + /** The reserved fields present in the record, keyed by reserved name. */ + reserved: Partial> + /** Every other key — the consumer's custom metadata, and nothing else. */ + custom: Record +} + +const RESERVED_ENTITY_SET: ReadonlySet = new Set(RESERVED_ENTITY_FIELDS) +const RESERVED_RELATION_SET: ReadonlySet = new Set(RESERVED_RELATION_FIELDS) + +/** + * @description Shared splitter — partitions a record's keys against a + * reserved-name set. `null`/`undefined` records split to two empty objects. + * @param record - The stored flat metadata record (reserved + custom keys mixed). + * @param reservedSet - The reserved-name set to partition against. + * @returns The `{ reserved, custom }` halves. + */ +function splitRecord( + record: Record | null | undefined, + reservedSet: ReadonlySet +): SplitMetadataRecord { + const reserved: Record = {} + const custom: Record = {} + if (record && typeof record === 'object') { + for (const [key, value] of Object.entries(record)) { + if (reservedSet.has(key)) { + reserved[key] = value + } else { + custom[key] = value + } + } + } + return { reserved: reserved as Partial>, custom } +} + +/** + * @description Split a stored entity (noun) flat metadata record into + * reserved fields and custom metadata — THE canonical read-side split. Every + * entity read path (live `get()`, batch reads, paginated listings, and + * historical `asOf()` materialization) goes through this function, so the + * reserved list can never drift between read paths. + * @param record - The stored flat metadata record. + * @returns `reserved` (Brainy-owned fields) and `custom` (the consumer's metadata bag). + * @example + * const { reserved, custom } = splitNounMetadataRecord(stored) + * // reserved.noun → entity.type, reserved.confidence → entity.confidence, … + * // custom → entity.metadata (custom fields only, always) + */ +export function splitNounMetadataRecord( + record: Record | null | undefined +): SplitMetadataRecord { + return splitRecord(record, RESERVED_ENTITY_SET) +} + +/** + * @description Split a stored relationship (verb) flat metadata record into + * reserved fields and custom metadata — the verb mirror of + * {@link splitNounMetadataRecord}, used by every relationship read path. + * @param record - The stored flat metadata record. + * @returns `reserved` (Brainy-owned fields) and `custom` (the consumer's metadata bag). + */ +export function splitVerbMetadataRecord( + record: Record | null | undefined +): SplitMetadataRecord { + return splitRecord(record, RESERVED_RELATION_SET) +} diff --git a/tests/configs/tsconfig.typecheck.json b/tests/configs/tsconfig.typecheck.json new file mode 100644 index 00000000..1d5d1ac1 --- /dev/null +++ b/tests/configs/tsconfig.typecheck.json @@ -0,0 +1,10 @@ +{ + "extends": "../../tsconfig.json", + "compilerOptions": { + "noEmit": true, + "declaration": false, + "sourceMap": false, + "rootDir": "../.." + }, + "include": ["../unit/types/**/*.test-d.ts", "../../src/**/*.d.ts"] +} diff --git a/tests/configs/vitest.unit.config.ts b/tests/configs/vitest.unit.config.ts index e67b48c0..dacb07dc 100644 --- a/tests/configs/vitest.unit.config.ts +++ b/tests/configs/vitest.unit.config.ts @@ -34,6 +34,16 @@ export default defineConfig({ 'node_modules/**' ], + // Compile-time tests (*.test-d.ts) — validates the reserved-metadata-key + // guard (@ts-expect-error assertions in tests/unit/types/) with tsc on + // every unit run. See src/types/reservedFields.ts for the contract. + typecheck: { + enabled: true, + checker: 'tsc', + include: ['tests/unit/types/**/*.test-d.ts'], + tsconfig: './tests/configs/tsconfig.typecheck.json' + }, + // Use 'forks' for process isolation — 'threads' causes vitest internal // "Timeout calling onTaskUpdate" RPC errors under heavy parallel load pool: 'forks', diff --git a/tests/unit/brainy/update-reserved-metadata-remap.test.ts b/tests/unit/brainy/update-reserved-metadata-remap.test.ts new file mode 100644 index 00000000..f0c6a4f4 --- /dev/null +++ b/tests/unit/brainy/update-reserved-metadata-remap.test.ts @@ -0,0 +1,393 @@ +/** + * @module tests/unit/brainy/update-reserved-metadata-remap + * @description Regression tests for the reserved-field metadata-bag trap, + * ported from the 7.x fix and extended to the full 8.0 contract. + * + * History: `add({metadata: {confidence}})` lifted reserved fields to their + * canonical top-level location, but `update({metadata: {confidence}})` + * silently dropped the same shape — the patch value survived the merge and + * was then clobbered by the preserve-existing spread. A production + * consumer's confidence-evolution writes no-oped for weeks before being + * caught by reading values back. + * + * 8.0 contract under test (every write path, entities AND relationships): + * - user-mutable reserved fields (`confidence`, `weight`, `subtype` — plus + * `service`/`createdBy` at add()/relate() time) remap from the metadata + * bag to their dedicated top-level param, with top-level winning when both + * are present; + * - system-managed reserved fields (`createdAt`, `_rev`, `noun`/`verb`, + * `data`, …) are dropped from the bag (one-shot warning); + * - the same normalization applies to `transact()` operations and `with()` + * speculative views; + * - reads NEVER echo a reserved field inside `metadata`. + * + * TypeScript callers can't write these shapes at all (compile-time guard on + * the metadata param types — see tests/unit/types/reserved-metadata-keys.test-d.ts); + * these tests simulate untyped (JavaScript) callers, hence the `as object` + * widenings on the metadata literals. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { Brainy } from '../../../src/index.js' +import { NounType, VerbType } from '../../../src/types/graphTypes.js' +import { createTestConfig } from '../../helpers/test-factory.js' + +describe('reserved-field metadata remap (8.0 contract)', () => { + let brain: Brainy + + beforeEach(async () => { + brain = new Brainy(createTestConfig()) + await brain.init() + }) + + afterEach(async () => { + await brain.close() + }) + + describe('update() — the ported 7.x regression', () => { + it('remaps metadata.confidence to the top-level field (the production repro)', async () => { + const id = await brain.add({ + type: NounType.Concept, + subtype: 'general', + data: 'x', + metadata: { confidence: 0.8 } as object + }) + + // Top-level write works (always did) + await brain.update({ id, confidence: 0.42 }) + let entity = await brain.get(id) + expect(entity?.confidence).toBe(0.42) + + // Metadata-patch write — silently dropped pre-fix, remapped now + await brain.update({ id, metadata: { confidence: 0.33 } as object }) + entity = await brain.get(id) + expect(entity?.confidence).toBe(0.33) + // The reserved key must not linger inside the metadata bag + expect((entity?.metadata as Record)?.confidence).toBeUndefined() + }) + + it('remaps metadata.weight and metadata.subtype the same way', async () => { + const id = await brain.add({ + type: NounType.Concept, + subtype: 'general', + data: 'y', + metadata: {} + }) + + await brain.update({ id, metadata: { weight: 0.7, subtype: 'specialized' } as object }) + const entity = await brain.get(id) + expect(entity?.weight).toBe(0.7) + expect(entity?.subtype).toBe('specialized') + expect((entity?.metadata as Record)?.weight).toBeUndefined() + expect((entity?.metadata as Record)?.subtype).toBeUndefined() + }) + + it('top-level param wins when both top-level and metadata-patch carry the field', async () => { + const id = await brain.add({ + type: NounType.Concept, + subtype: 'general', + data: 'z', + metadata: { confidence: 0.5 } as object + }) + + await brain.update({ id, confidence: 0.9, metadata: { confidence: 0.1 } as object }) + const entity = await brain.get(id) + expect(entity?.confidence).toBe(0.9) + }) + + it('drops system-managed fields from patches without corrupting the entity', async () => { + const id = await brain.add({ + type: NounType.Concept, + subtype: 'general', + data: 'w', + metadata: { keep: 'me' } + }) + const before = await brain.get(id) + + await brain.update({ + id, + metadata: { createdAt: 1, _rev: 999, noun: 'organization', other: 'applied' } as object + }) + const after = await brain.get(id) + + expect(after?.createdAt).toBe(before?.createdAt) // immutable + expect(after?.type).toBe('concept') // noun patch ignored + expect(after?._rev).toBe((before?._rev ?? 1) + 1) // _rev patch ignored; normal bump applied + expect((after?.metadata as Record)?.other).toBe('applied') // custom fields still merge + expect((after?.metadata as Record)?.keep).toBe('me') + expect((after?.metadata as Record)?._rev).toBeUndefined() + expect((after?.metadata as Record)?.createdAt).toBeUndefined() + expect((after?.metadata as Record)?.noun).toBeUndefined() + }) + + it('custom (non-reserved) metadata patches are unaffected by the remap', async () => { + const id = await brain.add({ + type: NounType.Concept, + subtype: 'general', + data: 'v', + metadata: { status: 'draft' } + }) + + await brain.update({ id, metadata: { status: 'reviewed', rating: 4.5 } }) + const entity = await brain.get(id) + expect((entity?.metadata as Record)?.status).toBe('reviewed') + expect((entity?.metadata as Record)?.rating).toBe(4.5) + }) + }) + + describe('add() — explicit lift, identical contract', () => { + it('lifts confidence/weight/subtype out of the bag to top level', async () => { + const id = await brain.add({ + type: NounType.Person, + data: 'lift check', + metadata: { confidence: 0.8, weight: 0.6, subtype: 'employee', dept: 'eng' } as object + }) + + const entity = await brain.get(id) + expect(entity?.confidence).toBe(0.8) + expect(entity?.weight).toBe(0.6) + expect(entity?.subtype).toBe('employee') + expect(entity?.metadata).toEqual({ dept: 'eng' }) + }) + + it('lifts service (settable at add time) and lets the top-level param win', async () => { + const lifted = await brain.add({ + type: NounType.Person, + subtype: 'employee', + data: 'service lift', + metadata: { service: 'orders' } as object + }) + expect((await brain.get(lifted))?.service).toBe('orders') + + const topLevelWins = await brain.add({ + type: NounType.Person, + subtype: 'employee', + data: 'service precedence', + service: 'billing', + metadata: { service: 'orders' } as object + }) + const entity = await brain.get(topLevelWins) + expect(entity?.service).toBe('billing') + expect((entity?.metadata as Record)?.service).toBeUndefined() + }) + + it('a remapped subtype satisfies subtype enforcement like a top-level one', async () => { + brain.requireSubtype(NounType.Document) + + // Top-level missing, but the bag carries it — must not throw. + const id = await brain.add({ + type: NounType.Document, + data: 'enforcement via remap', + metadata: { subtype: 'invoice' } as object + }) + expect((await brain.get(id))?.subtype).toBe('invoice') + + // Neither place carries it — must throw. + await expect( + brain.add({ type: NounType.Document, data: 'no subtype anywhere' }) + ).rejects.toThrow(/subtype/) + }) + }) + + describe('transact() — same remap on add and update ops', () => { + it('normalizes reserved fields in transact add + update ops', async () => { + const db1 = await brain.transact([ + { + op: 'add', + type: NounType.Concept, + subtype: 'general', + data: 'tx', + metadata: { confidence: 0.7, custom: 'a' } as object + } + ]) + const id = db1.receipt!.ids[0] + + let entity = await brain.get(id) + expect(entity?.confidence).toBe(0.7) + expect(entity?.metadata).toEqual({ custom: 'a' }) + + await brain.transact([ + { op: 'update', id, metadata: { confidence: 0.25, custom: 'b' } as object } + ]) + entity = await brain.get(id) + expect(entity?.confidence).toBe(0.25) + expect(entity?.metadata).toEqual({ custom: 'b' }) + expect((entity?.metadata as Record)?.confidence).toBeUndefined() + }) + + it('historical asOf() reads surface reserved fields ONLY top-level', async () => { + const db1 = await brain.transact([ + { + op: 'add', + type: NounType.Concept, + subtype: 'general', + data: 'historical', + metadata: { confidence: 0.9, custom: 'past' } as object + } + ]) + const id = db1.receipt!.ids[0] + + // Move the world forward so generation db1 is historical. + await brain.transact([{ op: 'update', id, confidence: 0.1, metadata: { custom: 'now' } }]) + + const past = await brain.asOf(db1.generation) + const historical = await past.get(id) + expect(historical?.confidence).toBe(0.9) + expect(historical?.metadata).toEqual({ custom: 'past' }) + await past.release() + }) + + it('with() speculative views apply the same normalization', async () => { + const base = await brain.now() + const speculative = await base.with([ + { + op: 'add', + id: 'spec-entity', + type: NounType.Concept, + subtype: 'general', + data: 'spec', + metadata: { confidence: 0.65, custom: 'spec' } as object + } + ]) + + const entity = await speculative.get('spec-entity') + expect(entity?.confidence).toBe(0.65) + expect(entity?.metadata).toEqual({ custom: 'spec' }) + await speculative.release() + await base.release() + }) + }) + + describe('read paths never echo reserved fields inside metadata', () => { + it('find() (storage pagination path) returns custom-only metadata with reserved fields top-level', async () => { + const id = await brain.add({ + type: NounType.Person, + subtype: 'employee', + data: 'pagination echo check', + confidence: 0.8, + weight: 0.6, + metadata: { dept: 'eng' } + }) + + // No query/filter → served by the direct storage pagination path + // (getNounsWithPagination), which historically echoed the full flat + // record (noun/subtype/createdAt/… inside metadata). + const results = await brain.find({ limit: 50 }) + const result = results.find((r) => r.id === id) + expect(result).toBeDefined() + expect(result?.entity.metadata).toEqual({ dept: 'eng' }) + expect(result?.entity.type).toBe(NounType.Person) + expect(result?.entity.subtype).toBe('employee') + expect(result?.entity.confidence).toBe(0.8) + expect(result?.entity.weight).toBe(0.6) + expect(typeof result?.entity.createdAt).toBe('number') + expect(result?.entity._rev).toBe(1) + }) + + it('getRelations() by target surfaces reserved fields top-level, custom-only metadata', async () => { + const a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'src' }) + const b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'tgt' }) + const relId = await brain.relate({ + from: a, + to: b, + type: VerbType.ReportsTo, + subtype: 'direct', + confidence: 0.9, + weight: 0.5, + service: 'orders', + metadata: { note: 'target path' } + }) + + const relations = await brain.getRelations({ to: b }) + const rel = relations.find((r) => r.id === relId) + expect(rel).toBeDefined() + expect(rel?.metadata).toEqual({ note: 'target path' }) + expect(rel?.subtype).toBe('direct') + expect(rel?.confidence).toBe(0.9) + expect(rel?.weight).toBe(0.5) + expect(rel?.service).toBe('orders') + expect(typeof rel?.createdAt).toBe('number') + }) + }) + + describe('relationships — relate() / updateRelation() mirror', () => { + let a: string + let b: string + + beforeEach(async () => { + a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'A' }) + b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'B' }) + }) + + it('relate() persists the top-level confidence and service params', async () => { + const relId = await brain.relate({ + from: a, + to: b, + type: VerbType.ReportsTo, + subtype: 'direct', + confidence: 0.77, + service: 'orders' + }) + + const relations = await brain.getRelations({ from: a }) + const rel = relations.find((r) => r.id === relId) + expect(rel?.confidence).toBe(0.77) + expect(rel?.service).toBe('orders') + }) + + it('relate() remaps reserved fields out of the metadata bag', async () => { + const relId = await brain.relate({ + from: a, + to: b, + type: VerbType.RelatedTo, + subtype: 'colleague', + metadata: { confidence: 0.4, weight: 0.3, role: 'peer' } as object + }) + + const relations = await brain.getRelations({ from: a }) + const rel = relations.find((r) => r.id === relId) + expect(rel?.confidence).toBe(0.4) + expect(rel?.weight).toBe(0.3) + expect(rel?.metadata).toEqual({ role: 'peer' }) + }) + + it('relation.metadata never echoes the verb type key', async () => { + const relId = await brain.relate({ + from: a, + to: b, + type: VerbType.RelatedTo, + subtype: 'colleague', + metadata: { note: 'no echo' } + }) + + const relations = await brain.getRelations({ from: a }) + const rel = relations.find((r) => r.id === relId) + expect(rel?.type).toBe(VerbType.RelatedTo) + expect((rel?.metadata as Record)?.verb).toBeUndefined() + expect(rel?.metadata).toEqual({ note: 'no echo' }) + }) + + it('updateRelation() remaps the user-mutable trio and preserves service', async () => { + const relId = await brain.relate({ + from: a, + to: b, + type: VerbType.ReportsTo, + subtype: 'direct', + service: 'orders', + metadata: { keep: 'me' } + }) + + await brain.updateRelation({ + id: relId, + metadata: { confidence: 0.55, subtype: 'dotted-line', extra: 'applied' } as object + }) + + const relations = await brain.getRelations({ from: a }) + const rel = relations.find((r) => r.id === relId) + expect(rel?.confidence).toBe(0.55) + expect(rel?.subtype).toBe('dotted-line') + expect(rel?.service).toBe('orders') // fixed at relate() time, never erased by updates + expect(rel?.metadata).toEqual({ keep: 'me', extra: 'applied' }) + }) + }) +}) diff --git a/tests/unit/types/reserved-metadata-keys.test-d.ts b/tests/unit/types/reserved-metadata-keys.test-d.ts new file mode 100644 index 00000000..37fceefa --- /dev/null +++ b/tests/unit/types/reserved-metadata-keys.test-d.ts @@ -0,0 +1,265 @@ +/** + * @module tests/unit/types/reserved-metadata-keys.test-d + * @description Compile-time tests for the reserved-field contract (layer 1 of + * three — see src/types/reservedFields.ts): a literal reserved key inside any + * `metadata` param is a TypeScript error, while the generic `T` ergonomics + * stay intact (typed bags, untyped brains, index-signature shapes, and the + * documented exemption for consumers who explicitly declare a reserved key in + * their own metadata type). + * + * Runs under vitest typecheck mode (`test.typecheck` in + * tests/configs/vitest.unit.config.ts) — these assertions are validated by + * `tsc`, never executed. The runtime half of the contract (the write-path + * remap for untyped callers) is pinned by + * tests/unit/brainy/update-reserved-metadata-remap.test.ts. + */ + +import { describe, it, assertType } from 'vitest' +import type { + AddParams, + UpdateParams, + RelateParams, + UpdateRelationParams, + TxOperation +} from '../../../src/index.js' +import { NounType, VerbType } from '../../../src/types/graphTypes.js' + +describe('reserved entity keys in metadata are compile errors', () => { + it('AddParams (untyped brain) rejects every reserved key but stays open for custom fields', () => { + // Custom fields of any shape remain legal — exactly the pre-8.0 latitude. + assertType({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + metadata: { dept: 'eng', level: 3, tags: ['a', 'b'], nested: { ok: true } } + }) + + assertType({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + // @ts-expect-error — 'noun' is reserved (the entity type travels via the top-level 'type' param) + metadata: { noun: 'organization' } + }) + assertType({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + // @ts-expect-error — 'subtype' is reserved (use the top-level 'subtype' param) + metadata: { subtype: 'contractor' } + }) + assertType({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + // @ts-expect-error — 'createdAt' is reserved (system-managed) + metadata: { createdAt: Date.now() } + }) + assertType({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + // @ts-expect-error — 'updatedAt' is reserved (system-managed) + metadata: { updatedAt: Date.now() } + }) + assertType({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + // @ts-expect-error — 'confidence' is reserved (use the top-level 'confidence' param) + metadata: { confidence: 0.8 } + }) + assertType({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + // @ts-expect-error — 'weight' is reserved (use the top-level 'weight' param) + metadata: { weight: 0.5 } + }) + assertType({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + // @ts-expect-error — 'service' is reserved (use the top-level 'service' param) + metadata: { service: 'orders' } + }) + assertType({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + // @ts-expect-error — 'data' is reserved (use the top-level 'data' param) + metadata: { data: 'content' } + }) + assertType({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + // @ts-expect-error — 'createdBy' is reserved (use the top-level 'createdBy' param) + metadata: { createdBy: { augmentation: 'importer', version: '1.0' } } + }) + assertType({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + // @ts-expect-error — '_rev' is reserved (system-managed revision counter) + metadata: { _rev: 7 } + }) + }) + + it('AddParams (typed brain) rejects reserved keys alongside the declared shape', () => { + interface EmployeeMeta { + dept: string + level: number + } + + assertType>({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + metadata: { dept: 'eng', level: 3 } + }) + + assertType>({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + // @ts-expect-error — 'confidence' is reserved even when T declares other fields + metadata: { dept: 'eng', level: 3, confidence: 0.8 } + }) + }) + + it('documented exemptions: T-declared reserved keys and index-signature shapes stay assignable', () => { + // A consumer who *explicitly* types a reserved key into their metadata + // shape keeps a working (if unwise) type — the guard exempts keyof T. + interface LegacyMeta { + confidence: number + note: string + } + assertType>({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + metadata: { confidence: 0.8, note: 'declared by the consumer type' } + }) + + // Index-signature metadata types (keyof T = string) remain fully open. + assertType>>({ + type: NounType.Person, + subtype: 'employee', + data: 'x', + metadata: { anything: 'goes', confidence: 0.8 } + }) + }) + + it('UpdateParams patch rejects reserved keys but accepts partial custom patches', () => { + interface EmployeeMeta { + dept: string + level: number + } + + // Partial patch of the declared shape is legal. + assertType>({ id: 'e1', metadata: { dept: 'sales' } }) + // Untyped patch with custom fields is legal. + assertType({ id: 'e1', metadata: { status: 'reviewed', rating: 4.5 } }) + + // @ts-expect-error — 'confidence' is reserved (use the top-level 'confidence' param) + assertType({ id: 'e1', metadata: { confidence: 0.33 } }) + // @ts-expect-error — 'subtype' is reserved (use the top-level 'subtype' param) + assertType({ id: 'e1', metadata: { subtype: 'specialized' } }) + // @ts-expect-error — '_rev' is reserved (pass 'ifRev' for optimistic concurrency) + assertType({ id: 'e1', metadata: { _rev: 3 } }) + // @ts-expect-error — 'confidence' is reserved even when T declares other fields + assertType>({ id: 'e1', metadata: { confidence: 0.1 } }) + }) +}) + +describe('reserved relationship keys in metadata are compile errors', () => { + it('RelateParams rejects reserved keys but stays open for custom edge fields', () => { + assertType({ + from: 'a', + to: 'b', + type: VerbType.ReportsTo, + subtype: 'direct', + metadata: { role: 'peer', since: 2024 } + }) + + assertType({ + from: 'a', + to: 'b', + type: VerbType.ReportsTo, + subtype: 'direct', + // @ts-expect-error — 'verb' is reserved (the relationship type travels via the top-level 'type' param) + metadata: { verb: 'relatedTo' } + }) + assertType({ + from: 'a', + to: 'b', + type: VerbType.ReportsTo, + subtype: 'direct', + // @ts-expect-error — 'confidence' is reserved (use the top-level 'confidence' param) + metadata: { confidence: 0.9 } + }) + assertType({ + from: 'a', + to: 'b', + type: VerbType.ReportsTo, + subtype: 'direct', + // @ts-expect-error — 'weight' is reserved (use the top-level 'weight' param) + metadata: { weight: 0.4 } + }) + assertType({ + from: 'a', + to: 'b', + type: VerbType.ReportsTo, + subtype: 'direct', + // @ts-expect-error — 'service' is reserved (use the top-level 'service' param) + metadata: { service: 'orders' } + }) + }) + + it('UpdateRelationParams patch rejects reserved keys', () => { + assertType({ id: 'r1', metadata: { note: 'fine' } }) + + // @ts-expect-error — 'confidence' is reserved (use the top-level 'confidence' param) + assertType({ id: 'r1', metadata: { confidence: 0.5 } }) + // @ts-expect-error — 'subtype' is reserved (use the top-level 'subtype' param) + assertType({ id: 'r1', metadata: { subtype: 'dotted-line' } }) + // @ts-expect-error — 'createdAt' is reserved (system-managed) + assertType({ id: 'r1', metadata: { createdAt: 1 } }) + }) +}) + +describe('transact() operations inherit the same guard', () => { + it('TxOperation add/update/relate metadata rejects reserved keys', () => { + assertType({ + op: 'add', + type: NounType.Concept, + subtype: 'general', + data: 'tx', + metadata: { custom: 'a' } + }) + assertType({ + op: 'add', + type: NounType.Concept, + subtype: 'general', + data: 'tx', + // @ts-expect-error — 'confidence' is reserved on transact add ops too + metadata: { confidence: 0.7 } + }) + assertType({ + op: 'update', + id: 'e1', + // @ts-expect-error — 'weight' is reserved on transact update ops too + metadata: { weight: 0.2 } + }) + assertType({ + op: 'relate', + from: 'a', + to: 'b', + type: VerbType.RelatedTo, + subtype: 'colleague', + // @ts-expect-error — 'verb' is reserved on transact relate ops too + metadata: { verb: 'contains' } + }) + }) +})