feat(8.0): reserved-field contract — one canonical location, typed prevention, unified read/write

Brainy-owned field names (noun/verb, subtype, createdAt, updatedAt,
confidence, weight, service, data, createdBy, _rev) now have exactly one
home — top level — enforced by three layers driven from a single source of
truth, src/types/reservedFields.ts (RESERVED_ENTITY_FIELDS /
RESERVED_RELATION_FIELDS, exported):

1. Compile time — AddParams/UpdateParams/RelateParams/UpdateRelationParams
   metadata (and the transact() ops that extend them) reject a literal
   reserved key as a TypeScript error while keeping generic T ergonomics
   (typed bags, untyped brains, index-signature shapes, and a documented
   exemption for T-declared reserved keys). Pinned by @ts-expect-error
   type tests run under vitest typecheck mode on every unit run.

2. Write time — the 7.x update() remap is ported to 8.0 and extended to
   every write path: add/update/relate/updateRelation, their transact()
   mirrors, and db.with() overlays. User-settable fields lift to their
   dedicated param (top-level wins when both are supplied — closes the 7.x
   trap where update({metadata:{confidence}}) silently no-oped), and
   system-managed fields drop with a one-shot warning naming the right
   path. A remapped subtype satisfies subtype-pairing enforcement exactly
   like a top-level one.

3. Read time — every storage combine goes through one canonical hydration
   helper (hydrateNounWithMetadata / hydrateVerbWithMetadata over
   splitNoun/VerbMetadataRecord), so reserved fields surface ONLY top-level
   and entity/relation.metadata carry ONLY custom fields on live reads,
   batch reads, paginated listings, getRelations by source/target, streamed
   verbs, and historical asOf() materialization alike.

Read-path echoes found and fixed (previously the full stored record —
including the verb type key — leaked inside metadata): noun pagination,
verb pagination, getVerbsBySource/ByTarget (adjacency + shard fallback),
getVerbsBySourceBatch (which also dropped subtype/data), and the
filesystem verb stream. getRelations() results now surface
confidence/updatedAt top-level via verbsToRelations, updateRelation() no
longer erases service/createdBy, relate() persists its top-level
confidence/service params, and the dead convertHNSWVerbToGraphVerb echo
path is deleted. Import paths (CLI extract, deduplicator, coordinators,
neural import) write confidence through the dedicated param instead of the
bag. UpdateRelationParams is now exported from the package root.

Documented for consumers in docs/concepts/consistency-model.md ("Reserved
fields") and RELEASES.md. Regression tests ported from the 7.x fix and
extended to the full 8.0 contract (17 runtime tests + 41 type-level
assertions); full unit suite 1427/1427, db-mvcc integration 24/24.

tests/unit/brainy/update-reserved-metadata-remap.test.ts

@@ -0,0 +1,393 @@
+/**
+ * @module tests/unit/brainy/update-reserved-metadata-remap
+ * @description Regression tests for the reserved-field metadata-bag trap,
+ * ported from the 7.x fix and extended to the full 8.0 contract.
+ *
+ * History: `add({metadata: {confidence}})` lifted reserved fields to their
+ * canonical top-level location, but `update({metadata: {confidence}})`
+ * silently dropped the same shape — the patch value survived the merge and
+ * was then clobbered by the preserve-existing spread. A production
+ * consumer's confidence-evolution writes no-oped for weeks before being
+ * caught by reading values back.
+ *
+ * 8.0 contract under test (every write path, entities AND relationships):
+ * - user-mutable reserved fields (`confidence`, `weight`, `subtype` — plus
+ *   `service`/`createdBy` at add()/relate() time) remap from the metadata
+ *   bag to their dedicated top-level param, with top-level winning when both
+ *   are present;
+ * - system-managed reserved fields (`createdAt`, `_rev`, `noun`/`verb`,
+ *   `data`, …) are dropped from the bag (one-shot warning);
+ * - the same normalization applies to `transact()` operations and `with()`
+ *   speculative views;
+ * - reads NEVER echo a reserved field inside `metadata`.
+ *
+ * TypeScript callers can't write these shapes at all (compile-time guard on
+ * the metadata param types — see tests/unit/types/reserved-metadata-keys.test-d.ts);
+ * these tests simulate untyped (JavaScript) callers, hence the `as object`
+ * widenings on the metadata literals.
+ */
+
+import { describe, it, expect, beforeEach, afterEach } from 'vitest'
+import { Brainy } from '../../../src/index.js'
+import { NounType, VerbType } from '../../../src/types/graphTypes.js'
+import { createTestConfig } from '../../helpers/test-factory.js'
+
+describe('reserved-field metadata remap (8.0 contract)', () => {
+  let brain: Brainy
+
+  beforeEach(async () => {
+    brain = new Brainy(createTestConfig())
+    await brain.init()
+  })
+
+  afterEach(async () => {
+    await brain.close()
+  })
+
+  describe('update() — the ported 7.x regression', () => {
+    it('remaps metadata.confidence to the top-level field (the production repro)', async () => {
+      const id = await brain.add({
+        type: NounType.Concept,
+        subtype: 'general',
+        data: 'x',
+        metadata: { confidence: 0.8 } as object
+      })
+
+      // Top-level write works (always did)
+      await brain.update({ id, confidence: 0.42 })
+      let entity = await brain.get(id)
+      expect(entity?.confidence).toBe(0.42)
+
+      // Metadata-patch write — silently dropped pre-fix, remapped now
+      await brain.update({ id, metadata: { confidence: 0.33 } as object })
+      entity = await brain.get(id)
+      expect(entity?.confidence).toBe(0.33)
+      // The reserved key must not linger inside the metadata bag
+      expect((entity?.metadata as Record<string, unknown>)?.confidence).toBeUndefined()
+    })
+
+    it('remaps metadata.weight and metadata.subtype the same way', async () => {
+      const id = await brain.add({
+        type: NounType.Concept,
+        subtype: 'general',
+        data: 'y',
+        metadata: {}
+      })
+
+      await brain.update({ id, metadata: { weight: 0.7, subtype: 'specialized' } as object })
+      const entity = await brain.get(id)
+      expect(entity?.weight).toBe(0.7)
+      expect(entity?.subtype).toBe('specialized')
+      expect((entity?.metadata as Record<string, unknown>)?.weight).toBeUndefined()
+      expect((entity?.metadata as Record<string, unknown>)?.subtype).toBeUndefined()
+    })
+
+    it('top-level param wins when both top-level and metadata-patch carry the field', async () => {
+      const id = await brain.add({
+        type: NounType.Concept,
+        subtype: 'general',
+        data: 'z',
+        metadata: { confidence: 0.5 } as object
+      })
+
+      await brain.update({ id, confidence: 0.9, metadata: { confidence: 0.1 } as object })
+      const entity = await brain.get(id)
+      expect(entity?.confidence).toBe(0.9)
+    })
+
+    it('drops system-managed fields from patches without corrupting the entity', async () => {
+      const id = await brain.add({
+        type: NounType.Concept,
+        subtype: 'general',
+        data: 'w',
+        metadata: { keep: 'me' }
+      })
+      const before = await brain.get(id)
+
+      await brain.update({
+        id,
+        metadata: { createdAt: 1, _rev: 999, noun: 'organization', other: 'applied' } as object
+      })
+      const after = await brain.get(id)
+
+      expect(after?.createdAt).toBe(before?.createdAt) // immutable
+      expect(after?.type).toBe('concept') // noun patch ignored
+      expect(after?._rev).toBe((before?._rev ?? 1) + 1) // _rev patch ignored; normal bump applied
+      expect((after?.metadata as Record<string, unknown>)?.other).toBe('applied') // custom fields still merge
+      expect((after?.metadata as Record<string, unknown>)?.keep).toBe('me')
+      expect((after?.metadata as Record<string, unknown>)?._rev).toBeUndefined()
+      expect((after?.metadata as Record<string, unknown>)?.createdAt).toBeUndefined()
+      expect((after?.metadata as Record<string, unknown>)?.noun).toBeUndefined()
+    })
+
+    it('custom (non-reserved) metadata patches are unaffected by the remap', async () => {
+      const id = await brain.add({
+        type: NounType.Concept,
+        subtype: 'general',
+        data: 'v',
+        metadata: { status: 'draft' }
+      })
+
+      await brain.update({ id, metadata: { status: 'reviewed', rating: 4.5 } })
+      const entity = await brain.get(id)
+      expect((entity?.metadata as Record<string, unknown>)?.status).toBe('reviewed')
+      expect((entity?.metadata as Record<string, unknown>)?.rating).toBe(4.5)
+    })
+  })
+
+  describe('add() — explicit lift, identical contract', () => {
+    it('lifts confidence/weight/subtype out of the bag to top level', async () => {
+      const id = await brain.add({
+        type: NounType.Person,
+        data: 'lift check',
+        metadata: { confidence: 0.8, weight: 0.6, subtype: 'employee', dept: 'eng' } as object
+      })
+
+      const entity = await brain.get(id)
+      expect(entity?.confidence).toBe(0.8)
+      expect(entity?.weight).toBe(0.6)
+      expect(entity?.subtype).toBe('employee')
+      expect(entity?.metadata).toEqual({ dept: 'eng' })
+    })
+
+    it('lifts service (settable at add time) and lets the top-level param win', async () => {
+      const lifted = await brain.add({
+        type: NounType.Person,
+        subtype: 'employee',
+        data: 'service lift',
+        metadata: { service: 'orders' } as object
+      })
+      expect((await brain.get(lifted))?.service).toBe('orders')
+
+      const topLevelWins = await brain.add({
+        type: NounType.Person,
+        subtype: 'employee',
+        data: 'service precedence',
+        service: 'billing',
+        metadata: { service: 'orders' } as object
+      })
+      const entity = await brain.get(topLevelWins)
+      expect(entity?.service).toBe('billing')
+      expect((entity?.metadata as Record<string, unknown>)?.service).toBeUndefined()
+    })
+
+    it('a remapped subtype satisfies subtype enforcement like a top-level one', async () => {
+      brain.requireSubtype(NounType.Document)
+
+      // Top-level missing, but the bag carries it — must not throw.
+      const id = await brain.add({
+        type: NounType.Document,
+        data: 'enforcement via remap',
+        metadata: { subtype: 'invoice' } as object
+      })
+      expect((await brain.get(id))?.subtype).toBe('invoice')
+
+      // Neither place carries it — must throw.
+      await expect(
+        brain.add({ type: NounType.Document, data: 'no subtype anywhere' })
+      ).rejects.toThrow(/subtype/)
+    })
+  })
+
+  describe('transact() — same remap on add and update ops', () => {
+    it('normalizes reserved fields in transact add + update ops', async () => {
+      const db1 = await brain.transact([
+        {
+          op: 'add',
+          type: NounType.Concept,
+          subtype: 'general',
+          data: 'tx',
+          metadata: { confidence: 0.7, custom: 'a' } as object
+        }
+      ])
+      const id = db1.receipt!.ids[0]
+
+      let entity = await brain.get(id)
+      expect(entity?.confidence).toBe(0.7)
+      expect(entity?.metadata).toEqual({ custom: 'a' })
+
+      await brain.transact([
+        { op: 'update', id, metadata: { confidence: 0.25, custom: 'b' } as object }
+      ])
+      entity = await brain.get(id)
+      expect(entity?.confidence).toBe(0.25)
+      expect(entity?.metadata).toEqual({ custom: 'b' })
+      expect((entity?.metadata as Record<string, unknown>)?.confidence).toBeUndefined()
+    })
+
+    it('historical asOf() reads surface reserved fields ONLY top-level', async () => {
+      const db1 = await brain.transact([
+        {
+          op: 'add',
+          type: NounType.Concept,
+          subtype: 'general',
+          data: 'historical',
+          metadata: { confidence: 0.9, custom: 'past' } as object
+        }
+      ])
+      const id = db1.receipt!.ids[0]
+
+      // Move the world forward so generation db1 is historical.
+      await brain.transact([{ op: 'update', id, confidence: 0.1, metadata: { custom: 'now' } }])
+
+      const past = await brain.asOf(db1.generation)
+      const historical = await past.get(id)
+      expect(historical?.confidence).toBe(0.9)
+      expect(historical?.metadata).toEqual({ custom: 'past' })
+      await past.release()
+    })
+
+    it('with() speculative views apply the same normalization', async () => {
+      const base = await brain.now()
+      const speculative = await base.with([
+        {
+          op: 'add',
+          id: 'spec-entity',
+          type: NounType.Concept,
+          subtype: 'general',
+          data: 'spec',
+          metadata: { confidence: 0.65, custom: 'spec' } as object
+        }
+      ])
+
+      const entity = await speculative.get('spec-entity')
+      expect(entity?.confidence).toBe(0.65)
+      expect(entity?.metadata).toEqual({ custom: 'spec' })
+      await speculative.release()
+      await base.release()
+    })
+  })
+
+  describe('read paths never echo reserved fields inside metadata', () => {
+    it('find() (storage pagination path) returns custom-only metadata with reserved fields top-level', async () => {
+      const id = await brain.add({
+        type: NounType.Person,
+        subtype: 'employee',
+        data: 'pagination echo check',
+        confidence: 0.8,
+        weight: 0.6,
+        metadata: { dept: 'eng' }
+      })
+
+      // No query/filter → served by the direct storage pagination path
+      // (getNounsWithPagination), which historically echoed the full flat
+      // record (noun/subtype/createdAt/… inside metadata).
+      const results = await brain.find({ limit: 50 })
+      const result = results.find((r) => r.id === id)
+      expect(result).toBeDefined()
+      expect(result?.entity.metadata).toEqual({ dept: 'eng' })
+      expect(result?.entity.type).toBe(NounType.Person)
+      expect(result?.entity.subtype).toBe('employee')
+      expect(result?.entity.confidence).toBe(0.8)
+      expect(result?.entity.weight).toBe(0.6)
+      expect(typeof result?.entity.createdAt).toBe('number')
+      expect(result?.entity._rev).toBe(1)
+    })
+
+    it('getRelations() by target surfaces reserved fields top-level, custom-only metadata', async () => {
+      const a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'src' })
+      const b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'tgt' })
+      const relId = await brain.relate({
+        from: a,
+        to: b,
+        type: VerbType.ReportsTo,
+        subtype: 'direct',
+        confidence: 0.9,
+        weight: 0.5,
+        service: 'orders',
+        metadata: { note: 'target path' }
+      })
+
+      const relations = await brain.getRelations({ to: b })
+      const rel = relations.find((r) => r.id === relId)
+      expect(rel).toBeDefined()
+      expect(rel?.metadata).toEqual({ note: 'target path' })
+      expect(rel?.subtype).toBe('direct')
+      expect(rel?.confidence).toBe(0.9)
+      expect(rel?.weight).toBe(0.5)
+      expect(rel?.service).toBe('orders')
+      expect(typeof rel?.createdAt).toBe('number')
+    })
+  })
+
+  describe('relationships — relate() / updateRelation() mirror', () => {
+    let a: string
+    let b: string
+
+    beforeEach(async () => {
+      a = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'A' })
+      b = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'B' })
+    })
+
+    it('relate() persists the top-level confidence and service params', async () => {
+      const relId = await brain.relate({
+        from: a,
+        to: b,
+        type: VerbType.ReportsTo,
+        subtype: 'direct',
+        confidence: 0.77,
+        service: 'orders'
+      })
+
+      const relations = await brain.getRelations({ from: a })
+      const rel = relations.find((r) => r.id === relId)
+      expect(rel?.confidence).toBe(0.77)
+      expect(rel?.service).toBe('orders')
+    })
+
+    it('relate() remaps reserved fields out of the metadata bag', async () => {
+      const relId = await brain.relate({
+        from: a,
+        to: b,
+        type: VerbType.RelatedTo,
+        subtype: 'colleague',
+        metadata: { confidence: 0.4, weight: 0.3, role: 'peer' } as object
+      })
+
+      const relations = await brain.getRelations({ from: a })
+      const rel = relations.find((r) => r.id === relId)
+      expect(rel?.confidence).toBe(0.4)
+      expect(rel?.weight).toBe(0.3)
+      expect(rel?.metadata).toEqual({ role: 'peer' })
+    })
+
+    it('relation.metadata never echoes the verb type key', async () => {
+      const relId = await brain.relate({
+        from: a,
+        to: b,
+        type: VerbType.RelatedTo,
+        subtype: 'colleague',
+        metadata: { note: 'no echo' }
+      })
+
+      const relations = await brain.getRelations({ from: a })
+      const rel = relations.find((r) => r.id === relId)
+      expect(rel?.type).toBe(VerbType.RelatedTo)
+      expect((rel?.metadata as Record<string, unknown>)?.verb).toBeUndefined()
+      expect(rel?.metadata).toEqual({ note: 'no echo' })
+    })
+
+    it('updateRelation() remaps the user-mutable trio and preserves service', async () => {
+      const relId = await brain.relate({
+        from: a,
+        to: b,
+        type: VerbType.ReportsTo,
+        subtype: 'direct',
+        service: 'orders',
+        metadata: { keep: 'me' }
+      })
+
+      await brain.updateRelation({
+        id: relId,
+        metadata: { confidence: 0.55, subtype: 'dotted-line', extra: 'applied' } as object
+      })
+
+      const relations = await brain.getRelations({ from: a })
+      const rel = relations.find((r) => r.id === relId)
+      expect(rel?.confidence).toBe(0.55)
+      expect(rel?.subtype).toBe('dotted-line')
+      expect(rel?.service).toBe('orders') // fixed at relate() time, never erased by updates
+      expect(rel?.metadata).toEqual({ keep: 'me', extra: 'applied' })
+    })
+  })
+})