open-brainy/docs/concepts/field-addressing.md
David Snelling 24bf6cdbc5
All checks were successful
CI / Node 22 (push) Successful in 12m9s
CI / Node 24 (push) Successful in 12m4s
CI / Bun (latest) (push) Successful in 12m52s
feat(namespace): NO SPECIAL NAMES + storage fidelity — the ruled completion of the field-addressing law
The write side of the law, ruled 2026-08-03: data is either in main space
where developers can use anything, or it is in system.*.

- The reserved-name write door DIES: add/update/relate/updateRelation
  metadata bags accept EVERY name (confidence, type, id, data, level,
  content, ...) as ordinary user fields — indexed, filterable, sortable,
  aggregatable, identical to any other field. The remap/enforce/warn
  machinery, the reservedFieldPolicy config (now a typed init refusal),
  and the compile-time metadata key bans are all removed. The one write
  refusal left: keys spelled 'system.*' (namespace forgery), now enforced
  on all four write doors.
- STORED RECORDS GO NESTED (v2): engine fields top-level, the user bag
  nested verbatim under 'metadata', sealed by a format stamp — by-name
  storage discrimination is unsound once colliders are admitted. Legacy
  flat records stay readable forever through the shape-aware splitters
  (sound for them: the old door refused colliders). Time travel rides the
  same split (generation store snapshots whole records).
- Name-based index exclusions DIE: user frame indexes every name; the
  excludeFields/indexedFields knobs and their silent-[] holes are gone;
  bulk-payload protection is value-shape only, uniform across names.
- Consumer-sweep findings fixed in the same wave: per-type counts read
  the frozen 'system.type' column (addToIndex sort, affinity tracking,
  cold-count rehydration, VFS type bitmaps — legacy 'noun' fallback for
  pre-rebuild reads); resolveHiddenIds addresses 'system.visibility'
  (bare 'visibility' was a silent no-op under the law — VFS/system
  entities leaked into default reads).
- Fidelity fallout fixed in the owning layers: readEntityFieldAddress
  reads the bag first (colliders were absent-shadowed by its own guard)
  and never serves system addresses from the bag; blob history refs read
  the bag shape-aware; migration transforms now receive ONE normalized
  view (engine fields + nested bag) regardless of stored era, and stray
  flat-habit keys refuse with the fix in the message.
- THE REOPEN-COLLIDER CONFORMANCE CASE (required before any RC counts as
  gates-green): all ten collider names + plumbing names written as user
  fields, verified verbatim + queryable across live reads, flush+reopen,
  a forced epoch rebuild, and asOf time travel; relation mirror; forgery
  refusals; legacy flat-record compat. 8/8 green.

Gates: unit 1901/1901 (exit 0) · integration 758 (exit 0) · conformance
27/27 (exit 0) · consumer test sweep migrated (10 files).
2026-08-03 16:59:32 -07:00

9.9 KiB
Raw Blame History


title: Field addressing: your fields and system fields slug: concepts/field-addressing public: true category: concepts template: concept order: 7 description: The one rule for every query-surface field name — a bare name always means your metadata, system. reaches the ten engine scalars explicitly, and anything else refuses by name. next:

  • concepts/consistency-model

Field addressing: your fields and system fields

Every query surface in Brainy — find()'s where, orderBy, aggregation groupBy, and aggregation source.where — resolves field names by one rule, with no exceptions:

A bare field name always means your metadata. system.<field> reaches an engine scalar, and only when you spell it explicitly.

await brain.find({ orderBy: 'level' })          // reads entity.metadata.level — YOUR field
await brain.find({ orderBy: 'system.createdAt' }) // reads the engine's createdAt scalar
await brain.find({ orderBy: 'metadata.level' })   // identical to bare 'level' — explicit scope

There is no priority list, no "try the system field, fall back to metadata" behavior, and no name that resolves differently depending on what else happens to exist on your entities. A field called level, score, createdAt, or type in your own metadata is read as your field, every time, by its bare name.

Why this rule exists

An internal report from a production deployment found that a user metadata field literally named level was being silently shadowed by the engine's own internal index layer field of the same name — every sort by level returned insertion order, with no error raised. This rule makes that class of bug structurally impossible: bare names belong to you, unconditionally, and anything that isn't yours has to be spelled out.

The system scalars

system.<field> addresses exactly ten scalars on an entity — no more, no fewer:

System field What it is
system.id The entity's id
system.type The entity's NounType
system.subtype The per-app sub-classification passed to add()
system.createdAt When the entity was created
system.updatedAt When the entity was last written
system.confidence The confidence param (01)
system.weight The weight param
system.visibility 'public' / 'internal' (see the visibility tiers in Consistency Model)
system.service The multi-tenancy service tag
system.createdBy Who/what created the entity

Relationships mirror the same eight shared scalars (subtype, createdAt, updatedAt, confidence, weight, visibility, service, createdBy) plus three of their own:

System field (relationship) What it is
system.verb The relationship's VerbType
system.sourceId The id of the entity the relationship starts from
system.targetId The id of the entity the relationship points to

Anything not on these two lists is not a system scalar — system.<name> for any other name refuses (see "Refusal semantics" below), even if that name sounds like it should be engine-owned.

Invisible plumbing — never addressable, in either spelling

Five names are pure engine internals. They are not reachable as a bare name, and not reachable as system.<name> either — they simply have no place on the query surface:

  • vector — the stored embedding. It participates in similarity search (query, near, vector find()), never in where/orderBy/groupBy.
  • connections — graph adjacency. Reached through connected and brain.related(), not through field addressing.
  • level — the internal index layer number used by the nearest-neighbor graph. It is pure index plumbing with no query-surface meaning at all — which is exactly why a user field of the same name must never be shadowed by it. level as a bare name is always yours; there is no engine-owned spelling of it to compete with.
  • data — your entity's content payload, not a scalar. It can be a string, a number, or an arbitrary object, so sorting or filtering it as a single comparable value would lie about its actual shape. Content is reached through the content/text-search APIs (query, searchMode: 'text'), not through where/orderBy.
  • _rev — the per-entity revision counter used for optimistic concurrency (ifRev). It is a CAS token, not a queryable dimension.

system.level, system.vector, and system.data all refuse for the same reason: they are not in the ten-scalar system map, full stop.

metadata.<field> — the explicit spelling of "mine"

Prefix any field with metadata. to say the same thing a bare name already says, spelled out. The two are interchangeable everywhere a field name is accepted, including orderBy:

await brain.find({ where: { 'customer.tier': 'gold' } })
await brain.find({ where: { 'metadata.customer.tier': 'gold' } })   // identical
await brain.find({ orderBy: 'metadata.score', order: 'desc' })      // identical to orderBy: 'score'

Reach for the explicit spelling when it reads more clearly next to a system. field in the same query — for example, sorting by your own score while filtering on system.confidence.

No special names — the write side

The same law governs writes:

Data is either in main space, where developers can use anything, or it is in system.*.

There are no reserved metadata names. A field called confidence, type, id, data, content, or anything else inside your metadata bag is an ordinary user field: it is stored verbatim, indexed, filterable, sortable, aggregatable, and it survives restarts, index rebuilds, and time-travel (asOf) reads exactly as written — even when an engine scalar shares its spelling. The engine's values are written only through their dedicated params (confidence, weight, subtype, visibility, …) and read at system.<field>; your bag can never touch them and they can never shadow your bag.

const id = await brain.add({
  data: 'Ada Lovelace',
  type: NounType.Person,
  confidence: 0.9,                       // the ENGINE scalar
  metadata: { confidence: 'self-rated' } // YOUR field, same spelling — both live
})

await brain.find({ where: { confidence: 'self-rated' } })      // finds it (yours)
await brain.find({ where: { 'system.confidence': 0.9 } })      // finds it (engine's)

The one spelling a write refuses is a metadata key that literally starts with system. — the explicit address namespace cannot be forged as a user field name. That refusal is typed and names the fix.

Value shape rules still apply uniformly to every name (they are not name carve-outs): arrays longer than 10 elements are not turned into posting-list scalars, and very long values are indexed by hash.

Refusal semantics

A name that resolves to neither your metadata nor a system scalar is a typed refusal, not a silent empty result and not a guess. Refusals name both candidates, so the fix is always in the error text:

await brain.find({ orderBy: 'createdAt' })
// UnresolvableFieldError: no metadata field 'createdAt' — did you mean
// system.createdAt or metadata.createdAt?

UnresolvableFieldError is exported from the package root:

import { UnresolvableFieldError } from '@soulcraft/brainy'

try {
  await brain.find({ orderBy: 'createdAt' })
} catch (err) {
  if (err instanceof UnresolvableFieldError) {
    // err.message names both candidates — usually enough to fix the call site.
  }
}

A handful of find() options are not implemented yet: cursor, includeRelations, and writeOnly. Rather than accepting them and quietly ignoring the option, find() refuses with UnsupportedFindOptionError — also exported from the package root — so a call site can never believe an unimplemented option took effect when it didn't.

The ordering contract

orderBy behaves identically regardless of which engine (the pure-TypeScript path or a native accelerator) is serving the query:

  • An entity missing the orderBy field, or holding null on it, sorts LAST — in both asc and desc. It is never treated as "smaller than everything" in one direction and "larger than everything" in the other; it is simply last, either way.
  • Rows are never dropped from an ordered read because they lack the field — a missing value changes position, never presence.
  • Ties on the orderBy field break by id ascending, regardless of the primary sort direction.
// employees: [{ score: 9 }, { score: 5 }, { /* no score field */ }]
await brain.find({ orderBy: 'score', order: 'desc' }) // [9, 5, missing] — missing is last
await brain.find({ orderBy: 'score', order: 'asc' })  // [5, 9, missing] — missing is STILL last

Migrating existing call sites

If you have call sites written before this rule shipped that rely on a bare system name — orderBy: 'createdAt', where: { confidence: { greaterThan: 0.8 } }, and similar — they now refuse instead of silently resolving to the engine field. The fix is always in the error: swap the bare name for system.<field> (or metadata.<field> if you actually meant your own field of that name, and it happens to share a name with a system scalar):

// Before: bare 'createdAt' silently meant the engine's timestamp.
await brain.find({ orderBy: 'createdAt' })

// After: say which one you meant.
await brain.find({ orderBy: 'system.createdAt' })   // the engine timestamp
await brain.find({ orderBy: 'metadata.createdAt' }) // your own field named createdAt, if you have one

There is no silent migration path by design — every ambiguous call site surfaces as a refusal naming its own fix, once, the first time it runs against the new rule.

Where to go next

  • Consistency Model — visibility tiers, revision counters, and the rest of the read/write contract this page's read-time addressing rule.