David Snelling
24bf6cdbc5
The write side of the law, ruled 2026-08-03: data is either in main space where developers can use anything, or it is in system.*. - The reserved-name write door DIES: add/update/relate/updateRelation metadata bags accept EVERY name (confidence, type, id, data, level, content, ...) as ordinary user fields — indexed, filterable, sortable, aggregatable, identical to any other field. The remap/enforce/warn machinery, the reservedFieldPolicy config (now a typed init refusal), and the compile-time metadata key bans are all removed. The one write refusal left: keys spelled 'system.*' (namespace forgery), now enforced on all four write doors. - STORED RECORDS GO NESTED (v2): engine fields top-level, the user bag nested verbatim under 'metadata', sealed by a format stamp — by-name storage discrimination is unsound once colliders are admitted. Legacy flat records stay readable forever through the shape-aware splitters (sound for them: the old door refused colliders). Time travel rides the same split (generation store snapshots whole records). - Name-based index exclusions DIE: user frame indexes every name; the excludeFields/indexedFields knobs and their silent-[] holes are gone; bulk-payload protection is value-shape only, uniform across names. - Consumer-sweep findings fixed in the same wave: per-type counts read the frozen 'system.type' column (addToIndex sort, affinity tracking, cold-count rehydration, VFS type bitmaps — legacy 'noun' fallback for pre-rebuild reads); resolveHiddenIds addresses 'system.visibility' (bare 'visibility' was a silent no-op under the law — VFS/system entities leaked into default reads). - Fidelity fallout fixed in the owning layers: readEntityFieldAddress reads the bag first (colliders were absent-shadowed by its own guard) and never serves system addresses from the bag; blob history refs read the bag shape-aware; migration transforms now receive ONE normalized view (engine fields + nested bag) regardless of stored era, and stray flat-habit keys refuse with the fix in the message. - THE REOPEN-COLLIDER CONFORMANCE CASE (required before any RC counts as gates-green): all ten collider names + plumbing names written as user fields, verified verbatim + queryable across live reads, flush+reopen, a forced epoch rebuild, and asOf time travel; relation mirror; forgery refusals; legacy flat-record compat. 8/8 green. Gates: unit 1901/1901 (exit 0) · integration 758 (exit 0) · conformance 27/27 (exit 0) · consumer test sweep migrated (10 files).
232 lines
9.9 KiB
Markdown
232 lines
9.9 KiB
Markdown
---
|
||
title: Field addressing: your fields and system fields
|
||
slug: concepts/field-addressing
|
||
public: true
|
||
category: concepts
|
||
template: concept
|
||
order: 7
|
||
description: The one rule for every query-surface field name — a bare name always means your metadata, system.<field> reaches the ten engine scalars explicitly, and anything else refuses by name.
|
||
next:
|
||
- concepts/consistency-model
|
||
---
|
||
|
||
# Field addressing: your fields and system fields
|
||
|
||
Every query surface in Brainy — `find()`'s `where`, `orderBy`, aggregation
|
||
`groupBy`, and aggregation `source.where` — resolves field names by one rule,
|
||
with no exceptions:
|
||
|
||
> **A bare field name always means your metadata. `system.<field>` reaches an
|
||
> engine scalar, and only when you spell it explicitly.**
|
||
|
||
```typescript
|
||
await brain.find({ orderBy: 'level' }) // reads entity.metadata.level — YOUR field
|
||
await brain.find({ orderBy: 'system.createdAt' }) // reads the engine's createdAt scalar
|
||
await brain.find({ orderBy: 'metadata.level' }) // identical to bare 'level' — explicit scope
|
||
```
|
||
|
||
There is no priority list, no "try the system field, fall back to metadata"
|
||
behavior, and no name that resolves differently depending on what else
|
||
happens to exist on your entities. A field called `level`, `score`,
|
||
`createdAt`, or `type` in your own `metadata` is read as *your* field, every
|
||
time, by its bare name.
|
||
|
||
## Why this rule exists
|
||
|
||
An internal report from a production deployment found that a user metadata
|
||
field literally named `level` was being silently shadowed by the engine's
|
||
own internal index layer field of the same name — every sort by `level`
|
||
returned insertion order, with no error raised. This rule makes that class of
|
||
bug structurally impossible: bare names belong to you, unconditionally, and
|
||
anything that isn't yours has to be spelled out.
|
||
|
||
## The system scalars
|
||
|
||
`system.<field>` addresses exactly ten scalars on an entity — no more, no
|
||
fewer:
|
||
|
||
| System field | What it is |
|
||
|---|---|
|
||
| `system.id` | The entity's id |
|
||
| `system.type` | The entity's `NounType` |
|
||
| `system.subtype` | The per-app sub-classification passed to `add()` |
|
||
| `system.createdAt` | When the entity was created |
|
||
| `system.updatedAt` | When the entity was last written |
|
||
| `system.confidence` | The `confidence` param (0–1) |
|
||
| `system.weight` | The `weight` param |
|
||
| `system.visibility` | `'public'` / `'internal'` (see the visibility tiers in [Consistency Model](./consistency-model.md)) |
|
||
| `system.service` | The multi-tenancy `service` tag |
|
||
| `system.createdBy` | Who/what created the entity |
|
||
|
||
Relationships mirror the same eight shared scalars (`subtype`, `createdAt`,
|
||
`updatedAt`, `confidence`, `weight`, `visibility`, `service`, `createdBy`)
|
||
plus three of their own:
|
||
|
||
| System field (relationship) | What it is |
|
||
|---|---|
|
||
| `system.verb` | The relationship's `VerbType` |
|
||
| `system.sourceId` | The id of the entity the relationship starts from |
|
||
| `system.targetId` | The id of the entity the relationship points to |
|
||
|
||
Anything not on these two lists is not a system scalar — `system.<name>` for
|
||
any other name refuses (see "Refusal semantics" below), even if that name
|
||
sounds like it should be engine-owned.
|
||
|
||
## Invisible plumbing — never addressable, in either spelling
|
||
|
||
Five names are pure engine internals. They are not reachable as a bare name,
|
||
and not reachable as `system.<name>` either — they simply have no place on
|
||
the query surface:
|
||
|
||
- **`vector`** — the stored embedding. It participates in similarity search
|
||
(`query`, `near`, vector `find()`), never in `where`/`orderBy`/`groupBy`.
|
||
- **`connections`** — graph adjacency. Reached through `connected` and
|
||
`brain.related()`, not through field addressing.
|
||
- **`level`** — the internal index layer number used by the nearest-neighbor
|
||
graph. It is pure index plumbing with no query-surface meaning at all —
|
||
which is exactly why a user field of the same name must never be shadowed
|
||
by it. `level` as a bare name is always yours; there is no engine-owned
|
||
spelling of it to compete with.
|
||
- **`data`** — your entity's content payload, not a scalar. It can be a
|
||
string, a number, or an arbitrary object, so sorting or filtering it as a
|
||
single comparable value would lie about its actual shape. Content is
|
||
reached through the content/text-search APIs (`query`, `searchMode:
|
||
'text'`), not through `where`/`orderBy`.
|
||
- **`_rev`** — the per-entity revision counter used for optimistic
|
||
concurrency (`ifRev`). It is a CAS token, not a queryable dimension.
|
||
|
||
`system.level`, `system.vector`, and `system.data` all refuse for the same
|
||
reason: they are not in the ten-scalar system map, full stop.
|
||
|
||
## `metadata.<field>` — the explicit spelling of "mine"
|
||
|
||
Prefix any field with `metadata.` to say the same thing a bare name already
|
||
says, spelled out. The two are interchangeable everywhere a field name is
|
||
accepted, including `orderBy`:
|
||
|
||
```typescript
|
||
await brain.find({ where: { 'customer.tier': 'gold' } })
|
||
await brain.find({ where: { 'metadata.customer.tier': 'gold' } }) // identical
|
||
await brain.find({ orderBy: 'metadata.score', order: 'desc' }) // identical to orderBy: 'score'
|
||
```
|
||
|
||
Reach for the explicit spelling when it reads more clearly next to a
|
||
`system.` field in the same query — for example, sorting by your own `score`
|
||
while filtering on `system.confidence`.
|
||
|
||
## No special names — the write side
|
||
|
||
The same law governs writes:
|
||
|
||
> **Data is either in main space, where developers can use anything, or it
|
||
> is in `system.*`.**
|
||
|
||
There are **no reserved metadata names**. A field called `confidence`,
|
||
`type`, `id`, `data`, `content`, or anything else inside your `metadata` bag
|
||
is an ordinary user field: it is stored verbatim, indexed, filterable,
|
||
sortable, aggregatable, and it survives restarts, index rebuilds, and
|
||
time-travel (`asOf`) reads exactly as written — even when an engine scalar
|
||
shares its spelling. The engine's values are written only through their
|
||
dedicated params (`confidence`, `weight`, `subtype`, `visibility`, …) and
|
||
read at `system.<field>`; your bag can never touch them and they can never
|
||
shadow your bag.
|
||
|
||
```typescript
|
||
const id = await brain.add({
|
||
data: 'Ada Lovelace',
|
||
type: NounType.Person,
|
||
confidence: 0.9, // the ENGINE scalar
|
||
metadata: { confidence: 'self-rated' } // YOUR field, same spelling — both live
|
||
})
|
||
|
||
await brain.find({ where: { confidence: 'self-rated' } }) // finds it (yours)
|
||
await brain.find({ where: { 'system.confidence': 0.9 } }) // finds it (engine's)
|
||
```
|
||
|
||
The one spelling a write refuses is a metadata key that literally starts
|
||
with `system.` — the explicit address namespace cannot be forged as a user
|
||
field name. That refusal is typed and names the fix.
|
||
|
||
Value **shape** rules still apply uniformly to every name (they are not name
|
||
carve-outs): arrays longer than 10 elements are not turned into posting-list
|
||
scalars, and very long values are indexed by hash.
|
||
|
||
## Refusal semantics
|
||
|
||
A name that resolves to neither your metadata nor a system scalar is a typed
|
||
refusal, not a silent empty result and not a guess. Refusals name **both**
|
||
candidates, so the fix is always in the error text:
|
||
|
||
```typescript
|
||
await brain.find({ orderBy: 'createdAt' })
|
||
// UnresolvableFieldError: no metadata field 'createdAt' — did you mean
|
||
// system.createdAt or metadata.createdAt?
|
||
```
|
||
|
||
`UnresolvableFieldError` is exported from the package root:
|
||
|
||
```typescript
|
||
import { UnresolvableFieldError } from '@soulcraft/brainy'
|
||
|
||
try {
|
||
await brain.find({ orderBy: 'createdAt' })
|
||
} catch (err) {
|
||
if (err instanceof UnresolvableFieldError) {
|
||
// err.message names both candidates — usually enough to fix the call site.
|
||
}
|
||
}
|
||
```
|
||
|
||
A handful of `find()` options are not implemented yet: `cursor`,
|
||
`includeRelations`, and `writeOnly`. Rather than accepting them and quietly
|
||
ignoring the option, `find()` refuses with `UnsupportedFindOptionError` —
|
||
also exported from the package root — so a call site can never believe an
|
||
unimplemented option took effect when it didn't.
|
||
|
||
## The ordering contract
|
||
|
||
`orderBy` behaves identically regardless of which engine (the pure-TypeScript
|
||
path or a native accelerator) is serving the query:
|
||
|
||
- An entity missing the `orderBy` field, or holding `null` on it, sorts
|
||
**LAST — in both `asc` and `desc`**. It is never treated as "smaller than
|
||
everything" in one direction and "larger than everything" in the other; it
|
||
is simply last, either way.
|
||
- Rows are **never dropped** from an ordered read because they lack the
|
||
field — a missing value changes position, never presence.
|
||
- Ties on the `orderBy` field break by **id ascending**, regardless of the
|
||
primary sort direction.
|
||
|
||
```typescript
|
||
// employees: [{ score: 9 }, { score: 5 }, { /* no score field */ }]
|
||
await brain.find({ orderBy: 'score', order: 'desc' }) // [9, 5, missing] — missing is last
|
||
await brain.find({ orderBy: 'score', order: 'asc' }) // [5, 9, missing] — missing is STILL last
|
||
```
|
||
|
||
## Migrating existing call sites
|
||
|
||
If you have call sites written before this rule shipped that rely on a bare
|
||
system name — `orderBy: 'createdAt'`, `where: { confidence: { greaterThan:
|
||
0.8 } }`, and similar — they now refuse instead of silently resolving to the
|
||
engine field. The fix is always in the error: swap the bare name for
|
||
`system.<field>` (or `metadata.<field>` if you actually meant your own field
|
||
of that name, and it happens to share a name with a system scalar):
|
||
|
||
```typescript
|
||
// Before: bare 'createdAt' silently meant the engine's timestamp.
|
||
await brain.find({ orderBy: 'createdAt' })
|
||
|
||
// After: say which one you meant.
|
||
await brain.find({ orderBy: 'system.createdAt' }) // the engine timestamp
|
||
await brain.find({ orderBy: 'metadata.createdAt' }) // your own field named createdAt, if you have one
|
||
```
|
||
|
||
There is no silent migration path by design — every ambiguous call site
|
||
surfaces as a refusal naming its own fix, once, the first time it runs
|
||
against the new rule.
|
||
|
||
## Where to go next
|
||
|
||
- [Consistency Model](./consistency-model.md) — visibility tiers, revision
|
||
counters, and the rest of the read/write contract this page's
|
||
read-time addressing rule.
|