feat(namespace): NO SPECIAL NAMES + storage fidelity — the ruled completion of the field-addressing law
All checks were successful
CI / Node 22 (push) Successful in 12m9s
CI / Node 24 (push) Successful in 12m4s
CI / Bun (latest) (push) Successful in 12m52s

The write side of the law, ruled 2026-08-03: data is either in main space
where developers can use anything, or it is in system.*.

- The reserved-name write door DIES: add/update/relate/updateRelation
  metadata bags accept EVERY name (confidence, type, id, data, level,
  content, ...) as ordinary user fields — indexed, filterable, sortable,
  aggregatable, identical to any other field. The remap/enforce/warn
  machinery, the reservedFieldPolicy config (now a typed init refusal),
  and the compile-time metadata key bans are all removed. The one write
  refusal left: keys spelled 'system.*' (namespace forgery), now enforced
  on all four write doors.
- STORED RECORDS GO NESTED (v2): engine fields top-level, the user bag
  nested verbatim under 'metadata', sealed by a format stamp — by-name
  storage discrimination is unsound once colliders are admitted. Legacy
  flat records stay readable forever through the shape-aware splitters
  (sound for them: the old door refused colliders). Time travel rides the
  same split (generation store snapshots whole records).
- Name-based index exclusions DIE: user frame indexes every name; the
  excludeFields/indexedFields knobs and their silent-[] holes are gone;
  bulk-payload protection is value-shape only, uniform across names.
- Consumer-sweep findings fixed in the same wave: per-type counts read
  the frozen 'system.type' column (addToIndex sort, affinity tracking,
  cold-count rehydration, VFS type bitmaps — legacy 'noun' fallback for
  pre-rebuild reads); resolveHiddenIds addresses 'system.visibility'
  (bare 'visibility' was a silent no-op under the law — VFS/system
  entities leaked into default reads).
- Fidelity fallout fixed in the owning layers: readEntityFieldAddress
  reads the bag first (colliders were absent-shadowed by its own guard)
  and never serves system addresses from the bag; blob history refs read
  the bag shape-aware; migration transforms now receive ONE normalized
  view (engine fields + nested bag) regardless of stored era, and stray
  flat-habit keys refuse with the fix in the message.
- THE REOPEN-COLLIDER CONFORMANCE CASE (required before any RC counts as
  gates-green): all ten collider names + plumbing names written as user
  fields, verified verbatim + queryable across live reads, flush+reopen,
  a forced epoch rebuild, and asOf time travel; relation mirror; forgery
  refusals; legacy flat-record compat. 8/8 green.

Gates: unit 1901/1901 (exit 0) · integration 758 (exit 0) · conformance
27/27 (exit 0) · consumer test sweep migrated (10 files).
This commit is contained in:
David Snelling 2026-08-03 16:59:13 -07:00
parent 48a6130a50
commit 24bf6cdbc5
32 changed files with 1355 additions and 1905 deletions

View file

@ -73,8 +73,11 @@ export interface MetadataIndexConfig {
maxIndexSize?: number // Max number of entries per field value (default: 10000)
rebuildThreshold?: number // Rebuild if index is this % stale (default: 0.1)
autoOptimize?: boolean // Auto-cleanup unused entries (default: true)
indexedFields?: string[] // Only index these fields (default: all)
excludeFields?: string[] // Never index these fields
// NOTE: the name-based indexedFields/excludeFields knobs died with the
// field-addressing law ("no special names"): EVERY user field indexes,
// whatever its name. Bulk-payload protection is value-SHAPE based and
// uniform across all names (large arrays never become posting scalars;
// long values index hashed) — shape is not a name carve-out.
}
export interface MetadataIndexOptions {
@ -185,31 +188,12 @@ export class MetadataIndexManager implements MetadataIndexProvider {
this.config = {
maxIndexSize: config.maxIndexSize ?? 10000,
rebuildThreshold: config.rebuildThreshold ?? 0.1,
autoOptimize: config.autoOptimize ?? true,
indexedFields: config.indexedFields ?? [],
excludeFields: config.excludeFields ?? [
// ONLY exclude truly un-indexable fields (binary data, large content)
// Timestamps are NOW indexed with automatic bucketing (prevents pollution)
// Vectors and embeddings (binary data, already have HNSW indexes)
'embedding',
'vector',
'embeddings',
'vectors',
// Large content fields (too large for metadata indexing)
'content',
'data',
'originalData',
'_data',
// Primary keys (use direct lookups instead)
'id'
// NOTE: 'accessed', 'modified', 'createdAt', etc. are NO LONGER excluded!
// They are now indexed with automatic 1-minute bucketing to prevent file pollution
// This enables range queries like: modified > yesterday
]
autoOptimize: config.autoOptimize ?? true
// No name-based exclude/allow lists — the field-addressing law: every
// user field indexes, whatever its name ('content', 'data', 'id',
// 'vector', … included). Bulk payloads are kept out by uniform value-
// SHAPE rules in extractIndexableFields (arrays >10 never become
// posting scalars; >100-char values index hashed), never by name.
}
// Initialize metadata cache with similar config to search cache
@ -301,7 +285,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
}
// Warm the cache with common fields (lazy loading optimization)
// This loads the 'noun' sparse index which is needed for type counts
// This loads the type column ('system.type') needed for type counts
await this.warmCache()
// Load type counts AFTER warmCache (sparse index is now cached)
@ -350,8 +334,9 @@ export class MetadataIndexManager implements MetadataIndexProvider {
* Target: >80% cache hit rate for typical workloads
*/
async warmCache(): Promise<void> {
// Common fields used in most queries
const commonFields = ['noun', 'type', 'service', 'createdAt']
// Common columns used in most queries — the frozen system keys, plus
// legacy spellings for a pre-epoch-3 brain read before its rebuild runs.
const commonFields = ['system.type', 'system.service', 'system.createdAt', 'noun']
prodLog.debug(`🔥 Warming metadata cache with common fields: ${commonFields.join(', ')}`)
@ -537,9 +522,11 @@ export class MetadataIndexManager implements MetadataIndexProvider {
}
/**
* Lazy load entity counts from the 'noun' field sparse index (O(n) where n = number of types)
* Lazy load entity counts from the type column (O(n) where n = number of
* types). The frozen key is 'system.type' (epoch 3); the legacy 'noun'
* column is read as a fallback for a pre-epoch-3 brain observed before its
* rebuild has run (e.g. a reader-mode open against an old writer).
* FIX: Previously read from stats.nounCount which was SERVICE-keyed, not TYPE-keyed
* Now computes counts from the sparse index which has the correct type information
*/
private async lazyLoadCounts(): Promise<void> {
try {
@ -549,23 +536,31 @@ export class MetadataIndexManager implements MetadataIndexProvider {
this.entityCountsByTypeFixed.fill(0)
this.verbCountsByTypeFixed.fill(0)
// PRIMARY (8.0+): rehydrate per-type counts from the column store's 'noun'
// field — the authoritative on-disk source after a cold reopen.
// PRIMARY (8.0+): rehydrate per-type counts from the column store's
// type column — the authoritative on-disk source after a cold reopen.
// Frozen key first ('system.type', epoch 3), legacy 'noun' as the
// pre-rebuild fallback.
//
// The chunked sparse-index WRITE path was removed in 7.20.0 (commit
// 11be039): new workspaces persist the 'noun' field ONLY to the column
// store, never to a `__sparse_index__noun` blob. So the legacy sparse
// path below finds nothing and leaves every count at 0 — which is exactly
// why counts.byType/byTypeEnum/topTypes/allNounTypeCounts all read empty
// 11be039): new workspaces persist the type column ONLY to the column
// store, never to a sparse-index blob. So the legacy sparse path below
// finds nothing and leaves every count at 0 — which is exactly why
// counts.byType/byTypeEnum/topTypes/allNounTypeCounts all read empty
// after close()+reopen while find()/getNounCount() (different sources)
// stay correct. The column store's per-value cardinality matches the warm
// `updateTypeFieldAffinity` counts EXACTLY because both are driven from the
// same `addToIndex` field set, in lockstep, with no visibility gate on
// either — so this rehydration reproduces the warm values precisely.
if (this.columnStore && this.columnStore.getIndexedFields().includes('noun')) {
const nounValues = await this.columnStore.getFilterValues('noun')
const indexedCols = this.columnStore ? this.columnStore.getIndexedFields() : []
const typeCol = indexedCols.includes('system.type')
? 'system.type'
: indexedCols.includes('noun')
? 'noun'
: null
if (this.columnStore && typeCol) {
const nounValues = await this.columnStore.getFilterValues(typeCol)
for (const value of nounValues) {
const bitmap = await this.columnStore.filter('noun', value)
const bitmap = await this.columnStore.filter(typeCol, value)
if (bitmap.size > 0) {
// Use the stored value directly as the key (the legacy sparse path
// did the same): it is already the normalized type string that
@ -580,16 +575,17 @@ export class MetadataIndexManager implements MetadataIndexProvider {
}
// LEGACY FALLBACK (pre-7.20.0 workspaces still on the chunked sparse index).
const nounSparseIndex = await this.loadSparseIndex('noun')
const sparseCol = (await this.loadSparseIndex('system.type')) ? 'system.type' : 'noun'
const nounSparseIndex = await this.loadSparseIndex(sparseCol)
if (!nounSparseIndex) {
// No column-store 'noun' field and no sparse index yet — counts will be
// No column-store type column and no sparse index yet — counts will be
// populated as entities are added.
return
}
// Iterate through all chunks and sum up bitmap sizes by type
for (const chunkId of nounSparseIndex.getAllChunkIds()) {
const chunk = await this.chunkManager.loadChunk('noun', chunkId)
const chunk = await this.chunkManager.loadChunk(sparseCol, chunkId)
if (chunk) {
for (const [type, bitmap] of chunk.entries) {
const currentCount = this.totalEntitiesByType.get(type) || 0
@ -1179,66 +1175,46 @@ export class MetadataIndexManager implements MetadataIndexProvider {
return `__HASH_${Math.abs(hash).toString(36)}`
}
/**
* Check if field should be indexed
*/
private shouldIndexField(field: string): boolean {
if (this.config.excludeFields.includes(field)) return false
if (this.config.indexedFields.length > 0) {
return this.config.indexedFields.includes(field)
}
return true
}
/**
* Extract indexable field-value pairs from entity or metadata
*
* Now handles BOTH entity structure (with top-level fields) AND plain metadata
* - Extracts from top-level fields (confidence, weight, timestamps, type, service, etc.)
* - Also extracts from nested metadata field (custom user fields)
* - Skips HNSW-specific fields (vector, connections, level, id)
* - Maps 'type' 'noun' for backward compatibility with existing indexes
*
* BUG FIX: Exclude vector embeddings and large arrays from indexing
* BUG FIX: Also exclude purely numeric field names (array indices)
* - Vector fields (384+ dimensions) were creating 825K chunk files for 1,144 entities
* - Arrays converted to objects with numeric keys were still being indexed
* Handles BOTH entity structure (with top-level fields) AND record shapes
* - Record-frame system scalars index under literal 'system.<field>' keys
* - The user's metadata bag indexes under bare keys EVERY name (the
* field-addressing law: no special names; 'level', 'data', 'id',
* 'content', 'vector' in a bag are ordinary user fields)
* - Record-frame plumbing (vector, connections, level, data, _rev, id)
* never indexes that is namespace routing, not a name carve-out
* - Value-SHAPE rules apply uniformly to all names: arrays >10 never
* become posting scalars; purely numeric key names (array indices)
* skip; >100-char values index hashed (normalizeValue)
*/
private extractIndexableFields(data: any): Array<{ field: string, value: any }> {
const fields: Array<{ field: string, value: any }> = []
// Fields that should NEVER be indexed: bulk structural payloads that would
// blow up the index (the 384-dim vector, embeddings, the adjacency list).
// These are also caught by the array-size guard below, but naming them is
// belt-and-suspenders. NOTE: `level` was previously here (an HNSW node's
// layer) but it never actually reaches this path — every caller passes a
// metadata bag or Entity record, neither of which carries the node's
// `level` — so its only effect was to silently drop a legitimate USER
// metadata field named `level` (log level, skill level, access level…),
// making `where: { level: … }` return nothing. Removed. (`id` stays: it is
// the reserved entity-identity field, resolved specially by find().)
const NEVER_INDEX = new Set(['vector', 'embedding', 'embeddings', 'connections', 'id'])
// RECORD-FRAME-ONLY plumbing guard: on an entity/stored-record frame
// these keys are the engine's structural payloads (the 384-dim vector,
// embeddings, the adjacency list, the identity field) and never index.
// This set is NEVER applied inside the user's metadata bag — under the
// field-addressing law every user name indexes; a real vector-sized
// value in a bag is kept out by the uniform array-size shape guard, not
// by its name.
const RECORD_PLUMBING = new Set(['vector', 'embedding', 'embeddings', 'connections', 'id'])
// THE FROZEN INDEX KEY FORMAT (cross-engine, sealed 2026-08-03; the native
// accelerator keys identically — epoch 3 rebuilds every brain onto it):
// user fields index under BARE keys exactly as the caller wrote them;
// the ten system scalars index under literal 'system.<field>' keys — the
// key IS the query address, so the two namespaces can never collide
// inside the index again. `origin` tracks which side of the record a key
// came from: 'record' = the entity/stored-record frame (system scalars,
// plumbing, and the metadata bag live here — the WRITE PATH's reserved-
// name remap guarantees a record-frame key matching a system name IS the
// system value); 'user' = inside the flattened metadata bag (everything
// is the user's, including natural names like `level` and `data`).
// Frame kinds: 'entity-record' = entityForIndexing shape (user fields
// nested under `metadata`; stray top-level keys are DROPPED, not guessed —
// epoch-3's rebuild-from-canonical normalizes historical shapes);
// 'flat-record' = the stored metadata-record shape (user fields FLAT
// beside the reserved ones — the write path's reserved-name remap
// guarantees a key matching a system name IS the system value, so
// non-system keys here are the user's and index bare); 'user' = inside
// the metadata bag (everything is the user's, including natural names
// like `level` and `data`).
// inside the index again.
// Frame kinds: 'entity-record' = entityForIndexing shape / v2 nested-bag
// stored record (user fields nested under `metadata`; stray top-level
// keys are DROPPED, not guessed); 'flat-record' = the LEGACY stored
// metadata-record shape (user fields flat beside the engine's — sound to
// split by name because the pre-law write door refused user metadata
// carrying engine names, so a flat key matching a system name IS the
// system value); 'user' = inside the metadata bag, where EVERY key is
// the user's and indexes bare — collider names included.
type Frame = 'entity-record' | 'flat-record' | 'user'
const extract = (obj: any, prefix = '', frame: Frame = 'entity-record'): void => {
for (const [key, value] of Object.entries(obj)) {
@ -1254,30 +1230,25 @@ export class MetadataIndexManager implements MetadataIndexProvider {
} else if (SYSTEM_ENTITY_SCALARS.has(key) && key !== 'id') {
fullKey = `system.${key}`
} else if (
key === 'data' || key === '_rev' || key === 'level' || NEVER_INDEX.has(key)
key === 'data' || key === '_rev' || key === 'level' || key === '_fmt' ||
RECORD_PLUMBING.has(key)
) {
continue // plumbing / identity / bulk payloads — never indexed from a record frame
continue // plumbing / identity / format stamp — never indexed from a record frame
} else if (frame === 'entity-record') {
continue // stray entity-frame key: dropped, not guessed
}
// flat-record fallthrough: a non-system, non-plumbing key IS a user
// field (flat beside the reserved ones) — indexes bare via fullKey.
} else if (!prefix && NEVER_INDEX.has(key)) {
// User frame: only the bulk-payload guards apply — natural names
// like `level` and `data` are real user fields here. (`id` as a
// user metadata field remains un-indexed this train — documented
// limitation; system.id resolves via the id mapper, never a column.)
continue
// field (flat beside the engine's, legacy shape) — indexes bare.
}
// User frame: NO name-based skips — every user field indexes, whatever
// its name (the field-addressing law). Only the uniform value-shape
// guards below apply.
// Skip purely numeric field names (array indices converted to object keys)
// Legitimate field names should never be purely numeric
// This catches vectors stored as objects: {0: 0.1, 1: 0.2, ...}
if (/^\d+$/.test(key)) continue
// Skip fields based on user configuration
if (!this.shouldIndexField(fullKey)) continue
// Skip large arrays (> 10 elements) - likely vectors or bulk data
if (Array.isArray(value) && value.length > 10) continue
@ -1510,10 +1481,11 @@ export class MetadataIndexManager implements MetadataIndexProvider {
prodLog.debug(`Entity ${id} has ${wordFields.length} indexed words (large document)`)
}
// Sort fields to process 'noun' field first for type-field affinity tracking
// Sort fields to process the type column first for type-field affinity
// tracking ('system.type' is the frozen key; 'noun' died at epoch 3).
fields.sort((a, b) => {
if (a.field === 'noun') return -1
if (b.field === 'noun') return 1
if (a.field === 'system.type') return -1
if (b.field === 'system.type') return 1
return 0
})
@ -2861,6 +2833,17 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// VFS Statistics Methods (uses existing Roaring bitmap infrastructure)
// ============================================================================
/**
* Read the type column's bitmap for one type value frozen key first
* ('system.type', epoch 3), legacy 'noun' as the pre-rebuild fallback.
*/
private async getTypeBitmap(type: string): Promise<RoaringBitmap32 | null> {
return (
(await this.getBitmapFromChunks('system.type', type)) ??
(await this.getBitmapFromChunks('noun', type))
)
}
/**
* Get VFS entity count for a specific type using Roaring bitmap intersection
* Uses hardware-accelerated SIMD operations (AVX2/SSE4.2)
@ -2869,7 +2852,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
*/
async getVFSEntityCountByType(type: string): Promise<number> {
const vfsBitmap = await this.getBitmapFromChunks('isVFSEntity', true)
const typeBitmap = await this.getBitmapFromChunks('noun', type)
const typeBitmap = await this.getTypeBitmap(type)
if (!vfsBitmap || !typeBitmap) return 0
@ -2892,7 +2875,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// Iterate through all known types and compute VFS count via intersection
for (const type of this.totalEntitiesByType.keys()) {
const typeBitmap = await this.getBitmapFromChunks('noun', type)
const typeBitmap = await this.getTypeBitmap(type)
if (typeBitmap) {
const intersection = RoaringBitmap32.and(vfsBitmap, typeBitmap)
if (intersection.size > 0) {
@ -3486,18 +3469,21 @@ export class MetadataIndexManager implements MetadataIndexProvider {
* Tracks which fields commonly appear with which entity types
*/
private updateTypeFieldAffinity(entityId: string, field: string, value: any, operation: 'add' | 'remove', metadata?: any): void {
// Only track affinity for non-system fields (but allow 'noun' for type detection)
if (this.config.excludeFields.includes(field) && field !== 'noun') return
// Only track affinity for user fields (plus the type column itself,
// which drives detection). Engine columns carry the literal 'system.'
// prefix under the frozen key format.
if (field.startsWith('system.') && field !== 'system.type') return
// For the 'noun' field, the value IS the entity type
// For the type column ('system.type'), the value IS the entity type
let entityType: string | null = null
if (field === 'noun') {
if (field === 'system.type') {
// This is the type definition itself
entityType = this.normalizeValue(value, field) // Pass field for bucketing!
} else if (metadata && metadata.noun) {
// Extract entity type from metadata
entityType = this.normalizeValue(metadata.noun, 'noun')
} else if (metadata && (metadata.noun ?? metadata.type)) {
// Extract entity type from the source shape: stored records carry it
// under 'noun', entity-for-indexing views under 'type'.
entityType = this.normalizeValue(metadata.noun ?? metadata.type, 'system.type')
} else {
// No type information available, skip affinity tracking
return
@ -3520,8 +3506,9 @@ export class MetadataIndexManager implements MetadataIndexProvider {
const currentCount = typeFields.get(field) || 0
typeFields.set(field, currentCount + 1)
// Update total entities of this type (only count once per entity)
if (field === 'noun') {
// Update total entities of this type (only count once per entity —
// the type column appears exactly once per entity)
if (field === 'system.type') {
const newCount = this.totalEntitiesByType.get(entityType)! + 1
this.totalEntitiesByType.set(entityType, newCount)
@ -3544,7 +3531,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
}
// Update total entities of this type
if (field === 'noun') {
if (field === 'system.type') {
const total = this.totalEntitiesByType.get(entityType)!
if (total > 1) {
const newCount = total - 1

View file

@ -618,6 +618,7 @@ export function validateUpdateParams(params: UpdateParams): void {
* Validate relate parameters
*/
export function validateRelateParams(params: RelateParams): void {
rejectForgedSystemKeys(params.metadata as Record<string, unknown> | undefined, 'relate()')
// 8.0 verb-id contract (L.7): verb ids are UUIDs, generated by brainy.
// RelateParams has no `id` field — an untyped caller passing one would
// previously have it silently ignored (a generated UUID was used instead).
@ -666,6 +667,7 @@ export function validateRelateParams(params: RelateParams): void {
* accepts type/subtype/weight/confidence/data/metadata changes.
*/
export function validateUpdateRelationParams(params: UpdateRelationParams): void {
rejectForgedSystemKeys(params.metadata as Record<string, unknown> | undefined, 'updateRelation()')
if (!params.id) {
throw new Error('id is required for updateRelation')
}