Commit graph

6 commits

Author SHA1 Message Date
b11b5f7e68 feat(namespace)!: NO SPECIAL NAMES + storage fidelity — the ruled completion of the field-addressing law
Some checks failed
CI / Node 22 (push) Has been cancelled
CI / Node 24 (push) Has been cancelled
CI / Bun (latest) (push) Has been cancelled
The write side of the law, ruled 2026-08-03: data is either in main space
where developers can use anything, or it is in system.*.

- The reserved-name write door DIES: add/update/relate/updateRelation
  metadata bags accept EVERY name (confidence, type, id, data, level,
  content, ...) as ordinary user fields — indexed, filterable, sortable,
  aggregatable, identical to any other field. The remap/enforce/warn
  machinery, the reservedFieldPolicy config (now a typed init refusal),
  and the compile-time metadata key bans are all removed. The one write
  refusal left: keys spelled 'system.*' (namespace forgery), now enforced
  on all four write doors.
- STORED RECORDS GO NESTED (v2): engine fields top-level, the user bag
  nested verbatim under 'metadata', sealed by a format stamp — by-name
  storage discrimination is unsound once colliders are admitted. Legacy
  flat records stay readable forever through the shape-aware splitters
  (sound for them: the old door refused colliders). Time travel rides the
  same split (generation store snapshots whole records).
- Name-based index exclusions DIE: user frame indexes every name; the
  excludeFields/indexedFields knobs and their silent-[] holes are gone;
  bulk-payload protection is value-shape only, uniform across names.
- Consumer-sweep findings fixed in the same wave: per-type counts read
  the frozen 'system.type' column (addToIndex sort, affinity tracking,
  cold-count rehydration, VFS type bitmaps — legacy 'noun' fallback for
  pre-rebuild reads); resolveHiddenIds addresses 'system.visibility'
  (bare 'visibility' was a silent no-op under the law — VFS/system
  entities leaked into default reads).
- Fidelity fallout fixed in the owning layers: readEntityFieldAddress
  reads the bag first (colliders were absent-shadowed by its own guard)
  and never serves system addresses from the bag; blob history refs read
  the bag shape-aware; migration transforms now receive ONE normalized
  view (engine fields + nested bag) regardless of stored era, and stray
  flat-habit keys refuse with the fix in the message.
- THE REOPEN-COLLIDER CONFORMANCE CASE (required before any RC counts as
  gates-green): all ten collider names + plumbing names written as user
  fields, verified verbatim + queryable across live reads, flush+reopen,
  a forced epoch rebuild, and asOf time travel; relation mirror; forgery
  refusals; legacy flat-record compat. 8/8 green.

Gates: unit 1901/1901 (exit 0) · integration 758 (exit 0) · conformance
27/27 (exit 0) · consumer test sweep migrated (10 files).
2026-08-03 16:59:13 -07:00
54c7c39669 feat(8.0): reserved-field enforcement — reservedFieldPolicy defaults to throw
An untyped (JS) caller that smuggles a Brainy-reserved field (confidence,
weight, subtype, visibility, service, createdBy, noun/verb, data, createdAt,
updatedAt, _rev) inside a write-path metadata bag previously got a silent
remap-or-drop — a class of bug where confidence-evolution writes no-oped for
weeks before being caught on read-back. 8.0 closes this with no silent failures.

- New BrainyConfig.reservedFieldPolicy: 'throw' | 'warn' | 'remap' (default 'throw').
  'throw' rejects the write naming every offending key + its correct write path;
  'warn' remaps with a one-shot per-key warning; 'remap' is the legacy silent path.
- Central enforceReservedPolicy gate wired into all four remap methods (add,
  update, relate, updateRelation) so live calls AND their transact()/with()
  mirrors honor it. Single-source reservedWritePath guidance shared by throw+warn.
- 'warn' now warns for EVERY reserved key (closes the gap where only
  system-managed fields warned). Dead warnDropped* helpers removed.
- Import pipeline migrated to route reserved values (confidence/weight/subtype)
  through dedicated params and strip reserved keys from extractor/customMetadata
  bags via the canonical split*MetadataRecord helpers — imports no longer trip
  the default throw.
- Tests: new reservedFieldPolicy matrix (throw/warn/remap across every write
  path + transact); remap-correctness suite reframed as opt-in 'remap'; shared
  test-factory no longer emits reserved keys in custom metadata.
2026-06-20 15:37:21 -07:00
1f7e365a4e chore(8.0): final pre-RC1 sweep — API consistency, named errors, orphans, zero-cast codebase 2026-06-11 14:51:00 -07:00
970e08c466 feat(8.0): reserved-field contract — one canonical location, typed prevention, unified read/write
Brainy-owned field names (noun/verb, subtype, createdAt, updatedAt,
confidence, weight, service, data, createdBy, _rev) now have exactly one
home — top level — enforced by three layers driven from a single source of
truth, src/types/reservedFields.ts (RESERVED_ENTITY_FIELDS /
RESERVED_RELATION_FIELDS, exported):

1. Compile time — AddParams/UpdateParams/RelateParams/UpdateRelationParams
   metadata (and the transact() ops that extend them) reject a literal
   reserved key as a TypeScript error while keeping generic T ergonomics
   (typed bags, untyped brains, index-signature shapes, and a documented
   exemption for T-declared reserved keys). Pinned by @ts-expect-error
   type tests run under vitest typecheck mode on every unit run.

2. Write time — the 7.x update() remap is ported to 8.0 and extended to
   every write path: add/update/relate/updateRelation, their transact()
   mirrors, and db.with() overlays. User-settable fields lift to their
   dedicated param (top-level wins when both are supplied — closes the 7.x
   trap where update({metadata:{confidence}}) silently no-oped), and
   system-managed fields drop with a one-shot warning naming the right
   path. A remapped subtype satisfies subtype-pairing enforcement exactly
   like a top-level one.

3. Read time — every storage combine goes through one canonical hydration
   helper (hydrateNounWithMetadata / hydrateVerbWithMetadata over
   splitNoun/VerbMetadataRecord), so reserved fields surface ONLY top-level
   and entity/relation.metadata carry ONLY custom fields on live reads,
   batch reads, paginated listings, getRelations by source/target, streamed
   verbs, and historical asOf() materialization alike.

Read-path echoes found and fixed (previously the full stored record —
including the verb type key — leaked inside metadata): noun pagination,
verb pagination, getVerbsBySource/ByTarget (adjacency + shard fallback),
getVerbsBySourceBatch (which also dropped subtype/data), and the
filesystem verb stream. getRelations() results now surface
confidence/updatedAt top-level via verbsToRelations, updateRelation() no
longer erases service/createdBy, relate() persists its top-level
confidence/service params, and the dead convertHNSWVerbToGraphVerb echo
path is deleted. Import paths (CLI extract, deduplicator, coordinators,
neural import) write confidence through the dedicated param instead of the
bag. UpdateRelationParams is now exported from the package root.

Documented for consumers in docs/concepts/consistency-model.md ("Reserved
fields") and RELEASES.md. Regression tests ported from the 7.x fix and
extended to the full 8.0 contract (17 runtime tests + 41 type-level
assertions); full unit suite 1427/1427, db-mvcc integration 24/24.
2026-06-11 13:13:09 -07:00
5f3a2ca7d5 fix: internal subtype consistency + brain.audit() diagnostic + improved enforcement errors
Brainy 7.30 shipped opt-in subtype enforcement; SDK 3.20.0 then registered
SDK_CORE_VOCABULARY on every consumer's brain (Event, Collection, Message,
Contract, Media, Document NounTypes). On 2026-06-08 Venue's /book flow went 500
because their brain.add({ type: NounType.Event, ... }) call sites lacked
subtype. An audit of Brainy's OWN source revealed 14 HIGH-risk internal write
paths that also omit subtype — any consumer running the same vocabulary would
have hit Brainy's infrastructure paths next. 7.30.1 closes both gaps before
8.0 makes strict mode the default.

Additive across the board. Zero behavior change for consumers not using strict
mode. Every change is JS-side — Cortex needs no work for 7.30.1.

NEW — brain.audit() diagnostic
- Read-only method walking storage.getNouns() / getVerbs() pagination
- Returns { entitiesWithoutSubtype: { type: count }, relationshipsWithoutSubtype,
  total, scanned, recommendation }
- VFS infrastructure entities excluded by default (they bypass enforcement via
  isVFSEntity marker); pass { includeVFS: true } to surface them
- The companion to migrateField (7.x) and fillSubtypes (8.0): tells consumers
  exactly what would break under strict enforcement, deterministically

NEW — Improved enforcement error messages
- Caller's source location extracted from Error().stack so users see their own
  call site, not a Brainy internal frame
- Specific guidance branches: registered vocabulary → "Pass one of: a, b, c";
  brain-wide strict mode → mentions the except clause; otherwise → registration
  recipe via brain.requireSubtype()
- Documentation link to the canonical migration recipe
- Same shape for noun and verb enforcement

NEW — CLI --subtype flag
- brainy add and brainy relate gain -s/--subtype <value>
- Defaults to 'cli-add' / 'cli-relate' so the CLI works against strict-mode
  brains without the user needing to know the vocabulary in advance

INTERNAL — every Brainy write path now sets subtype
- VFS Contains edges (5 sites at lines 503/905/1694/1772/1886) → 'vfs-contains'
- VFS symlink entity → 'vfs-symlink' (NEW — distinct from 'vfs-file')
- VFS copy-file → preserves source subtype, falls back to 'vfs-file'
- VFS symlink also adopts the isVFSEntity infrastructure marker so it bypasses
  enforcement in strict mode
- Aggregation materializer (Measurement entities) → 'materialized-aggregate'
- ImportCoordinator (3 sites): document → 'import-source'; entities →
  options.defaultSubtype ?? 'imported'; placeholder → 'import-placeholder'
- SmartImportOrchestrator (4 entity sites + 2 batch relate sites): same
  precedence (extractor → options.defaultSubtype → 'imported')
- EntityDeduplicator → candidate.subtype ?? 'imported'
- UniversalImportAPI → extractor → 'extracted' for both entities and relations
- NeuralImport → adds defaultSubtype to NeuralImportOptions; precedence same
- GoogleSheetsIntegration → request body 'subtype' ?? 'imported-from-sheets'
- ODataIntegration → request body 'Subtype' ?? 'imported-from-odata'
- MCP client message storage → 'mcp-message' (also fixes pre-existing missing
  data field and missing type by aliasing from the prior text field)

Side-effect fix: storage.getNouns() paginated now surfaces subtype to top-level
- Single-noun getNoun() already did this in 7.30; the paginated path was missed
- Without this fix brain.audit() saw missing subtype on entities that actually
  had one (caught by the strict-mode self-test before release)

NEW — tests/integration/strict-mode-self-test.test.ts (13 tests)
- Creates a brain under the exact SDK_CORE_VOCABULARY shape Venue hit + brain-
  wide strict mode
- Exercises every internal Brainy path: VFS root + mkdir + writeFile + cp + mv
  + ln + symlink; aggregation engine; audit diagnostic with includeVFS toggle
- Validates error message UX: caller location, vocabulary guidance, brain-wide
  strict mode guidance, off-vocabulary value reporting

Docs
- New "Strict mode in practice" section in docs/guides/subtypes-and-facets.md
  covering the SDK_CORE_VOCABULARY pattern, 4-step migration recipe
  (audit → migrateField → hand-fix → re-audit), the Brainy-internal label
  reference table, and an 8.0 forward-look on fillSubtypes()
- docs/api/README.md: new audit() entry, strict-mode tips on add() and relate()
- RELEASES.md: full 7.30.1 entry

Cortex parity (forward-looking, not blocking 7.30.1)
- 6th open question added to .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md: native
  fast path for audit() and fillSubtypes() via column-store null-subtype
  bitmap for billion-scale brains
- Cortex should add a parity test mirroring strict-mode-self-test.test.ts
  against their native paths to catch any latent bug where native writes
  bypass JS validation
- Brainy-internal subtype labels become a documented part of the 8.0 contract
  (useful for Cortex telemetry surfacing Brainy-managed infrastructure %)

Verification
- npx tsc --noEmit: clean
- npm test: 1468/1468 unit
- 7.29 noun integration suite: 26/26 (no regression)
- 7.30 verb subtype + enforcement integration suite: 30/30 (no regression)
- New strict-mode-self-test integration suite: 13/13
- npm run build: clean
- Closed-source product reference audit: clean

Addresses VE-SUBTYPE-MIGRATION (Venue's reported request) and ships internal
labels Venue did NOT ask for but that would have broken them next under their
own vocabulary registration.
2026-06-08 11:31:47 -07:00
36db644eca refactor: remove src/cortex/ directory and fix README claims
- Move neuralImport.ts and neuralImportAugmentation.ts to src/neural/
- Delete 3 dead files (healthCheck, backupRestore, performanceMonitor)
- Remove cortex entries from package.json browser field
- Fix unsubstantiated performance claims in README (add PROJECTED labels)
- Delete stale dist/cortex/ artifacts
2026-02-01 11:07:26 -08:00
Renamed from src/cortex/neuralImport.ts (Browse further)