diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 84bd25e3..54d4f784 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -64,6 +64,10 @@ see `package.json` for `test:integration`, `test:coverage`, and friends. store, measured solo under an exclusive lock" tells a reader everything the number depends on; the machine it ran on and whose data it was tell them nothing except where somebody's infrastructure lives. +- **Documents that answer or reference a confidential specification never enter + this repository, even summarized.** The public docs describe THIS engine and + the published contract, and nothing else — a summary of a private document is + still that document's contents. ## License diff --git a/docs/api-contract.json b/docs/api-contract.json index 9dadcc0e..aafd838a 100644 --- a/docs/api-contract.json +++ b/docs/api-contract.json @@ -1,7 +1,6 @@ { "contractVersion": 1, "engine": "@soulcraftlabs/brainy", - "prose": "docs/contract-1-ratification.md", "compatibility": { "minor": "additive — a new optional door, a new served operator, a new error class; every existing implementation still conforms", "major": "breaking — a door removed, an answer narrowed, an ordering law changed, an optional door promoted to required, or an operator moved from served to refused" diff --git a/docs/canonical-layout-ratification.md b/docs/canonical-layout-ratification.md deleted file mode 100644 index 3c7cf8db..00000000 --- a/docs/canonical-layout-ratification.md +++ /dev/null @@ -1,415 +0,0 @@ -# The canonical layout — ratification - -Open Brainy's answer to the canonical layout specification written by the -accelerated engine's reader team. It reads that document's rules against THIS -repository's writer, at `10.4.3` plus the changes on -`next/open-lazy-open-and-counts`, and for each one says **confirmed**, -**corrected**, or **changed under you** — with the line of code that proves it. - -The specification was written against `10.4.1`. Three of its rules have moved -since, and one of its open questions has already been answered by a fix that -shipped in `10.4.2`. Those are called out first, because a reader -implementation built to the draft would be wrong about them today. - -*Internal engineering document — no frontmatter, not published.* - ---- - -## What moved since the draft was written - -| # | Draft says | Today | Where | -|---|---|---|---| -| A | `scanCanonicalEntities()` counts id DIRECTORIES, so a legacy ledger's derivation includes scars and orphans (§4, open question 6) | **CORRECTED IN 10.4.2.** The scan counts one entity per IDENTITY RECORD, sharing its predicate with the orphan pruner | `src/storage/adapters/fileSystemStorage.ts` `scanCanonicalEntities()` / `hasMetadataContentLeg()` | -| B | An already-derived legacy denominator is persisted "once and never rescanned" (§4, §8) | **CORRECTED on this branch.** A ledger without the `identity-record` stamp is marked suspect at open AND re-derived honestly in the background, then persisted stamped | `scheduleCountLedgerDerivation()`, same file | -| C | `counts.json` has a zero-byte window because `persistCounts()` uses a bare `fs.writeFile` (§8.1, open question 5) | **FIXED on this branch.** Written temp+rename like every other object | `persistCounts()`, same file | - -Everything else below stands as the draft states it, except where marked. - ---- - -## §1 — the tree is a projection, not the source of truth - -**CONFIRMED.** The generation log under `_generations/` is canonical and the -entity tree is a materialization of it; `_system/generation.json[.gz]` carries -`{ generation, updatedAt }` for the projection. `src/db/generationStore.ts` -owns the counter and its manifest; `readBrainFormat` / the entity-tree stamp -are read at open in `src/brainy.ts`'s generation-store phase. - -Two walks minutes apart may legitimately disagree, and a caller that needs a -point-in-time answer must pin the generation. Confirmed as a rule of this -engine, not merely of the reader. - ---- - -## §2 — paths and sharding - -**CONFIRMED, all three clauses.** - -- Two lowercase hex characters, 256 buckets, and an id that normalises to 32 - hex characters buckets by its first byte: `src/storage/sharding.ts` - `getShardId()` — `const normalized = id.toLowerCase().replace(/-/g, '')`, - `if (/^[0-9a-f]{32}$/.test(normalized)) return normalized.substring(0, 2)`. -- Every other id hashes: `hashToShardId(id)`, FNV-1a over **UTF-16 code - units** (`id.charCodeAt(i)`), folded to the low byte. The UTF-16 detail is - load-bearing above the BMP and is part of the on-disk contract — changing it - relocates every application-id record. -- `hnsw/` and `metadata/` under `entities//` are vestigial and the write - path never populates them. Every scan in this repository excludes them with - the same two-hex test (`/^[0-9a-f]{2}$/i`) — see `scanCanonicalEntities()` - and `scanVectoredNounCount()`. **A reader that counts directory entries - reports 258 shards where there are 256** is correct and worth keeping in the - spec. - ---- - -## §3 — encodings - -**CONFIRMED.** - -- The compressed spelling APPENDS: `const compressedPath = \`${fullPath}.gz\`` - (`fileSystemStorage.ts`, the object read and write paths). Deriving it by - replacing the extension finds a file that never exists. -- Read order is `.gz` first, then plain — same call sites. -- The three outcomes are distinct in this engine too: absence is `null`, an - undecodable body raises `TornRecordError` (`src/storage/tornRecordError.ts`, - `registerTornRecordEncounter`), and an IO fault is rethrown unreshaped. -- The legacy exception the draft names — a torn `.gz` with a decodable plain - twin returns the recovered plain object, loudly — is real and deliberate; the - code comment at the read path states it in those terms. - -**Divergence (torn rows): RATIFIED AS PERMANENT for the current contract.** -This engine's paginated walks skip-and-count a torn row so enumeration heals -past it; the reader raises with a resume cursor. Both are defensible and they -are not reconcilable by picking one: an enumeration that raises cannot serve a -heal, and a heal that skips silently is how corruption becomes invisible. The -reader's `resume_cursor` gives a caller the skip behaviour deliberately, which -is the better shape. **This engine will not change its walks to raise** — its -walks exist to heal — and the two enumerations are therefore allowed to report -different populations over a DAMAGED store, never over a healthy one. Any -consumer comparing the two must first establish the store is not damaged. - ---- - -## §4 — the population law - -**CONFIRMED as the law. CORRECTED as a description of this engine's code.** - -The law is right and is this engine's own: *the identity record is the -population; the id directory is not.* `pruneOrphanedEntities()` states it and -`hasMetadataContentLeg()` is the single predicate both it and the count scan -use, so the two agree by construction: - -```ts -private hasMetadataContentLeg(legs: string[]): boolean { - return legs.some((f) => f.startsWith('metadata.json')) -} -``` - -**The draft's finding that `scanCanonicalEntities()` counts directories is out -of date.** It was true of `10.4.1`, which the draft was written against; it was -fixed in `10.4.2` ("derive the canonical count ledger from identity records, -stamp the derivation rule, and mark legacy-derived ledgers suspect at load"). -The scan now `continue`s on any container with no metadata content leg. - -**But the draft's MEASUREMENTS remain valid, and the defect they name was only -half-fixed until this branch.** A ledger DERIVED under the old rule was -adopted at load with its wrong numbers and merely flagged `allCountsSuspect`, -and nothing corrected it short of an operator running `repairIndex()`. That is -why the same frozen archive still reads 14,231 or 14,081 against 14,056 -identity records depending on which copy you open, and why a graph heal -subtracted 72,679 walked rows from a 72,729 denominator and reported -`remaining: 50` — the store's 50 verb scar directories. The heal was behaving -correctly against a lying denominator. - -**On this branch the ledger heals itself**: a load that finds no -`identity-record` stamp marks the scalars suspect, schedules an honest -derivation in the BACKGROUND (never blocking the open — the same walk is part -of why a 24,898-id store opened in silence), and persists the corrected -scalars stamped. A derivation that raced a write refuses to stamp its number -and leaves the ledger suspect, naming `repairIndex()` as the door that -recounts under a barrier. Pinned in -`tests/integration/count-ledger-identity-record.test.ts`, including two copies -of one archive deriving the same number. - -**Answer to open question 6, in full:** - -1. *Should the derivation require an identity leg?* **Yes, and it does** — since - `10.4.2`, sharing the pruner's predicate. -2. *Should already-persisted denominators on affected stores be re-derived?* - **Yes, and they now are, automatically, in the background.** An operator no - longer has to know. -3. *Which scalar may a consumer subtract against?* See the next section — the - answer is narrower than "the corrected one". - ---- - -## §8 / open question 6 — WHICH SCALAR A CONSUMER MAY SUBTRACT AGAINST - -The specification asked this to be stated in the ratification. Stated: - -| scalar | what it is | safe to subtract an enumeration against? | -|---|---|---| -| `totalNounCount` / `totalVerbCount` | the **counted tiers only** (`isCountedVisibility` excludes `internal` and `system`) | **NO.** It is routinely far below the live population by design. A full-tier walk measured against it reads permanently over. | -| `totalNounCountAll` / `totalVerbCountAll` **with `allCountsDerivedBy: 'identity-record'` and `allCountsSuspect: false`** | the all-tier identity-record population | **YES.** This is the only pair a coverage verdict may use. | -| the same scalars **without the stamp, or with `allCountsSuspect: true`** | an unverified number: either derived under the old container rule, or made unprovable by a delete that could not read its record | **NO.** Read the flag, refuse the verdict, and say so. On this branch the engine is already correcting it behind you; wait for the correction rather than subtracting against the interim. | -| `totalVectoredNounCount` | nouns holding a real, non-empty, non-zero-norm vector, derived by reading record CONTENT | **YES for the vector family only.** It never had the container defect. It shares `allCountsSuspect`, so the same flag rule applies. | - -`getCanonicalCounts()` (`src/storage/adapters/baseStorageAdapter.ts`) returns -all of them with the suspect flag in one O(1) call, and the flag is the whole -contract: **a consumer that ignores `suspect` is computing a verdict the engine -has told it not to compute.** - -The draft's §8 rules are otherwise **CONFIRMED**: the ledger is counted-tier, -an absent key is a legacy file and not a zero, and an absent `counts.json` is a -real condition (this engine rebuilds it from disk on the next open — see -`initializeCountsFromDisk()`, which on this branch narrates that it is doing so -and why it cannot be backgrounded). - -**Open question 3 — the ledger drifted from the public tier by 3 nouns and 1 -verb.** Not ratified as expected drift. The counted-tier scalars are maintained -incrementally and can be left inflated by a delete whose decrement was skipped; -that is exactly what `allCountsSuspect` exists to record and what -`rebuildTypeCounts()` (the sanctioned recount, `src/storage/baseStorage.ts`) -corrects. A drift of 3 against 8,663 on a store with a long delete history is -consistent with that mechanism, not with a fresh defect — but it is a REAL -inaccuracy, not a definition, and the cure is a recount. **Finding for the -spec: do not describe counted-tier drift as normal; describe it as an -uncorrected incremental counter.** - ---- - -## §5.1 — the identity leg - -**CONFIRMED, every clause.** - -- Two shapes coexist in one store: `_fmt: 2` nested-bag, and the legacy flat - shape. `src/types/reservedFields.ts` — `METADATA_RECORD_FORMAT_KEY = '_fmt'`, - `NESTED_BAG_FORMAT`. -- **The stamp alone does not decide it.** `isNestedBagRecord()` requires the - stamp AND `typeof record.metadata === 'object'` AND non-null AND - `!Array.isArray(record.metadata)`. A reader that trusts the stamp alone - mis-splits a pre-law record carrying a user field named `metadata`. The draft - is right to make this a rule. -- Engine fields sit at top level in BOTH shapes, so extraction is - shape-independent: `RESERVED_ENTITY_FIELDS` / `RESERVED_RELATIONSHIP_FIELDS`. - The entity type is under `noun` (verbs: `verb`), never `type`. -- **No BigInt at this boundary.** Confirmed — every numeric in a metadata - record is a JSON number. - -**Divergence (timestamps): RATIFIED IN THE READER'S FAVOUR, and this engine -should follow.** The draft is correct that this engine substitutes `Date.now()` -where a stored timestamp is unusable. That is inventing data: a row whose -`updatedAt` could not be read is not a row that was updated now. The reader's -`null` is right. **This is a finding against this engine, filed here rather -than fixed on this branch** — changing hydration's timestamp fallback touches -every read path and belongs in its own change with its own pins, not in a -branch about opens and counters. Until then, a consumer must treat a timestamp -equal to read-time as suspect. - ---- - -## §5.2 — the vector leg - -**CONFIRMED, every clause**, including the three that are easiest to get wrong: - -- It is **JSON**, not an mmap-able buffer; raw blobs live in `_blobs/` behind - `getBinaryBlobPath()` — a different door entirely. -- **The five-way vector state** is this engine's own law: `absent-leg`, - `absent-field`, `empty` (deferred embed — `vector: []` written at `add()` - time), `zero-norm`, `real`. `isZeroNormVector()` (`src/utils/distance.ts`) is - the shared predicate, and `scanVectoredNounCount()` applies exactly it. A - zero-norm vector **is not a vector** — the 10.4.2 unvector work is precisely - this rule, canonical side included. -- **`connections` on disk is not index state.** Confirmed. Nothing may infer - that a store is indexed, or how, from that field or from the vestigial - `hnsw/` directories. - -**Open question 4 — vector legs carrying stale identity copies.** **CONFIRMED -NEVER AUTHORITATIVE.** Type, subtype and visibility are read from the identity -leg only; the copies in a vector leg are residue from an older writer. This -engine will not start reading them and any reader that does will disagree with -it on rows the two writers touched at different times. Whether Stage 2 drops -them is the accelerated engine's format decision, not this engine's — from -here, dropping them is safe. - -**The `float_roundtrip` finding is accepted and worth restating as a rule for -any future reader in any language:** the leg is JSON text, so every component -is a double, and a decimal has exactly one nearest `f64`. A parser one ULP off -builds a different index — different distances, different neighbours — and -makes any parity claim between two engines false. This engine gets it right -only because `JSON.parse` is correctly rounded by specification; nothing in -this repository would have caught the divergence, and the conformance suite's -whole-vector deep-equal is the right instrument. - ---- - -## §6 — verb endpoints - -**CONFIRMED, exactly as stated, and it is a defect of shape rather than of -data.** - -`saveVerb_internal()` (`src/storage/baseStorage.ts`) writes the whole -`HNSWVerb` — `{ id, vector, connections, verb, sourceId, targetId }` — to -`entities/verbs///vectors.json`, and `saveVerbMetadata()` writes the -engine scalars and user bag to `metadata.json` with **no endpoints**. `getVerb()` -reassembles the two. - -The consequences the draft draws are all correct, including the one that costs -the most: **a graph heal cannot be driven from identity records alone**, so -every edge's vector leg is read purely for its structure even though a verb's -`vector` is always `[]`. This engine confirms the measurement in principle — -every verb it has ever written carries its endpoints in the vector leg and none -in the identity leg. - -The door contract that returns `(id, sourceId, targetId, verb, subtype)` -regardless of which leg holds the bytes is the right shape and this engine -endorses it as the migration seam. **Commitment: when Stage 2 moves endpoints -onto the verb record, this engine's reader side must accept BOTH placements for -the life of contract 1** — a store written by either engine must be readable by -the other. - ---- - -## §7 — `_system/` - -**CONFIRMED**, with one clause now needing a footnote. - -- `_system/` mixes persisted state with live protocol and must never be - enumerated as data. `idx/` and `family-stamps/` are subtrees. -- `tx-log.jsonl` is mutated in place and must be byte-copied, not hard-linked, - by a snapshot. -- `/locks/` is live coordination — `_writer.lock` and the flush-request - protocol. Reading them as records is meaningless; writing or deleting them - interferes with a live writer. **Footnote: this branch adds one more file - there, `_writer.close`** — the clean-close record naming the lock generation - a writer released, consumed by the next claim. Same rule applies: it is - protocol, not data. A reader must ignore it, and must not treat its presence - or absence as a fact about the store's contents. -- **The encoding split is real and remains real:** everything in `_system/` - goes through the compressing writer and is `.gz` **except `counts.json`**, - which is plain JSON. This branch made that write ATOMIC but deliberately did - NOT make it `.gz` — the ledger is the one file an operator reads with `cat` - during an incident, and a reader implementation already handles the - exception. Try both spellings, as the draft says. - ---- - -## §8.1 — the zero-byte ledger window - -**ACCEPTED AS A DEFECT AND FIXED ON THIS BRANCH.** - -Cause 1 is confirmed exactly as written: `persistCounts()` used a bare -`fs.writeFile`, which truncates first, so the ledger was empty for the whole -write while every other object in the store was written atomically. It now goes -through the same temp+rename path (`writeFileAtomic`), so a reader sees the old -ledger or the new one and never neither. - -Cause 2 — "the final write is not awaited by the closing path" — **NOT -REPRODUCED at 10.4.3**, and the mechanism named does not exist here: the -filesystem adapter's count persist is write-through, not debounced -(`scheduleCountPersist()` sets the flag and immediately `await`s -`flushCounts()`), and `close()` awaits `flushCounts()` in its first flush -phase. The ~750 ms the draft measured is more likely the tail of an in-flight -write-through persist observed through the truncation window of cause 1, which -is now closed. **Finding: re-measure on 10.4.4 before keeping cause 2 in the -spec.** - -**Answer to open question 5:** yes, fixed; and yes, a reader should keep -classifying a zero-byte ledger as `TORN_RECORD` — the classification is right -on its own terms (a file that exists and holds nothing IS torn), it is the only -honest reading of a store damaged by an older release, and stores written by -every version before 10.4.4 can still carry one. - -Pinned here by `tests/integration/count-ledger-identity-record.test.ts` — -40 consecutive persists watched at 1 ms and never once unparseable. - ---- - -## §9 — the ordering law - -**CONFIRMED, including the two clauses most likely to be "simplified" by a -future implementer.** - -- Shard `00` → `ff` ascending, then id ascending in **UTF-16 code-unit order** — - this engine's `.sort()` with an `a.id < b.id` comparator, which is exactly - UTF-16 code-unit order, not UTF-8 byte order and not code-point order. They - diverge above the BMP. -- Cursor tokens: `cn1::` and `cv1::`, shard in DECIMAL - (it is the loop index, not the directory name), id last so an id containing - `:` survives the round trip. `src/storage/baseStorage.ts` — - `` return `cn1:${shard}:${id}` `` and `` return `cv1:${shard}:${id}` ``. -- A supplied-but-undecodable cursor must FAIL, never restart at offset 0. - Confirmed as a rule: silently restarting turns a `while (hasMore)` loop into - an unbounded one. -- The stability properties (a consistent prefix per shard, not a snapshot; - appends behind the cursor missed; deletes ahead take effect) are this - engine's own behaviour, confirmed. - ---- - -## §10 — read depth - -**CONFIRMED as a design, with one clause this engine wants on the record:** -*read depth bounds what corruption can be seen.* An ids-only enumeration opens -nothing and therefore reports a clean population over a store of undecodable -bodies. That is the honest consequence of reading exactly what was asked for. -A caller asking "is this store healthy?" from the cheap level is asking the -wrong door — health needs a decoding read. This engine's health reporting -follows the same rule and should say so as plainly. - ---- - -## §11 — cost - -**NOT RATIFIED — NOT THIS ENGINE'S TO RATIFY.** The comparative walls are -measurements of two implementations on one box; nothing in this repository can -confirm or refute them. Two observations that ARE this engine's: - -1. The structural explanation offered for the verb gap is consistent with this - engine's code: the noun walk hydrates in batches, the verb walk does not, - and a cursored resume re-lists and re-sorts the cursor's shard on every - page. That is a real asymmetry in `src/storage/baseStorage.ts` and it is - fair to name it. -2. The draft's own honesty about the applier holding 98% of the verb wall after - fan-out is the part worth keeping. **Nobody may read "8 workers" as "8×".** - ---- - -## §12 — Stage 1 - -**NOTED, NOT RATIFIED.** Which read path the accelerated engine's walks use is -its own decision. Two things this engine confirms because they are claims about -THIS code: - -- **The membership-predicate correction is right.** This engine's membership - is the IDENTITY RECORD: an id directory holding a vector leg and no identity - record is enumerated by neither `getNouns()` nor - `getNounIdsWithPagination()` and is absent from `getCanonicalCounts()`; an - identity-only row IS enumerated and IS counted. The draft's earlier statement - that the doors key on `vectors.json` was wrong and its correction is accepted. -- **Read-your-writes is confirmed.** There is no write-behind buffer between an - awaited `saveNoun()` / `saveNounMetadata()` and the bytes on disk; the leg is - `stat`-able before the returned promise resolves. **Commitment: this engine - will not introduce a write-behind buffer in the filesystem adapter without - declaring a flush contract alongside it** — the law "anything not yet awaited - is not yet canonical" is only useful if the converse holds. - ---- - -## Findings this ratification files back - -1. **§4 / open question 6 is out of date** — the directory-counting scan was - fixed in 10.4.2. Re-read the spec's §4 against 10.4.3 before Stage 2 builds - on it. The MEASUREMENTS stay valid; the code claim does not. -2. **§8.1 cause 2 is not reproducible at 10.4.3** — the count persist is - write-through and `close()` awaits it. Re-measure before republishing. -3. **§8's counted-tier drift (open question 3) should not be described as - expected** — it is an uncorrected incremental counter, and `repairIndex()` - is its cure. -4. **The timestamp divergence is a defect on THIS engine's side**, not a - difference of opinion: substituting `Date.now()` for an unreadable - timestamp invents data. Filed, not fixed on this branch. -5. **`locks/_writer.close` is new** — a reader must ignore it like every other - file under `locks/`. -6. **`counts.json` is now written atomically and stays plain JSON** — the - `_system/` encoding exception is deliberate and permanent for contract 1. diff --git a/docs/contract-1-ratification.md b/docs/contract-1-ratification.md deleted file mode 100644 index 87df7b1f..00000000 --- a/docs/contract-1-ratification.md +++ /dev/null @@ -1,244 +0,0 @@ -# Contract 1 — ratification - -Open Brainy's answer to the API contract published by the accelerated engine -(`docs/api-contract.md` + `docs/api-contract.json`, contract version 1). Each -item is answered with the code line that proves it, and each promise is stated -as a promise rather than a description. - -*Internal engineering document — no frontmatter, not published.* - ---- - -## 1. Contract version — DECLARED - -`package.json` carries `"brainyContract": 1`, and the engine states its own: - -```ts -export const BRAINY_CONTRACT_VERSION = 1 as const -export function contractVersion(): number { return BRAINY_CONTRACT_VERSION } -``` - -`src/utils/version.ts`, re-exported from `src/index.ts`. Two engines can now -compare an integer instead of probing prototypes, and a tool can read the -package field without importing the engine. Pinned in -`tests/integration/filter-operator-conformance.test.ts` ("declares its contract -version in code and in package.json") — the code value and the package field -can never drift apart silently. - ---- - -## 2. The REQUIRED / OPTIONAL split — RATIFIED, WITH A COMMITMENT - -**Ratified: 41 required doors of 57.** The promise, stated plainly: - -> **A REQUIRED door is never removed, never narrowed, and never made optional -> without a MAJOR contract bump.** "Narrowed" includes: refusing an input it -> used to accept, returning less than it used to return, and changing an -> ordering, a cursor encoding, or a refusal's typed code. An OPTIONAL door may -> be added in a minor; an optional door **promoted to required** is a major, -> because a consumer that relied on feature-detecting it now has a hard -> dependency. - -Two clarifications this engine attaches, so the promise means the same thing -on both sides: - -1. **A refusal is part of the door.** Contract 1 includes not only that - `find({ where })` answers, but that it REFUSES by name for the operators - listed as refused. Turning a refusal into a silent empty answer is a - narrowing, not a relaxation — the same class of change as removing the door. -2. **Deprecation is not removal.** This engine may mark a required door - deprecated in a minor (documented, warned) as long as it keeps working. Only - its removal is a major. - ---- - -## 3. `is` / `isNot` / `greaterEqual` / `lessEqual` — A FINDING AGAINST THE SPEC - -**The specification is wrong about these four, and this engine has never -served them.** `docs/filter-operator-conformance.md` (in the accelerated -engine's repository — not editable from here) lists them as served aliases. -The accepted set is defined in one place: - -```ts -// src/utils/metadataFilter.ts -const VALUE_OPERATORS = new Set([ - 'equals', 'eq', 'notEquals', 'ne', - 'greaterThan', 'gt', 'greaterThanOrEqual', 'gte', - 'lessThan', 'lt', 'lessThanOrEqual', 'lte', - 'between', 'oneOf', 'in', 'noneOf', - 'contains', 'excludes', 'hasAll', 'length', - 'exists', 'missing', 'matches', 'startsWith', 'endsWith' -]) -``` - -25 tokens. None of the four appears; `validateWhereFilter()` raises -`BrainyError('INVALID_QUERY')` naming the bad operator and listing the valid -set, before any index read. A consumer following the spec would have written a -filter this engine rejects outright. - -**Action taken here, since the prose lives in the other repository:** the truth -is made machine-checkable rather than re-asserted in another document. The -accepted set is asserted token-for-token in -`tests/integration/filter-operator-conformance.test.ts`, read out of the -engine's own refusal message, and the same set is emitted into -`docs/api-contract.json` (item 8). Diff the manifests; the prose can then be -corrected from a fact. - ---- - -## 4. The serving-withholding invariant list — CONFIRMED IDENTICAL - -`index-initialized · durable-state-present · manifest-residency · -replay-clean · strand-latch`. Confirmed as this engine's list, and confirmed -EXHAUSTIVE for contract 1: these are the only invariants whose failure may -withhold serving. Everything else a health report can fail is a `warn` — it -names damage without closing a door. - -The mechanism on this side: `assessProviderHealth()` -(`src/utils/indexReadiness.ts`) treats the provider's own `serving` verdict as -authoritative and verbatim; an UNLEDGERED family never flips a serving provider -to not-ready and never flips a not-serving provider to ready. The read gate -refuses PER FAMILY — a metadata read is never refused by an unserving vector -leg (`src/brainy.ts`, `ensureFamiliesServing`). - -**One addition this engine is making, declared here because it changes what a -refusal MEANS:** a provider may now report `rebuildInProgress()` — it is -rebuilding ITSELF, online, and its doors refuse by name with progress until it -is whole. This does not add a withholding invariant (the provider's own -`serving: false` is still what withholds); it adds a REASON attached to that -withholding, so a caller can tell "temporarily closed, opens by itself" from -"broken, needs repairIndex()". Additive, hence a minor. - ---- - -## 5. The compatibility rule — ADOPTED - -**Minor = additive. Major = breaking.** Adopted verbatim, with the -announcement duty attached: - -> **Every public-surface addition is announced.** The accelerated engine's -> package re-exports this engine's surface by enumeration, so it goes red on -> any new export BY DESIGN — that redness is the announcement mechanism -> working, not a build break to route around. - -The mechanics that make this checkable rather than remembered: -`scripts/emit-contract-manifest.mjs --check` fails when the committed -`docs/api-contract.json` no longer matches the built surface. A new export is -therefore a red check with a message naming what to do: re-emit and announce. - ---- - -## 6. The 30 storage seam methods — SUPPORTED SURFACE, COMMITTED - -**Committed: every method in `docs/api-contract.md` §15 is supported surface -until Stage 2, and none is removed without a contract major.** They are the -seam the accelerated engine's storage adapter implements and the seam its -reader replaces piece by piece; removing one mid-programme would break a -working pair for no gain. - -Two qualifications, both stated so neither side is surprised: - -1. **Supported ≠ frozen in behaviour.** A seam method may become FASTER, may - narrate more, and may start refusing an input that was previously an - undefined-behaviour footgun — the last of those is announced as a divergence - here before it ships, not discovered by the other engine. -2. **`counts.json`'s ledger is the one seam value that is not an - enumeration.** See `docs/canonical-layout-ratification.md` §8: only the - all-tier pair carrying `allCountsDerivedBy: 'identity-record'` with - `allCountsSuspect: false` may be subtracted against. That rule is part of - this commitment. - ---- - -## 7. `hasAll` / `noneOf` / `excludes` — SERVED, NOT RATIFIED AS A DIVERGENCE - -The accelerated engine was right that it was the correct side, and the -divergence is now closed in the right direction: **this engine serves all three -on the index path.** - -The defect underneath was worse than a divergence. The metadata index's -operator switch (`src/utils/metadataIndex.ts`) had **no default case**, so any -operator without a `case` left the field's match set at its initial `[]` and -`find()` returned an empty page. `hasAll`, `noneOf` and `excludes` are -documented, accepted by the validator, and implemented in the in-memory -matcher — and they answered silently wrong through an index-backed find. - -- **`hasAll: [a, b]`** — the intersection of each element's posting set. An - empty operand array is vacuously true of every row that HAS the field. -- **`noneOf: [a, b]`** — the complement of the union of their posting sets. -- **`excludes: v`** — the complement of `contains`. - -**And the other four are now REFUSED BY NAME rather than answered empty.** -`startsWith`, `endsWith`, `matches` and `length` cannot be evaluated by an -equality/range posting index without reading every row, which is the cost this -path exists to avoid. They raise `BrainyError('INVALID_QUERY')` naming the -operator, the field, and the reason. This matches the accelerated engine's -behaviour for the same four tokens, so the two engines now AGREE on all 25: - -| class | tokens | -|---|---| -| served on the index path | between, contains, eq, equals, excludes, exists, greaterThan, greaterThanOrEqual, gt, gte, hasAll, in, lessThan, lessThanOrEqual, lt, lte, missing, ne, noneOf, notEquals, oneOf | -| refused by name | endsWith, length, matches, startsWith | - -Pinned in `tests/integration/filter-operator-conformance.test.ts`: the exact -25-token accepted set, the three now served with their real answers (including -an honest zero), and each of the four refusing by name. - -**This is a behaviour change for any consumer today calling the four refused -operators through `find({ where })`.** They received an empty page; they now -receive a typed refusal. Converting a wrong answer into a loud refusal is this -engine's own law, and the previous behaviour was not a contract anyone could -have relied on deliberately — but it is a change, and it is named here rather -than discovered. - -**`knownDivergences` after this change:** the entry -`served-beyond-baseline` is RESOLVED (both engines serve all three). The entry -`refused-operators-answer-differently` is RESOLVED (both engines refuse the -same four by name). Contract 1 has no remaining operator divergence. - ---- - -## 8. This engine's own manifest — EMITTED - -`docs/api-contract.json`, generated by `scripts/emit-contract-manifest.mjs` -from the BUILT surface: the prototype's own methods and accessors, the exported -error classes, the operator sets read out of their single definitions, the -field-addressing vocabulary read out of `src/db/fieldAddressing.ts`, and the -health verdicts. Nothing in it is hand-maintained, so a diff between the two -manifests is a diff between two engines rather than between two authors. - -`node scripts/emit-contract-manifest.mjs --check` fails when the committed -manifest is stale — the announcement duty of item 5, made mechanical. - -**What the manifest deliberately does NOT carry: requirement marking.** Whether -a door is required is a commitment, not a property of the surface; it is item 2 -of this document. The diff the two sides want — "does Open Brainy still expose -every door contract 1 requires?" — is a set-membership check between their -`doors[].name` where `requirement === 'required'` and this manifest's -`doors[].name`. - ---- - -## A defect this work surfaced but did not fix - -`src/vfs/VirtualFileSystem.ts` builds a path-prefix filter as -`path: { $startsWith: options.path }` — with a `$` prefix. No operator in this -engine carries a `$`, so `validateWhereFilter()` rejects it with -`INVALID_QUERY` before any index read: **`vfs.searchFiles({ path })` throws -today, on every call that passes a path.** It is pre-existing and unrelated to -the operator work above (the validator refuses it before the index path is -reached), and it is left as a filing rather than fixed here, because the right -answer is a design question — a path-prefix search cannot be served by an -equality/range index, so it needs either a path-segment index or an explicit -in-memory narrow, not a spelling correction. - ---- - -## Summary of what changed in code for this ratification - -| item | change | -|---|---| -| 1 | `"brainyContract": 1` in package.json; `contractVersion()` / `BRAINY_CONTRACT_VERSION` exported | -| 3 | the accepted 25-token set asserted from the engine's own refusal message, and emitted into the manifest | -| 7 | `hasAll` / `noneOf` / `excludes` served on the index path; `startsWith` / `endsWith` / `matches` / `length` refused by name instead of answered empty | -| 8 | `scripts/emit-contract-manifest.mjs` + the generated `docs/api-contract.json`, with a `--check` mode | diff --git a/scripts/emit-contract-manifest.mjs b/scripts/emit-contract-manifest.mjs index 13a9bc6d..be73d4ca 100644 --- a/scripts/emit-contract-manifest.mjs +++ b/scripts/emit-contract-manifest.mjs @@ -10,8 +10,8 @@ * between two engines, never between two authors. * * Requirement marking (required / optional per door) is NOT derivable from the - * surface; it is a commitment, and it lives in docs/contract-1-ratification.md. - * This manifest carries the surface; that document carries the promise. + * surface — it is a commitment, recorded with the contract's owner rather than + * here. This manifest carries the surface; the promise lives with the contract. * * Usage: node scripts/emit-contract-manifest.mjs [--check] * --check exits non-zero when the committed manifest is stale. @@ -68,7 +68,6 @@ const servedOnIndex = accepted.filter((op) => !refusedByIndex.includes(op)) const manifest = { contractVersion: versionModule.contractVersion(), engine: '@soulcraftlabs/brainy', - prose: 'docs/contract-1-ratification.md', compatibility: { minor: 'additive — a new optional door, a new served operator, a new error class; every existing implementation still conforms',