diff --git a/docs/canonical-layout-ratification.md b/docs/canonical-layout-ratification.md new file mode 100644 index 00000000..3c7cf8db --- /dev/null +++ b/docs/canonical-layout-ratification.md @@ -0,0 +1,415 @@ +# The canonical layout — ratification + +Open Brainy's answer to the canonical layout specification written by the +accelerated engine's reader team. It reads that document's rules against THIS +repository's writer, at `10.4.3` plus the changes on +`next/open-lazy-open-and-counts`, and for each one says **confirmed**, +**corrected**, or **changed under you** — with the line of code that proves it. + +The specification was written against `10.4.1`. Three of its rules have moved +since, and one of its open questions has already been answered by a fix that +shipped in `10.4.2`. Those are called out first, because a reader +implementation built to the draft would be wrong about them today. + +*Internal engineering document — no frontmatter, not published.* + +--- + +## What moved since the draft was written + +| # | Draft says | Today | Where | +|---|---|---|---| +| A | `scanCanonicalEntities()` counts id DIRECTORIES, so a legacy ledger's derivation includes scars and orphans (§4, open question 6) | **CORRECTED IN 10.4.2.** The scan counts one entity per IDENTITY RECORD, sharing its predicate with the orphan pruner | `src/storage/adapters/fileSystemStorage.ts` `scanCanonicalEntities()` / `hasMetadataContentLeg()` | +| B | An already-derived legacy denominator is persisted "once and never rescanned" (§4, §8) | **CORRECTED on this branch.** A ledger without the `identity-record` stamp is marked suspect at open AND re-derived honestly in the background, then persisted stamped | `scheduleCountLedgerDerivation()`, same file | +| C | `counts.json` has a zero-byte window because `persistCounts()` uses a bare `fs.writeFile` (§8.1, open question 5) | **FIXED on this branch.** Written temp+rename like every other object | `persistCounts()`, same file | + +Everything else below stands as the draft states it, except where marked. + +--- + +## §1 — the tree is a projection, not the source of truth + +**CONFIRMED.** The generation log under `_generations/` is canonical and the +entity tree is a materialization of it; `_system/generation.json[.gz]` carries +`{ generation, updatedAt }` for the projection. `src/db/generationStore.ts` +owns the counter and its manifest; `readBrainFormat` / the entity-tree stamp +are read at open in `src/brainy.ts`'s generation-store phase. + +Two walks minutes apart may legitimately disagree, and a caller that needs a +point-in-time answer must pin the generation. Confirmed as a rule of this +engine, not merely of the reader. + +--- + +## §2 — paths and sharding + +**CONFIRMED, all three clauses.** + +- Two lowercase hex characters, 256 buckets, and an id that normalises to 32 + hex characters buckets by its first byte: `src/storage/sharding.ts` + `getShardId()` — `const normalized = id.toLowerCase().replace(/-/g, '')`, + `if (/^[0-9a-f]{32}$/.test(normalized)) return normalized.substring(0, 2)`. +- Every other id hashes: `hashToShardId(id)`, FNV-1a over **UTF-16 code + units** (`id.charCodeAt(i)`), folded to the low byte. The UTF-16 detail is + load-bearing above the BMP and is part of the on-disk contract — changing it + relocates every application-id record. +- `hnsw/` and `metadata/` under `entities//` are vestigial and the write + path never populates them. Every scan in this repository excludes them with + the same two-hex test (`/^[0-9a-f]{2}$/i`) — see `scanCanonicalEntities()` + and `scanVectoredNounCount()`. **A reader that counts directory entries + reports 258 shards where there are 256** is correct and worth keeping in the + spec. + +--- + +## §3 — encodings + +**CONFIRMED.** + +- The compressed spelling APPENDS: `const compressedPath = \`${fullPath}.gz\`` + (`fileSystemStorage.ts`, the object read and write paths). Deriving it by + replacing the extension finds a file that never exists. +- Read order is `.gz` first, then plain — same call sites. +- The three outcomes are distinct in this engine too: absence is `null`, an + undecodable body raises `TornRecordError` (`src/storage/tornRecordError.ts`, + `registerTornRecordEncounter`), and an IO fault is rethrown unreshaped. +- The legacy exception the draft names — a torn `.gz` with a decodable plain + twin returns the recovered plain object, loudly — is real and deliberate; the + code comment at the read path states it in those terms. + +**Divergence (torn rows): RATIFIED AS PERMANENT for the current contract.** +This engine's paginated walks skip-and-count a torn row so enumeration heals +past it; the reader raises with a resume cursor. Both are defensible and they +are not reconcilable by picking one: an enumeration that raises cannot serve a +heal, and a heal that skips silently is how corruption becomes invisible. The +reader's `resume_cursor` gives a caller the skip behaviour deliberately, which +is the better shape. **This engine will not change its walks to raise** — its +walks exist to heal — and the two enumerations are therefore allowed to report +different populations over a DAMAGED store, never over a healthy one. Any +consumer comparing the two must first establish the store is not damaged. + +--- + +## §4 — the population law + +**CONFIRMED as the law. CORRECTED as a description of this engine's code.** + +The law is right and is this engine's own: *the identity record is the +population; the id directory is not.* `pruneOrphanedEntities()` states it and +`hasMetadataContentLeg()` is the single predicate both it and the count scan +use, so the two agree by construction: + +```ts +private hasMetadataContentLeg(legs: string[]): boolean { + return legs.some((f) => f.startsWith('metadata.json')) +} +``` + +**The draft's finding that `scanCanonicalEntities()` counts directories is out +of date.** It was true of `10.4.1`, which the draft was written against; it was +fixed in `10.4.2` ("derive the canonical count ledger from identity records, +stamp the derivation rule, and mark legacy-derived ledgers suspect at load"). +The scan now `continue`s on any container with no metadata content leg. + +**But the draft's MEASUREMENTS remain valid, and the defect they name was only +half-fixed until this branch.** A ledger DERIVED under the old rule was +adopted at load with its wrong numbers and merely flagged `allCountsSuspect`, +and nothing corrected it short of an operator running `repairIndex()`. That is +why the same frozen archive still reads 14,231 or 14,081 against 14,056 +identity records depending on which copy you open, and why a graph heal +subtracted 72,679 walked rows from a 72,729 denominator and reported +`remaining: 50` — the store's 50 verb scar directories. The heal was behaving +correctly against a lying denominator. + +**On this branch the ledger heals itself**: a load that finds no +`identity-record` stamp marks the scalars suspect, schedules an honest +derivation in the BACKGROUND (never blocking the open — the same walk is part +of why a 24,898-id store opened in silence), and persists the corrected +scalars stamped. A derivation that raced a write refuses to stamp its number +and leaves the ledger suspect, naming `repairIndex()` as the door that +recounts under a barrier. Pinned in +`tests/integration/count-ledger-identity-record.test.ts`, including two copies +of one archive deriving the same number. + +**Answer to open question 6, in full:** + +1. *Should the derivation require an identity leg?* **Yes, and it does** — since + `10.4.2`, sharing the pruner's predicate. +2. *Should already-persisted denominators on affected stores be re-derived?* + **Yes, and they now are, automatically, in the background.** An operator no + longer has to know. +3. *Which scalar may a consumer subtract against?* See the next section — the + answer is narrower than "the corrected one". + +--- + +## §8 / open question 6 — WHICH SCALAR A CONSUMER MAY SUBTRACT AGAINST + +The specification asked this to be stated in the ratification. Stated: + +| scalar | what it is | safe to subtract an enumeration against? | +|---|---|---| +| `totalNounCount` / `totalVerbCount` | the **counted tiers only** (`isCountedVisibility` excludes `internal` and `system`) | **NO.** It is routinely far below the live population by design. A full-tier walk measured against it reads permanently over. | +| `totalNounCountAll` / `totalVerbCountAll` **with `allCountsDerivedBy: 'identity-record'` and `allCountsSuspect: false`** | the all-tier identity-record population | **YES.** This is the only pair a coverage verdict may use. | +| the same scalars **without the stamp, or with `allCountsSuspect: true`** | an unverified number: either derived under the old container rule, or made unprovable by a delete that could not read its record | **NO.** Read the flag, refuse the verdict, and say so. On this branch the engine is already correcting it behind you; wait for the correction rather than subtracting against the interim. | +| `totalVectoredNounCount` | nouns holding a real, non-empty, non-zero-norm vector, derived by reading record CONTENT | **YES for the vector family only.** It never had the container defect. It shares `allCountsSuspect`, so the same flag rule applies. | + +`getCanonicalCounts()` (`src/storage/adapters/baseStorageAdapter.ts`) returns +all of them with the suspect flag in one O(1) call, and the flag is the whole +contract: **a consumer that ignores `suspect` is computing a verdict the engine +has told it not to compute.** + +The draft's §8 rules are otherwise **CONFIRMED**: the ledger is counted-tier, +an absent key is a legacy file and not a zero, and an absent `counts.json` is a +real condition (this engine rebuilds it from disk on the next open — see +`initializeCountsFromDisk()`, which on this branch narrates that it is doing so +and why it cannot be backgrounded). + +**Open question 3 — the ledger drifted from the public tier by 3 nouns and 1 +verb.** Not ratified as expected drift. The counted-tier scalars are maintained +incrementally and can be left inflated by a delete whose decrement was skipped; +that is exactly what `allCountsSuspect` exists to record and what +`rebuildTypeCounts()` (the sanctioned recount, `src/storage/baseStorage.ts`) +corrects. A drift of 3 against 8,663 on a store with a long delete history is +consistent with that mechanism, not with a fresh defect — but it is a REAL +inaccuracy, not a definition, and the cure is a recount. **Finding for the +spec: do not describe counted-tier drift as normal; describe it as an +uncorrected incremental counter.** + +--- + +## §5.1 — the identity leg + +**CONFIRMED, every clause.** + +- Two shapes coexist in one store: `_fmt: 2` nested-bag, and the legacy flat + shape. `src/types/reservedFields.ts` — `METADATA_RECORD_FORMAT_KEY = '_fmt'`, + `NESTED_BAG_FORMAT`. +- **The stamp alone does not decide it.** `isNestedBagRecord()` requires the + stamp AND `typeof record.metadata === 'object'` AND non-null AND + `!Array.isArray(record.metadata)`. A reader that trusts the stamp alone + mis-splits a pre-law record carrying a user field named `metadata`. The draft + is right to make this a rule. +- Engine fields sit at top level in BOTH shapes, so extraction is + shape-independent: `RESERVED_ENTITY_FIELDS` / `RESERVED_RELATIONSHIP_FIELDS`. + The entity type is under `noun` (verbs: `verb`), never `type`. +- **No BigInt at this boundary.** Confirmed — every numeric in a metadata + record is a JSON number. + +**Divergence (timestamps): RATIFIED IN THE READER'S FAVOUR, and this engine +should follow.** The draft is correct that this engine substitutes `Date.now()` +where a stored timestamp is unusable. That is inventing data: a row whose +`updatedAt` could not be read is not a row that was updated now. The reader's +`null` is right. **This is a finding against this engine, filed here rather +than fixed on this branch** — changing hydration's timestamp fallback touches +every read path and belongs in its own change with its own pins, not in a +branch about opens and counters. Until then, a consumer must treat a timestamp +equal to read-time as suspect. + +--- + +## §5.2 — the vector leg + +**CONFIRMED, every clause**, including the three that are easiest to get wrong: + +- It is **JSON**, not an mmap-able buffer; raw blobs live in `_blobs/` behind + `getBinaryBlobPath()` — a different door entirely. +- **The five-way vector state** is this engine's own law: `absent-leg`, + `absent-field`, `empty` (deferred embed — `vector: []` written at `add()` + time), `zero-norm`, `real`. `isZeroNormVector()` (`src/utils/distance.ts`) is + the shared predicate, and `scanVectoredNounCount()` applies exactly it. A + zero-norm vector **is not a vector** — the 10.4.2 unvector work is precisely + this rule, canonical side included. +- **`connections` on disk is not index state.** Confirmed. Nothing may infer + that a store is indexed, or how, from that field or from the vestigial + `hnsw/` directories. + +**Open question 4 — vector legs carrying stale identity copies.** **CONFIRMED +NEVER AUTHORITATIVE.** Type, subtype and visibility are read from the identity +leg only; the copies in a vector leg are residue from an older writer. This +engine will not start reading them and any reader that does will disagree with +it on rows the two writers touched at different times. Whether Stage 2 drops +them is the accelerated engine's format decision, not this engine's — from +here, dropping them is safe. + +**The `float_roundtrip` finding is accepted and worth restating as a rule for +any future reader in any language:** the leg is JSON text, so every component +is a double, and a decimal has exactly one nearest `f64`. A parser one ULP off +builds a different index — different distances, different neighbours — and +makes any parity claim between two engines false. This engine gets it right +only because `JSON.parse` is correctly rounded by specification; nothing in +this repository would have caught the divergence, and the conformance suite's +whole-vector deep-equal is the right instrument. + +--- + +## §6 — verb endpoints + +**CONFIRMED, exactly as stated, and it is a defect of shape rather than of +data.** + +`saveVerb_internal()` (`src/storage/baseStorage.ts`) writes the whole +`HNSWVerb` — `{ id, vector, connections, verb, sourceId, targetId }` — to +`entities/verbs///vectors.json`, and `saveVerbMetadata()` writes the +engine scalars and user bag to `metadata.json` with **no endpoints**. `getVerb()` +reassembles the two. + +The consequences the draft draws are all correct, including the one that costs +the most: **a graph heal cannot be driven from identity records alone**, so +every edge's vector leg is read purely for its structure even though a verb's +`vector` is always `[]`. This engine confirms the measurement in principle — +every verb it has ever written carries its endpoints in the vector leg and none +in the identity leg. + +The door contract that returns `(id, sourceId, targetId, verb, subtype)` +regardless of which leg holds the bytes is the right shape and this engine +endorses it as the migration seam. **Commitment: when Stage 2 moves endpoints +onto the verb record, this engine's reader side must accept BOTH placements for +the life of contract 1** — a store written by either engine must be readable by +the other. + +--- + +## §7 — `_system/` + +**CONFIRMED**, with one clause now needing a footnote. + +- `_system/` mixes persisted state with live protocol and must never be + enumerated as data. `idx/` and `family-stamps/` are subtrees. +- `tx-log.jsonl` is mutated in place and must be byte-copied, not hard-linked, + by a snapshot. +- `/locks/` is live coordination — `_writer.lock` and the flush-request + protocol. Reading them as records is meaningless; writing or deleting them + interferes with a live writer. **Footnote: this branch adds one more file + there, `_writer.close`** — the clean-close record naming the lock generation + a writer released, consumed by the next claim. Same rule applies: it is + protocol, not data. A reader must ignore it, and must not treat its presence + or absence as a fact about the store's contents. +- **The encoding split is real and remains real:** everything in `_system/` + goes through the compressing writer and is `.gz` **except `counts.json`**, + which is plain JSON. This branch made that write ATOMIC but deliberately did + NOT make it `.gz` — the ledger is the one file an operator reads with `cat` + during an incident, and a reader implementation already handles the + exception. Try both spellings, as the draft says. + +--- + +## §8.1 — the zero-byte ledger window + +**ACCEPTED AS A DEFECT AND FIXED ON THIS BRANCH.** + +Cause 1 is confirmed exactly as written: `persistCounts()` used a bare +`fs.writeFile`, which truncates first, so the ledger was empty for the whole +write while every other object in the store was written atomically. It now goes +through the same temp+rename path (`writeFileAtomic`), so a reader sees the old +ledger or the new one and never neither. + +Cause 2 — "the final write is not awaited by the closing path" — **NOT +REPRODUCED at 10.4.3**, and the mechanism named does not exist here: the +filesystem adapter's count persist is write-through, not debounced +(`scheduleCountPersist()` sets the flag and immediately `await`s +`flushCounts()`), and `close()` awaits `flushCounts()` in its first flush +phase. The ~750 ms the draft measured is more likely the tail of an in-flight +write-through persist observed through the truncation window of cause 1, which +is now closed. **Finding: re-measure on 10.4.4 before keeping cause 2 in the +spec.** + +**Answer to open question 5:** yes, fixed; and yes, a reader should keep +classifying a zero-byte ledger as `TORN_RECORD` — the classification is right +on its own terms (a file that exists and holds nothing IS torn), it is the only +honest reading of a store damaged by an older release, and stores written by +every version before 10.4.4 can still carry one. + +Pinned here by `tests/integration/count-ledger-identity-record.test.ts` — +40 consecutive persists watched at 1 ms and never once unparseable. + +--- + +## §9 — the ordering law + +**CONFIRMED, including the two clauses most likely to be "simplified" by a +future implementer.** + +- Shard `00` → `ff` ascending, then id ascending in **UTF-16 code-unit order** — + this engine's `.sort()` with an `a.id < b.id` comparator, which is exactly + UTF-16 code-unit order, not UTF-8 byte order and not code-point order. They + diverge above the BMP. +- Cursor tokens: `cn1::` and `cv1::`, shard in DECIMAL + (it is the loop index, not the directory name), id last so an id containing + `:` survives the round trip. `src/storage/baseStorage.ts` — + `` return `cn1:${shard}:${id}` `` and `` return `cv1:${shard}:${id}` ``. +- A supplied-but-undecodable cursor must FAIL, never restart at offset 0. + Confirmed as a rule: silently restarting turns a `while (hasMore)` loop into + an unbounded one. +- The stability properties (a consistent prefix per shard, not a snapshot; + appends behind the cursor missed; deletes ahead take effect) are this + engine's own behaviour, confirmed. + +--- + +## §10 — read depth + +**CONFIRMED as a design, with one clause this engine wants on the record:** +*read depth bounds what corruption can be seen.* An ids-only enumeration opens +nothing and therefore reports a clean population over a store of undecodable +bodies. That is the honest consequence of reading exactly what was asked for. +A caller asking "is this store healthy?" from the cheap level is asking the +wrong door — health needs a decoding read. This engine's health reporting +follows the same rule and should say so as plainly. + +--- + +## §11 — cost + +**NOT RATIFIED — NOT THIS ENGINE'S TO RATIFY.** The comparative walls are +measurements of two implementations on one box; nothing in this repository can +confirm or refute them. Two observations that ARE this engine's: + +1. The structural explanation offered for the verb gap is consistent with this + engine's code: the noun walk hydrates in batches, the verb walk does not, + and a cursored resume re-lists and re-sorts the cursor's shard on every + page. That is a real asymmetry in `src/storage/baseStorage.ts` and it is + fair to name it. +2. The draft's own honesty about the applier holding 98% of the verb wall after + fan-out is the part worth keeping. **Nobody may read "8 workers" as "8×".** + +--- + +## §12 — Stage 1 + +**NOTED, NOT RATIFIED.** Which read path the accelerated engine's walks use is +its own decision. Two things this engine confirms because they are claims about +THIS code: + +- **The membership-predicate correction is right.** This engine's membership + is the IDENTITY RECORD: an id directory holding a vector leg and no identity + record is enumerated by neither `getNouns()` nor + `getNounIdsWithPagination()` and is absent from `getCanonicalCounts()`; an + identity-only row IS enumerated and IS counted. The draft's earlier statement + that the doors key on `vectors.json` was wrong and its correction is accepted. +- **Read-your-writes is confirmed.** There is no write-behind buffer between an + awaited `saveNoun()` / `saveNounMetadata()` and the bytes on disk; the leg is + `stat`-able before the returned promise resolves. **Commitment: this engine + will not introduce a write-behind buffer in the filesystem adapter without + declaring a flush contract alongside it** — the law "anything not yet awaited + is not yet canonical" is only useful if the converse holds. + +--- + +## Findings this ratification files back + +1. **§4 / open question 6 is out of date** — the directory-counting scan was + fixed in 10.4.2. Re-read the spec's §4 against 10.4.3 before Stage 2 builds + on it. The MEASUREMENTS stay valid; the code claim does not. +2. **§8.1 cause 2 is not reproducible at 10.4.3** — the count persist is + write-through and `close()` awaits it. Re-measure before republishing. +3. **§8's counted-tier drift (open question 3) should not be described as + expected** — it is an uncorrected incremental counter, and `repairIndex()` + is its cure. +4. **The timestamp divergence is a defect on THIS engine's side**, not a + difference of opinion: substituting `Date.now()` for an unreadable + timestamp invents data. Filed, not fixed on this branch. +5. **`locks/_writer.close` is new** — a reader must ignore it like every other + file under `locks/`. +6. **`counts.json` is now written atomically and stays plain JSON** — the + `_system/` encoding exception is deliberate and permanent for contract 1.