docs: ratify the canonical layout specification against this engine's writer

Rule by rule, with the line of code that proves each: confirmed, corrected, or
changed-under-you. Written against 10.4.3 plus this branch.

Three rules moved since the draft was written against 10.4.1, and a reader
built to the draft would be wrong about them: the identity-record count fix
landed in 10.4.2 (the draft's open question 6 was already half-answered), the
legacy denominator now heals itself in the background instead of lying for the
life of the store, and counts.json is written atomically.

Answers the six open questions, states which scalar a consumer may subtract an
enumeration against (only the all-tier pair carrying the identity-record stamp
with suspect false; the counted-tier scalars never), ratifies the torn-row and
timestamp divergences — the timestamp one AGAINST this engine, which invents a
Date.now() where a stored value is unreadable — and files six findings back,
including that the spec's §4 code claim and §8.1 cause 2 are both out of date.

Internal document: no frontmatter, not published.

docs/canonical-layout-ratification.md

@@ -0,0 +1,415 @@
+# The canonical layout — ratification
+
+Open Brainy's answer to the canonical layout specification written by the
+accelerated engine's reader team. It reads that document's rules against THIS
+repository's writer, at `10.4.3` plus the changes on
+`next/open-lazy-open-and-counts`, and for each one says **confirmed**,
+**corrected**, or **changed under you** — with the line of code that proves it.
+
+The specification was written against `10.4.1`. Three of its rules have moved
+since, and one of its open questions has already been answered by a fix that
+shipped in `10.4.2`. Those are called out first, because a reader
+implementation built to the draft would be wrong about them today.
+
+*Internal engineering document — no frontmatter, not published.*
+
+---
+
+## What moved since the draft was written
+
+| # | Draft says | Today | Where |
+|---|---|---|---|
+| A | `scanCanonicalEntities()` counts id DIRECTORIES, so a legacy ledger's derivation includes scars and orphans (§4, open question 6) | **CORRECTED IN 10.4.2.** The scan counts one entity per IDENTITY RECORD, sharing its predicate with the orphan pruner | `src/storage/adapters/fileSystemStorage.ts` `scanCanonicalEntities()` / `hasMetadataContentLeg()` |
+| B | An already-derived legacy denominator is persisted "once and never rescanned" (§4, §8) | **CORRECTED on this branch.** A ledger without the `identity-record` stamp is marked suspect at open AND re-derived honestly in the background, then persisted stamped | `scheduleCountLedgerDerivation()`, same file |
+| C | `counts.json` has a zero-byte window because `persistCounts()` uses a bare `fs.writeFile` (§8.1, open question 5) | **FIXED on this branch.** Written temp+rename like every other object | `persistCounts()`, same file |
+
+Everything else below stands as the draft states it, except where marked.
+
+---
+
+## §1 — the tree is a projection, not the source of truth
+
+**CONFIRMED.** The generation log under `_generations/` is canonical and the
+entity tree is a materialization of it; `_system/generation.json[.gz]` carries
+`{ generation, updatedAt }` for the projection. `src/db/generationStore.ts`
+owns the counter and its manifest; `readBrainFormat` / the entity-tree stamp
+are read at open in `src/brainy.ts`'s generation-store phase.
+
+Two walks minutes apart may legitimately disagree, and a caller that needs a
+point-in-time answer must pin the generation. Confirmed as a rule of this
+engine, not merely of the reader.
+
+---
+
+## §2 — paths and sharding
+
+**CONFIRMED, all three clauses.**
+
+- Two lowercase hex characters, 256 buckets, and an id that normalises to 32
+  hex characters buckets by its first byte: `src/storage/sharding.ts`
+  `getShardId()` — `const normalized = id.toLowerCase().replace(/-/g, '')`,
+  `if (/^[0-9a-f]{32}$/.test(normalized)) return normalized.substring(0, 2)`.
+- Every other id hashes: `hashToShardId(id)`, FNV-1a over **UTF-16 code
+  units** (`id.charCodeAt(i)`), folded to the low byte. The UTF-16 detail is
+  load-bearing above the BMP and is part of the on-disk contract — changing it
+  relocates every application-id record.
+- `hnsw/` and `metadata/` under `entities/<kind>/` are vestigial and the write
+  path never populates them. Every scan in this repository excludes them with
+  the same two-hex test (`/^[0-9a-f]{2}$/i`) — see `scanCanonicalEntities()`
+  and `scanVectoredNounCount()`. **A reader that counts directory entries
+  reports 258 shards where there are 256** is correct and worth keeping in the
+  spec.
+
+---
+
+## §3 — encodings
+
+**CONFIRMED.**
+
+- The compressed spelling APPENDS: `const compressedPath = \`${fullPath}.gz\``
+  (`fileSystemStorage.ts`, the object read and write paths). Deriving it by
+  replacing the extension finds a file that never exists.
+- Read order is `.gz` first, then plain — same call sites.
+- The three outcomes are distinct in this engine too: absence is `null`, an
+  undecodable body raises `TornRecordError` (`src/storage/tornRecordError.ts`,
+  `registerTornRecordEncounter`), and an IO fault is rethrown unreshaped.
+- The legacy exception the draft names — a torn `.gz` with a decodable plain
+  twin returns the recovered plain object, loudly — is real and deliberate; the
+  code comment at the read path states it in those terms.
+
+**Divergence (torn rows): RATIFIED AS PERMANENT for the current contract.**
+This engine's paginated walks skip-and-count a torn row so enumeration heals
+past it; the reader raises with a resume cursor. Both are defensible and they
+are not reconcilable by picking one: an enumeration that raises cannot serve a
+heal, and a heal that skips silently is how corruption becomes invisible. The
+reader's `resume_cursor` gives a caller the skip behaviour deliberately, which
+is the better shape. **This engine will not change its walks to raise** — its
+walks exist to heal — and the two enumerations are therefore allowed to report
+different populations over a DAMAGED store, never over a healthy one. Any
+consumer comparing the two must first establish the store is not damaged.
+
+---
+
+## §4 — the population law
+
+**CONFIRMED as the law. CORRECTED as a description of this engine's code.**
+
+The law is right and is this engine's own: *the identity record is the
+population; the id directory is not.* `pruneOrphanedEntities()` states it and
+`hasMetadataContentLeg()` is the single predicate both it and the count scan
+use, so the two agree by construction:
+
+```ts
+private hasMetadataContentLeg(legs: string[]): boolean {
+  return legs.some((f) => f.startsWith('metadata.json'))
+}
+```
+
+**The draft's finding that `scanCanonicalEntities()` counts directories is out
+of date.** It was true of `10.4.1`, which the draft was written against; it was
+fixed in `10.4.2` ("derive the canonical count ledger from identity records,
+stamp the derivation rule, and mark legacy-derived ledgers suspect at load").
+The scan now `continue`s on any container with no metadata content leg.
+
+**But the draft's MEASUREMENTS remain valid, and the defect they name was only
+half-fixed until this branch.** A ledger DERIVED under the old rule was
+adopted at load with its wrong numbers and merely flagged `allCountsSuspect`,
+and nothing corrected it short of an operator running `repairIndex()`. That is
+why the same frozen archive still reads 14,231 or 14,081 against 14,056
+identity records depending on which copy you open, and why a graph heal
+subtracted 72,679 walked rows from a 72,729 denominator and reported
+`remaining: 50` — the store's 50 verb scar directories. The heal was behaving
+correctly against a lying denominator.
+
+**On this branch the ledger heals itself**: a load that finds no
+`identity-record` stamp marks the scalars suspect, schedules an honest
+derivation in the BACKGROUND (never blocking the open — the same walk is part
+of why a 24,898-id store opened in silence), and persists the corrected
+scalars stamped. A derivation that raced a write refuses to stamp its number
+and leaves the ledger suspect, naming `repairIndex()` as the door that
+recounts under a barrier. Pinned in
+`tests/integration/count-ledger-identity-record.test.ts`, including two copies
+of one archive deriving the same number.
+
+**Answer to open question 6, in full:**
+
+1. *Should the derivation require an identity leg?* **Yes, and it does** — since
+   `10.4.2`, sharing the pruner's predicate.
+2. *Should already-persisted denominators on affected stores be re-derived?*
+   **Yes, and they now are, automatically, in the background.** An operator no
+   longer has to know.
+3. *Which scalar may a consumer subtract against?* See the next section — the
+   answer is narrower than "the corrected one".
+
+---
+
+## §8 / open question 6 — WHICH SCALAR A CONSUMER MAY SUBTRACT AGAINST
+
+The specification asked this to be stated in the ratification. Stated:
+
+| scalar | what it is | safe to subtract an enumeration against? |
+|---|---|---|
+| `totalNounCount` / `totalVerbCount` | the **counted tiers only** (`isCountedVisibility` excludes `internal` and `system`) | **NO.** It is routinely far below the live population by design. A full-tier walk measured against it reads permanently over. |
+| `totalNounCountAll` / `totalVerbCountAll` **with `allCountsDerivedBy: 'identity-record'` and `allCountsSuspect: false`** | the all-tier identity-record population | **YES.** This is the only pair a coverage verdict may use. |
+| the same scalars **without the stamp, or with `allCountsSuspect: true`** | an unverified number: either derived under the old container rule, or made unprovable by a delete that could not read its record | **NO.** Read the flag, refuse the verdict, and say so. On this branch the engine is already correcting it behind you; wait for the correction rather than subtracting against the interim. |
+| `totalVectoredNounCount` | nouns holding a real, non-empty, non-zero-norm vector, derived by reading record CONTENT | **YES for the vector family only.** It never had the container defect. It shares `allCountsSuspect`, so the same flag rule applies. |
+
+`getCanonicalCounts()` (`src/storage/adapters/baseStorageAdapter.ts`) returns
+all of them with the suspect flag in one O(1) call, and the flag is the whole
+contract: **a consumer that ignores `suspect` is computing a verdict the engine
+has told it not to compute.**
+
+The draft's §8 rules are otherwise **CONFIRMED**: the ledger is counted-tier,
+an absent key is a legacy file and not a zero, and an absent `counts.json` is a
+real condition (this engine rebuilds it from disk on the next open — see
+`initializeCountsFromDisk()`, which on this branch narrates that it is doing so
+and why it cannot be backgrounded).
+
+**Open question 3 — the ledger drifted from the public tier by 3 nouns and 1
+verb.** Not ratified as expected drift. The counted-tier scalars are maintained
+incrementally and can be left inflated by a delete whose decrement was skipped;
+that is exactly what `allCountsSuspect` exists to record and what
+`rebuildTypeCounts()` (the sanctioned recount, `src/storage/baseStorage.ts`)
+corrects. A drift of 3 against 8,663 on a store with a long delete history is
+consistent with that mechanism, not with a fresh defect — but it is a REAL
+inaccuracy, not a definition, and the cure is a recount. **Finding for the
+spec: do not describe counted-tier drift as normal; describe it as an
+uncorrected incremental counter.**
+
+---
+
+## §5.1 — the identity leg
+
+**CONFIRMED, every clause.**
+
+- Two shapes coexist in one store: `_fmt: 2` nested-bag, and the legacy flat
+  shape. `src/types/reservedFields.ts` — `METADATA_RECORD_FORMAT_KEY = '_fmt'`,
+  `NESTED_BAG_FORMAT`.
+- **The stamp alone does not decide it.** `isNestedBagRecord()` requires the
+  stamp AND `typeof record.metadata === 'object'` AND non-null AND
+  `!Array.isArray(record.metadata)`. A reader that trusts the stamp alone
+  mis-splits a pre-law record carrying a user field named `metadata`. The draft
+  is right to make this a rule.
+- Engine fields sit at top level in BOTH shapes, so extraction is
+  shape-independent: `RESERVED_ENTITY_FIELDS` / `RESERVED_RELATIONSHIP_FIELDS`.
+  The entity type is under `noun` (verbs: `verb`), never `type`.
+- **No BigInt at this boundary.** Confirmed — every numeric in a metadata
+  record is a JSON number.
+
+**Divergence (timestamps): RATIFIED IN THE READER'S FAVOUR, and this engine
+should follow.** The draft is correct that this engine substitutes `Date.now()`
+where a stored timestamp is unusable. That is inventing data: a row whose
+`updatedAt` could not be read is not a row that was updated now. The reader's
+`null` is right. **This is a finding against this engine, filed here rather
+than fixed on this branch** — changing hydration's timestamp fallback touches
+every read path and belongs in its own change with its own pins, not in a
+branch about opens and counters. Until then, a consumer must treat a timestamp
+equal to read-time as suspect.
+
+---
+
+## §5.2 — the vector leg
+
+**CONFIRMED, every clause**, including the three that are easiest to get wrong:
+
+- It is **JSON**, not an mmap-able buffer; raw blobs live in `_blobs/` behind
+  `getBinaryBlobPath()` — a different door entirely.
+- **The five-way vector state** is this engine's own law: `absent-leg`,
+  `absent-field`, `empty` (deferred embed — `vector: []` written at `add()`
+  time), `zero-norm`, `real`. `isZeroNormVector()` (`src/utils/distance.ts`) is
+  the shared predicate, and `scanVectoredNounCount()` applies exactly it. A
+  zero-norm vector **is not a vector** — the 10.4.2 unvector work is precisely
+  this rule, canonical side included.
+- **`connections` on disk is not index state.** Confirmed. Nothing may infer
+  that a store is indexed, or how, from that field or from the vestigial
+  `hnsw/` directories.
+
+**Open question 4 — vector legs carrying stale identity copies.** **CONFIRMED
+NEVER AUTHORITATIVE.** Type, subtype and visibility are read from the identity
+leg only; the copies in a vector leg are residue from an older writer. This
+engine will not start reading them and any reader that does will disagree with
+it on rows the two writers touched at different times. Whether Stage 2 drops
+them is the accelerated engine's format decision, not this engine's — from
+here, dropping them is safe.
+
+**The `float_roundtrip` finding is accepted and worth restating as a rule for
+any future reader in any language:** the leg is JSON text, so every component
+is a double, and a decimal has exactly one nearest `f64`. A parser one ULP off
+builds a different index — different distances, different neighbours — and
+makes any parity claim between two engines false. This engine gets it right
+only because `JSON.parse` is correctly rounded by specification; nothing in
+this repository would have caught the divergence, and the conformance suite's
+whole-vector deep-equal is the right instrument.
+
+---
+
+## §6 — verb endpoints
+
+**CONFIRMED, exactly as stated, and it is a defect of shape rather than of
+data.**
+
+`saveVerb_internal()` (`src/storage/baseStorage.ts`) writes the whole
+`HNSWVerb` — `{ id, vector, connections, verb, sourceId, targetId }` — to
+`entities/verbs/<shard>/<id>/vectors.json`, and `saveVerbMetadata()` writes the
+engine scalars and user bag to `metadata.json` with **no endpoints**. `getVerb()`
+reassembles the two.
+
+The consequences the draft draws are all correct, including the one that costs
+the most: **a graph heal cannot be driven from identity records alone**, so
+every edge's vector leg is read purely for its structure even though a verb's
+`vector` is always `[]`. This engine confirms the measurement in principle —
+every verb it has ever written carries its endpoints in the vector leg and none
+in the identity leg.
+
+The door contract that returns `(id, sourceId, targetId, verb, subtype)`
+regardless of which leg holds the bytes is the right shape and this engine
+endorses it as the migration seam. **Commitment: when Stage 2 moves endpoints
+onto the verb record, this engine's reader side must accept BOTH placements for
+the life of contract 1** — a store written by either engine must be readable by
+the other.
+
+---
+
+## §7 — `_system/`
+
+**CONFIRMED**, with one clause now needing a footnote.
+
+- `_system/` mixes persisted state with live protocol and must never be
+  enumerated as data. `idx/` and `family-stamps/` are subtrees.
+- `tx-log.jsonl` is mutated in place and must be byte-copied, not hard-linked,
+  by a snapshot.
+- `<root>/locks/` is live coordination — `_writer.lock` and the flush-request
+  protocol. Reading them as records is meaningless; writing or deleting them
+  interferes with a live writer. **Footnote: this branch adds one more file
+  there, `_writer.close`** — the clean-close record naming the lock generation
+  a writer released, consumed by the next claim. Same rule applies: it is
+  protocol, not data. A reader must ignore it, and must not treat its presence
+  or absence as a fact about the store's contents.
+- **The encoding split is real and remains real:** everything in `_system/`
+  goes through the compressing writer and is `.gz` **except `counts.json`**,
+  which is plain JSON. This branch made that write ATOMIC but deliberately did
+  NOT make it `.gz` — the ledger is the one file an operator reads with `cat`
+  during an incident, and a reader implementation already handles the
+  exception. Try both spellings, as the draft says.
+
+---
+
+## §8.1 — the zero-byte ledger window
+
+**ACCEPTED AS A DEFECT AND FIXED ON THIS BRANCH.**
+
+Cause 1 is confirmed exactly as written: `persistCounts()` used a bare
+`fs.writeFile`, which truncates first, so the ledger was empty for the whole
+write while every other object in the store was written atomically. It now goes
+through the same temp+rename path (`writeFileAtomic`), so a reader sees the old
+ledger or the new one and never neither.
+
+Cause 2 — "the final write is not awaited by the closing path" — **NOT
+REPRODUCED at 10.4.3**, and the mechanism named does not exist here: the
+filesystem adapter's count persist is write-through, not debounced
+(`scheduleCountPersist()` sets the flag and immediately `await`s
+`flushCounts()`), and `close()` awaits `flushCounts()` in its first flush
+phase. The ~750 ms the draft measured is more likely the tail of an in-flight
+write-through persist observed through the truncation window of cause 1, which
+is now closed. **Finding: re-measure on 10.4.4 before keeping cause 2 in the
+spec.**
+
+**Answer to open question 5:** yes, fixed; and yes, a reader should keep
+classifying a zero-byte ledger as `TORN_RECORD` — the classification is right
+on its own terms (a file that exists and holds nothing IS torn), it is the only
+honest reading of a store damaged by an older release, and stores written by
+every version before 10.4.4 can still carry one.
+
+Pinned here by `tests/integration/count-ledger-identity-record.test.ts` —
+40 consecutive persists watched at 1 ms and never once unparseable.
+
+---
+
+## §9 — the ordering law
+
+**CONFIRMED, including the two clauses most likely to be "simplified" by a
+future implementer.**
+
+- Shard `00` → `ff` ascending, then id ascending in **UTF-16 code-unit order** —
+  this engine's `.sort()` with an `a.id < b.id` comparator, which is exactly
+  UTF-16 code-unit order, not UTF-8 byte order and not code-point order. They
+  diverge above the BMP.
+- Cursor tokens: `cn1:<shard>:<id>` and `cv1:<shard>:<id>`, shard in DECIMAL
+  (it is the loop index, not the directory name), id last so an id containing
+  `:` survives the round trip. `src/storage/baseStorage.ts` —
+  `` return `cn1:${shard}:${id}` `` and `` return `cv1:${shard}:${id}` ``.
+- A supplied-but-undecodable cursor must FAIL, never restart at offset 0.
+  Confirmed as a rule: silently restarting turns a `while (hasMore)` loop into
+  an unbounded one.
+- The stability properties (a consistent prefix per shard, not a snapshot;
+  appends behind the cursor missed; deletes ahead take effect) are this
+  engine's own behaviour, confirmed.
+
+---
+
+## §10 — read depth
+
+**CONFIRMED as a design, with one clause this engine wants on the record:**
+*read depth bounds what corruption can be seen.* An ids-only enumeration opens
+nothing and therefore reports a clean population over a store of undecodable
+bodies. That is the honest consequence of reading exactly what was asked for.
+A caller asking "is this store healthy?" from the cheap level is asking the
+wrong door — health needs a decoding read. This engine's health reporting
+follows the same rule and should say so as plainly.
+
+---
+
+## §11 — cost
+
+**NOT RATIFIED — NOT THIS ENGINE'S TO RATIFY.** The comparative walls are
+measurements of two implementations on one box; nothing in this repository can
+confirm or refute them. Two observations that ARE this engine's:
+
+1. The structural explanation offered for the verb gap is consistent with this
+   engine's code: the noun walk hydrates in batches, the verb walk does not,
+   and a cursored resume re-lists and re-sorts the cursor's shard on every
+   page. That is a real asymmetry in `src/storage/baseStorage.ts` and it is
+   fair to name it.
+2. The draft's own honesty about the applier holding 98% of the verb wall after
+   fan-out is the part worth keeping. **Nobody may read "8 workers" as "8×".**
+
+---
+
+## §12 — Stage 1
+
+**NOTED, NOT RATIFIED.** Which read path the accelerated engine's walks use is
+its own decision. Two things this engine confirms because they are claims about
+THIS code:
+
+- **The membership-predicate correction is right.** This engine's membership
+  is the IDENTITY RECORD: an id directory holding a vector leg and no identity
+  record is enumerated by neither `getNouns()` nor
+  `getNounIdsWithPagination()` and is absent from `getCanonicalCounts()`; an
+  identity-only row IS enumerated and IS counted. The draft's earlier statement
+  that the doors key on `vectors.json` was wrong and its correction is accepted.
+- **Read-your-writes is confirmed.** There is no write-behind buffer between an
+  awaited `saveNoun()` / `saveNounMetadata()` and the bytes on disk; the leg is
+  `stat`-able before the returned promise resolves. **Commitment: this engine
+  will not introduce a write-behind buffer in the filesystem adapter without
+  declaring a flush contract alongside it** — the law "anything not yet awaited
+  is not yet canonical" is only useful if the converse holds.
+
+---
+
+## Findings this ratification files back
+
+1. **§4 / open question 6 is out of date** — the directory-counting scan was
+   fixed in 10.4.2. Re-read the spec's §4 against 10.4.3 before Stage 2 builds
+   on it. The MEASUREMENTS stay valid; the code claim does not.
+2. **§8.1 cause 2 is not reproducible at 10.4.3** — the count persist is
+   write-through and `close()` awaits it. Re-measure before republishing.
+3. **§8's counted-tier drift (open question 3) should not be described as
+   expected** — it is an uncorrected incremental counter, and `repairIndex()`
+   is its cure.
+4. **The timestamp divergence is a defect on THIS engine's side**, not a
+   difference of opinion: substituting `Date.now()` for an unreadable
+   timestamp invents data. Filed, not fixed on this branch.
+5. **`locks/_writer.close` is new** — a reader must ignore it like every other
+   file under `locks/`.
+6. **`counts.json` is now written atomically and stays plain JSON** — the
+   `_system/` encoding exception is deliberate and permanent for contract 1.