# The canonical layout — ratification Open Brainy's answer to the canonical layout specification written by the accelerated engine's reader team. It reads that document's rules against THIS repository's writer, at `10.4.3` plus the changes on `next/open-lazy-open-and-counts`, and for each one says **confirmed**, **corrected**, or **changed under you** — with the line of code that proves it. The specification was written against `10.4.1`. Three of its rules have moved since, and one of its open questions has already been answered by a fix that shipped in `10.4.2`. Those are called out first, because a reader implementation built to the draft would be wrong about them today. *Internal engineering document — no frontmatter, not published.* --- ## What moved since the draft was written | # | Draft says | Today | Where | |---|---|---|---| | A | `scanCanonicalEntities()` counts id DIRECTORIES, so a legacy ledger's derivation includes scars and orphans (§4, open question 6) | **CORRECTED IN 10.4.2.** The scan counts one entity per IDENTITY RECORD, sharing its predicate with the orphan pruner | `src/storage/adapters/fileSystemStorage.ts` `scanCanonicalEntities()` / `hasMetadataContentLeg()` | | B | An already-derived legacy denominator is persisted "once and never rescanned" (§4, §8) | **CORRECTED on this branch.** A ledger without the `identity-record` stamp is marked suspect at open AND re-derived honestly in the background, then persisted stamped | `scheduleCountLedgerDerivation()`, same file | | C | `counts.json` has a zero-byte window because `persistCounts()` uses a bare `fs.writeFile` (§8.1, open question 5) | **FIXED on this branch.** Written temp+rename like every other object | `persistCounts()`, same file | Everything else below stands as the draft states it, except where marked. --- ## §1 — the tree is a projection, not the source of truth **CONFIRMED.** The generation log under `_generations/` is canonical and the entity tree is a materialization of it; `_system/generation.json[.gz]` carries `{ generation, updatedAt }` for the projection. `src/db/generationStore.ts` owns the counter and its manifest; `readBrainFormat` / the entity-tree stamp are read at open in `src/brainy.ts`'s generation-store phase. Two walks minutes apart may legitimately disagree, and a caller that needs a point-in-time answer must pin the generation. Confirmed as a rule of this engine, not merely of the reader. --- ## §2 — paths and sharding **CONFIRMED, all three clauses.** - Two lowercase hex characters, 256 buckets, and an id that normalises to 32 hex characters buckets by its first byte: `src/storage/sharding.ts` `getShardId()` — `const normalized = id.toLowerCase().replace(/-/g, '')`, `if (/^[0-9a-f]{32}$/.test(normalized)) return normalized.substring(0, 2)`. - Every other id hashes: `hashToShardId(id)`, FNV-1a over **UTF-16 code units** (`id.charCodeAt(i)`), folded to the low byte. The UTF-16 detail is load-bearing above the BMP and is part of the on-disk contract — changing it relocates every application-id record. - `hnsw/` and `metadata/` under `entities//` are vestigial and the write path never populates them. Every scan in this repository excludes them with the same two-hex test (`/^[0-9a-f]{2}$/i`) — see `scanCanonicalEntities()` and `scanVectoredNounCount()`. **A reader that counts directory entries reports 258 shards where there are 256** is correct and worth keeping in the spec. --- ## §3 — encodings **CONFIRMED.** - The compressed spelling APPENDS: `const compressedPath = \`${fullPath}.gz\`` (`fileSystemStorage.ts`, the object read and write paths). Deriving it by replacing the extension finds a file that never exists. - Read order is `.gz` first, then plain — same call sites. - The three outcomes are distinct in this engine too: absence is `null`, an undecodable body raises `TornRecordError` (`src/storage/tornRecordError.ts`, `registerTornRecordEncounter`), and an IO fault is rethrown unreshaped. - The legacy exception the draft names — a torn `.gz` with a decodable plain twin returns the recovered plain object, loudly — is real and deliberate; the code comment at the read path states it in those terms. **Divergence (torn rows): RATIFIED AS PERMANENT for the current contract.** This engine's paginated walks skip-and-count a torn row so enumeration heals past it; the reader raises with a resume cursor. Both are defensible and they are not reconcilable by picking one: an enumeration that raises cannot serve a heal, and a heal that skips silently is how corruption becomes invisible. The reader's `resume_cursor` gives a caller the skip behaviour deliberately, which is the better shape. **This engine will not change its walks to raise** — its walks exist to heal — and the two enumerations are therefore allowed to report different populations over a DAMAGED store, never over a healthy one. Any consumer comparing the two must first establish the store is not damaged. --- ## §4 — the population law **CONFIRMED as the law. CORRECTED as a description of this engine's code.** The law is right and is this engine's own: *the identity record is the population; the id directory is not.* `pruneOrphanedEntities()` states it and `hasMetadataContentLeg()` is the single predicate both it and the count scan use, so the two agree by construction: ```ts private hasMetadataContentLeg(legs: string[]): boolean { return legs.some((f) => f.startsWith('metadata.json')) } ``` **The draft's finding that `scanCanonicalEntities()` counts directories is out of date.** It was true of `10.4.1`, which the draft was written against; it was fixed in `10.4.2` ("derive the canonical count ledger from identity records, stamp the derivation rule, and mark legacy-derived ledgers suspect at load"). The scan now `continue`s on any container with no metadata content leg. **But the draft's MEASUREMENTS remain valid, and the defect they name was only half-fixed until this branch.** A ledger DERIVED under the old rule was adopted at load with its wrong numbers and merely flagged `allCountsSuspect`, and nothing corrected it short of an operator running `repairIndex()`. That is why the same frozen archive still reads 14,231 or 14,081 against 14,056 identity records depending on which copy you open, and why a graph heal subtracted 72,679 walked rows from a 72,729 denominator and reported `remaining: 50` — the store's 50 verb scar directories. The heal was behaving correctly against a lying denominator. **On this branch the ledger heals itself**: a load that finds no `identity-record` stamp marks the scalars suspect, schedules an honest derivation in the BACKGROUND (never blocking the open — the same walk is part of why a 24,898-id store opened in silence), and persists the corrected scalars stamped. A derivation that raced a write refuses to stamp its number and leaves the ledger suspect, naming `repairIndex()` as the door that recounts under a barrier. Pinned in `tests/integration/count-ledger-identity-record.test.ts`, including two copies of one archive deriving the same number. **Answer to open question 6, in full:** 1. *Should the derivation require an identity leg?* **Yes, and it does** — since `10.4.2`, sharing the pruner's predicate. 2. *Should already-persisted denominators on affected stores be re-derived?* **Yes, and they now are, automatically, in the background.** An operator no longer has to know. 3. *Which scalar may a consumer subtract against?* See the next section — the answer is narrower than "the corrected one". --- ## §8 / open question 6 — WHICH SCALAR A CONSUMER MAY SUBTRACT AGAINST The specification asked this to be stated in the ratification. Stated: | scalar | what it is | safe to subtract an enumeration against? | |---|---|---| | `totalNounCount` / `totalVerbCount` | the **counted tiers only** (`isCountedVisibility` excludes `internal` and `system`) | **NO.** It is routinely far below the live population by design. A full-tier walk measured against it reads permanently over. | | `totalNounCountAll` / `totalVerbCountAll` **with `allCountsDerivedBy: 'identity-record'` and `allCountsSuspect: false`** | the all-tier identity-record population | **YES.** This is the only pair a coverage verdict may use. | | the same scalars **without the stamp, or with `allCountsSuspect: true`** | an unverified number: either derived under the old container rule, or made unprovable by a delete that could not read its record | **NO.** Read the flag, refuse the verdict, and say so. On this branch the engine is already correcting it behind you; wait for the correction rather than subtracting against the interim. | | `totalVectoredNounCount` | nouns holding a real, non-empty, non-zero-norm vector, derived by reading record CONTENT | **YES for the vector family only.** It never had the container defect. It shares `allCountsSuspect`, so the same flag rule applies. | `getCanonicalCounts()` (`src/storage/adapters/baseStorageAdapter.ts`) returns all of them with the suspect flag in one O(1) call, and the flag is the whole contract: **a consumer that ignores `suspect` is computing a verdict the engine has told it not to compute.** The draft's §8 rules are otherwise **CONFIRMED**: the ledger is counted-tier, an absent key is a legacy file and not a zero, and an absent `counts.json` is a real condition (this engine rebuilds it from disk on the next open — see `initializeCountsFromDisk()`, which on this branch narrates that it is doing so and why it cannot be backgrounded). **Open question 3 — the ledger drifted from the public tier by 3 nouns and 1 verb.** Not ratified as expected drift. The counted-tier scalars are maintained incrementally and can be left inflated by a delete whose decrement was skipped; that is exactly what `allCountsSuspect` exists to record and what `rebuildTypeCounts()` (the sanctioned recount, `src/storage/baseStorage.ts`) corrects. A drift of 3 against 8,663 on a store with a long delete history is consistent with that mechanism, not with a fresh defect — but it is a REAL inaccuracy, not a definition, and the cure is a recount. **Finding for the spec: do not describe counted-tier drift as normal; describe it as an uncorrected incremental counter.** --- ## §5.1 — the identity leg **CONFIRMED, every clause.** - Two shapes coexist in one store: `_fmt: 2` nested-bag, and the legacy flat shape. `src/types/reservedFields.ts` — `METADATA_RECORD_FORMAT_KEY = '_fmt'`, `NESTED_BAG_FORMAT`. - **The stamp alone does not decide it.** `isNestedBagRecord()` requires the stamp AND `typeof record.metadata === 'object'` AND non-null AND `!Array.isArray(record.metadata)`. A reader that trusts the stamp alone mis-splits a pre-law record carrying a user field named `metadata`. The draft is right to make this a rule. - Engine fields sit at top level in BOTH shapes, so extraction is shape-independent: `RESERVED_ENTITY_FIELDS` / `RESERVED_RELATIONSHIP_FIELDS`. The entity type is under `noun` (verbs: `verb`), never `type`. - **No BigInt at this boundary.** Confirmed — every numeric in a metadata record is a JSON number. **Divergence (timestamps): RATIFIED IN THE READER'S FAVOUR, and this engine should follow.** The draft is correct that this engine substitutes `Date.now()` where a stored timestamp is unusable. That is inventing data: a row whose `updatedAt` could not be read is not a row that was updated now. The reader's `null` is right. **This is a finding against this engine, filed here rather than fixed on this branch** — changing hydration's timestamp fallback touches every read path and belongs in its own change with its own pins, not in a branch about opens and counters. Until then, a consumer must treat a timestamp equal to read-time as suspect. --- ## §5.2 — the vector leg **CONFIRMED, every clause**, including the three that are easiest to get wrong: - It is **JSON**, not an mmap-able buffer; raw blobs live in `_blobs/` behind `getBinaryBlobPath()` — a different door entirely. - **The five-way vector state** is this engine's own law: `absent-leg`, `absent-field`, `empty` (deferred embed — `vector: []` written at `add()` time), `zero-norm`, `real`. `isZeroNormVector()` (`src/utils/distance.ts`) is the shared predicate, and `scanVectoredNounCount()` applies exactly it. A zero-norm vector **is not a vector** — the 10.4.2 unvector work is precisely this rule, canonical side included. - **`connections` on disk is not index state.** Confirmed. Nothing may infer that a store is indexed, or how, from that field or from the vestigial `hnsw/` directories. **Open question 4 — vector legs carrying stale identity copies.** **CONFIRMED NEVER AUTHORITATIVE.** Type, subtype and visibility are read from the identity leg only; the copies in a vector leg are residue from an older writer. This engine will not start reading them and any reader that does will disagree with it on rows the two writers touched at different times. Whether Stage 2 drops them is the accelerated engine's format decision, not this engine's — from here, dropping them is safe. **The `float_roundtrip` finding is accepted and worth restating as a rule for any future reader in any language:** the leg is JSON text, so every component is a double, and a decimal has exactly one nearest `f64`. A parser one ULP off builds a different index — different distances, different neighbours — and makes any parity claim between two engines false. This engine gets it right only because `JSON.parse` is correctly rounded by specification; nothing in this repository would have caught the divergence, and the conformance suite's whole-vector deep-equal is the right instrument. --- ## §6 — verb endpoints **CONFIRMED, exactly as stated, and it is a defect of shape rather than of data.** `saveVerb_internal()` (`src/storage/baseStorage.ts`) writes the whole `HNSWVerb` — `{ id, vector, connections, verb, sourceId, targetId }` — to `entities/verbs///vectors.json`, and `saveVerbMetadata()` writes the engine scalars and user bag to `metadata.json` with **no endpoints**. `getVerb()` reassembles the two. The consequences the draft draws are all correct, including the one that costs the most: **a graph heal cannot be driven from identity records alone**, so every edge's vector leg is read purely for its structure even though a verb's `vector` is always `[]`. This engine confirms the measurement in principle — every verb it has ever written carries its endpoints in the vector leg and none in the identity leg. The door contract that returns `(id, sourceId, targetId, verb, subtype)` regardless of which leg holds the bytes is the right shape and this engine endorses it as the migration seam. **Commitment: when Stage 2 moves endpoints onto the verb record, this engine's reader side must accept BOTH placements for the life of contract 1** — a store written by either engine must be readable by the other. --- ## §7 — `_system/` **CONFIRMED**, with one clause now needing a footnote. - `_system/` mixes persisted state with live protocol and must never be enumerated as data. `idx/` and `family-stamps/` are subtrees. - `tx-log.jsonl` is mutated in place and must be byte-copied, not hard-linked, by a snapshot. - `/locks/` is live coordination — `_writer.lock` and the flush-request protocol. Reading them as records is meaningless; writing or deleting them interferes with a live writer. **Footnote: this branch adds one more file there, `_writer.close`** — the clean-close record naming the lock generation a writer released, consumed by the next claim. Same rule applies: it is protocol, not data. A reader must ignore it, and must not treat its presence or absence as a fact about the store's contents. - **The encoding split is real and remains real:** everything in `_system/` goes through the compressing writer and is `.gz` **except `counts.json`**, which is plain JSON. This branch made that write ATOMIC but deliberately did NOT make it `.gz` — the ledger is the one file an operator reads with `cat` during an incident, and a reader implementation already handles the exception. Try both spellings, as the draft says. --- ## §8.1 — the zero-byte ledger window **ACCEPTED AS A DEFECT AND FIXED ON THIS BRANCH.** Cause 1 is confirmed exactly as written: `persistCounts()` used a bare `fs.writeFile`, which truncates first, so the ledger was empty for the whole write while every other object in the store was written atomically. It now goes through the same temp+rename path (`writeFileAtomic`), so a reader sees the old ledger or the new one and never neither. Cause 2 — "the final write is not awaited by the closing path" — **NOT REPRODUCED at 10.4.3**, and the mechanism named does not exist here: the filesystem adapter's count persist is write-through, not debounced (`scheduleCountPersist()` sets the flag and immediately `await`s `flushCounts()`), and `close()` awaits `flushCounts()` in its first flush phase. The ~750 ms the draft measured is more likely the tail of an in-flight write-through persist observed through the truncation window of cause 1, which is now closed. **Finding: re-measure on 10.4.4 before keeping cause 2 in the spec.** **Answer to open question 5:** yes, fixed; and yes, a reader should keep classifying a zero-byte ledger as `TORN_RECORD` — the classification is right on its own terms (a file that exists and holds nothing IS torn), it is the only honest reading of a store damaged by an older release, and stores written by every version before 10.4.4 can still carry one. Pinned here by `tests/integration/count-ledger-identity-record.test.ts` — 40 consecutive persists watched at 1 ms and never once unparseable. --- ## §9 — the ordering law **CONFIRMED, including the two clauses most likely to be "simplified" by a future implementer.** - Shard `00` → `ff` ascending, then id ascending in **UTF-16 code-unit order** — this engine's `.sort()` with an `a.id < b.id` comparator, which is exactly UTF-16 code-unit order, not UTF-8 byte order and not code-point order. They diverge above the BMP. - Cursor tokens: `cn1::` and `cv1::`, shard in DECIMAL (it is the loop index, not the directory name), id last so an id containing `:` survives the round trip. `src/storage/baseStorage.ts` — `` return `cn1:${shard}:${id}` `` and `` return `cv1:${shard}:${id}` ``. - A supplied-but-undecodable cursor must FAIL, never restart at offset 0. Confirmed as a rule: silently restarting turns a `while (hasMore)` loop into an unbounded one. - The stability properties (a consistent prefix per shard, not a snapshot; appends behind the cursor missed; deletes ahead take effect) are this engine's own behaviour, confirmed. --- ## §10 — read depth **CONFIRMED as a design, with one clause this engine wants on the record:** *read depth bounds what corruption can be seen.* An ids-only enumeration opens nothing and therefore reports a clean population over a store of undecodable bodies. That is the honest consequence of reading exactly what was asked for. A caller asking "is this store healthy?" from the cheap level is asking the wrong door — health needs a decoding read. This engine's health reporting follows the same rule and should say so as plainly. --- ## §11 — cost **NOT RATIFIED — NOT THIS ENGINE'S TO RATIFY.** The comparative walls are measurements of two implementations on one box; nothing in this repository can confirm or refute them. Two observations that ARE this engine's: 1. The structural explanation offered for the verb gap is consistent with this engine's code: the noun walk hydrates in batches, the verb walk does not, and a cursored resume re-lists and re-sorts the cursor's shard on every page. That is a real asymmetry in `src/storage/baseStorage.ts` and it is fair to name it. 2. The draft's own honesty about the applier holding 98% of the verb wall after fan-out is the part worth keeping. **Nobody may read "8 workers" as "8×".** --- ## §12 — Stage 1 **NOTED, NOT RATIFIED.** Which read path the accelerated engine's walks use is its own decision. Two things this engine confirms because they are claims about THIS code: - **The membership-predicate correction is right.** This engine's membership is the IDENTITY RECORD: an id directory holding a vector leg and no identity record is enumerated by neither `getNouns()` nor `getNounIdsWithPagination()` and is absent from `getCanonicalCounts()`; an identity-only row IS enumerated and IS counted. The draft's earlier statement that the doors key on `vectors.json` was wrong and its correction is accepted. - **Read-your-writes is confirmed.** There is no write-behind buffer between an awaited `saveNoun()` / `saveNounMetadata()` and the bytes on disk; the leg is `stat`-able before the returned promise resolves. **Commitment: this engine will not introduce a write-behind buffer in the filesystem adapter without declaring a flush contract alongside it** — the law "anything not yet awaited is not yet canonical" is only useful if the converse holds. --- ## Findings this ratification files back 1. **§4 / open question 6 is out of date** — the directory-counting scan was fixed in 10.4.2. Re-read the spec's §4 against 10.4.3 before Stage 2 builds on it. The MEASUREMENTS stay valid; the code claim does not. 2. **§8.1 cause 2 is not reproducible at 10.4.3** — the count persist is write-through and `close()` awaits it. Re-measure before republishing. 3. **§8's counted-tier drift (open question 3) should not be described as expected** — it is an uncorrected incremental counter, and `repairIndex()` is its cure. 4. **The timestamp divergence is a defect on THIS engine's side**, not a difference of opinion: substituting `Date.now()` for an unreadable timestamp invents data. Filed, not fixed on this branch. 5. **`locks/_writer.close` is new** — a reader must ignore it like every other file under `locks/`. 6. **`counts.json` is now written atomically and stays plain JSON** — the `_system/` encoding exception is deliberate and permanent for contract 1.