open-brainy/docs/canonical-layout-ratification.md

# The canonical layout — ratification

Open Brainy's answer to the canonical layout specification written by the
accelerated engine's reader team. It reads that document's rules against THIS
repository's writer, at `10.4.3` plus the changes on
`next/open-lazy-open-and-counts`, and for each one says **confirmed**,
**corrected**, or **changed under you** — with the line of code that proves it.

The specification was written against `10.4.1`. Three of its rules have moved
since, and one of its open questions has already been answered by a fix that
shipped in `10.4.2`. Those are called out first, because a reader
implementation built to the draft would be wrong about them today.

*Internal engineering document — no frontmatter, not published.*

---

## What moved since the draft was written

| # | Draft says | Today | Where |
|---|---|---|---|
| A | `scanCanonicalEntities()` counts id DIRECTORIES, so a legacy ledger's derivation includes scars and orphans (§4, open question 6) | **CORRECTED IN 10.4.2.** The scan counts one entity per IDENTITY RECORD, sharing its predicate with the orphan pruner | `src/storage/adapters/fileSystemStorage.ts` `scanCanonicalEntities()` / `hasMetadataContentLeg()` |
| B | An already-derived legacy denominator is persisted "once and never rescanned" (§4, §8) | **CORRECTED on this branch.** A ledger without the `identity-record` stamp is marked suspect at open AND re-derived honestly in the background, then persisted stamped | `scheduleCountLedgerDerivation()`, same file |
| C | `counts.json` has a zero-byte window because `persistCounts()` uses a bare `fs.writeFile` (§8.1, open question 5) | **FIXED on this branch.** Written temp+rename like every other object | `persistCounts()`, same file |

Everything else below stands as the draft states it, except where marked.

---

## §1 — the tree is a projection, not the source of truth

**CONFIRMED.** The generation log under `_generations/` is canonical and the
entity tree is a materialization of it; `_system/generation.json[.gz]` carries
`{ generation, updatedAt }` for the projection. `src/db/generationStore.ts`
owns the counter and its manifest; `readBrainFormat` / the entity-tree stamp
are read at open in `src/brainy.ts`'s generation-store phase.

Two walks minutes apart may legitimately disagree, and a caller that needs a
point-in-time answer must pin the generation. Confirmed as a rule of this
engine, not merely of the reader.

---

## §2 — paths and sharding

**CONFIRMED, all three clauses.**

- Two lowercase hex characters, 256 buckets, and an id that normalises to 32
  hex characters buckets by its first byte: `src/storage/sharding.ts`
  `getShardId()` — `const normalized = id.toLowerCase().replace(/-/g, '')`,
  `if (/^[0-9a-f]{32}$/.test(normalized)) return normalized.substring(0, 2)`.
- Every other id hashes: `hashToShardId(id)`, FNV-1a over **UTF-16 code
  units** (`id.charCodeAt(i)`), folded to the low byte. The UTF-16 detail is
  load-bearing above the BMP and is part of the on-disk contract — changing it
  relocates every application-id record.
- `hnsw/` and `metadata/` under `entities/<kind>/` are vestigial and the write
  path never populates them. Every scan in this repository excludes them with
  the same two-hex test (`/^[0-9a-f]{2}$/i`) — see `scanCanonicalEntities()`
  and `scanVectoredNounCount()`. **A reader that counts directory entries
  reports 258 shards where there are 256** is correct and worth keeping in the
  spec.

---

## §3 — encodings

**CONFIRMED.**

- The compressed spelling APPENDS: `const compressedPath = \`${fullPath}.gz\``
  (`fileSystemStorage.ts`, the object read and write paths). Deriving it by
  replacing the extension finds a file that never exists.
- Read order is `.gz` first, then plain — same call sites.
- The three outcomes are distinct in this engine too: absence is `null`, an
  undecodable body raises `TornRecordError` (`src/storage/tornRecordError.ts`,
  `registerTornRecordEncounter`), and an IO fault is rethrown unreshaped.
- The legacy exception the draft names — a torn `.gz` with a decodable plain
  twin returns the recovered plain object, loudly — is real and deliberate; the
  code comment at the read path states it in those terms.

**Divergence (torn rows): RATIFIED AS PERMANENT for the current contract.**
This engine's paginated walks skip-and-count a torn row so enumeration heals
past it; the reader raises with a resume cursor. Both are defensible and they
are not reconcilable by picking one: an enumeration that raises cannot serve a
heal, and a heal that skips silently is how corruption becomes invisible. The
reader's `resume_cursor` gives a caller the skip behaviour deliberately, which
is the better shape. **This engine will not change its walks to raise** — its
walks exist to heal — and the two enumerations are therefore allowed to report
different populations over a DAMAGED store, never over a healthy one. Any
consumer comparing the two must first establish the store is not damaged.

---

## §4 — the population law

**CONFIRMED as the law. CORRECTED as a description of this engine's code.**

The law is right and is this engine's own: *the identity record is the
population; the id directory is not.* `pruneOrphanedEntities()` states it and
`hasMetadataContentLeg()` is the single predicate both it and the count scan
use, so the two agree by construction:

```ts
private hasMetadataContentLeg(legs: string[]): boolean {
  return legs.some((f) => f.startsWith('metadata.json'))
}
```

**The draft's finding that `scanCanonicalEntities()` counts directories is out
of date.** It was true of `10.4.1`, which the draft was written against; it was
fixed in `10.4.2` ("derive the canonical count ledger from identity records,
stamp the derivation rule, and mark legacy-derived ledgers suspect at load").
The scan now `continue`s on any container with no metadata content leg.

**But the draft's MEASUREMENTS remain valid, and the defect they name was only
half-fixed until this branch.** A ledger DERIVED under the old rule was
adopted at load with its wrong numbers and merely flagged `allCountsSuspect`,
and nothing corrected it short of an operator running `repairIndex()`. That is
why the same frozen archive still reads 14,231 or 14,081 against 14,056
identity records depending on which copy you open, and why a graph heal
subtracted 72,679 walked rows from a 72,729 denominator and reported
`remaining: 50` — the store's 50 verb scar directories. The heal was behaving
correctly against a lying denominator.

**On this branch the ledger heals itself**: a load that finds no
`identity-record` stamp marks the scalars suspect, schedules an honest
derivation in the BACKGROUND (never blocking the open — the same walk is part
of why a 24,898-id store opened in silence), and persists the corrected
scalars stamped. A derivation that raced a write refuses to stamp its number
and leaves the ledger suspect, naming `repairIndex()` as the door that
recounts under a barrier. Pinned in
`tests/integration/count-ledger-identity-record.test.ts`, including two copies
of one archive deriving the same number.

**Answer to open question 6, in full:**

1. *Should the derivation require an identity leg?* **Yes, and it does** — since
   `10.4.2`, sharing the pruner's predicate.
2. *Should already-persisted denominators on affected stores be re-derived?*
   **Yes, and they now are, automatically, in the background.** An operator no
   longer has to know.
3. *Which scalar may a consumer subtract against?* See the next section — the
   answer is narrower than "the corrected one".

---

## §8 / open question 6 — WHICH SCALAR A CONSUMER MAY SUBTRACT AGAINST

The specification asked this to be stated in the ratification. Stated:

| scalar | what it is | safe to subtract an enumeration against? |
|---|---|---|
| `totalNounCount` / `totalVerbCount` | the **counted tiers only** (`isCountedVisibility` excludes `internal` and `system`) | **NO.** It is routinely far below the live population by design. A full-tier walk measured against it reads permanently over. |
| `totalNounCountAll` / `totalVerbCountAll` **with `allCountsDerivedBy: 'identity-record'` and `allCountsSuspect: false`** | the all-tier identity-record population | **YES.** This is the only pair a coverage verdict may use. |
| the same scalars **without the stamp, or with `allCountsSuspect: true`** | an unverified number: either derived under the old container rule, or made unprovable by a delete that could not read its record | **NO.** Read the flag, refuse the verdict, and say so. On this branch the engine is already correcting it behind you; wait for the correction rather than subtracting against the interim. |
| `totalVectoredNounCount` | nouns holding a real, non-empty, non-zero-norm vector, derived by reading record CONTENT | **YES for the vector family only.** It never had the container defect. It shares `allCountsSuspect`, so the same flag rule applies. |

`getCanonicalCounts()` (`src/storage/adapters/baseStorageAdapter.ts`) returns
all of them with the suspect flag in one O(1) call, and the flag is the whole
contract: **a consumer that ignores `suspect` is computing a verdict the engine
has told it not to compute.**

The draft's §8 rules are otherwise **CONFIRMED**: the ledger is counted-tier,
an absent key is a legacy file and not a zero, and an absent `counts.json` is a
real condition (this engine rebuilds it from disk on the next open — see
`initializeCountsFromDisk()`, which on this branch narrates that it is doing so
and why it cannot be backgrounded).

**Open question 3 — the ledger drifted from the public tier by 3 nouns and 1
verb.** Not ratified as expected drift. The counted-tier scalars are maintained
incrementally and can be left inflated by a delete whose decrement was skipped;
that is exactly what `allCountsSuspect` exists to record and what
`rebuildTypeCounts()` (the sanctioned recount, `src/storage/baseStorage.ts`)
corrects. A drift of 3 against 8,663 on a store with a long delete history is
consistent with that mechanism, not with a fresh defect — but it is a REAL
inaccuracy, not a definition, and the cure is a recount. **Finding for the
spec: do not describe counted-tier drift as normal; describe it as an
uncorrected incremental counter.**

---

## §5.1 — the identity leg

**CONFIRMED, every clause.**

- Two shapes coexist in one store: `_fmt: 2` nested-bag, and the legacy flat
  shape. `src/types/reservedFields.ts` — `METADATA_RECORD_FORMAT_KEY = '_fmt'`,
  `NESTED_BAG_FORMAT`.
- **The stamp alone does not decide it.** `isNestedBagRecord()` requires the
  stamp AND `typeof record.metadata === 'object'` AND non-null AND
  `!Array.isArray(record.metadata)`. A reader that trusts the stamp alone
  mis-splits a pre-law record carrying a user field named `metadata`. The draft
  is right to make this a rule.
- Engine fields sit at top level in BOTH shapes, so extraction is
  shape-independent: `RESERVED_ENTITY_FIELDS` / `RESERVED_RELATIONSHIP_FIELDS`.
  The entity type is under `noun` (verbs: `verb`), never `type`.
- **No BigInt at this boundary.** Confirmed — every numeric in a metadata
  record is a JSON number.

**Divergence (timestamps): RATIFIED IN THE READER'S FAVOUR, and this engine
should follow.** The draft is correct that this engine substitutes `Date.now()`
where a stored timestamp is unusable. That is inventing data: a row whose
`updatedAt` could not be read is not a row that was updated now. The reader's
`null` is right. **This is a finding against this engine, filed here rather
than fixed on this branch** — changing hydration's timestamp fallback touches
every read path and belongs in its own change with its own pins, not in a
branch about opens and counters. Until then, a consumer must treat a timestamp
equal to read-time as suspect.

---

## §5.2 — the vector leg

**CONFIRMED, every clause**, including the three that are easiest to get wrong:

- It is **JSON**, not an mmap-able buffer; raw blobs live in `_blobs/` behind
  `getBinaryBlobPath()` — a different door entirely.
- **The five-way vector state** is this engine's own law: `absent-leg`,
  `absent-field`, `empty` (deferred embed — `vector: []` written at `add()`
  time), `zero-norm`, `real`. `isZeroNormVector()` (`src/utils/distance.ts`) is
  the shared predicate, and `scanVectoredNounCount()` applies exactly it. A
  zero-norm vector **is not a vector** — the 10.4.2 unvector work is precisely
  this rule, canonical side included.
- **`connections` on disk is not index state.** Confirmed. Nothing may infer
  that a store is indexed, or how, from that field or from the vestigial
  `hnsw/` directories.

**Open question 4 — vector legs carrying stale identity copies.** **CONFIRMED
NEVER AUTHORITATIVE.** Type, subtype and visibility are read from the identity
leg only; the copies in a vector leg are residue from an older writer. This
engine will not start reading them and any reader that does will disagree with
it on rows the two writers touched at different times. Whether Stage 2 drops
them is the accelerated engine's format decision, not this engine's — from
here, dropping them is safe.

**The `float_roundtrip` finding is accepted and worth restating as a rule for
any future reader in any language:** the leg is JSON text, so every component
is a double, and a decimal has exactly one nearest `f64`. A parser one ULP off
builds a different index — different distances, different neighbours — and
makes any parity claim between two engines false. This engine gets it right
only because `JSON.parse` is correctly rounded by specification; nothing in
this repository would have caught the divergence, and the conformance suite's
whole-vector deep-equal is the right instrument.

---

## §6 — verb endpoints

**CONFIRMED, exactly as stated, and it is a defect of shape rather than of
data.**

`saveVerb_internal()` (`src/storage/baseStorage.ts`) writes the whole
`HNSWVerb` — `{ id, vector, connections, verb, sourceId, targetId }` — to
`entities/verbs/<shard>/<id>/vectors.json`, and `saveVerbMetadata()` writes the
engine scalars and user bag to `metadata.json` with **no endpoints**. `getVerb()`
reassembles the two.

The consequences the draft draws are all correct, including the one that costs
the most: **a graph heal cannot be driven from identity records alone**, so
every edge's vector leg is read purely for its structure even though a verb's
`vector` is always `[]`. This engine confirms the measurement in principle —
every verb it has ever written carries its endpoints in the vector leg and none
in the identity leg.

The door contract that returns `(id, sourceId, targetId, verb, subtype)`
regardless of which leg holds the bytes is the right shape and this engine
endorses it as the migration seam. **Commitment: when Stage 2 moves endpoints
onto the verb record, this engine's reader side must accept BOTH placements for
the life of contract 1** — a store written by either engine must be readable by
the other.

---

## §7 — `_system/`

**CONFIRMED**, with one clause now needing a footnote.

- `_system/` mixes persisted state with live protocol and must never be
  enumerated as data. `idx/` and `family-stamps/` are subtrees.
- `tx-log.jsonl` is mutated in place and must be byte-copied, not hard-linked,
  by a snapshot.
- `<root>/locks/` is live coordination — `_writer.lock` and the flush-request
  protocol. Reading them as records is meaningless; writing or deleting them
  interferes with a live writer. **Footnote: this branch adds one more file
  there, `_writer.close`** — the clean-close record naming the lock generation
  a writer released, consumed by the next claim. Same rule applies: it is
  protocol, not data. A reader must ignore it, and must not treat its presence
  or absence as a fact about the store's contents.
- **The encoding split is real and remains real:** everything in `_system/`
  goes through the compressing writer and is `.gz` **except `counts.json`**,
  which is plain JSON. This branch made that write ATOMIC but deliberately did
  NOT make it `.gz` — the ledger is the one file an operator reads with `cat`
  during an incident, and a reader implementation already handles the
  exception. Try both spellings, as the draft says.

---

## §8.1 — the zero-byte ledger window

**ACCEPTED AS A DEFECT AND FIXED ON THIS BRANCH.**

Cause 1 is confirmed exactly as written: `persistCounts()` used a bare
`fs.writeFile`, which truncates first, so the ledger was empty for the whole
write while every other object in the store was written atomically. It now goes
through the same temp+rename path (`writeFileAtomic`), so a reader sees the old
ledger or the new one and never neither.

Cause 2 — "the final write is not awaited by the closing path" — **NOT
REPRODUCED at 10.4.3**, and the mechanism named does not exist here: the
filesystem adapter's count persist is write-through, not debounced
(`scheduleCountPersist()` sets the flag and immediately `await`s
`flushCounts()`), and `close()` awaits `flushCounts()` in its first flush
phase. The ~750 ms the draft measured is more likely the tail of an in-flight
write-through persist observed through the truncation window of cause 1, which
is now closed. **Finding: re-measure on 10.4.4 before keeping cause 2 in the
spec.**

**Answer to open question 5:** yes, fixed; and yes, a reader should keep
classifying a zero-byte ledger as `TORN_RECORD` — the classification is right
on its own terms (a file that exists and holds nothing IS torn), it is the only
honest reading of a store damaged by an older release, and stores written by
every version before 10.4.4 can still carry one.

Pinned here by `tests/integration/count-ledger-identity-record.test.ts` —
40 consecutive persists watched at 1 ms and never once unparseable.

---

## §9 — the ordering law

**CONFIRMED, including the two clauses most likely to be "simplified" by a
future implementer.**

- Shard `00` → `ff` ascending, then id ascending in **UTF-16 code-unit order** —
  this engine's `.sort()` with an `a.id < b.id` comparator, which is exactly
  UTF-16 code-unit order, not UTF-8 byte order and not code-point order. They
  diverge above the BMP.
- Cursor tokens: `cn1:<shard>:<id>` and `cv1:<shard>:<id>`, shard in DECIMAL
  (it is the loop index, not the directory name), id last so an id containing
  `:` survives the round trip. `src/storage/baseStorage.ts` —
  `` return `cn1:${shard}:${id}` `` and `` return `cv1:${shard}:${id}` ``.
- A supplied-but-undecodable cursor must FAIL, never restart at offset 0.
  Confirmed as a rule: silently restarting turns a `while (hasMore)` loop into
  an unbounded one.
- The stability properties (a consistent prefix per shard, not a snapshot;
  appends behind the cursor missed; deletes ahead take effect) are this
  engine's own behaviour, confirmed.

---

## §10 — read depth

**CONFIRMED as a design, with one clause this engine wants on the record:**
*read depth bounds what corruption can be seen.* An ids-only enumeration opens
nothing and therefore reports a clean population over a store of undecodable
bodies. That is the honest consequence of reading exactly what was asked for.
A caller asking "is this store healthy?" from the cheap level is asking the
wrong door — health needs a decoding read. This engine's health reporting
follows the same rule and should say so as plainly.

---

## §11 — cost

**NOT RATIFIED — NOT THIS ENGINE'S TO RATIFY.** The comparative walls are
measurements of two implementations on one box; nothing in this repository can
confirm or refute them. Two observations that ARE this engine's:

1. The structural explanation offered for the verb gap is consistent with this
   engine's code: the noun walk hydrates in batches, the verb walk does not,
   and a cursored resume re-lists and re-sorts the cursor's shard on every
   page. That is a real asymmetry in `src/storage/baseStorage.ts` and it is
   fair to name it.
2. The draft's own honesty about the applier holding 98% of the verb wall after
   fan-out is the part worth keeping. **Nobody may read "8 workers" as "8×".**

---

## §12 — Stage 1

**NOTED, NOT RATIFIED.** Which read path the accelerated engine's walks use is
its own decision. Two things this engine confirms because they are claims about
THIS code:

- **The membership-predicate correction is right.** This engine's membership
  is the IDENTITY RECORD: an id directory holding a vector leg and no identity
  record is enumerated by neither `getNouns()` nor
  `getNounIdsWithPagination()` and is absent from `getCanonicalCounts()`; an
  identity-only row IS enumerated and IS counted. The draft's earlier statement
  that the doors key on `vectors.json` was wrong and its correction is accepted.
- **Read-your-writes is confirmed.** There is no write-behind buffer between an
  awaited `saveNoun()` / `saveNounMetadata()` and the bytes on disk; the leg is
  `stat`-able before the returned promise resolves. **Commitment: this engine
  will not introduce a write-behind buffer in the filesystem adapter without
  declaring a flush contract alongside it** — the law "anything not yet awaited
  is not yet canonical" is only useful if the converse holds.

---

## Findings this ratification files back

1. **§4 / open question 6 is out of date** — the directory-counting scan was
   fixed in 10.4.2. Re-read the spec's §4 against 10.4.3 before Stage 2 builds
   on it. The MEASUREMENTS stay valid; the code claim does not.
2. **§8.1 cause 2 is not reproducible at 10.4.3** — the count persist is
   write-through and `close()` awaits it. Re-measure before republishing.
3. **§8's counted-tier drift (open question 3) should not be described as
   expected** — it is an uncorrected incremental counter, and `repairIndex()`
   is its cure.
4. **The timestamp divergence is a defect on THIS engine's side**, not a
   difference of opinion: substituting `Date.now()` for an unreadable
   timestamp invents data. Filed, not fixed on this branch.
5. **`locks/_writer.close` is new** — a reader must ignore it like every other
   file under `locks/`.
6. **`counts.json` is now written atomically and stays plain JSON** — the
   `_system/` encoding exception is deliberate and permanent for contract 1.