open-brainy/docs/canonical-layout-ratification.md

The canonical layout — ratification

Open Brainy's answer to the canonical layout specification written by the accelerated engine's reader team. It reads that document's rules against THIS repository's writer, at 10.4.3 plus the changes on next/open-lazy-open-and-counts, and for each one says confirmed, corrected, or changed under you — with the line of code that proves it.

The specification was written against 10.4.1. Three of its rules have moved since, and one of its open questions has already been answered by a fix that shipped in 10.4.2. Those are called out first, because a reader implementation built to the draft would be wrong about them today.

Internal engineering document — no frontmatter, not published.

---

What moved since the draft was written

#	Draft says	Today	Where
A	scanCanonicalEntities() counts id DIRECTORIES, so a legacy ledger's derivation includes scars and orphans (§4, open question 6)	CORRECTED IN 10.4.2. The scan counts one entity per IDENTITY RECORD, sharing its predicate with the orphan pruner	src/storage/adapters/fileSystemStorage.ts scanCanonicalEntities() / hasMetadataContentLeg()
B	An already-derived legacy denominator is persisted "once and never rescanned" (§4, §8)	CORRECTED on this branch. A ledger without the identity-record stamp is marked suspect at open AND re-derived honestly in the background, then persisted stamped	scheduleCountLedgerDerivation(), same file
C	counts.json has a zero-byte window because persistCounts() uses a bare fs.writeFile (§8.1, open question 5)	FIXED on this branch. Written temp+rename like every other object	persistCounts(), same file

Everything else below stands as the draft states it, except where marked.

---

§1 — the tree is a projection, not the source of truth

CONFIRMED. The generation log under _generations/ is canonical and the entity tree is a materialization of it; _system/generation.json[.gz] carries { generation, updatedAt } for the projection. src/db/generationStore.ts owns the counter and its manifest; readBrainFormat / the entity-tree stamp are read at open in src/brainy.ts's generation-store phase.

Two walks minutes apart may legitimately disagree, and a caller that needs a point-in-time answer must pin the generation. Confirmed as a rule of this engine, not merely of the reader.

---

§2 — paths and sharding

CONFIRMED, all three clauses.

• Two lowercase hex characters, 256 buckets, and an id that normalises to 32 hex characters buckets by its first byte: src/storage/sharding.ts getShardId() — const normalized = id.toLowerCase().replace(/-/g, ''), if (/^[0-9a-f]{32}$/.test(normalized)) return normalized.substring(0, 2).
• Every other id hashes: hashToShardId(id), FNV-1a over UTF-16 code units (id.charCodeAt(i)), folded to the low byte. The UTF-16 detail is load-bearing above the BMP and is part of the on-disk contract — changing it relocates every application-id record.
• hnsw/ and metadata/ under entities/<kind>/ are vestigial and the write path never populates them. Every scan in this repository excludes them with the same two-hex test (/^[0-9a-f]{2}$/i) — see scanCanonicalEntities() and scanVectoredNounCount(). A reader that counts directory entries reports 258 shards where there are 256 is correct and worth keeping in the spec.

---

§3 — encodings

CONFIRMED.

• The compressed spelling APPENDS: const compressedPath = \${fullPath}.gz` (fileSystemStorage.ts`, the object read and write paths). Deriving it by replacing the extension finds a file that never exists.
• Read order is .gz first, then plain — same call sites.
• The three outcomes are distinct in this engine too: absence is null, an undecodable body raises TornRecordError (src/storage/tornRecordError.ts, registerTornRecordEncounter), and an IO fault is rethrown unreshaped.
• The legacy exception the draft names — a torn .gz with a decodable plain twin returns the recovered plain object, loudly — is real and deliberate; the code comment at the read path states it in those terms.

Divergence (torn rows): RATIFIED AS PERMANENT for the current contract. This engine's paginated walks skip-and-count a torn row so enumeration heals past it; the reader raises with a resume cursor. Both are defensible and they are not reconcilable by picking one: an enumeration that raises cannot serve a heal, and a heal that skips silently is how corruption becomes invisible. The reader's resume_cursor gives a caller the skip behaviour deliberately, which is the better shape. This engine will not change its walks to raise — its walks exist to heal — and the two enumerations are therefore allowed to report different populations over a DAMAGED store, never over a healthy one. Any consumer comparing the two must first establish the store is not damaged.

---

§4 — the population law

CONFIRMED as the law. CORRECTED as a description of this engine's code.

The law is right and is this engine's own: the identity record is the population; the id directory is not. pruneOrphanedEntities() states it and hasMetadataContentLeg() is the single predicate both it and the count scan use, so the two agree by construction:

private hasMetadataContentLeg(legs: string[]): boolean {
  return legs.some((f) => f.startsWith('metadata.json'))
}

The draft's finding that scanCanonicalEntities() counts directories is out of date. It was true of 10.4.1, which the draft was written against; it was fixed in 10.4.2 ("derive the canonical count ledger from identity records, stamp the derivation rule, and mark legacy-derived ledgers suspect at load"). The scan now continues on any container with no metadata content leg.

But the draft's MEASUREMENTS remain valid, and the defect they name was only half-fixed until this branch. A ledger DERIVED under the old rule was adopted at load with its wrong numbers and merely flagged allCountsSuspect, and nothing corrected it short of an operator running repairIndex(). That is why the same frozen archive still reads 14,231 or 14,081 against 14,056 identity records depending on which copy you open, and why a graph heal subtracted 72,679 walked rows from a 72,729 denominator and reported remaining: 50 — the store's 50 verb scar directories. The heal was behaving correctly against a lying denominator.

On this branch the ledger heals itself: a load that finds no identity-record stamp marks the scalars suspect, schedules an honest derivation in the BACKGROUND (never blocking the open — the same walk is part of why a 24,898-id store opened in silence), and persists the corrected scalars stamped. A derivation that raced a write refuses to stamp its number and leaves the ledger suspect, naming repairIndex() as the door that recounts under a barrier. Pinned in tests/integration/count-ledger-identity-record.test.ts, including two copies of one archive deriving the same number.

Answer to open question 6, in full:

1. Should the derivation require an identity leg? Yes, and it does — since 10.4.2, sharing the pruner's predicate.
2. Should already-persisted denominators on affected stores be re-derived? Yes, and they now are, automatically, in the background. An operator no longer has to know.
3. Which scalar may a consumer subtract against? See the next section — the answer is narrower than "the corrected one".

---

§8 / open question 6 — WHICH SCALAR A CONSUMER MAY SUBTRACT AGAINST

The specification asked this to be stated in the ratification. Stated:

scalar	what it is	safe to subtract an enumeration against?
totalNounCount / totalVerbCount	the counted tiers only (isCountedVisibility excludes internal and system)	NO. It is routinely far below the live population by design. A full-tier walk measured against it reads permanently over.
totalNounCountAll / totalVerbCountAll with allCountsDerivedBy: 'identity-record' and allCountsSuspect: false	the all-tier identity-record population	YES. This is the only pair a coverage verdict may use.
the same scalars without the stamp, or with allCountsSuspect: true	an unverified number: either derived under the old container rule, or made unprovable by a delete that could not read its record	NO. Read the flag, refuse the verdict, and say so. On this branch the engine is already correcting it behind you; wait for the correction rather than subtracting against the interim.
totalVectoredNounCount	nouns holding a real, non-empty, non-zero-norm vector, derived by reading record CONTENT	YES for the vector family only. It never had the container defect. It shares allCountsSuspect, so the same flag rule applies.

getCanonicalCounts() (src/storage/adapters/baseStorageAdapter.ts) returns all of them with the suspect flag in one O(1) call, and the flag is the whole contract: a consumer that ignores suspect is computing a verdict the engine has told it not to compute.

The draft's §8 rules are otherwise CONFIRMED: the ledger is counted-tier, an absent key is a legacy file and not a zero, and an absent counts.json is a real condition (this engine rebuilds it from disk on the next open — see initializeCountsFromDisk(), which on this branch narrates that it is doing so and why it cannot be backgrounded).

Open question 3 — the ledger drifted from the public tier by 3 nouns and 1 verb. Not ratified as expected drift. The counted-tier scalars are maintained incrementally and can be left inflated by a delete whose decrement was skipped; that is exactly what allCountsSuspect exists to record and what rebuildTypeCounts() (the sanctioned recount, src/storage/baseStorage.ts) corrects. A drift of 3 against 8,663 on a store with a long delete history is consistent with that mechanism, not with a fresh defect — but it is a REAL inaccuracy, not a definition, and the cure is a recount. Finding for the spec: do not describe counted-tier drift as normal; describe it as an uncorrected incremental counter.

---

§5.1 — the identity leg

CONFIRMED, every clause.

• Two shapes coexist in one store: _fmt: 2 nested-bag, and the legacy flat shape. src/types/reservedFields.ts — METADATA_RECORD_FORMAT_KEY = '_fmt', NESTED_BAG_FORMAT.
• The stamp alone does not decide it. isNestedBagRecord() requires the stamp AND typeof record.metadata === 'object' AND non-null AND !Array.isArray(record.metadata). A reader that trusts the stamp alone mis-splits a pre-law record carrying a user field named metadata. The draft is right to make this a rule.
• Engine fields sit at top level in BOTH shapes, so extraction is shape-independent: RESERVED_ENTITY_FIELDS / RESERVED_RELATIONSHIP_FIELDS. The entity type is under noun (verbs: verb), never type.
• No BigInt at this boundary. Confirmed — every numeric in a metadata record is a JSON number.

Divergence (timestamps): RATIFIED IN THE READER'S FAVOUR, and this engine should follow. The draft is correct that this engine substitutes Date.now() where a stored timestamp is unusable. That is inventing data: a row whose updatedAt could not be read is not a row that was updated now. The reader's null is right. This is a finding against this engine, filed here rather than fixed on this branch — changing hydration's timestamp fallback touches every read path and belongs in its own change with its own pins, not in a branch about opens and counters. Until then, a consumer must treat a timestamp equal to read-time as suspect.

---

§5.2 — the vector leg

CONFIRMED, every clause, including the three that are easiest to get wrong:

• It is JSON, not an mmap-able buffer; raw blobs live in _blobs/ behind getBinaryBlobPath() — a different door entirely.
• The five-way vector state is this engine's own law: absent-leg, absent-field, empty (deferred embed — vector: [] written at add() time), zero-norm, real. isZeroNormVector() (src/utils/distance.ts) is the shared predicate, and scanVectoredNounCount() applies exactly it. A zero-norm vector is not a vector — the 10.4.2 unvector work is precisely this rule, canonical side included.
• connections on disk is not index state. Confirmed. Nothing may infer that a store is indexed, or how, from that field or from the vestigial hnsw/ directories.

Open question 4 — vector legs carrying stale identity copies. CONFIRMED NEVER AUTHORITATIVE. Type, subtype and visibility are read from the identity leg only; the copies in a vector leg are residue from an older writer. This engine will not start reading them and any reader that does will disagree with it on rows the two writers touched at different times. Whether Stage 2 drops them is the accelerated engine's format decision, not this engine's — from here, dropping them is safe.

The float_roundtrip finding is accepted and worth restating as a rule for any future reader in any language: the leg is JSON text, so every component is a double, and a decimal has exactly one nearest f64. A parser one ULP off builds a different index — different distances, different neighbours — and makes any parity claim between two engines false. This engine gets it right only because JSON.parse is correctly rounded by specification; nothing in this repository would have caught the divergence, and the conformance suite's whole-vector deep-equal is the right instrument.

---

§6 — verb endpoints

CONFIRMED, exactly as stated, and it is a defect of shape rather than of data.

saveVerb_internal() (src/storage/baseStorage.ts) writes the whole HNSWVerb — { id, vector, connections, verb, sourceId, targetId } — to entities/verbs/<shard>/<id>/vectors.json, and saveVerbMetadata() writes the engine scalars and user bag to metadata.json with no endpoints. getVerb() reassembles the two.

The consequences the draft draws are all correct, including the one that costs the most: a graph heal cannot be driven from identity records alone, so every edge's vector leg is read purely for its structure even though a verb's vector is always []. This engine confirms the measurement in principle — every verb it has ever written carries its endpoints in the vector leg and none in the identity leg.

The door contract that returns (id, sourceId, targetId, verb, subtype) regardless of which leg holds the bytes is the right shape and this engine endorses it as the migration seam. Commitment: when Stage 2 moves endpoints onto the verb record, this engine's reader side must accept BOTH placements for the life of contract 1 — a store written by either engine must be readable by the other.

---

§7 — _system/

CONFIRMED, with one clause now needing a footnote.

• _system/ mixes persisted state with live protocol and must never be enumerated as data. idx/ and family-stamps/ are subtrees.
• tx-log.jsonl is mutated in place and must be byte-copied, not hard-linked, by a snapshot.
• <root>/locks/ is live coordination — _writer.lock and the flush-request protocol. Reading them as records is meaningless; writing or deleting them interferes with a live writer. Footnote: this branch adds one more file there, _writer.close — the clean-close record naming the lock generation a writer released, consumed by the next claim. Same rule applies: it is protocol, not data. A reader must ignore it, and must not treat its presence or absence as a fact about the store's contents.
• The encoding split is real and remains real: everything in _system/ goes through the compressing writer and is .gz except counts.json, which is plain JSON. This branch made that write ATOMIC but deliberately did NOT make it .gz — the ledger is the one file an operator reads with cat during an incident, and a reader implementation already handles the exception. Try both spellings, as the draft says.

---

§8.1 — the zero-byte ledger window

ACCEPTED AS A DEFECT AND FIXED ON THIS BRANCH.

Cause 1 is confirmed exactly as written: persistCounts() used a bare fs.writeFile, which truncates first, so the ledger was empty for the whole write while every other object in the store was written atomically. It now goes through the same temp+rename path (writeFileAtomic), so a reader sees the old ledger or the new one and never neither.

Cause 2 — "the final write is not awaited by the closing path" — NOT REPRODUCED at 10.4.3, and the mechanism named does not exist here: the filesystem adapter's count persist is write-through, not debounced (scheduleCountPersist() sets the flag and immediately awaits flushCounts()), and close() awaits flushCounts() in its first flush phase. The ~750 ms the draft measured is more likely the tail of an in-flight write-through persist observed through the truncation window of cause 1, which is now closed. Finding: re-measure on 10.4.4 before keeping cause 2 in the spec.

Answer to open question 5: yes, fixed; and yes, a reader should keep classifying a zero-byte ledger as TORN_RECORD — the classification is right on its own terms (a file that exists and holds nothing IS torn), it is the only honest reading of a store damaged by an older release, and stores written by every version before 10.4.4 can still carry one.

Pinned here by tests/integration/count-ledger-identity-record.test.ts — 40 consecutive persists watched at 1 ms and never once unparseable.

---

§9 — the ordering law

CONFIRMED, including the two clauses most likely to be "simplified" by a future implementer.

• Shard 00 → ff ascending, then id ascending in UTF-16 code-unit order — this engine's .sort() with an a.id < b.id comparator, which is exactly UTF-16 code-unit order, not UTF-8 byte order and not code-point order. They diverge above the BMP.
• Cursor tokens: cn1:<shard>:<id> and cv1:<shard>:<id>, shard in DECIMAL (it is the loop index, not the directory name), id last so an id containing : survives the round trip. src/storage/baseStorage.ts — return `cn1:${shard}:${id}` and return `cv1:${shard}:${id}`.
• A supplied-but-undecodable cursor must FAIL, never restart at offset 0. Confirmed as a rule: silently restarting turns a while (hasMore) loop into an unbounded one.
• The stability properties (a consistent prefix per shard, not a snapshot; appends behind the cursor missed; deletes ahead take effect) are this engine's own behaviour, confirmed.

---

§10 — read depth

CONFIRMED as a design, with one clause this engine wants on the record: read depth bounds what corruption can be seen. An ids-only enumeration opens nothing and therefore reports a clean population over a store of undecodable bodies. That is the honest consequence of reading exactly what was asked for. A caller asking "is this store healthy?" from the cheap level is asking the wrong door — health needs a decoding read. This engine's health reporting follows the same rule and should say so as plainly.

---

§11 — cost

NOT RATIFIED — NOT THIS ENGINE'S TO RATIFY. The comparative walls are measurements of two implementations on one box; nothing in this repository can confirm or refute them. Two observations that ARE this engine's:

1. The structural explanation offered for the verb gap is consistent with this engine's code: the noun walk hydrates in batches, the verb walk does not, and a cursored resume re-lists and re-sorts the cursor's shard on every page. That is a real asymmetry in src/storage/baseStorage.ts and it is fair to name it.
2. The draft's own honesty about the applier holding 98% of the verb wall after fan-out is the part worth keeping. Nobody may read "8 workers" as "8×".

---

§12 — Stage 1

NOTED, NOT RATIFIED. Which read path the accelerated engine's walks use is its own decision. Two things this engine confirms because they are claims about THIS code:

• The membership-predicate correction is right. This engine's membership is the IDENTITY RECORD: an id directory holding a vector leg and no identity record is enumerated by neither getNouns() nor getNounIdsWithPagination() and is absent from getCanonicalCounts(); an identity-only row IS enumerated and IS counted. The draft's earlier statement that the doors key on vectors.json was wrong and its correction is accepted.
• Read-your-writes is confirmed. There is no write-behind buffer between an awaited saveNoun() / saveNounMetadata() and the bytes on disk; the leg is stat-able before the returned promise resolves. Commitment: this engine will not introduce a write-behind buffer in the filesystem adapter without declaring a flush contract alongside it — the law "anything not yet awaited is not yet canonical" is only useful if the converse holds.

---

Findings this ratification files back

1. §4 / open question 6 is out of date — the directory-counting scan was fixed in 10.4.2. Re-read the spec's §4 against 10.4.3 before Stage 2 builds on it. The MEASUREMENTS stay valid; the code claim does not.
2. §8.1 cause 2 is not reproducible at 10.4.3 — the count persist is write-through and close() awaits it. Re-measure before republishing.
3. §8's counted-tier drift (open question 3) should not be described as expected — it is an uncorrected incremental counter, and repairIndex() is its cure.
4. The timestamp divergence is a defect on THIS engine's side, not a difference of opinion: substituting Date.now() for an unreadable timestamp invents data. Filed, not fixed on this branch.
5. locks/_writer.close is new — a reader must ignore it like every other file under locks/.
6. counts.json is now written atomically and stays plain JSON — the _system/ encoding exception is deliberate and permanent for contract 1.