Compare commits
17 commits
a95cc5849a
...
a8c724a202
| Author | SHA1 | Date | |
|---|---|---|---|
| a8c724a202 | |||
| 61a469270e | |||
| 02c6163637 | |||
| 2cf3801007 | |||
| 5c22f9500c | |||
| 16d2e1a97e | |||
| fb1da1c56d | |||
| 417ddb5143 | |||
| 9dd399216b | |||
| e4c27fbca8 | |||
| 5a091ccad9 | |||
| 4a67aa0fb9 | |||
| c1f0972395 | |||
| 48802ba385 | |||
| 50676c02f4 | |||
| 131daa08cd | |||
| f5a6cb3f61 |
5 changed files with 6 additions and 663 deletions
|
|
@ -64,6 +64,10 @@ see `package.json` for `test:integration`, `test:coverage`, and friends.
|
||||||
store, measured solo under an exclusive lock" tells a reader everything the
|
store, measured solo under an exclusive lock" tells a reader everything the
|
||||||
number depends on; the machine it ran on and whose data it was tell them
|
number depends on; the machine it ran on and whose data it was tell them
|
||||||
nothing except where somebody's infrastructure lives.
|
nothing except where somebody's infrastructure lives.
|
||||||
|
- **Documents that answer or reference a confidential specification never enter
|
||||||
|
this repository, even summarized.** The public docs describe THIS engine and
|
||||||
|
the published contract, and nothing else — a summary of a private document is
|
||||||
|
still that document's contents.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,6 @@
|
||||||
{
|
{
|
||||||
"contractVersion": 1,
|
"contractVersion": 1,
|
||||||
"engine": "@soulcraftlabs/brainy",
|
"engine": "@soulcraftlabs/brainy",
|
||||||
"prose": "docs/contract-1-ratification.md",
|
|
||||||
"compatibility": {
|
"compatibility": {
|
||||||
"minor": "additive — a new optional door, a new served operator, a new error class; every existing implementation still conforms",
|
"minor": "additive — a new optional door, a new served operator, a new error class; every existing implementation still conforms",
|
||||||
"major": "breaking — a door removed, an answer narrowed, an ordering law changed, an optional door promoted to required, or an operator moved from served to refused"
|
"major": "breaking — a door removed, an answer narrowed, an ordering law changed, an optional door promoted to required, or an operator moved from served to refused"
|
||||||
|
|
|
||||||
|
|
@ -1,415 +0,0 @@
|
||||||
# The canonical layout — ratification
|
|
||||||
|
|
||||||
Open Brainy's answer to the canonical layout specification written by the
|
|
||||||
accelerated engine's reader team. It reads that document's rules against THIS
|
|
||||||
repository's writer, at `10.4.3` plus the changes on
|
|
||||||
`next/open-lazy-open-and-counts`, and for each one says **confirmed**,
|
|
||||||
**corrected**, or **changed under you** — with the line of code that proves it.
|
|
||||||
|
|
||||||
The specification was written against `10.4.1`. Three of its rules have moved
|
|
||||||
since, and one of its open questions has already been answered by a fix that
|
|
||||||
shipped in `10.4.2`. Those are called out first, because a reader
|
|
||||||
implementation built to the draft would be wrong about them today.
|
|
||||||
|
|
||||||
*Internal engineering document — no frontmatter, not published.*
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## What moved since the draft was written
|
|
||||||
|
|
||||||
| # | Draft says | Today | Where |
|
|
||||||
|---|---|---|---|
|
|
||||||
| A | `scanCanonicalEntities()` counts id DIRECTORIES, so a legacy ledger's derivation includes scars and orphans (§4, open question 6) | **CORRECTED IN 10.4.2.** The scan counts one entity per IDENTITY RECORD, sharing its predicate with the orphan pruner | `src/storage/adapters/fileSystemStorage.ts` `scanCanonicalEntities()` / `hasMetadataContentLeg()` |
|
|
||||||
| B | An already-derived legacy denominator is persisted "once and never rescanned" (§4, §8) | **CORRECTED on this branch.** A ledger without the `identity-record` stamp is marked suspect at open AND re-derived honestly in the background, then persisted stamped | `scheduleCountLedgerDerivation()`, same file |
|
|
||||||
| C | `counts.json` has a zero-byte window because `persistCounts()` uses a bare `fs.writeFile` (§8.1, open question 5) | **FIXED on this branch.** Written temp+rename like every other object | `persistCounts()`, same file |
|
|
||||||
|
|
||||||
Everything else below stands as the draft states it, except where marked.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## §1 — the tree is a projection, not the source of truth
|
|
||||||
|
|
||||||
**CONFIRMED.** The generation log under `_generations/` is canonical and the
|
|
||||||
entity tree is a materialization of it; `_system/generation.json[.gz]` carries
|
|
||||||
`{ generation, updatedAt }` for the projection. `src/db/generationStore.ts`
|
|
||||||
owns the counter and its manifest; `readBrainFormat` / the entity-tree stamp
|
|
||||||
are read at open in `src/brainy.ts`'s generation-store phase.
|
|
||||||
|
|
||||||
Two walks minutes apart may legitimately disagree, and a caller that needs a
|
|
||||||
point-in-time answer must pin the generation. Confirmed as a rule of this
|
|
||||||
engine, not merely of the reader.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## §2 — paths and sharding
|
|
||||||
|
|
||||||
**CONFIRMED, all three clauses.**
|
|
||||||
|
|
||||||
- Two lowercase hex characters, 256 buckets, and an id that normalises to 32
|
|
||||||
hex characters buckets by its first byte: `src/storage/sharding.ts`
|
|
||||||
`getShardId()` — `const normalized = id.toLowerCase().replace(/-/g, '')`,
|
|
||||||
`if (/^[0-9a-f]{32}$/.test(normalized)) return normalized.substring(0, 2)`.
|
|
||||||
- Every other id hashes: `hashToShardId(id)`, FNV-1a over **UTF-16 code
|
|
||||||
units** (`id.charCodeAt(i)`), folded to the low byte. The UTF-16 detail is
|
|
||||||
load-bearing above the BMP and is part of the on-disk contract — changing it
|
|
||||||
relocates every application-id record.
|
|
||||||
- `hnsw/` and `metadata/` under `entities/<kind>/` are vestigial and the write
|
|
||||||
path never populates them. Every scan in this repository excludes them with
|
|
||||||
the same two-hex test (`/^[0-9a-f]{2}$/i`) — see `scanCanonicalEntities()`
|
|
||||||
and `scanVectoredNounCount()`. **A reader that counts directory entries
|
|
||||||
reports 258 shards where there are 256** is correct and worth keeping in the
|
|
||||||
spec.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## §3 — encodings
|
|
||||||
|
|
||||||
**CONFIRMED.**
|
|
||||||
|
|
||||||
- The compressed spelling APPENDS: `const compressedPath = \`${fullPath}.gz\``
|
|
||||||
(`fileSystemStorage.ts`, the object read and write paths). Deriving it by
|
|
||||||
replacing the extension finds a file that never exists.
|
|
||||||
- Read order is `.gz` first, then plain — same call sites.
|
|
||||||
- The three outcomes are distinct in this engine too: absence is `null`, an
|
|
||||||
undecodable body raises `TornRecordError` (`src/storage/tornRecordError.ts`,
|
|
||||||
`registerTornRecordEncounter`), and an IO fault is rethrown unreshaped.
|
|
||||||
- The legacy exception the draft names — a torn `.gz` with a decodable plain
|
|
||||||
twin returns the recovered plain object, loudly — is real and deliberate; the
|
|
||||||
code comment at the read path states it in those terms.
|
|
||||||
|
|
||||||
**Divergence (torn rows): RATIFIED AS PERMANENT for the current contract.**
|
|
||||||
This engine's paginated walks skip-and-count a torn row so enumeration heals
|
|
||||||
past it; the reader raises with a resume cursor. Both are defensible and they
|
|
||||||
are not reconcilable by picking one: an enumeration that raises cannot serve a
|
|
||||||
heal, and a heal that skips silently is how corruption becomes invisible. The
|
|
||||||
reader's `resume_cursor` gives a caller the skip behaviour deliberately, which
|
|
||||||
is the better shape. **This engine will not change its walks to raise** — its
|
|
||||||
walks exist to heal — and the two enumerations are therefore allowed to report
|
|
||||||
different populations over a DAMAGED store, never over a healthy one. Any
|
|
||||||
consumer comparing the two must first establish the store is not damaged.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## §4 — the population law
|
|
||||||
|
|
||||||
**CONFIRMED as the law. CORRECTED as a description of this engine's code.**
|
|
||||||
|
|
||||||
The law is right and is this engine's own: *the identity record is the
|
|
||||||
population; the id directory is not.* `pruneOrphanedEntities()` states it and
|
|
||||||
`hasMetadataContentLeg()` is the single predicate both it and the count scan
|
|
||||||
use, so the two agree by construction:
|
|
||||||
|
|
||||||
```ts
|
|
||||||
private hasMetadataContentLeg(legs: string[]): boolean {
|
|
||||||
return legs.some((f) => f.startsWith('metadata.json'))
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**The draft's finding that `scanCanonicalEntities()` counts directories is out
|
|
||||||
of date.** It was true of `10.4.1`, which the draft was written against; it was
|
|
||||||
fixed in `10.4.2` ("derive the canonical count ledger from identity records,
|
|
||||||
stamp the derivation rule, and mark legacy-derived ledgers suspect at load").
|
|
||||||
The scan now `continue`s on any container with no metadata content leg.
|
|
||||||
|
|
||||||
**But the draft's MEASUREMENTS remain valid, and the defect they name was only
|
|
||||||
half-fixed until this branch.** A ledger DERIVED under the old rule was
|
|
||||||
adopted at load with its wrong numbers and merely flagged `allCountsSuspect`,
|
|
||||||
and nothing corrected it short of an operator running `repairIndex()`. That is
|
|
||||||
why the same frozen archive still reads 14,231 or 14,081 against 14,056
|
|
||||||
identity records depending on which copy you open, and why a graph heal
|
|
||||||
subtracted 72,679 walked rows from a 72,729 denominator and reported
|
|
||||||
`remaining: 50` — the store's 50 verb scar directories. The heal was behaving
|
|
||||||
correctly against a lying denominator.
|
|
||||||
|
|
||||||
**On this branch the ledger heals itself**: a load that finds no
|
|
||||||
`identity-record` stamp marks the scalars suspect, schedules an honest
|
|
||||||
derivation in the BACKGROUND (never blocking the open — the same walk is part
|
|
||||||
of why a 24,898-id store opened in silence), and persists the corrected
|
|
||||||
scalars stamped. A derivation that raced a write refuses to stamp its number
|
|
||||||
and leaves the ledger suspect, naming `repairIndex()` as the door that
|
|
||||||
recounts under a barrier. Pinned in
|
|
||||||
`tests/integration/count-ledger-identity-record.test.ts`, including two copies
|
|
||||||
of one archive deriving the same number.
|
|
||||||
|
|
||||||
**Answer to open question 6, in full:**
|
|
||||||
|
|
||||||
1. *Should the derivation require an identity leg?* **Yes, and it does** — since
|
|
||||||
`10.4.2`, sharing the pruner's predicate.
|
|
||||||
2. *Should already-persisted denominators on affected stores be re-derived?*
|
|
||||||
**Yes, and they now are, automatically, in the background.** An operator no
|
|
||||||
longer has to know.
|
|
||||||
3. *Which scalar may a consumer subtract against?* See the next section — the
|
|
||||||
answer is narrower than "the corrected one".
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## §8 / open question 6 — WHICH SCALAR A CONSUMER MAY SUBTRACT AGAINST
|
|
||||||
|
|
||||||
The specification asked this to be stated in the ratification. Stated:
|
|
||||||
|
|
||||||
| scalar | what it is | safe to subtract an enumeration against? |
|
|
||||||
|---|---|---|
|
|
||||||
| `totalNounCount` / `totalVerbCount` | the **counted tiers only** (`isCountedVisibility` excludes `internal` and `system`) | **NO.** It is routinely far below the live population by design. A full-tier walk measured against it reads permanently over. |
|
|
||||||
| `totalNounCountAll` / `totalVerbCountAll` **with `allCountsDerivedBy: 'identity-record'` and `allCountsSuspect: false`** | the all-tier identity-record population | **YES.** This is the only pair a coverage verdict may use. |
|
|
||||||
| the same scalars **without the stamp, or with `allCountsSuspect: true`** | an unverified number: either derived under the old container rule, or made unprovable by a delete that could not read its record | **NO.** Read the flag, refuse the verdict, and say so. On this branch the engine is already correcting it behind you; wait for the correction rather than subtracting against the interim. |
|
|
||||||
| `totalVectoredNounCount` | nouns holding a real, non-empty, non-zero-norm vector, derived by reading record CONTENT | **YES for the vector family only.** It never had the container defect. It shares `allCountsSuspect`, so the same flag rule applies. |
|
|
||||||
|
|
||||||
`getCanonicalCounts()` (`src/storage/adapters/baseStorageAdapter.ts`) returns
|
|
||||||
all of them with the suspect flag in one O(1) call, and the flag is the whole
|
|
||||||
contract: **a consumer that ignores `suspect` is computing a verdict the engine
|
|
||||||
has told it not to compute.**
|
|
||||||
|
|
||||||
The draft's §8 rules are otherwise **CONFIRMED**: the ledger is counted-tier,
|
|
||||||
an absent key is a legacy file and not a zero, and an absent `counts.json` is a
|
|
||||||
real condition (this engine rebuilds it from disk on the next open — see
|
|
||||||
`initializeCountsFromDisk()`, which on this branch narrates that it is doing so
|
|
||||||
and why it cannot be backgrounded).
|
|
||||||
|
|
||||||
**Open question 3 — the ledger drifted from the public tier by 3 nouns and 1
|
|
||||||
verb.** Not ratified as expected drift. The counted-tier scalars are maintained
|
|
||||||
incrementally and can be left inflated by a delete whose decrement was skipped;
|
|
||||||
that is exactly what `allCountsSuspect` exists to record and what
|
|
||||||
`rebuildTypeCounts()` (the sanctioned recount, `src/storage/baseStorage.ts`)
|
|
||||||
corrects. A drift of 3 against 8,663 on a store with a long delete history is
|
|
||||||
consistent with that mechanism, not with a fresh defect — but it is a REAL
|
|
||||||
inaccuracy, not a definition, and the cure is a recount. **Finding for the
|
|
||||||
spec: do not describe counted-tier drift as normal; describe it as an
|
|
||||||
uncorrected incremental counter.**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## §5.1 — the identity leg
|
|
||||||
|
|
||||||
**CONFIRMED, every clause.**
|
|
||||||
|
|
||||||
- Two shapes coexist in one store: `_fmt: 2` nested-bag, and the legacy flat
|
|
||||||
shape. `src/types/reservedFields.ts` — `METADATA_RECORD_FORMAT_KEY = '_fmt'`,
|
|
||||||
`NESTED_BAG_FORMAT`.
|
|
||||||
- **The stamp alone does not decide it.** `isNestedBagRecord()` requires the
|
|
||||||
stamp AND `typeof record.metadata === 'object'` AND non-null AND
|
|
||||||
`!Array.isArray(record.metadata)`. A reader that trusts the stamp alone
|
|
||||||
mis-splits a pre-law record carrying a user field named `metadata`. The draft
|
|
||||||
is right to make this a rule.
|
|
||||||
- Engine fields sit at top level in BOTH shapes, so extraction is
|
|
||||||
shape-independent: `RESERVED_ENTITY_FIELDS` / `RESERVED_RELATIONSHIP_FIELDS`.
|
|
||||||
The entity type is under `noun` (verbs: `verb`), never `type`.
|
|
||||||
- **No BigInt at this boundary.** Confirmed — every numeric in a metadata
|
|
||||||
record is a JSON number.
|
|
||||||
|
|
||||||
**Divergence (timestamps): RATIFIED IN THE READER'S FAVOUR, and this engine
|
|
||||||
should follow.** The draft is correct that this engine substitutes `Date.now()`
|
|
||||||
where a stored timestamp is unusable. That is inventing data: a row whose
|
|
||||||
`updatedAt` could not be read is not a row that was updated now. The reader's
|
|
||||||
`null` is right. **This is a finding against this engine, filed here rather
|
|
||||||
than fixed on this branch** — changing hydration's timestamp fallback touches
|
|
||||||
every read path and belongs in its own change with its own pins, not in a
|
|
||||||
branch about opens and counters. Until then, a consumer must treat a timestamp
|
|
||||||
equal to read-time as suspect.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## §5.2 — the vector leg
|
|
||||||
|
|
||||||
**CONFIRMED, every clause**, including the three that are easiest to get wrong:
|
|
||||||
|
|
||||||
- It is **JSON**, not an mmap-able buffer; raw blobs live in `_blobs/` behind
|
|
||||||
`getBinaryBlobPath()` — a different door entirely.
|
|
||||||
- **The five-way vector state** is this engine's own law: `absent-leg`,
|
|
||||||
`absent-field`, `empty` (deferred embed — `vector: []` written at `add()`
|
|
||||||
time), `zero-norm`, `real`. `isZeroNormVector()` (`src/utils/distance.ts`) is
|
|
||||||
the shared predicate, and `scanVectoredNounCount()` applies exactly it. A
|
|
||||||
zero-norm vector **is not a vector** — the 10.4.2 unvector work is precisely
|
|
||||||
this rule, canonical side included.
|
|
||||||
- **`connections` on disk is not index state.** Confirmed. Nothing may infer
|
|
||||||
that a store is indexed, or how, from that field or from the vestigial
|
|
||||||
`hnsw/` directories.
|
|
||||||
|
|
||||||
**Open question 4 — vector legs carrying stale identity copies.** **CONFIRMED
|
|
||||||
NEVER AUTHORITATIVE.** Type, subtype and visibility are read from the identity
|
|
||||||
leg only; the copies in a vector leg are residue from an older writer. This
|
|
||||||
engine will not start reading them and any reader that does will disagree with
|
|
||||||
it on rows the two writers touched at different times. Whether Stage 2 drops
|
|
||||||
them is the accelerated engine's format decision, not this engine's — from
|
|
||||||
here, dropping them is safe.
|
|
||||||
|
|
||||||
**The `float_roundtrip` finding is accepted and worth restating as a rule for
|
|
||||||
any future reader in any language:** the leg is JSON text, so every component
|
|
||||||
is a double, and a decimal has exactly one nearest `f64`. A parser one ULP off
|
|
||||||
builds a different index — different distances, different neighbours — and
|
|
||||||
makes any parity claim between two engines false. This engine gets it right
|
|
||||||
only because `JSON.parse` is correctly rounded by specification; nothing in
|
|
||||||
this repository would have caught the divergence, and the conformance suite's
|
|
||||||
whole-vector deep-equal is the right instrument.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## §6 — verb endpoints
|
|
||||||
|
|
||||||
**CONFIRMED, exactly as stated, and it is a defect of shape rather than of
|
|
||||||
data.**
|
|
||||||
|
|
||||||
`saveVerb_internal()` (`src/storage/baseStorage.ts`) writes the whole
|
|
||||||
`HNSWVerb` — `{ id, vector, connections, verb, sourceId, targetId }` — to
|
|
||||||
`entities/verbs/<shard>/<id>/vectors.json`, and `saveVerbMetadata()` writes the
|
|
||||||
engine scalars and user bag to `metadata.json` with **no endpoints**. `getVerb()`
|
|
||||||
reassembles the two.
|
|
||||||
|
|
||||||
The consequences the draft draws are all correct, including the one that costs
|
|
||||||
the most: **a graph heal cannot be driven from identity records alone**, so
|
|
||||||
every edge's vector leg is read purely for its structure even though a verb's
|
|
||||||
`vector` is always `[]`. This engine confirms the measurement in principle —
|
|
||||||
every verb it has ever written carries its endpoints in the vector leg and none
|
|
||||||
in the identity leg.
|
|
||||||
|
|
||||||
The door contract that returns `(id, sourceId, targetId, verb, subtype)`
|
|
||||||
regardless of which leg holds the bytes is the right shape and this engine
|
|
||||||
endorses it as the migration seam. **Commitment: when Stage 2 moves endpoints
|
|
||||||
onto the verb record, this engine's reader side must accept BOTH placements for
|
|
||||||
the life of contract 1** — a store written by either engine must be readable by
|
|
||||||
the other.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## §7 — `_system/`
|
|
||||||
|
|
||||||
**CONFIRMED**, with one clause now needing a footnote.
|
|
||||||
|
|
||||||
- `_system/` mixes persisted state with live protocol and must never be
|
|
||||||
enumerated as data. `idx/` and `family-stamps/` are subtrees.
|
|
||||||
- `tx-log.jsonl` is mutated in place and must be byte-copied, not hard-linked,
|
|
||||||
by a snapshot.
|
|
||||||
- `<root>/locks/` is live coordination — `_writer.lock` and the flush-request
|
|
||||||
protocol. Reading them as records is meaningless; writing or deleting them
|
|
||||||
interferes with a live writer. **Footnote: this branch adds one more file
|
|
||||||
there, `_writer.close`** — the clean-close record naming the lock generation
|
|
||||||
a writer released, consumed by the next claim. Same rule applies: it is
|
|
||||||
protocol, not data. A reader must ignore it, and must not treat its presence
|
|
||||||
or absence as a fact about the store's contents.
|
|
||||||
- **The encoding split is real and remains real:** everything in `_system/`
|
|
||||||
goes through the compressing writer and is `.gz` **except `counts.json`**,
|
|
||||||
which is plain JSON. This branch made that write ATOMIC but deliberately did
|
|
||||||
NOT make it `.gz` — the ledger is the one file an operator reads with `cat`
|
|
||||||
during an incident, and a reader implementation already handles the
|
|
||||||
exception. Try both spellings, as the draft says.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## §8.1 — the zero-byte ledger window
|
|
||||||
|
|
||||||
**ACCEPTED AS A DEFECT AND FIXED ON THIS BRANCH.**
|
|
||||||
|
|
||||||
Cause 1 is confirmed exactly as written: `persistCounts()` used a bare
|
|
||||||
`fs.writeFile`, which truncates first, so the ledger was empty for the whole
|
|
||||||
write while every other object in the store was written atomically. It now goes
|
|
||||||
through the same temp+rename path (`writeFileAtomic`), so a reader sees the old
|
|
||||||
ledger or the new one and never neither.
|
|
||||||
|
|
||||||
Cause 2 — "the final write is not awaited by the closing path" — **NOT
|
|
||||||
REPRODUCED at 10.4.3**, and the mechanism named does not exist here: the
|
|
||||||
filesystem adapter's count persist is write-through, not debounced
|
|
||||||
(`scheduleCountPersist()` sets the flag and immediately `await`s
|
|
||||||
`flushCounts()`), and `close()` awaits `flushCounts()` in its first flush
|
|
||||||
phase. The ~750 ms the draft measured is more likely the tail of an in-flight
|
|
||||||
write-through persist observed through the truncation window of cause 1, which
|
|
||||||
is now closed. **Finding: re-measure on 10.4.4 before keeping cause 2 in the
|
|
||||||
spec.**
|
|
||||||
|
|
||||||
**Answer to open question 5:** yes, fixed; and yes, a reader should keep
|
|
||||||
classifying a zero-byte ledger as `TORN_RECORD` — the classification is right
|
|
||||||
on its own terms (a file that exists and holds nothing IS torn), it is the only
|
|
||||||
honest reading of a store damaged by an older release, and stores written by
|
|
||||||
every version before 10.4.4 can still carry one.
|
|
||||||
|
|
||||||
Pinned here by `tests/integration/count-ledger-identity-record.test.ts` —
|
|
||||||
40 consecutive persists watched at 1 ms and never once unparseable.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## §9 — the ordering law
|
|
||||||
|
|
||||||
**CONFIRMED, including the two clauses most likely to be "simplified" by a
|
|
||||||
future implementer.**
|
|
||||||
|
|
||||||
- Shard `00` → `ff` ascending, then id ascending in **UTF-16 code-unit order** —
|
|
||||||
this engine's `.sort()` with an `a.id < b.id` comparator, which is exactly
|
|
||||||
UTF-16 code-unit order, not UTF-8 byte order and not code-point order. They
|
|
||||||
diverge above the BMP.
|
|
||||||
- Cursor tokens: `cn1:<shard>:<id>` and `cv1:<shard>:<id>`, shard in DECIMAL
|
|
||||||
(it is the loop index, not the directory name), id last so an id containing
|
|
||||||
`:` survives the round trip. `src/storage/baseStorage.ts` —
|
|
||||||
`` return `cn1:${shard}:${id}` `` and `` return `cv1:${shard}:${id}` ``.
|
|
||||||
- A supplied-but-undecodable cursor must FAIL, never restart at offset 0.
|
|
||||||
Confirmed as a rule: silently restarting turns a `while (hasMore)` loop into
|
|
||||||
an unbounded one.
|
|
||||||
- The stability properties (a consistent prefix per shard, not a snapshot;
|
|
||||||
appends behind the cursor missed; deletes ahead take effect) are this
|
|
||||||
engine's own behaviour, confirmed.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## §10 — read depth
|
|
||||||
|
|
||||||
**CONFIRMED as a design, with one clause this engine wants on the record:**
|
|
||||||
*read depth bounds what corruption can be seen.* An ids-only enumeration opens
|
|
||||||
nothing and therefore reports a clean population over a store of undecodable
|
|
||||||
bodies. That is the honest consequence of reading exactly what was asked for.
|
|
||||||
A caller asking "is this store healthy?" from the cheap level is asking the
|
|
||||||
wrong door — health needs a decoding read. This engine's health reporting
|
|
||||||
follows the same rule and should say so as plainly.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## §11 — cost
|
|
||||||
|
|
||||||
**NOT RATIFIED — NOT THIS ENGINE'S TO RATIFY.** The comparative walls are
|
|
||||||
measurements of two implementations on one box; nothing in this repository can
|
|
||||||
confirm or refute them. Two observations that ARE this engine's:
|
|
||||||
|
|
||||||
1. The structural explanation offered for the verb gap is consistent with this
|
|
||||||
engine's code: the noun walk hydrates in batches, the verb walk does not,
|
|
||||||
and a cursored resume re-lists and re-sorts the cursor's shard on every
|
|
||||||
page. That is a real asymmetry in `src/storage/baseStorage.ts` and it is
|
|
||||||
fair to name it.
|
|
||||||
2. The draft's own honesty about the applier holding 98% of the verb wall after
|
|
||||||
fan-out is the part worth keeping. **Nobody may read "8 workers" as "8×".**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## §12 — Stage 1
|
|
||||||
|
|
||||||
**NOTED, NOT RATIFIED.** Which read path the accelerated engine's walks use is
|
|
||||||
its own decision. Two things this engine confirms because they are claims about
|
|
||||||
THIS code:
|
|
||||||
|
|
||||||
- **The membership-predicate correction is right.** This engine's membership
|
|
||||||
is the IDENTITY RECORD: an id directory holding a vector leg and no identity
|
|
||||||
record is enumerated by neither `getNouns()` nor
|
|
||||||
`getNounIdsWithPagination()` and is absent from `getCanonicalCounts()`; an
|
|
||||||
identity-only row IS enumerated and IS counted. The draft's earlier statement
|
|
||||||
that the doors key on `vectors.json` was wrong and its correction is accepted.
|
|
||||||
- **Read-your-writes is confirmed.** There is no write-behind buffer between an
|
|
||||||
awaited `saveNoun()` / `saveNounMetadata()` and the bytes on disk; the leg is
|
|
||||||
`stat`-able before the returned promise resolves. **Commitment: this engine
|
|
||||||
will not introduce a write-behind buffer in the filesystem adapter without
|
|
||||||
declaring a flush contract alongside it** — the law "anything not yet awaited
|
|
||||||
is not yet canonical" is only useful if the converse holds.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Findings this ratification files back
|
|
||||||
|
|
||||||
1. **§4 / open question 6 is out of date** — the directory-counting scan was
|
|
||||||
fixed in 10.4.2. Re-read the spec's §4 against 10.4.3 before Stage 2 builds
|
|
||||||
on it. The MEASUREMENTS stay valid; the code claim does not.
|
|
||||||
2. **§8.1 cause 2 is not reproducible at 10.4.3** — the count persist is
|
|
||||||
write-through and `close()` awaits it. Re-measure before republishing.
|
|
||||||
3. **§8's counted-tier drift (open question 3) should not be described as
|
|
||||||
expected** — it is an uncorrected incremental counter, and `repairIndex()`
|
|
||||||
is its cure.
|
|
||||||
4. **The timestamp divergence is a defect on THIS engine's side**, not a
|
|
||||||
difference of opinion: substituting `Date.now()` for an unreadable
|
|
||||||
timestamp invents data. Filed, not fixed on this branch.
|
|
||||||
5. **`locks/_writer.close` is new** — a reader must ignore it like every other
|
|
||||||
file under `locks/`.
|
|
||||||
6. **`counts.json` is now written atomically and stays plain JSON** — the
|
|
||||||
`_system/` encoding exception is deliberate and permanent for contract 1.
|
|
||||||
|
|
@ -1,244 +0,0 @@
|
||||||
# Contract 1 — ratification
|
|
||||||
|
|
||||||
Open Brainy's answer to the API contract published by the accelerated engine
|
|
||||||
(`docs/api-contract.md` + `docs/api-contract.json`, contract version 1). Each
|
|
||||||
item is answered with the code line that proves it, and each promise is stated
|
|
||||||
as a promise rather than a description.
|
|
||||||
|
|
||||||
*Internal engineering document — no frontmatter, not published.*
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1. Contract version — DECLARED
|
|
||||||
|
|
||||||
`package.json` carries `"brainyContract": 1`, and the engine states its own:
|
|
||||||
|
|
||||||
```ts
|
|
||||||
export const BRAINY_CONTRACT_VERSION = 1 as const
|
|
||||||
export function contractVersion(): number { return BRAINY_CONTRACT_VERSION }
|
|
||||||
```
|
|
||||||
|
|
||||||
`src/utils/version.ts`, re-exported from `src/index.ts`. Two engines can now
|
|
||||||
compare an integer instead of probing prototypes, and a tool can read the
|
|
||||||
package field without importing the engine. Pinned in
|
|
||||||
`tests/integration/filter-operator-conformance.test.ts` ("declares its contract
|
|
||||||
version in code and in package.json") — the code value and the package field
|
|
||||||
can never drift apart silently.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. The REQUIRED / OPTIONAL split — RATIFIED, WITH A COMMITMENT
|
|
||||||
|
|
||||||
**Ratified: 41 required doors of 57.** The promise, stated plainly:
|
|
||||||
|
|
||||||
> **A REQUIRED door is never removed, never narrowed, and never made optional
|
|
||||||
> without a MAJOR contract bump.** "Narrowed" includes: refusing an input it
|
|
||||||
> used to accept, returning less than it used to return, and changing an
|
|
||||||
> ordering, a cursor encoding, or a refusal's typed code. An OPTIONAL door may
|
|
||||||
> be added in a minor; an optional door **promoted to required** is a major,
|
|
||||||
> because a consumer that relied on feature-detecting it now has a hard
|
|
||||||
> dependency.
|
|
||||||
|
|
||||||
Two clarifications this engine attaches, so the promise means the same thing
|
|
||||||
on both sides:
|
|
||||||
|
|
||||||
1. **A refusal is part of the door.** Contract 1 includes not only that
|
|
||||||
`find({ where })` answers, but that it REFUSES by name for the operators
|
|
||||||
listed as refused. Turning a refusal into a silent empty answer is a
|
|
||||||
narrowing, not a relaxation — the same class of change as removing the door.
|
|
||||||
2. **Deprecation is not removal.** This engine may mark a required door
|
|
||||||
deprecated in a minor (documented, warned) as long as it keeps working. Only
|
|
||||||
its removal is a major.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. `is` / `isNot` / `greaterEqual` / `lessEqual` — A FINDING AGAINST THE SPEC
|
|
||||||
|
|
||||||
**The specification is wrong about these four, and this engine has never
|
|
||||||
served them.** `docs/filter-operator-conformance.md` (in the accelerated
|
|
||||||
engine's repository — not editable from here) lists them as served aliases.
|
|
||||||
The accepted set is defined in one place:
|
|
||||||
|
|
||||||
```ts
|
|
||||||
// src/utils/metadataFilter.ts
|
|
||||||
const VALUE_OPERATORS = new Set<string>([
|
|
||||||
'equals', 'eq', 'notEquals', 'ne',
|
|
||||||
'greaterThan', 'gt', 'greaterThanOrEqual', 'gte',
|
|
||||||
'lessThan', 'lt', 'lessThanOrEqual', 'lte',
|
|
||||||
'between', 'oneOf', 'in', 'noneOf',
|
|
||||||
'contains', 'excludes', 'hasAll', 'length',
|
|
||||||
'exists', 'missing', 'matches', 'startsWith', 'endsWith'
|
|
||||||
])
|
|
||||||
```
|
|
||||||
|
|
||||||
25 tokens. None of the four appears; `validateWhereFilter()` raises
|
|
||||||
`BrainyError('INVALID_QUERY')` naming the bad operator and listing the valid
|
|
||||||
set, before any index read. A consumer following the spec would have written a
|
|
||||||
filter this engine rejects outright.
|
|
||||||
|
|
||||||
**Action taken here, since the prose lives in the other repository:** the truth
|
|
||||||
is made machine-checkable rather than re-asserted in another document. The
|
|
||||||
accepted set is asserted token-for-token in
|
|
||||||
`tests/integration/filter-operator-conformance.test.ts`, read out of the
|
|
||||||
engine's own refusal message, and the same set is emitted into
|
|
||||||
`docs/api-contract.json` (item 8). Diff the manifests; the prose can then be
|
|
||||||
corrected from a fact.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. The serving-withholding invariant list — CONFIRMED IDENTICAL
|
|
||||||
|
|
||||||
`index-initialized · durable-state-present · manifest-residency ·
|
|
||||||
replay-clean · strand-latch`. Confirmed as this engine's list, and confirmed
|
|
||||||
EXHAUSTIVE for contract 1: these are the only invariants whose failure may
|
|
||||||
withhold serving. Everything else a health report can fail is a `warn` — it
|
|
||||||
names damage without closing a door.
|
|
||||||
|
|
||||||
The mechanism on this side: `assessProviderHealth()`
|
|
||||||
(`src/utils/indexReadiness.ts`) treats the provider's own `serving` verdict as
|
|
||||||
authoritative and verbatim; an UNLEDGERED family never flips a serving provider
|
|
||||||
to not-ready and never flips a not-serving provider to ready. The read gate
|
|
||||||
refuses PER FAMILY — a metadata read is never refused by an unserving vector
|
|
||||||
leg (`src/brainy.ts`, `ensureFamiliesServing`).
|
|
||||||
|
|
||||||
**One addition this engine is making, declared here because it changes what a
|
|
||||||
refusal MEANS:** a provider may now report `rebuildInProgress()` — it is
|
|
||||||
rebuilding ITSELF, online, and its doors refuse by name with progress until it
|
|
||||||
is whole. This does not add a withholding invariant (the provider's own
|
|
||||||
`serving: false` is still what withholds); it adds a REASON attached to that
|
|
||||||
withholding, so a caller can tell "temporarily closed, opens by itself" from
|
|
||||||
"broken, needs repairIndex()". Additive, hence a minor.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. The compatibility rule — ADOPTED
|
|
||||||
|
|
||||||
**Minor = additive. Major = breaking.** Adopted verbatim, with the
|
|
||||||
announcement duty attached:
|
|
||||||
|
|
||||||
> **Every public-surface addition is announced.** The accelerated engine's
|
|
||||||
> package re-exports this engine's surface by enumeration, so it goes red on
|
|
||||||
> any new export BY DESIGN — that redness is the announcement mechanism
|
|
||||||
> working, not a build break to route around.
|
|
||||||
|
|
||||||
The mechanics that make this checkable rather than remembered:
|
|
||||||
`scripts/emit-contract-manifest.mjs --check` fails when the committed
|
|
||||||
`docs/api-contract.json` no longer matches the built surface. A new export is
|
|
||||||
therefore a red check with a message naming what to do: re-emit and announce.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6. The 30 storage seam methods — SUPPORTED SURFACE, COMMITTED
|
|
||||||
|
|
||||||
**Committed: every method in `docs/api-contract.md` §15 is supported surface
|
|
||||||
until Stage 2, and none is removed without a contract major.** They are the
|
|
||||||
seam the accelerated engine's storage adapter implements and the seam its
|
|
||||||
reader replaces piece by piece; removing one mid-programme would break a
|
|
||||||
working pair for no gain.
|
|
||||||
|
|
||||||
Two qualifications, both stated so neither side is surprised:
|
|
||||||
|
|
||||||
1. **Supported ≠ frozen in behaviour.** A seam method may become FASTER, may
|
|
||||||
narrate more, and may start refusing an input that was previously an
|
|
||||||
undefined-behaviour footgun — the last of those is announced as a divergence
|
|
||||||
here before it ships, not discovered by the other engine.
|
|
||||||
2. **`counts.json`'s ledger is the one seam value that is not an
|
|
||||||
enumeration.** See `docs/canonical-layout-ratification.md` §8: only the
|
|
||||||
all-tier pair carrying `allCountsDerivedBy: 'identity-record'` with
|
|
||||||
`allCountsSuspect: false` may be subtracted against. That rule is part of
|
|
||||||
this commitment.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 7. `hasAll` / `noneOf` / `excludes` — SERVED, NOT RATIFIED AS A DIVERGENCE
|
|
||||||
|
|
||||||
The accelerated engine was right that it was the correct side, and the
|
|
||||||
divergence is now closed in the right direction: **this engine serves all three
|
|
||||||
on the index path.**
|
|
||||||
|
|
||||||
The defect underneath was worse than a divergence. The metadata index's
|
|
||||||
operator switch (`src/utils/metadataIndex.ts`) had **no default case**, so any
|
|
||||||
operator without a `case` left the field's match set at its initial `[]` and
|
|
||||||
`find()` returned an empty page. `hasAll`, `noneOf` and `excludes` are
|
|
||||||
documented, accepted by the validator, and implemented in the in-memory
|
|
||||||
matcher — and they answered silently wrong through an index-backed find.
|
|
||||||
|
|
||||||
- **`hasAll: [a, b]`** — the intersection of each element's posting set. An
|
|
||||||
empty operand array is vacuously true of every row that HAS the field.
|
|
||||||
- **`noneOf: [a, b]`** — the complement of the union of their posting sets.
|
|
||||||
- **`excludes: v`** — the complement of `contains`.
|
|
||||||
|
|
||||||
**And the other four are now REFUSED BY NAME rather than answered empty.**
|
|
||||||
`startsWith`, `endsWith`, `matches` and `length` cannot be evaluated by an
|
|
||||||
equality/range posting index without reading every row, which is the cost this
|
|
||||||
path exists to avoid. They raise `BrainyError('INVALID_QUERY')` naming the
|
|
||||||
operator, the field, and the reason. This matches the accelerated engine's
|
|
||||||
behaviour for the same four tokens, so the two engines now AGREE on all 25:
|
|
||||||
|
|
||||||
| class | tokens |
|
|
||||||
|---|---|
|
|
||||||
| served on the index path | between, contains, eq, equals, excludes, exists, greaterThan, greaterThanOrEqual, gt, gte, hasAll, in, lessThan, lessThanOrEqual, lt, lte, missing, ne, noneOf, notEquals, oneOf |
|
|
||||||
| refused by name | endsWith, length, matches, startsWith |
|
|
||||||
|
|
||||||
Pinned in `tests/integration/filter-operator-conformance.test.ts`: the exact
|
|
||||||
25-token accepted set, the three now served with their real answers (including
|
|
||||||
an honest zero), and each of the four refusing by name.
|
|
||||||
|
|
||||||
**This is a behaviour change for any consumer today calling the four refused
|
|
||||||
operators through `find({ where })`.** They received an empty page; they now
|
|
||||||
receive a typed refusal. Converting a wrong answer into a loud refusal is this
|
|
||||||
engine's own law, and the previous behaviour was not a contract anyone could
|
|
||||||
have relied on deliberately — but it is a change, and it is named here rather
|
|
||||||
than discovered.
|
|
||||||
|
|
||||||
**`knownDivergences` after this change:** the entry
|
|
||||||
`served-beyond-baseline` is RESOLVED (both engines serve all three). The entry
|
|
||||||
`refused-operators-answer-differently` is RESOLVED (both engines refuse the
|
|
||||||
same four by name). Contract 1 has no remaining operator divergence.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 8. This engine's own manifest — EMITTED
|
|
||||||
|
|
||||||
`docs/api-contract.json`, generated by `scripts/emit-contract-manifest.mjs`
|
|
||||||
from the BUILT surface: the prototype's own methods and accessors, the exported
|
|
||||||
error classes, the operator sets read out of their single definitions, the
|
|
||||||
field-addressing vocabulary read out of `src/db/fieldAddressing.ts`, and the
|
|
||||||
health verdicts. Nothing in it is hand-maintained, so a diff between the two
|
|
||||||
manifests is a diff between two engines rather than between two authors.
|
|
||||||
|
|
||||||
`node scripts/emit-contract-manifest.mjs --check` fails when the committed
|
|
||||||
manifest is stale — the announcement duty of item 5, made mechanical.
|
|
||||||
|
|
||||||
**What the manifest deliberately does NOT carry: requirement marking.** Whether
|
|
||||||
a door is required is a commitment, not a property of the surface; it is item 2
|
|
||||||
of this document. The diff the two sides want — "does Open Brainy still expose
|
|
||||||
every door contract 1 requires?" — is a set-membership check between their
|
|
||||||
`doors[].name` where `requirement === 'required'` and this manifest's
|
|
||||||
`doors[].name`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## A defect this work surfaced but did not fix
|
|
||||||
|
|
||||||
`src/vfs/VirtualFileSystem.ts` builds a path-prefix filter as
|
|
||||||
`path: { $startsWith: options.path }` — with a `$` prefix. No operator in this
|
|
||||||
engine carries a `$`, so `validateWhereFilter()` rejects it with
|
|
||||||
`INVALID_QUERY` before any index read: **`vfs.searchFiles({ path })` throws
|
|
||||||
today, on every call that passes a path.** It is pre-existing and unrelated to
|
|
||||||
the operator work above (the validator refuses it before the index path is
|
|
||||||
reached), and it is left as a filing rather than fixed here, because the right
|
|
||||||
answer is a design question — a path-prefix search cannot be served by an
|
|
||||||
equality/range index, so it needs either a path-segment index or an explicit
|
|
||||||
in-memory narrow, not a spelling correction.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Summary of what changed in code for this ratification
|
|
||||||
|
|
||||||
| item | change |
|
|
||||||
|---|---|
|
|
||||||
| 1 | `"brainyContract": 1` in package.json; `contractVersion()` / `BRAINY_CONTRACT_VERSION` exported |
|
|
||||||
| 3 | the accepted 25-token set asserted from the engine's own refusal message, and emitted into the manifest |
|
|
||||||
| 7 | `hasAll` / `noneOf` / `excludes` served on the index path; `startsWith` / `endsWith` / `matches` / `length` refused by name instead of answered empty |
|
|
||||||
| 8 | `scripts/emit-contract-manifest.mjs` + the generated `docs/api-contract.json`, with a `--check` mode |
|
|
||||||
|
|
@ -10,8 +10,8 @@
|
||||||
* between two engines, never between two authors.
|
* between two engines, never between two authors.
|
||||||
*
|
*
|
||||||
* Requirement marking (required / optional per door) is NOT derivable from the
|
* Requirement marking (required / optional per door) is NOT derivable from the
|
||||||
* surface; it is a commitment, and it lives in docs/contract-1-ratification.md.
|
* surface — it is a commitment, recorded with the contract's owner rather than
|
||||||
* This manifest carries the surface; that document carries the promise.
|
* here. This manifest carries the surface; the promise lives with the contract.
|
||||||
*
|
*
|
||||||
* Usage: node scripts/emit-contract-manifest.mjs [--check]
|
* Usage: node scripts/emit-contract-manifest.mjs [--check]
|
||||||
* --check exits non-zero when the committed manifest is stale.
|
* --check exits non-zero when the committed manifest is stale.
|
||||||
|
|
@ -68,7 +68,6 @@ const servedOnIndex = accepted.filter((op) => !refusedByIndex.includes(op))
|
||||||
const manifest = {
|
const manifest = {
|
||||||
contractVersion: versionModule.contractVersion(),
|
contractVersion: versionModule.contractVersion(),
|
||||||
engine: '@soulcraftlabs/brainy',
|
engine: '@soulcraftlabs/brainy',
|
||||||
prose: 'docs/contract-1-ratification.md',
|
|
||||||
compatibility: {
|
compatibility: {
|
||||||
minor:
|
minor:
|
||||||
'additive — a new optional door, a new served operator, a new error class; every existing implementation still conforms',
|
'additive — a new optional door, a new served operator, a new error class; every existing implementation still conforms',
|
||||||
|
|
|
||||||
Reference in a new issue