feat(storage): the canonical count ledger — ALL-visibility scalars, unclamped totals, suspect-on-unprovable-delete
Some checks failed
CI / Node 22 (push) Has been cancelled
CI / Node 24 (push) Has been cancelled
CI / Integration + conformance (Node 22) (push) Has been cancelled
CI / Bun (latest) (push) Has been cancelled

The storage-level unfiltered getNouns()/getVerbs() walks enumerate every
tier, but their totalCount reported the user-facing scalar, which skips
system/internal records on the write path — so a derived-index coverage
ledger comparing its posted count against that total would read
"over-posted by N" on every store with a VFS. This adds the ledger's real
denominators:

- totalNounCountAll / totalVerbCountAll: +1 for every new canonical record
  regardless of tier, −1 for every PROVEN delete (record read, or the
  caller's prior image), persisted in counts.json beside the counted
  scalars, recomputed by the sanctioned recount (rebuildTypeCounts).
- The unfiltered storage-level totalCount is now the ALL scalar and is
  never clamped: Math.max(scalar, scanned) could only move a scalar up, so
  an inflated counter hid forever; a divergence is now visible and healed
  by repairIndex().
- A delete that cannot prove the record existed never decrements on faith:
  it marks the ledger SUSPECT (persisted, narrated once per session) and
  the recount clears the flag with proof.
- getCanonicalCounts() on StorageAdapter (optional) exposes {counted, all}
  per family plus the suspect flag — O(1), no I/O.
- A counts.json written before the ledger existed derives both scalars
  once from the canonical id tree at open and persists them; absent keys
  are a legacy file, never a zero.

User-facing getNounCount()/getVerbCount() are unchanged.

Pinned in tests/integration/canonical-count-ledger.test.ts (5 laws).
This commit is contained in:
David Snelling 2026-08-24 09:49:29 -07:00
parent 607e9f5492
commit 7c8c8be30c
6 changed files with 391 additions and 15 deletions

View file

@ -2561,6 +2561,33 @@ export class FileSystemStorage extends BaseStorage {
this.totalNounCount = counts.totalNounCount || 0
this.totalVerbCount = counts.totalVerbCount || 0
// The ALL-visibility scalars (ledger denominators). A counts.json
// written before they existed carries neither key: derive both ONCE
// from the canonical id tree (an id-directory listing — O(ids), no
// record reads), persist, and never scan again. Absent keys are a
// legacy file, not a zero — a zero here would make every provider's
// coverage ledger read "over-posted" on a populated store.
if (
typeof counts.totalNounCountAll === 'number' &&
typeof counts.totalVerbCountAll === 'number'
) {
this.totalNounCountAll = counts.totalNounCountAll
this.totalVerbCountAll = counts.totalVerbCountAll
this.allCountsSuspect = counts.allCountsSuspect === true
} else {
const nouns = await this.scanCanonicalEntities('nouns')
const verbs = await this.scanCanonicalEntities('verbs')
this.totalNounCountAll = nouns.count
this.totalVerbCountAll = verbs.count
this.allCountsSuspect = false
console.warn(
`[FileSystemStorage] counts.json predates the ALL-visibility count ledger — ` +
`derived once from the canonical id tree (${nouns.count} nouns, ${verbs.count} verbs, ` +
`every tier) and persisted; no further scan.`
)
await this.persistCounts()
}
// Also populate the cache for backward compatibility
this.countCache.set('nouns_count', {
count: this.totalNounCount,
@ -2596,6 +2623,10 @@ export class FileSystemStorage extends BaseStorage {
this.totalNounCount = nouns.count
const verbs = await this.scanCanonicalEntities('verbs')
this.totalVerbCount = verbs.count
// The id-tree scan counts every tier — it IS the ALL-visibility ledger.
this.totalNounCountAll = nouns.count
this.totalVerbCountAll = verbs.count
this.allCountsSuspect = false
// Sample some entities for the type distribution (don't read all).
// Read the metadata files DIRECTLY with fs — this runs inside init(),
@ -2693,6 +2724,11 @@ export class FileSystemStorage extends BaseStorage {
verbCounts: Object.fromEntries(this.verbCounts),
totalNounCount: this.totalNounCount,
totalVerbCount: this.totalVerbCount,
// ALL-visibility ledger scalars (+ the suspect flag) — absent in files
// written before the ledger existed; initializeCounts() derives them once.
totalNounCountAll: this.totalNounCountAll,
totalVerbCountAll: this.totalVerbCountAll,
allCountsSuspect: this.allCountsSuspect,
lastUpdated: new Date().toISOString()
}