feat(storage): the canonical count ledger — ALL-visibility scalars, unclamped totals, suspect-on-unprovable-delete
The storage-level unfiltered getNouns()/getVerbs() walks enumerate every
tier, but their totalCount reported the user-facing scalar, which skips
system/internal records on the write path — so a derived-index coverage
ledger comparing its posted count against that total would read
"over-posted by N" on every store with a VFS. This adds the ledger's real
denominators:
- totalNounCountAll / totalVerbCountAll: +1 for every new canonical record
regardless of tier, −1 for every PROVEN delete (record read, or the
caller's prior image), persisted in counts.json beside the counted
scalars, recomputed by the sanctioned recount (rebuildTypeCounts).
- The unfiltered storage-level totalCount is now the ALL scalar and is
never clamped: Math.max(scalar, scanned) could only move a scalar up, so
an inflated counter hid forever; a divergence is now visible and healed
by repairIndex().
- A delete that cannot prove the record existed never decrements on faith:
it marks the ledger SUSPECT (persisted, narrated once per session) and
the recount clears the flag with proof.
- getCanonicalCounts() on StorageAdapter (optional) exposes {counted, all}
per family plus the suspect flag — O(1), no I/O.
- A counts.json written before the ledger existed derives both scalars
once from the canonical id tree at open and persists them; absent keys
are a legacy file, never a zero.
User-facing getNounCount()/getVerbCount() are unchanged.
Pinned in tests/integration/canonical-count-ledger.test.ts (5 laws).
This commit is contained in:
parent
607e9f5492
commit
7c8c8be30c
6 changed files with 391 additions and 15 deletions
|
|
@ -2561,6 +2561,33 @@ export class FileSystemStorage extends BaseStorage {
|
|||
this.totalNounCount = counts.totalNounCount || 0
|
||||
this.totalVerbCount = counts.totalVerbCount || 0
|
||||
|
||||
// The ALL-visibility scalars (ledger denominators). A counts.json
|
||||
// written before they existed carries neither key: derive both ONCE
|
||||
// from the canonical id tree (an id-directory listing — O(ids), no
|
||||
// record reads), persist, and never scan again. Absent keys are a
|
||||
// legacy file, not a zero — a zero here would make every provider's
|
||||
// coverage ledger read "over-posted" on a populated store.
|
||||
if (
|
||||
typeof counts.totalNounCountAll === 'number' &&
|
||||
typeof counts.totalVerbCountAll === 'number'
|
||||
) {
|
||||
this.totalNounCountAll = counts.totalNounCountAll
|
||||
this.totalVerbCountAll = counts.totalVerbCountAll
|
||||
this.allCountsSuspect = counts.allCountsSuspect === true
|
||||
} else {
|
||||
const nouns = await this.scanCanonicalEntities('nouns')
|
||||
const verbs = await this.scanCanonicalEntities('verbs')
|
||||
this.totalNounCountAll = nouns.count
|
||||
this.totalVerbCountAll = verbs.count
|
||||
this.allCountsSuspect = false
|
||||
console.warn(
|
||||
`[FileSystemStorage] counts.json predates the ALL-visibility count ledger — ` +
|
||||
`derived once from the canonical id tree (${nouns.count} nouns, ${verbs.count} verbs, ` +
|
||||
`every tier) and persisted; no further scan.`
|
||||
)
|
||||
await this.persistCounts()
|
||||
}
|
||||
|
||||
// Also populate the cache for backward compatibility
|
||||
this.countCache.set('nouns_count', {
|
||||
count: this.totalNounCount,
|
||||
|
|
@ -2596,6 +2623,10 @@ export class FileSystemStorage extends BaseStorage {
|
|||
this.totalNounCount = nouns.count
|
||||
const verbs = await this.scanCanonicalEntities('verbs')
|
||||
this.totalVerbCount = verbs.count
|
||||
// The id-tree scan counts every tier — it IS the ALL-visibility ledger.
|
||||
this.totalNounCountAll = nouns.count
|
||||
this.totalVerbCountAll = verbs.count
|
||||
this.allCountsSuspect = false
|
||||
|
||||
// Sample some entities for the type distribution (don't read all).
|
||||
// Read the metadata files DIRECTLY with fs — this runs inside init(),
|
||||
|
|
@ -2693,6 +2724,11 @@ export class FileSystemStorage extends BaseStorage {
|
|||
verbCounts: Object.fromEntries(this.verbCounts),
|
||||
totalNounCount: this.totalNounCount,
|
||||
totalVerbCount: this.totalVerbCount,
|
||||
// ALL-visibility ledger scalars (+ the suspect flag) — absent in files
|
||||
// written before the ledger existed; initializeCounts() derives them once.
|
||||
totalNounCountAll: this.totalNounCountAll,
|
||||
totalVerbCountAll: this.totalVerbCountAll,
|
||||
allCountsSuspect: this.allCountsSuspect,
|
||||
lastUpdated: new Date().toISOString()
|
||||
}
|
||||
|
||||
|
|
|
|||
Reference in a new issue