fix: honest counters — removal never re-reads the removed record + repairIndex recounts and persists all rollups

Persisted entity totals inflated permanently: a delete's count decrement
was sourced from re-reading the record being removed, so a null read
(replace race, or a ghost left by a pre-8.3.1 partial delete) silently
skipped the decrement while the paired add had counted — and
Math.max(persistedTotal, scanned) meant no disk cleanup could ever
lower the reported total. Reported by a production deployment with a
write→delete→re-create repro minting drift within hours.

- The caller's pre-delete read now rides through the entire delete path
  (remove()/removeMany()/plan → DeleteNoun/DeleteVerb operations →
  deleteNoun/deleteVerb → deleteNounMetadata/deleteVerbMetadata): a
  null internal read falls back to the known prior record instead of
  skipping the decrement. StorageAdapter.deleteNoun/deleteVerb gain an
  optional priorMetadata parameter (additive).
- rebuildTypeCounts() rebuilt only the type-statistics arrays and
  computed the total just to LOG it — the persisted scalar
  (counts.json) survived every rebuild. One canonical walk now rebuilds
  every counter rollup (scalar totals + per-type maps + type
  statistics) and persists them; repairIndex() runs the recount
  unconditionally, since counters can be inflated over clean shelves.
This commit is contained in:
David Snelling 2026-07-14 11:51:44 -07:00
parent d9fa3be648
commit 2e2ba9c9aa
5 changed files with 237 additions and 28 deletions

View file

@ -127,22 +127,33 @@ export class DeleteNounMetadataOperation implements Operation {
constructor(
private readonly storage: StorageAdapter,
private readonly id: string
private readonly id: string,
/**
* OPTIONAL already-known metadata of the entity being removed (the caller's
* pre-delete read). Removal must never REQUIRE re-reading the thing being
* removed: if the reads here return null (replace race, or a ghost left by
* an earlier version), the count decrement downstream falls back to this
* record instead of being silently skipped the skip minted permanent
* counter inflation (adds counted, paired removals not decremented).
*/
private readonly priorMetadata?: NounMetadata | null
) {}
async execute(): Promise<RollbackAction> {
// Capture the FULL before-image (both legs) so the undo restores the whole
// entity — a metadata-only rollback would leave the vector leg unrestored.
// A null metadata read falls back to the caller's pre-delete read.
const previousNoun = await this.storage.getNoun(this.id)
const previousMetadata = await this.storage.getNounMetadata(this.id)
const previousMetadata = (await this.storage.getNounMetadata(this.id)) ?? this.priorMetadata ?? null
if (!previousNoun && !previousMetadata) {
// Nothing to delete - no rollback needed
return async () => {}
}
// Full removal: both canonical legs + the entity container + count decrement.
await this.storage.deleteNoun(this.id)
// Full removal: both canonical legs + the entity container + count decrement
// (the prior record keeps the decrement honest on a null canonical read).
await this.storage.deleteNoun(this.id, previousMetadata)
// Return rollback action
return async () => {
@ -268,9 +279,9 @@ export class DeleteVerbMetadataOperation implements Operation {
return async () => {}
}
// Delete verb (metadata + vector)
// Note: StorageAdapter has deleteVerb but not deleteVerbMetadata
await this.storage.deleteVerb(this.id)
// Delete verb (metadata + vector). The pre-read rides along so the count
// decrement never depends on re-reading the record being removed.
await this.storage.deleteVerb(this.id, previousMetadata)
// Return rollback action
return async () => {