fix: count recovery scans the canonical layout; remove the dead 7.x hnsw sharding machinery

Sweeping the vestigial hnsw sharding machinery surfaced two real defects on
the counts-recovery path (the code that rebuilds totalNounCount/totalVerbCount
when counts.json is lost or corrupted — container restarts, partial copies):

- initializeCountsFromDisk counted files in entities/*/hnsw/ — directories the
  8.0 write path never populates — so an established store recovered to ZERO
  counts on real data: wrong getNounCount()/stats, a "New installation"-style
  boot log, and a mis-sized rebuild-strategy decision at open. The scan now
  walks the canonical entities/<kind>/<shard>/<id>/ tree (one entity per id
  directory), with the same sampled type-distribution estimate as before.
- The sampler called getNounMetadata — whose ensureInitialized() re-enters
  init() — from INSIDE init(), a latent deadlock reachable the moment the scan
  found anything to sample. The sampled metadata files are now read directly
  with fs (gz-transparent), no guarded accessors inside init.

The dead machinery itself (verified zero callers by reachability analysis
before deletion): the nine 7.x hnsw-layout entity/edge CRUD methods, the
sharding-depth prober + its depth-migration engine (unreachable — the probe
always returned null on 8.0 stores), an orphaned streaming verb paginator,
their path/scan helpers, and the now-unused type aliases and fields. The init
block keeps the truthful new-vs-established boot log introduced in 8.0.13,
now decided directly from the canonical layout. Net -1,138 lines; no public
API touched.

Adds a regression test: delete counts.json from a populated store, reopen,
counts recover exactly from the canonical tree.
This commit is contained in:
David Snelling 2026-07-08 15:49:19 -07:00
parent 716a8513bf
commit 352e2da88f
2 changed files with 127 additions and 1150 deletions

File diff suppressed because it is too large Load diff

View file

@ -87,6 +87,29 @@ describe('FileSystemStorage boot log: an established brain is not "New installat
await teardown(second)
})
it('counts RECOVER from the canonical layout when counts.json is lost (container-restart case)', async () => {
const dir = mkTmp()
// Establish a store with 3 entities (canonical layout + counts.json).
const first: any = new FileSystemStorage(dir)
await first.init()
for (const c of ['a', 'b', 'c']) await seedOne(first, c.repeat(32))
await teardown(first)
// Simulate the lost/corrupted counts.json a container restart can leave.
for (const f of ['counts.json', 'counts.json.gz']) {
rmSync(join(dir, '_system', f), { force: true })
}
// Reopen: the recovery scan must count the CANONICAL tree. The removed
// implementation scanned the vestigial `entities/*/hnsw` dirs and
// re-initialized every counter to ZERO on real data.
const second: any = new FileSystemStorage(dir)
await second.init()
expect(await second.getNounCount()).toBe(3)
await teardown(second)
})
it('hasCanonicalEntities() sees the canonical shard tree — independent of counts.json', async () => {
const dir = mkTmp()
const s: any = new FileSystemStorage(dir)