fix: an established store no longer boot-logs "New installation"

Every persisted 8.0 store logged "📁 New installation: using depth 1 sharding"
on every open — even brains holding thousands of entities — which is alarming to
read during a restart or incident. 8.0 stores nouns in the canonical
`entities/nouns/<shard>/<id>/vectors.json` layout (the path `saveNoun`/`getNouns`
use), but the legacy sharding probe `detectExistingShardingDepth()` inspects
`entities/nouns/hnsw` — a 7.x directory the 8.0 write path never populates (its
only writer, `saveNode`, is dead code with zero callers). So the probe returned
null for every 8.0 store and concluded "new".

Drive the new-vs-existing log from the layout the database actually reads and
writes: a new `hasCanonicalEntities()` checks for a real 2-hex shard directory
under `entities/nouns/`, and the known noun count short-circuits it. An
established store now logs "Using depth 1 sharding (N entities)"; only a genuinely
empty store reports a new installation. Behavior is otherwise unchanged — the
probe only ever set a log line, never triggered a rebuild or migration.
This commit is contained in:
David Snelling 2026-07-07 15:49:49 -07:00
parent 4341272c56
commit 308691603f
2 changed files with 151 additions and 2 deletions

View file

@ -265,8 +265,19 @@ export class FileSystemStorage extends BaseStorage {
console.log(`✅ Migration complete - now using depth ${this.SHARDING_DEPTH} sharding`)
} else if (detectedDepth === null) {
// New installation
console.log(`📁 New installation: using depth ${this.SHARDING_DEPTH} sharding (optimal for 1-2.5M entities)`)
// The legacy sharding probe inspects `entities/nouns/hnsw` — a 7.x
// directory the 8.0 write path never populates (8.0 stores entities in
// the canonical `entities/nouns/<shard>/<id>/` layout), so it returns
// null for EVERY 8.0 store, new or not. Decide new-vs-existing from the
// canonical layout the DB actually reads/writes so an established brain
// is not mislabeled "New installation" on every boot.
const established =
this.totalNounCount > 0 || (await this.hasCanonicalEntities())
console.log(
established
? `📁 Using depth ${this.SHARDING_DEPTH} sharding (${this.totalNounCount} entities)`
: `📁 New installation: using depth ${this.SHARDING_DEPTH} sharding (optimal for 1-2.5M entities)`
)
} else {
// Already using correct depth
console.log(`📁 Using depth ${this.SHARDING_DEPTH} sharding (${this.totalNounCount} entities)`)
@ -2737,6 +2748,39 @@ export class FileSystemStorage extends BaseStorage {
}
}
/**
* Whether this store already holds canonical 8.0 entities.
*
* 8.0 writes nouns to `entities/nouns/<shard>/<id>/vectors.json` (see
* `getNounVectorPath`), but the legacy sharding probe
* ({@link detectExistingShardingDepth}) inspects `entities/nouns/hnsw` a 7.x
* directory the 8.0 write path never populates. The probe therefore returns
* null for every 8.0 store and cannot tell an established brain from a fresh
* one, which mislabels established stores "New installation" on every boot.
* This checks the canonical shard tree the DB actually reads and writes (the
* same `entities/nouns/<hex>` shards `getNounsWithPagination` walks) so boot
* logs are truthful.
*
* @returns true if at least one 2-hex shard directory (00ff) exists under
* `entities/nouns/`, i.e. the store has previously persisted entities.
*/
private async hasCanonicalEntities(): Promise<boolean> {
const canonicalNounsDir = path.join(this.rootDir, 'entities', 'nouns')
try {
const entries = await fs.promises.readdir(canonicalNounsDir, {
withFileTypes: true
})
// A populated store has ≥1 two-hex shard dir (00ff). The vestigial
// `hnsw` subdir is 4 chars and correctly excluded by the hex test.
return entries.some(
(e: any) => e.isDirectory() && /^[0-9a-f]{2}$/i.test(e.name)
)
} catch {
// Directory absent (fresh store) or unreadable → no persisted entities.
return false
}
}
/**
* Get sharding depth
* Always returns 1 (single-level sharding) for optimal balance of