perf(generations): discover generations by directory name, not by walking the log
MEASURED on a production-shaped store (14,056 nouns / 72,679 verbs, an 11 GB
generation history), measured solo under an exclusive lock: the generation-store phase cost 55,538 ms of a WARM
REOPEN after a clean close — with the fold correctly skipped, so nothing in
that phase's name explained it.
This is what it was doing. Discovering which generations exist on disk called
listRawObjects('_generations'), which RECURSES the whole tree and returns
every file in every generation directory — to extract a set of integers that
the top-level directory NAMES already spell out. The cost scales with the
entire history, is paid on every open, warm or cold, and grows for the life of
the store.
A one-level door — listRawPrefixes(prefix), the immediate child directory
names — is added to the storage seam. The filesystem adapter answers it with a
single readdir; BaseStorage derives it from the recursive listing, so an
adapter without a cheap implementation is never wrong, only never faster; and
the generation store falls back to the old listing when the door is absent.
One behavioural difference, stated: an EMPTY generation directory is now
discovered where the file listing could not see it. Above the committed
watermark that is a crash scar, and recovery already has an explicit branch
for it ("indeterminate partial dir" — dropped, narrated). Below it, it becomes
a resolvable generation holding no records, which is what an empty generation
means.
Suites: the durability kill matrix (15), db-mvcc (30), history repacking (4),
rollback trapdoor (3), entity-tree stamp (4) and the full unit suite (2,105)
all green.
This commit is contained in:
parent
e4c27fbca8
commit
9dd399216b
4 changed files with 84 additions and 5 deletions
|
|
@ -1437,6 +1437,29 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
return this.listObjectsUnderPath(prefix)
|
||||
}
|
||||
|
||||
/**
|
||||
* @description The IMMEDIATE child directory names under a prefix — one
|
||||
* level, no recursion. See the seam's JSDoc (`db/types.ts`) for why a
|
||||
* separate door exists. This default derives them from the recursive
|
||||
* listing, so it is never WRONG, only never faster; the filesystem adapter
|
||||
* overrides it with a single directory read.
|
||||
* @param prefix - Storage-root-relative directory prefix.
|
||||
* @returns The child directory names (not paths), in listing order.
|
||||
*/
|
||||
public async listRawPrefixes(prefix: string): Promise<string[]> {
|
||||
await this.ensureInitialized()
|
||||
const paths = await this.listObjectsUnderPath(prefix)
|
||||
const normalizedPrefix = prefix.endsWith('/') ? prefix : `${prefix}/`
|
||||
const names = new Set<string>()
|
||||
for (const p of paths) {
|
||||
const rest = p.startsWith(normalizedPrefix) ? p.slice(normalizedPrefix.length) : null
|
||||
if (rest === null) continue
|
||||
const slash = rest.search(/[/\\]/)
|
||||
if (slash > 0) names.add(rest.slice(0, slash))
|
||||
}
|
||||
return [...names]
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove every object under a storage-root-relative prefix. The filesystem
|
||||
* adapter overrides this with a recursive directory removal; this default
|
||||
|
|
|
|||
Reference in a new issue