perf(generations): discover generations by directory name, not by walking the log
MEASURED on a production-shaped store (14,056 nouns / 72,679 verbs, an 11 GB
generation history), measured solo under an exclusive lock: the generation-store phase cost 55,538 ms of a WARM
REOPEN after a clean close — with the fold correctly skipped, so nothing in
that phase's name explained it.
This is what it was doing. Discovering which generations exist on disk called
listRawObjects('_generations'), which RECURSES the whole tree and returns
every file in every generation directory — to extract a set of integers that
the top-level directory NAMES already spell out. The cost scales with the
entire history, is paid on every open, warm or cold, and grows for the life of
the store.
A one-level door — listRawPrefixes(prefix), the immediate child directory
names — is added to the storage seam. The filesystem adapter answers it with a
single readdir; BaseStorage derives it from the recursive listing, so an
adapter without a cheap implementation is never wrong, only never faster; and
the generation store falls back to the old listing when the door is absent.
One behavioural difference, stated: an EMPTY generation directory is now
discovered where the file listing could not see it. Above the committed
watermark that is a crash scar, and recovery already has an explicit branch
for it ("indeterminate partial dir" — dropped, narrated). Below it, it becomes
a resolvable generation holding no records, which is what an empty generation
means.
Suites: the durability kill matrix (15), db-mvcc (30), history repacking (4),
rollback trapdoor (3), entity-tree stamp (4) and the full unit suite (2,105)
all green.
This commit is contained in:
parent
e4c27fbca8
commit
9dd399216b
4 changed files with 84 additions and 5 deletions
|
|
@ -686,6 +686,30 @@ export class FileSystemStorage extends BaseStorage {
|
|||
return pruned
|
||||
}
|
||||
|
||||
/**
|
||||
* @description The IMMEDIATE child directory names under a prefix — ONE
|
||||
* `readdir`, no recursion, no file paths. See the seam's JSDoc
|
||||
* (`src/db/types.ts`) for what this replaced: discovering the generations on
|
||||
* disk walked the entire generation log on every open, reading out every
|
||||
* file in every generation, to learn the set of integers the top-level
|
||||
* directory names already spell.
|
||||
* @param prefix - Storage-root-relative directory prefix.
|
||||
* @returns The child directory names (not paths); empty when the prefix does
|
||||
* not exist.
|
||||
*/
|
||||
public override async listRawPrefixes(prefix: string): Promise<string[]> {
|
||||
await this.ensureInitialized()
|
||||
const fullPath = path.join(this.rootDir, prefix)
|
||||
try {
|
||||
const entries = await fs.promises.readdir(fullPath, { withFileTypes: true })
|
||||
return entries.filter((e: { isDirectory: () => boolean }) => e.isDirectory())
|
||||
.map((e: { name: string }) => e.name)
|
||||
} catch (error: any) {
|
||||
if (error?.code === 'ENOENT') return []
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Primitive operation: List objects under path prefix
|
||||
* All metadata operations use this internally via base class routing
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue