feat(open): the open narrates itself, on a channel production cannot clamp
Some checks failed
CI / Node 22 (push) Has been cancelled
CI / Node 24 (push) Has been cancelled
CI / Integration + conformance (Node 22) (push) Has been cancelled
CI / Bun (latest) (push) Has been cancelled

An operator watched a production service open a 16 GB store and print nothing
for three minutes before its first line of work. Two defects, both fixed here.

The narration was written to `prodLog.warn`, and every environment that looks
like production clamps the logger to ERROR — so the phase breakdown that would
have named the slow phase was composed and thrown away. `prodLog.narrate` is
always visible, like `error`: it carries the two things an operator is
entitled to hear from a database regardless of a cost setting — why it is slow
and what it is doing about it. `silent: true` still silences it; that is a
request, not a default.

And nothing spoke DURING a phase, only after the whole open. init() now runs
an unref'd heartbeat that every 5s names the phase currently running, its
elapsed wall and what it is paying for, plus one line per phase as it ends for
any phase over 2s. The generation-log fold's own progress and completion lines
move to the same channel and now carry their wall — they were invisible in
production, which is how an operator came to restart a converging fold three
times.

Pins: tests/integration/open-narration.test.ts — narrate() survives the clamp
that silences warn(); a 6.5s storage-init produces a heartbeat naming the
phase and a completion line naming its wall, with the logger clamped to ERROR.
This commit is contained in:
David Snelling 2026-08-28 10:19:55 -07:00
parent e652162c1f
commit afe08a1ff9
4 changed files with 202 additions and 15 deletions

View file

@ -652,6 +652,7 @@ export class GenerationStore {
: 'WHOLE-LOG fold'
: 'above-manifest replay'
let replayed = 0
const foldStartedAt = Date.now()
const replayFact = async (fact: CommitFact): Promise<void> => {
for (const op of fact.ops) {
let image: { metadata: unknown | null; vector: unknown | null }
@ -697,9 +698,10 @@ export class GenerationStore {
}
replayed++
if (replayed % 1000 === 0) {
prodLog.warn(
prodLog.narrate(
`[GenerationStore] recovery fold in progress — ${replayed} fact(s) folded ` +
`(at generation ${fact.generation}); do not restart, the fold is finite`
`in ${Date.now() - foldStartedAt}ms (at generation ${fact.generation}); ` +
`do not restart, the fold is finite`
)
}
if (fact.generation > this.committed) {
@ -714,7 +716,7 @@ export class GenerationStore {
}
}
if (uncleanOpen) {
prodLog.warn(
prodLog.narrate(
`[GenerationStore] log-authority recovery: ${foldKind} beginning ` +
`(unclean shutdown detected) — streaming replay, bounded memory, ` +
`progress every 1000 facts. Do not restart the process; a restart ` +
@ -737,9 +739,10 @@ export class GenerationStore {
}
await this.storage.writeRawObject(MANIFEST_PATH, manifest)
await this.storage.syncRawObjects([MANIFEST_PATH])
prodLog.warn(
prodLog.narrate(
`[GenerationStore] log-authority recovery replayed ${replayed} fact(s) into ` +
`canonical (${foldKind}; committed at ${this.committed}) — an acked write is never lost`
`canonical in ${Date.now() - foldStartedAt}ms (${foldKind}; committed at ` +
`${this.committed}) — an acked write is never lost`
)
}
// A recovery fold re-applied (and the barrier below re-syncs) every