feat(open): the open narrates itself, on a channel production cannot clamp
An operator watched a production service open a 16 GB store and print nothing for three minutes before its first line of work. Two defects, both fixed here. The narration was written to `prodLog.warn`, and every environment that looks like production clamps the logger to ERROR — so the phase breakdown that would have named the slow phase was composed and thrown away. `prodLog.narrate` is always visible, like `error`: it carries the two things an operator is entitled to hear from a database regardless of a cost setting — why it is slow and what it is doing about it. `silent: true` still silences it; that is a request, not a default. And nothing spoke DURING a phase, only after the whole open. init() now runs an unref'd heartbeat that every 5s names the phase currently running, its elapsed wall and what it is paying for, plus one line per phase as it ends for any phase over 2s. The generation-log fold's own progress and completion lines move to the same channel and now carry their wall — they were invisible in production, which is how an operator came to restart a converging fold three times. Pins: tests/integration/open-narration.test.ts — narrate() survives the clamp that silences warn(); a 6.5s storage-init produces a heartbeat naming the phase and a completion line naming its wall, with the logger clamped to ERROR.
This commit is contained in:
parent
e652162c1f
commit
afe08a1ff9
4 changed files with 202 additions and 15 deletions
|
|
@ -652,6 +652,7 @@ export class GenerationStore {
|
|||
: 'WHOLE-LOG fold'
|
||||
: 'above-manifest replay'
|
||||
let replayed = 0
|
||||
const foldStartedAt = Date.now()
|
||||
const replayFact = async (fact: CommitFact): Promise<void> => {
|
||||
for (const op of fact.ops) {
|
||||
let image: { metadata: unknown | null; vector: unknown | null }
|
||||
|
|
@ -697,9 +698,10 @@ export class GenerationStore {
|
|||
}
|
||||
replayed++
|
||||
if (replayed % 1000 === 0) {
|
||||
prodLog.warn(
|
||||
prodLog.narrate(
|
||||
`[GenerationStore] recovery fold in progress — ${replayed} fact(s) folded ` +
|
||||
`(at generation ${fact.generation}); do not restart, the fold is finite`
|
||||
`in ${Date.now() - foldStartedAt}ms (at generation ${fact.generation}); ` +
|
||||
`do not restart, the fold is finite`
|
||||
)
|
||||
}
|
||||
if (fact.generation > this.committed) {
|
||||
|
|
@ -714,7 +716,7 @@ export class GenerationStore {
|
|||
}
|
||||
}
|
||||
if (uncleanOpen) {
|
||||
prodLog.warn(
|
||||
prodLog.narrate(
|
||||
`[GenerationStore] log-authority recovery: ${foldKind} beginning ` +
|
||||
`(unclean shutdown detected) — streaming replay, bounded memory, ` +
|
||||
`progress every 1000 facts. Do not restart the process; a restart ` +
|
||||
|
|
@ -737,9 +739,10 @@ export class GenerationStore {
|
|||
}
|
||||
await this.storage.writeRawObject(MANIFEST_PATH, manifest)
|
||||
await this.storage.syncRawObjects([MANIFEST_PATH])
|
||||
prodLog.warn(
|
||||
prodLog.narrate(
|
||||
`[GenerationStore] log-authority recovery replayed ${replayed} fact(s) into ` +
|
||||
`canonical (${foldKind}; committed at ${this.committed}) — an acked write is never lost`
|
||||
`canonical in ${Date.now() - foldStartedAt}ms (${foldKind}; committed at ` +
|
||||
`${this.committed}) — an acked write is never lost`
|
||||
)
|
||||
}
|
||||
// A recovery fold re-applied (and the barrier below re-syncs) every
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue