feat: generation-segment store — the D1+D3 packed-tier file format

First stage of the co-frozen D1+D3+repacking unit: the format core,
self-contained under _generations/segments/.

- seg-<firstGen,20pad>.bgs: append-once packs of consecutive
  generations (magic BGS1; frame = u32 len + u32 crc32c + msgpack
  [generation, timestamp, delta, records, flags]; flags reserves
  compressed-payload evolution without a format break). Sealed
  segments are immutable — fold refuses overlap with sealed ranges.
- seg-<firstGen>.idx: DERIVED sidecar (per-generation frame offsets +
  per-id generation postings + checksums); lost/corrupt sidecars
  rebuild from their segment loudly; a damaged segment (frame CRC
  mismatch) fails loudly, never serves wrong bytes.
- manifest.json: the one discovery path — open() reads it and never
  lists the packed backlog (the scan-wedge class's cure); refuses a
  newer manifest version rather than serving partial history.
- D3 semantics: dropSegmentsBelow reclaims WHOLE segments at
  boundaries only and bumps compactedBelow durably; archival-profile
  enforcement stays with the caller per the co-freeze.
- D8 rider: digestThroughPacked(g) — deterministic crc32c chain over
  sealed-segment checksums (+ frame-level prefix mid-segment),
  O(segments), reopen-stable.

Also fixes a cross-adapter contract bug the suite caught: memory
storage's deleteObjectFromPath ignored the raw-bytes store, so
deleteRawObject on a raw-bytes path (fact-log or segment files)
silently no-op'd — deletes now match filesystem unlink semantics.

Six pins. Wiring into GenerationStore (two-tier reads, the repacker,
cold-open manifest path) lands with the rest of the unit before its
release; cortex's fact-record/stamp shapes reconcile the sidecar
keying when they post.
This commit is contained in:
David Snelling 2026-07-19 15:14:27 -07:00
parent f8e6da2b66
commit d8acb3776b
3 changed files with 614 additions and 0 deletions

View file

@ -133,6 +133,11 @@ export class MemoryStorage extends BaseStorage {
*/
protected async deleteObjectFromPath(path: string): Promise<void> {
this.objectStore.delete(path)
// Filesystem parity: on disk, objects and raw BYTE files are both just
// files — unlink removes whichever exists. Without this, deleteRawObject
// on a raw-bytes path (fact-log/generation segments) silently no-ops on
// memory storage: the delete "succeeds" and the bytes remain.
this.rawBytesStore.delete(path)
}
/**