David Snelling
38b0041464
Every committed generation now also appends a FACT — an after-image commit record (what each touched entity/relationship became, or a body-less tombstone for a removal) — to an append-only, crc32c-framed segment log under _generations/facts/. The before-image history and the canonical tree remain authoritative; the fact log gives consumers ONE sequential, self-verifying stream (index heals, incremental replays) in place of a per-entity directory walk. - Wire format: positional msgpack facts [generation, timestamp, ops, meta, blobHashes]; op = [kind u8, id bin16, record | nil tombstone]; 32-byte segment header (magic, formatVersion, firstGeneration, zeroed+verified reserved); length+crc32c frame per fact; zero-padded segment names so lexicographic order == generation order; JSON manifest with an atomic rename flip, manifest-first rotation. - Commit protocol: facts append+fsync BEFORE the commit point inside the existing durability window, so a crash can only leave the log AHEAD of committed truth — open() truncates back (torn tails detected by CRC). Absent generation = never committed; a scan can never see an uncommitted fact. transact() facts are durable-on-return; single-op facts ride the group-commit flush exactly like buffered history. A fact-append failure fails the write, loudly — a silent gap would be a lie a later replay discovers. - New public surface: brain.scanFacts() (sequential batches with heal telemetry: head/segments/approx up front, per-batch generation range + bytes + segment id, loud abort on gaps, summary cross-check) and brain.factSegmentPaths() (immutable sealed segments for zero-copy consumers; the mutable tail excluded). Exported types CommitFact, FactOp, FactScanBatch, FactScanHandle. - Storage: optional binary raw-byte primitives (appendRawBytes, readRawBytes, writeRawBytes, rawByteSize) on StorageAdapter — feature-detected; filesystem + memory adapters implement them; an adapter without them hosts no fact log. Fact segments are byte-copied (never hard-linked) into snapshots. The _generations/facts/ namespace is registered as a protected family (rebuildable: false): no sweeper or GC may delete under it. - New crc32c (Castagnoli) utility with RFC known-answer tests.
43 lines
1.5 KiB
TypeScript
43 lines
1.5 KiB
TypeScript
/**
|
|
* @module utils/crc32c
|
|
* @description CRC-32C (Castagnoli, polynomial 0x1EDC6F41, reflected 0x82F63B78)
|
|
* — the storage-industry frame checksum (ext4, iSCSI, SCTP, LSM segment files).
|
|
* Used to frame generation-fact segments: every appended record carries the
|
|
* CRC-32C of its payload, so a torn tail (crash mid-append) or bit rot is
|
|
* DETECTED at scan time and never silently read as data.
|
|
*
|
|
* Table-driven, dependency-free reference implementation. Native providers may
|
|
* substitute a hardware-accelerated (SSE4.2 / ARMv8 CRC) implementation — the
|
|
* polynomial is the contract, byte-identical results required.
|
|
*/
|
|
|
|
/** The 256-entry lookup table for the reflected CRC-32C polynomial. */
|
|
const TABLE: Uint32Array = (() => {
|
|
const table = new Uint32Array(256)
|
|
for (let n = 0; n < 256; n++) {
|
|
let c = n
|
|
for (let k = 0; k < 8; k++) {
|
|
c = c & 1 ? 0x82f63b78 ^ (c >>> 1) : c >>> 1
|
|
}
|
|
table[n] = c >>> 0
|
|
}
|
|
return table
|
|
})()
|
|
|
|
/**
|
|
* Compute the CRC-32C checksum of a byte buffer.
|
|
*
|
|
* Known-answer vectors (RFC 3720 appendix / the standard test suite):
|
|
* - ASCII "123456789" → 0xE3069283
|
|
* - 32 zero bytes → 0x8A9136AA
|
|
*
|
|
* @param bytes - The payload to checksum.
|
|
* @returns The CRC-32C as an unsigned 32-bit integer.
|
|
*/
|
|
export function crc32c(bytes: Uint8Array): number {
|
|
let crc = 0xffffffff
|
|
for (let i = 0; i < bytes.length; i++) {
|
|
crc = TABLE[(crc ^ bytes[i]) & 0xff] ^ (crc >>> 8)
|
|
}
|
|
return (crc ^ 0xffffffff) >>> 0
|
|
}
|