feat(storage): add raw binary-blob primitive to every storage adapter

Introduce a first-class binary-blob storage primitive on the StorageAdapter
contract and implement it across all storage backends. This stores opaque byte
payloads verbatim instead of base64-in-JSON, eliminating the ~33% inflation and
full-materialization cost of the JSON envelope. It unblocks zero-copy,
mmap-able column-store segments and batch vector I/O at billion scale.

New methods (declared abstract on BaseStorageAdapter, the class that implements
StorageAdapter, and added to the StorageAdapter interface):

  saveBinaryBlob(key, data)    raw write, atomic on real filesystems
  loadBinaryBlob(key)          exact bytes, or null if absent
  deleteBinaryBlob(key)        idempotent (missing is ignored)
  getBinaryBlobPath(key)       real local fs path where one exists, else null

Shared key -> location convention across every adapter: the key's
"/"-separated segments nest under a `_blobs/` prefix and are suffixed with
`.bin`, e.g. "graph-lsm/source/sstable-123" ->
"<root>/_blobs/graph-lsm/source/sstable-123.bin". Blobs are not branch-scoped
(COW): they are immutable producer-managed segments.

Per-adapter behavior:
- FileSystemStorage: writes under <rootDir>/_blobs via tmp+rename; returns the
  real on-disk path so native code can mmap it directly. Path convention matches
  the existing MmapFileSystemStorage subclass byte-for-byte.
- S3CompatibleStorage / R2Storage / GcsStorage / AzureBlobStorage: put/get/delete
  raw octet-stream objects; getBinaryBlobPath returns null (remote stores have no
  local path).
- MemoryStorage: defensive-copied Map<string, Buffer>; null path; cleared on
  clear().
- OPFSStorage: stores raw bytes in the OPFS tree; null path.
- HistoricalStorageAdapter: read-only — save/delete throw; load resolves the
  blob from the historical commit tree; null path.

Tests: tests/unit/storage/binaryBlob.test.ts exercises save/load round-trip
(byte-identical, incl. non-UTF8 bytes), overwrite, delete-then-load, load-missing,
and getBinaryBlobPath behavior for all eight adapters. Cloud adapters run against
in-memory client fakes that drive the real adapter code; OPFS runs against an
in-memory FileSystem Access API mock; the historical adapter commits a blob into
a real COW tree. 59 new tests; full unit suite (1398 tests) green.
This commit is contained in:
David Snelling 2026-05-27 11:49:49 -07:00
parent 547721ae14
commit 298b572671
12 changed files with 1552 additions and 0 deletions

View file

@ -725,6 +725,110 @@ export class R2Storage extends BaseStorage {
}
}
// ===========================================================================
// Raw binary-blob primitive
// ===========================================================================
/**
* Map a blob key to its R2 object key under the shared `_blobs/` prefix, e.g.
* `"graph-lsm/source/sstable-123"` `"_blobs/graph-lsm/source/sstable-123.bin"`.
*
* @param key - The blob key.
* @returns The R2 object key for the blob.
* @private
*/
private blobObjectKey(key: string): string {
return `_blobs/${key}.bin`
}
/**
* Store a raw binary blob as an R2 object, writing the bytes verbatim with
* `application/octet-stream` content type. Overwrites any existing blob at the
* same key.
*
* @param key - The blob key.
* @param data - The exact bytes to store.
*/
public async saveBinaryBlob(key: string, data: Buffer): Promise<void> {
await this.ensureInitialized()
const { PutObjectCommand } = await import('@aws-sdk/client-s3')
await this.s3Client!.send(
new PutObjectCommand({
Bucket: this.bucketName,
Key: this.blobObjectKey(key),
Body: data,
ContentType: 'application/octet-stream'
})
)
}
/**
* Load the raw bytes of the R2 blob object for `key`, or `null` if it does not
* exist.
*
* @param key - The blob key.
* @returns The blob bytes, or `null` if absent.
*/
public async loadBinaryBlob(key: string): Promise<Buffer | null> {
await this.ensureInitialized()
try {
const { GetObjectCommand } = await import('@aws-sdk/client-s3')
const response = await this.s3Client!.send(
new GetObjectCommand({
Bucket: this.bucketName,
Key: this.blobObjectKey(key)
})
)
if (!response || !response.Body) {
return null
}
const bytes = await response.Body.transformToByteArray()
return Buffer.from(bytes)
} catch (error: any) {
if (error.name === 'NoSuchKey' || error.$metadata?.httpStatusCode === 404) {
return null
}
throw BrainyError.fromError(error, `loadBinaryBlob(${key})`)
}
}
/**
* Delete the R2 blob object for `key`. Missing objects are ignored.
*
* @param key - The blob key.
*/
public async deleteBinaryBlob(key: string): Promise<void> {
await this.ensureInitialized()
try {
const { DeleteObjectCommand } = await import('@aws-sdk/client-s3')
await this.s3Client!.send(
new DeleteObjectCommand({
Bucket: this.bucketName,
Key: this.blobObjectKey(key)
})
)
} catch (error: any) {
if (error.name === 'NoSuchKey' || error.$metadata?.httpStatusCode === 404) {
return
}
throw new Error(`Failed to delete blob ${key}: ${error}`)
}
}
/**
* R2 is a remote object store with no local filesystem path to mmap, so this
* always returns `null`. Callers must use {@link loadBinaryBlob} instead.
*
* @param _key - The blob key (unused).
* @returns Always `null`.
*/
public getBinaryBlobPath(_key: string): string | null {
return null
}
/**
* List all objects under a specific prefix in R2
*/