David Snelling
298b572671
Introduce a first-class binary-blob storage primitive on the StorageAdapter contract and implement it across all storage backends. This stores opaque byte payloads verbatim instead of base64-in-JSON, eliminating the ~33% inflation and full-materialization cost of the JSON envelope. It unblocks zero-copy, mmap-able column-store segments and batch vector I/O at billion scale. New methods (declared abstract on BaseStorageAdapter, the class that implements StorageAdapter, and added to the StorageAdapter interface): saveBinaryBlob(key, data) raw write, atomic on real filesystems loadBinaryBlob(key) exact bytes, or null if absent deleteBinaryBlob(key) idempotent (missing is ignored) getBinaryBlobPath(key) real local fs path where one exists, else null Shared key -> location convention across every adapter: the key's "/"-separated segments nest under a `_blobs/` prefix and are suffixed with `.bin`, e.g. "graph-lsm/source/sstable-123" -> "<root>/_blobs/graph-lsm/source/sstable-123.bin". Blobs are not branch-scoped (COW): they are immutable producer-managed segments. Per-adapter behavior: - FileSystemStorage: writes under <rootDir>/_blobs via tmp+rename; returns the real on-disk path so native code can mmap it directly. Path convention matches the existing MmapFileSystemStorage subclass byte-for-byte. - S3CompatibleStorage / R2Storage / GcsStorage / AzureBlobStorage: put/get/delete raw octet-stream objects; getBinaryBlobPath returns null (remote stores have no local path). - MemoryStorage: defensive-copied Map<string, Buffer>; null path; cleared on clear(). - OPFSStorage: stores raw bytes in the OPFS tree; null path. - HistoricalStorageAdapter: read-only — save/delete throw; load resolves the blob from the historical commit tree; null path. Tests: tests/unit/storage/binaryBlob.test.ts exercises save/load round-trip (byte-identical, incl. non-UTF8 bytes), overwrite, delete-then-load, load-missing, and getBinaryBlobPath behavior for all eight adapters. Cloud adapters run against in-memory client fakes that drive the real adapter code; OPFS runs against an in-memory FileSystem Access API mock; the historical adapter commits a blob into a real COW tree. 59 new tests; full unit suite (1398 tests) green. |
||
|---|---|---|
| .. | ||
| aggregation | ||
| brainy | ||
| graph | ||
| hnsw | ||
| import | ||
| indexes/columnStore | ||
| neural | ||
| performance | ||
| storage | ||
| types | ||
| utils | ||
| versioning | ||
| vfs | ||
| brainy-core.unit.test.ts | ||
| brainy-get-optimization.test.ts | ||
| create-entities-default.test.ts | ||
| fieldTypeInference.test.ts | ||
| plugin.test.ts | ||
| type-filtering.unit.test.ts | ||
| vfs-restart-fix.test.ts | ||
| workshop-vfs-diagnostic.test.ts | ||