149 lines
5.3 KiB
Markdown
149 lines
5.3 KiB
Markdown
---
|
|
title: Storage Adapters
|
|
slug: guides/storage-adapters
|
|
public: true
|
|
category: guides
|
|
template: guide
|
|
order: 2
|
|
description: "Two adapters cover every deployment: in-memory for tests + ephemeral workloads, filesystem for everything that needs to persist. Both share one on-disk contract, including generational history and snapshots. Cloud backup is operator tooling, not a built-in adapter."
|
|
next:
|
|
- guides/plugins
|
|
- concepts/consistency-model
|
|
---
|
|
|
|
# Storage Adapters
|
|
|
|
Brainy 8.0 ships **two storage adapters**:
|
|
|
|
- **`FileSystemStorage`** — persistent on-disk storage. The default for any
|
|
deployment that needs to survive a restart. Runs on Node.js, Bun, and Deno.
|
|
- **`MemoryStorage`** — in-memory only. The right choice for tests, ephemeral
|
|
workloads, and short-lived demos.
|
|
|
|
Both implement the same `StorageAdapter` interface, support the full Db API
|
|
(generational history, snapshots, restore — see the
|
|
[consistency model](../concepts/consistency-model.md)), and use the same
|
|
on-disk layout (memory's "disk" is a JS Map).
|
|
|
|
## Quick start
|
|
|
|
```ts
|
|
import { Brainy } from '@soulcraft/brainy'
|
|
|
|
// Filesystem (recommended for any persistent workload):
|
|
const brain = new Brainy({
|
|
storage: { type: 'filesystem', rootDirectory: './brainy-data' }
|
|
})
|
|
|
|
// Memory (tests, ephemeral):
|
|
const brainMem = new Brainy({ storage: { type: 'memory' } })
|
|
|
|
// Auto-detect (filesystem on Node-like runtimes, memory in browsers):
|
|
const brainAuto = new Brainy({ storage: { type: 'auto' } })
|
|
```
|
|
|
|
## When to use which
|
|
|
|
| Use case | Adapter | Why |
|
|
|---|---|---|
|
|
| Production app | `filesystem` | Durable, snapshot-able, mmap-able |
|
|
| Tests, CI | `memory` | No disk teardown; fast |
|
|
| Short-lived data pipeline | `memory` | No persistence needed |
|
|
| In-browser demo | `memory` | Filesystem unavailable in browsers |
|
|
| Cloud deployment | `filesystem` on local disk + operator backup | See "Cloud backup" below |
|
|
|
|
## Cloud backup — operator tooling, not a built-in
|
|
|
|
Brainy 8.0 deliberately ships **no cloud storage adapters**. Cloud backup is
|
|
handled at the operator layer with standard tooling, the same pattern every
|
|
production database uses (Postgres, SQLite, Redis):
|
|
|
|
```bash
|
|
# After a brainy flush, sync the on-disk artefact to your cloud of choice.
|
|
gsutil rsync -r /var/lib/brainy gs://my-backup-bucket/brainy/
|
|
# or:
|
|
aws s3 sync /var/lib/brainy s3://my-backup-bucket/brainy/
|
|
# or:
|
|
rclone sync /var/lib/brainy remote:brainy-backups/
|
|
# or:
|
|
azcopy sync /var/lib/brainy "https://account.blob.core.windows.net/brainy?sv=..."
|
|
```
|
|
|
|
Brainy's filesystem layout is sync-friendly:
|
|
- Atomic writes (temp + rename) — readers never see torn files
|
|
- Per-shard files — `rsync`-style incremental sync works well
|
|
- Immutable generation records (`_generations/`) — append-only, cache-friendly
|
|
|
|
For point-in-time backups, take a filesystem snapshot (ZFS, btrfs, LVM, EBS,
|
|
etc.) or use `brain.now().persist(path)` to write a self-contained snapshot
|
|
you can sync independently of the live brain — see
|
|
[Snapshots & Time Travel](./snapshots-and-time-travel.md).
|
|
|
|
## Why no cloud adapters in 8.0?
|
|
|
|
Cloud storage adapters lived in Brainy 4.x-7.x. They were dropped in 8.0
|
|
because:
|
|
|
|
- Zero production consumers used them at scale — every known production
|
|
deployment ran on local filesystem.
|
|
- Cloud-storage HNSW / DiskANN doesn't perform — vector indexes need
|
|
low-latency random reads that S3 / GCS / R2 / Azure can't provide
|
|
consistently.
|
|
- Bundling cloud SDKs into the library cost ~3000-5000 LOC + 4-7 transitive
|
|
dependencies for a feature nobody used.
|
|
- Cloud backup via operator tooling is strictly more reliable than in-app
|
|
upload (better retry semantics, better observability, better cost control).
|
|
|
|
Brainy 8.0 is smaller, faster to install, and clearer about what it does.
|
|
|
|
## Configuration
|
|
|
|
```ts
|
|
// BrainyConfig['storage'] — either a config object or a pre-constructed adapter:
|
|
storage?:
|
|
| {
|
|
// The adapter type. Defaults to 'auto'
|
|
// (filesystem on Node-like runtimes, memory otherwise).
|
|
type: 'auto' | 'memory' | 'filesystem'
|
|
|
|
// Root directory for filesystem storage. Passed through to storage
|
|
// factories, including plugin-provided ones.
|
|
rootDirectory?: string
|
|
|
|
// Adapter-specific options.
|
|
options?: any
|
|
}
|
|
| StorageAdapter // e.g. storage: new MemoryStorage()
|
|
```
|
|
|
|
## Direct construction
|
|
|
|
If you want to skip the factory:
|
|
|
|
```ts
|
|
import { FileSystemStorage, MemoryStorage } from '@soulcraft/brainy'
|
|
|
|
const fsStorage = new FileSystemStorage('./brainy-data')
|
|
const memStorage = new MemoryStorage()
|
|
|
|
const brain = new Brainy({ storage: fsStorage })
|
|
```
|
|
|
|
## Migration from 7.x cloud adapters
|
|
|
|
7.x consumers of `OPFSStorage`, `GcsStorage`, `R2Storage`, `S3CompatibleStorage`,
|
|
or `AzureBlobStorage` need to migrate to `FileSystemStorage` plus operator
|
|
backup tooling. The recipe:
|
|
|
|
1. On the host running Brainy, mount a local disk (NVMe recommended). Cloud
|
|
providers all expose persistent local disks: GCP Persistent Disk, AWS EBS,
|
|
Azure Managed Disks.
|
|
2. Set `storage: { type: 'filesystem', rootDirectory: '/mnt/brainy-data' }`.
|
|
3. Run your existing data import once into the new local store.
|
|
4. Set up an operator backup job using `gsutil` / `aws s3` / `rclone` /
|
|
`azcopy` on a cron — hourly or whatever your RPO requires. Point it at
|
|
the brainy data dir.
|
|
5. For point-in-time backups, use filesystem snapshots or
|
|
`brain.now().persist(path)`.
|
|
|
|
Same data, same APIs, no library-side cloud code.
|