feat: portable graph export()/import() (BackupData v1) on brain.data()
brain.data() now serializes part or all of a brain to a versioned, portable
JSON document (BackupData) and restores it — the path for partial backups,
cross-environment moves, and 7.x→8.0 migration.
- export(selector?, options?): select by ids, collection (+transitive Contains),
connected neighbourhood, vfsPath subtree, predicate, or whole brain; structural
and predicate selectors compose. Options: includeVectors, includeContent (VFS
blobs), includeSystem, edges ('induced'|'incident'|'none').
- import(backup, options?): onConflict 'merge' (dedup-by-id) | 'replace' | 'skip';
reembed 'auto' (re-embed from data when no vector carried) | 'never'; remapIds
to clone a subgraph under fresh ids.
- BackupData v1: format/formatVersion/brainyVersion/createdAt/embedding/entities/
relations/blobs?/danglingIds?/stats. Reserved fields (subtype, data, confidence,
weight, service) top-level; metadata custom-only. Current-state, no generations.
- Replaces the prior flat-entity export() (dropped relations) with a graph-complete
document. Distinct from brain.import(file) ingestion, which is unchanged.
- Export BackupData/BackupEntity/BackupRelation/ExportSelector/ExportOptions/
ImportOptions/ImportResult/DataAPI from the package root. CLI `brainy export`
now writes a BackupData document.
Guide: docs/guides/backup-and-export.md. Tests: tests/unit/api/data-backup.test.ts.
This commit is contained in:
parent
89c6d043ba
commit
a408d3799d
11 changed files with 1393 additions and 382 deletions
47
RELEASES.md
47
RELEASES.md
|
|
@ -10,6 +10,53 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## v7.32.0 — 2026-06-16
|
||||||
|
|
||||||
|
**Affected products:** anyone who needs a **portable graph backup**, a partial export, or a
|
||||||
|
**7.x → 8.0 migration path**. Purely additive — no API removed, no data change. Drop-in.
|
||||||
|
|
||||||
|
### New — portable graph `export()` / `import()` on `brain.data()`
|
||||||
|
|
||||||
|
`brain.data()` gains a portable backup/restore pair that serializes part or all of a brain
|
||||||
|
to a single versioned JSON document (`BackupData`) and restores it. This is the format that
|
||||||
|
embeds inside portable artifacts (e.g. a serialized subgraph carried by a higher-level file
|
||||||
|
type) and the recommended way to move a graph between environments or **upgrade an existing
|
||||||
|
7.x brain into an 8.0 brain**.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const data = await brain.data()
|
||||||
|
|
||||||
|
const backup = await data.export() // whole brain → BackupData
|
||||||
|
const subset = await data.export({ ids }, { includeVectors: true })
|
||||||
|
await otherBrain.data().then(d => d.import(subset, { onConflict: 'merge' }))
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Selectors** — `export(selector?, options?)` picks the node set: `{ ids }`, `{ collection }`
|
||||||
|
(alias `memberOf`) + transitive `Contains` members, `{ connected: { from, depth, verbs?, direction? } }`,
|
||||||
|
`{ vfsPath, recursive? }`, a predicate (`{ type, subtype, where, service }`), or the whole brain
|
||||||
|
(omit). Structural + predicate selectors **compose**.
|
||||||
|
- **Options** — `includeVectors` (default off → re-embed on import), `includeContent` (VFS file
|
||||||
|
bytes in `blobs`), `includeSystem` (include the VFS root etc.), `edges` (`'induced'` default /
|
||||||
|
`'incident'` / `'none'`).
|
||||||
|
- **Import** — `import(backup, options?)` → `{ imported, merged, skipped, reembedded, blobsWritten, errors }`.
|
||||||
|
`onConflict` defaults to `'merge'` (dedup-by-id, so you can assemble many documents into one
|
||||||
|
graph); `reembed` defaults to `'auto'` (re-embed from `data` when no vector is carried);
|
||||||
|
`remapIds` clones a subgraph under fresh ids.
|
||||||
|
- **Format** — `{ format:'brainy-backup', formatVersion:1, brainyVersion, createdAt, embedding,
|
||||||
|
entities, relations, blobs?, danglingIds?, stats }`. Entities carry `subtype` and the standard
|
||||||
|
reserved fields at the top level; `metadata` is custom-only. **Current-state** (no generation
|
||||||
|
history). `formatVersion` gates cross-version import (a 7.x document imports into 8.0).
|
||||||
|
- **Types** — `BackupData`, `BackupEntity`, `BackupRelation`, `ExportSelector`, `ExportOptions`,
|
||||||
|
`ImportOptions`, `ImportResult`, and the `DataAPI` class are now exported from the package root.
|
||||||
|
|
||||||
|
This replaces the previous `brain.data().export()` flat-entity array (which dropped relations)
|
||||||
|
with a graph-complete document. Distinct from `brain.import(file)` (CSV/PDF/Excel/JSON ingestion),
|
||||||
|
which is unchanged. The CLI `brainy export` now writes a `BackupData` document (json/jsonl/csv).
|
||||||
|
|
||||||
|
Guide: `docs/guides/export-and-import.md`. Tested in `tests/unit/api/data-backup.test.ts`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## v7.31.8 — 2026-06-16
|
## v7.31.8 — 2026-06-16
|
||||||
|
|
||||||
**Affected products:** every consumer on a bare VM or `mmap-filesystem` storage — **upgrade recommended** (Memory, Workshop on 7.31.x; Venue on 7.28.x). Two platform-wide production fixes. Drop-in; no API or data changes.
|
**Affected products:** every consumer on a bare VM or `mmap-filesystem` storage — **upgrade recommended** (Memory, Workshop on 7.31.x; Venue on 7.28.x). Two platform-wide production fixes. Drop-in; no API or data changes.
|
||||||
|
|
|
||||||
|
|
@ -1765,20 +1765,33 @@ await brain.import('https://api.example.com/data.json')
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### Export & Snapshots
|
### Export & Import (portable backup)
|
||||||
|
|
||||||
|
`brain.data()` exposes a portable graph backup/restore API. `export(selector?, options?)`
|
||||||
|
serializes part or all of the graph to a versioned, portable `BackupData` document;
|
||||||
|
`import(backup, options?)` restores it (dedup-by-id merge by default, re-embedding when
|
||||||
|
vectors are absent).
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// Export to file
|
const data = await brain.data()
|
||||||
await brain.export('/path/to/backup.brainy')
|
|
||||||
|
|
||||||
// Create instant snapshot using COW fork
|
// Whole brain → a portable BackupData document
|
||||||
await brain.fork('backup-2025-01-19')
|
const backup = await data.export()
|
||||||
|
|
||||||
// Time-travel to specific commit
|
// Just one workbench's members, with vectors, then restore elsewhere (merge by id)
|
||||||
const snapshot = await brain.asOf(commitId)
|
const subset = await data.export({ ids }, { includeVectors: true })
|
||||||
const entities = await snapshot.find({ limit: 100 })
|
await otherBrain.data().then(d => d.import(subset, { onConflict: 'merge' }))
|
||||||
|
|
||||||
|
// A collection + its children; a connected neighbourhood; a VFS subtree (+ bytes)
|
||||||
|
await data.export({ collection: collectionId })
|
||||||
|
await data.export({ connected: { from: id, depth: 2 } })
|
||||||
|
await data.export({ vfsPath: '/docs' }, { includeContent: true })
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The format is versioned (`formatVersion`) and current-state (no generation history) — see
|
||||||
|
the **[Export & Import guide →](../guides/export-and-import.md)**. This is distinct from
|
||||||
|
`brain.import(file)` (CSV/PDF/Excel/JSON ingestion).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
|
||||||
|
|
@ -457,23 +457,19 @@ await brain.storage.withLock('resource-id', async () => {
|
||||||
|
|
||||||
### Export Data
|
### Export Data
|
||||||
```typescript
|
```typescript
|
||||||
// Export entire database
|
// Export the whole brain to a portable BackupData document
|
||||||
const backup = await brain.export({
|
const backup = await brain.data().then(d => d.export(undefined, { includeVectors: true }))
|
||||||
format: 'json',
|
|
||||||
includeVectors: true,
|
|
||||||
includeIndexes: false
|
|
||||||
})
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Import Data
|
### Import Data
|
||||||
```typescript
|
```typescript
|
||||||
// Import from backup
|
// Restore a BackupData document (dedup-by-id merge by default)
|
||||||
await brain.import(backup, {
|
await brain.data().then(d => d.import(backup, { onConflict: 'merge' }))
|
||||||
mode: 'merge', // or 'replace'
|
|
||||||
validateSchema: true
|
|
||||||
})
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
See the [Export & Import guide](../guides/export-and-import.md) for partial exports
|
||||||
|
(by id, collection, connected neighbourhood, VFS subtree, or predicate).
|
||||||
|
|
||||||
### Storage Migration
|
### Storage Migration
|
||||||
```typescript
|
```typescript
|
||||||
// Migrate between storage types
|
// Migrate between storage types
|
||||||
|
|
@ -483,9 +479,9 @@ const newBrain = new Brainy({ storage: { type: 's3' } })
|
||||||
await oldBrain.init()
|
await oldBrain.init()
|
||||||
await newBrain.init()
|
await newBrain.init()
|
||||||
|
|
||||||
// Transfer all data
|
// Transfer all data via a portable backup
|
||||||
const data = await oldBrain.export()
|
const data = await oldBrain.data().then(d => d.export(undefined, { includeVectors: true }))
|
||||||
await newBrain.import(data)
|
await newBrain.data().then(d => d.import(data))
|
||||||
```
|
```
|
||||||
|
|
||||||
## Performance Tuning
|
## Performance Tuning
|
||||||
|
|
|
||||||
|
|
@ -546,11 +546,9 @@ const appComponent = await brain.findOne({ type: 'ReactComponent', name: 'App' }
|
||||||
const imports = await brain.getRelated(appComponent.id, 'Imports')
|
const imports = await brain.getRelated(appComponent.id, 'Imports')
|
||||||
console.log(`App component imports:`, imports.map(c => c.name))
|
console.log(`App component imports:`, imports.map(c => c.name))
|
||||||
|
|
||||||
// Export diagram
|
// Export the analyzed component graph as a portable BackupData document
|
||||||
const diagram = await brain.export({
|
const graph = await brain.data().then(d => d.export({ type: 'ReactComponent' }))
|
||||||
format: 'react-diagram'
|
console.log(`Exported ${graph.stats.entityCount} components, ${graph.stats.relationCount} edges`)
|
||||||
})
|
|
||||||
console.log(diagram.diagram) // Mermaid diagram
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Step 4: Premium Licensing (Optional)
|
### Step 4: Premium Licensing (Optional)
|
||||||
|
|
|
||||||
173
docs/guides/export-and-import.md
Normal file
173
docs/guides/export-and-import.md
Normal file
|
|
@ -0,0 +1,173 @@
|
||||||
|
---
|
||||||
|
title: Export & Import (portable graph)
|
||||||
|
slug: guides/export-and-import
|
||||||
|
public: true
|
||||||
|
category: guides
|
||||||
|
template: guide
|
||||||
|
order: 9
|
||||||
|
description: Export part or all of a brain to a portable, versioned BackupData document and import it back — by id, collection, connected neighbourhood, VFS subtree, predicate, or the whole brain. Covers vectors, edge policy, conflict handling, and id remapping.
|
||||||
|
next:
|
||||||
|
- guides/subtypes-and-facets
|
||||||
|
- api/README
|
||||||
|
---
|
||||||
|
|
||||||
|
# Export & Import (portable graph)
|
||||||
|
|
||||||
|
`brain.data()` exposes a **portable graph export/import** API. One method serializes a
|
||||||
|
graph — an item, a collection, a connected neighbourhood, a VFS subtree, a predicate
|
||||||
|
match, or the whole brain — into a single versioned JSON document (`BackupData`); the
|
||||||
|
inverse restores it.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const data = await brain.data()
|
||||||
|
|
||||||
|
const backup = await data.export() // whole brain → BackupData
|
||||||
|
await data.import(backup) // restore (merge by id, re-embed if no vectors)
|
||||||
|
```
|
||||||
|
|
||||||
|
It is **portable** (human-readable JSON), **versioned** (`formatVersion`, so a 7.x export
|
||||||
|
imports cleanly into 8.0), and **current-state** (the entities and edges as they are now —
|
||||||
|
no generation history). Use it for portable artifacts, partial backups, cross-environment
|
||||||
|
moves, and version upgrades.
|
||||||
|
|
||||||
|
## When to use which
|
||||||
|
|
||||||
|
| You want… | Use |
|
||||||
|
|-----------|-----|
|
||||||
|
| A portable, partial-or-whole, cross-version graph document | **`brain.data().export()` / `import()`** (this guide) |
|
||||||
|
| To ingest a CSV / PDF / Excel / JSON **file** as new entities | `brain.import(file)` — see [Import Anything](./import-anything.md) |
|
||||||
|
|
||||||
|
The two are different operations that happen to share a verb: `import(file)` parses a
|
||||||
|
foreign document into new entities; `data().import(backup)` restores a graph this API
|
||||||
|
exported.
|
||||||
|
|
||||||
|
## Exporting
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
export(selector?, options?): Promise<BackupData>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Selectors — *what* to export
|
||||||
|
|
||||||
|
Omit the selector to export the whole brain. Otherwise pick a node set:
|
||||||
|
|
||||||
|
| Scenario | Selector |
|
||||||
|
|----------|----------|
|
||||||
|
| Just an item (or items) | `{ ids: ['a', 'b'] }` |
|
||||||
|
| A collection + its children | `{ collection: collectionId }` (alias `memberOf`) |
|
||||||
|
| A connected neighbourhood | `{ connected: { from: id, depth: 2, verbs?, direction? } }` |
|
||||||
|
| A VFS directory / file (+ subtree) | `{ vfsPath: '/docs', recursive?: true }` |
|
||||||
|
| Everything matching a predicate | `{ type, subtype, where, service }` |
|
||||||
|
| The whole brain | *(omit)* |
|
||||||
|
|
||||||
|
The selector reuses `find()`'s grammar — *"export what `find()` would match, minus ranking
|
||||||
|
and limit."* Structural and predicate selectors **compose** — a structural selector picks
|
||||||
|
the nodes, and predicate keys then filter them:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Members of a collection whose status is "open"
|
||||||
|
await data.export({ collection: collectionId, where: { status: 'open' } })
|
||||||
|
```
|
||||||
|
|
||||||
|
If you already have results from `find()`, export exactly those with the `ids` selector:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const hits = await brain.find({ type: NounType.Document })
|
||||||
|
await data.export({ ids: hits.map(r => r.id) })
|
||||||
|
```
|
||||||
|
|
||||||
|
### Options — *how* to serialize
|
||||||
|
|
||||||
|
| Option | Default | Effect |
|
||||||
|
|--------|---------|--------|
|
||||||
|
| `includeVectors` | `false` | Carry embedding vectors verbatim. Off ⇒ `import()` re-embeds from `data`. |
|
||||||
|
| `includeContent` | `false` | Include VFS file bytes in `blobs` so files round-trip byte-identically. |
|
||||||
|
| `includeSystem` | `false` | Include `system` entities such as the VFS root. |
|
||||||
|
| `edges` | `'induced'` | `'induced'` (both endpoints in the set), `'incident'` (also dangling edges, recorded in `danglingIds`), or `'none'` (nodes only). |
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// A self-contained subgraph with vectors, ready to restore elsewhere
|
||||||
|
const subset = await data.export({ ids }, { includeVectors: true })
|
||||||
|
|
||||||
|
// A VFS subtree including file bytes
|
||||||
|
const tree = await data.export({ vfsPath: '/docs' }, { includeContent: true })
|
||||||
|
```
|
||||||
|
|
||||||
|
## Importing
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import(backup, options?): Promise<ImportResult>
|
||||||
|
```
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const result = await brain.data().then(d => d.import(backup, { onConflict: 'merge' }))
|
||||||
|
// → { imported, merged, skipped, reembedded, blobsWritten, errors }
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Default | Effect |
|
||||||
|
|--------|---------|--------|
|
||||||
|
| `onConflict` | `'merge'` | `'merge'` (update existing id in place — assemble many backups), `'replace'` (delete + recreate), or `'skip'`. |
|
||||||
|
| `reembed` | `'auto'` | `'auto'` (use the carried vector, else re-embed from `data`) or `'never'` (require a carried vector; record an error if absent). |
|
||||||
|
| `remapIds` | — | Rewrite every id on the way in, e.g. to clone a template subgraph under fresh ids. |
|
||||||
|
|
||||||
|
The default `onConflict: 'merge'` is what lets you assemble one working graph from many
|
||||||
|
exported documents that share entity ids — re-importing an id merges rather than duplicates.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Clone a subgraph under fresh ids (a copy, not a move)
|
||||||
|
const remap = new Map(backup.entities.map(e => [e.id, crypto.randomUUID()]))
|
||||||
|
await data.import(backup, { remapIds: id => remap.get(id) ?? id })
|
||||||
|
```
|
||||||
|
|
||||||
|
## The `BackupData` format
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
{
|
||||||
|
"format": "brainy-backup",
|
||||||
|
"formatVersion": 1, // import gates on this (cross-version migration)
|
||||||
|
"brainyVersion": "7.32.0",
|
||||||
|
"createdAt": "2026-06-16T…Z",
|
||||||
|
"embedding": { "model": "all-MiniLM-L6-v2", "dimensions": 384 },
|
||||||
|
"selector": { … }, // echoes what was exported (provenance)
|
||||||
|
"entities": [
|
||||||
|
{
|
||||||
|
"id": "…", "type": "Document", "subtype": "invoice",
|
||||||
|
"data": "…", // the embedding source
|
||||||
|
"confidence": 1, "weight": 1, "service": "…",
|
||||||
|
"vector": [ … ], // only with includeVectors
|
||||||
|
"metadata": { … } // custom fields only (reserved fields are top-level)
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"relations": [
|
||||||
|
{ "id":"…", "from":"…", "to":"…", "type":"Contains", "subtype":"…",
|
||||||
|
"weight":1, "confidence":1, "metadata": { … } }
|
||||||
|
],
|
||||||
|
"blobs": { "<sha256>": "<base64>" }, // only with includeContent
|
||||||
|
"danglingIds": [ "…" ], // only with edges:'incident'
|
||||||
|
"stats": { "entityCount": 0, "relationCount": 0, "blobCount": 0, "vectorDimensions": 384 }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Standard fields (`subtype`, `data`, `confidence`, `weight`, `service`) sit at the top level
|
||||||
|
of each entity; `metadata` holds **only** custom user fields — mirroring the in-memory
|
||||||
|
`Entity` shape, so `import()` maps each field to its dedicated parameter.
|
||||||
|
|
||||||
|
## Cross-version (7.x → 8.0)
|
||||||
|
|
||||||
|
Because the document is shared and versioned, a backup written by 7.x imports into 8.0:
|
||||||
|
`formatVersion` is read forward, `subtype` is carried so 8.0 re-types correctly, and the
|
||||||
|
same 384-dimension model on both lines means `includeVectors:false` re-embeds identically
|
||||||
|
(or `true` carries vectors verbatim). The format is **current-state** — if you need a
|
||||||
|
whole-brain snapshot *with* generation history, that is a separate native facility
|
||||||
|
(`db.persist()` / `Brainy.load()` on 8.0).
|
||||||
|
|
||||||
|
## VFS
|
||||||
|
|
||||||
|
VFS directories are `Collection` entities and files are entities linked by `Contains`, so
|
||||||
|
the whole filesystem (or any subtree) exports through the `vfsPath` selector:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
await data.export({ vfsPath: '/' }, { includeContent: true }) // all VFS + bytes
|
||||||
|
await data.export({ vfsPath: '/docs' }, { includeContent: true }) // one directory
|
||||||
|
await data.export({ vfsPath: '/a/b.txt' }, { includeContent: true }) // one file
|
||||||
|
```
|
||||||
|
|
@ -540,11 +540,11 @@ export async function generateStaticProps() {
|
||||||
})
|
})
|
||||||
await brain.init()
|
await brain.init()
|
||||||
|
|
||||||
// Build search index
|
// Build a search index — export the whole brain as a portable BackupData document
|
||||||
const allContent = await brain.export()
|
const backup = await brain.data().then(d => d.export())
|
||||||
|
|
||||||
return {
|
return {
|
||||||
props: { searchIndex: allContent }
|
props: { searchIndex: backup.entities }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
|
||||||
1082
src/api/DataAPI.ts
1082
src/api/DataAPI.ts
File diff suppressed because it is too large
Load diff
|
|
@ -5770,12 +5770,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
*/
|
*/
|
||||||
async data() {
|
async data() {
|
||||||
const { DataAPI } = await import('./api/DataAPI.js')
|
const { DataAPI } = await import('./api/DataAPI.js')
|
||||||
return new DataAPI(
|
return new DataAPI(this.storage, this)
|
||||||
this.storage,
|
|
||||||
(id: string) => this.get(id),
|
|
||||||
undefined, // No getRelation method yet
|
|
||||||
this
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
|
||||||
|
|
@ -876,60 +876,66 @@ export const coreCommands = {
|
||||||
const brain = getBrainy()
|
const brain = getBrainy()
|
||||||
const format = options.format || 'json'
|
const format = options.format || 'json'
|
||||||
|
|
||||||
// Export all data
|
// Export the whole brain as a portable BackupData document (vectors + VFS file bytes).
|
||||||
const dataApi = await brain.data()
|
const dataApi = await brain.data()
|
||||||
const data = await dataApi.export({ format: 'json' })
|
const backup = await dataApi.export(undefined, {
|
||||||
|
includeVectors: true,
|
||||||
|
includeContent: true
|
||||||
|
})
|
||||||
let output = ''
|
let output = ''
|
||||||
|
|
||||||
|
const csvCell = (v: any): string => {
|
||||||
|
if (v === undefined || v === null) return ''
|
||||||
|
const s = typeof v === 'object' ? JSON.stringify(v) : String(v)
|
||||||
|
return /[",\n]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s
|
||||||
|
}
|
||||||
|
|
||||||
switch (format) {
|
switch (format) {
|
||||||
case 'json':
|
case 'json':
|
||||||
output = options.pretty
|
output = options.pretty
|
||||||
? JSON.stringify(data, null, 2)
|
? JSON.stringify(backup, null, 2)
|
||||||
: JSON.stringify(data)
|
: JSON.stringify(backup)
|
||||||
break
|
break
|
||||||
|
|
||||||
case 'jsonl':
|
case 'jsonl': {
|
||||||
if (Array.isArray(data)) {
|
// NDJSON: a header line, then one line per entity, then one line per relation.
|
||||||
output = data.map(item => JSON.stringify(item)).join('\n')
|
const { entities, relations, ...header } = backup
|
||||||
} else {
|
const lines: string[] = [JSON.stringify({ kind: 'header', ...header })]
|
||||||
output = JSON.stringify(data)
|
for (const e of entities) lines.push(JSON.stringify({ kind: 'entity', ...e }))
|
||||||
}
|
for (const r of relations) lines.push(JSON.stringify({ kind: 'relation', ...r }))
|
||||||
|
output = lines.join('\n')
|
||||||
break
|
break
|
||||||
|
}
|
||||||
case 'csv':
|
|
||||||
if (Array.isArray(data) && data.length > 0) {
|
case 'csv': {
|
||||||
// Get all unique keys for headers
|
// CSV represents entities only — a graph's edges and blobs can't be tabular.
|
||||||
|
const entities = backup.entities
|
||||||
|
if (entities.length > 0) {
|
||||||
const headers = new Set<string>()
|
const headers = new Set<string>()
|
||||||
data.forEach(item => {
|
entities.forEach((e: Record<string, any>) => Object.keys(e).forEach(k => headers.add(k)))
|
||||||
Object.keys(item).forEach(key => headers.add(key))
|
|
||||||
})
|
|
||||||
const headerArray = Array.from(headers)
|
const headerArray = Array.from(headers)
|
||||||
|
|
||||||
// Create CSV
|
|
||||||
output = headerArray.join(',') + '\n'
|
output = headerArray.join(',') + '\n'
|
||||||
output += data.map(item => {
|
output += entities
|
||||||
return headerArray.map(h => {
|
.map((e: Record<string, any>) => headerArray.map(h => csvCell(e[h])).join(','))
|
||||||
const value = item[h]
|
.join('\n')
|
||||||
if (typeof value === 'object') {
|
|
||||||
return JSON.stringify(value)
|
|
||||||
}
|
|
||||||
return value || ''
|
|
||||||
}).join(',')
|
|
||||||
}).join('\n')
|
|
||||||
}
|
}
|
||||||
break
|
break
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (file) {
|
if (file) {
|
||||||
writeFileSync(file, output)
|
writeFileSync(file, output)
|
||||||
spinner.succeed(`Exported to ${file}`)
|
spinner.succeed(`Exported to ${file}`)
|
||||||
|
|
||||||
if (!options.json) {
|
if (!options.json) {
|
||||||
console.log(chalk.green(`✓ Successfully exported database to ${file}`))
|
console.log(chalk.green(`✓ Successfully exported database to ${file}`))
|
||||||
console.log(chalk.dim(` Format: ${format}`))
|
console.log(chalk.dim(` Format: ${format}`))
|
||||||
console.log(chalk.dim(` Items: ${Array.isArray(data) ? data.length : 1}`))
|
console.log(chalk.dim(` Entities: ${backup.stats.entityCount} Relations: ${backup.stats.relationCount}`))
|
||||||
} else {
|
} else {
|
||||||
formatOutput({ file, format, count: Array.isArray(data) ? data.length : 1 }, options)
|
formatOutput(
|
||||||
|
{ file, format, entityCount: backup.stats.entityCount, relationCount: backup.stats.relationCount },
|
||||||
|
options
|
||||||
|
)
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
spinner.succeed('Export complete')
|
spinner.succeed('Export complete')
|
||||||
|
|
|
||||||
12
src/index.ts
12
src/index.ts
|
|
@ -116,6 +116,18 @@ export {
|
||||||
// Export version utilities
|
// Export version utilities
|
||||||
export { getBrainyVersion } from './utils/version.js'
|
export { getBrainyVersion } from './utils/version.js'
|
||||||
|
|
||||||
|
// Export portable graph backup/restore API (brain.data())
|
||||||
|
export { DataAPI } from './api/DataAPI.js'
|
||||||
|
export type {
|
||||||
|
BackupData,
|
||||||
|
BackupEntity,
|
||||||
|
BackupRelation,
|
||||||
|
ExportSelector,
|
||||||
|
ExportOptions,
|
||||||
|
ImportOptions,
|
||||||
|
ImportResult
|
||||||
|
} from './api/DataAPI.js'
|
||||||
|
|
||||||
// Export plugin system
|
// Export plugin system
|
||||||
export type { BrainyPlugin, BrainyPluginContext, StorageAdapterFactory } from './plugin.js'
|
export type { BrainyPlugin, BrainyPluginContext, StorageAdapterFactory } from './plugin.js'
|
||||||
export { PluginRegistry } from './plugin.js'
|
export { PluginRegistry } from './plugin.js'
|
||||||
|
|
|
||||||
311
tests/unit/api/data-backup.test.ts
Normal file
311
tests/unit/api/data-backup.test.ts
Normal file
|
|
@ -0,0 +1,311 @@
|
||||||
|
/**
|
||||||
|
* Unit tests for the portable graph backup/restore API (brain.data()).
|
||||||
|
*
|
||||||
|
* Exercises BackupData v1 export()/import(): real round-trips against in-memory
|
||||||
|
* storage (no mocks of the API under test) — selectors, edge policy, vectors,
|
||||||
|
* conflict handling, id remapping, subtype fidelity, and cross-version guards.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||||
|
import { randomUUID } from 'node:crypto'
|
||||||
|
import { Brainy } from '../../../src/brainy'
|
||||||
|
import { createTestConfig } from '../../helpers/test-factory'
|
||||||
|
import { NounType, VerbType } from '../../../src/types/graphTypes'
|
||||||
|
import type { BackupData } from '../../../src/api/DataAPI'
|
||||||
|
|
||||||
|
const VFS_ROOT_ID = '00000000-0000-0000-0000-000000000000'
|
||||||
|
|
||||||
|
describe('DataAPI — portable graph backup/restore (BackupData v1)', () => {
|
||||||
|
let brain: Brainy
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
brain = new Brainy(createTestConfig())
|
||||||
|
await brain.init()
|
||||||
|
})
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('format + whole-brain round-trip', () => {
|
||||||
|
it('exports a self-describing, versioned BackupData document', async () => {
|
||||||
|
const a = await brain.add({ data: 'Alice', type: NounType.Person, subtype: 'employee' })
|
||||||
|
const b = await brain.add({ data: 'Acme', type: NounType.Organization })
|
||||||
|
await brain.relate({ from: a, to: b, type: VerbType.WorksWith, subtype: 'full-time' })
|
||||||
|
|
||||||
|
const backup = await brain.data().then((d) => d.export())
|
||||||
|
|
||||||
|
expect(backup.format).toBe('brainy-backup')
|
||||||
|
expect(backup.formatVersion).toBe(1)
|
||||||
|
expect(typeof backup.brainyVersion).toBe('string')
|
||||||
|
expect(backup.brainyVersion.length).toBeGreaterThan(0)
|
||||||
|
expect(typeof backup.createdAt).toBe('string')
|
||||||
|
expect(backup.embedding.dimensions).toBeGreaterThan(0)
|
||||||
|
expect(backup.stats.entityCount).toBe(backup.entities.length)
|
||||||
|
expect(backup.stats.relationCount).toBe(backup.relations.length)
|
||||||
|
// Two user entities, one relation (VFS root excluded by default).
|
||||||
|
expect(backup.entities.map((e) => e.id).sort()).toEqual([a, b].sort())
|
||||||
|
expect(backup.relations).toHaveLength(1)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('round-trips entities and relations through clear() + import()', async () => {
|
||||||
|
const a = await brain.add({ data: 'Alice', type: NounType.Person })
|
||||||
|
const b = await brain.add({ data: 'Bob', type: NounType.Person })
|
||||||
|
const c = await brain.add({ data: 'Carol', type: NounType.Person })
|
||||||
|
await brain.relate({ from: a, to: b, type: VerbType.FriendOf })
|
||||||
|
await brain.relate({ from: b, to: c, type: VerbType.FriendOf })
|
||||||
|
|
||||||
|
const backup = await brain.data().then((d) => d.export({}, { includeVectors: true }))
|
||||||
|
|
||||||
|
await brain.data().then((d) => d.clear())
|
||||||
|
const afterClear = await brain.find({ limit: 1000 })
|
||||||
|
expect(afterClear.filter((r) => [a, b, c].includes(r.id))).toHaveLength(0)
|
||||||
|
|
||||||
|
const result = await brain.data().then((d) => d.import(backup))
|
||||||
|
expect(result.imported).toBe(3)
|
||||||
|
expect(result.errors).toHaveLength(0)
|
||||||
|
|
||||||
|
const ra = await brain.get(a)
|
||||||
|
expect(ra?.id).toBe(a)
|
||||||
|
const rels = await brain.getRelations({ from: a })
|
||||||
|
expect(rels.some((r) => r.to === b && r.type === VerbType.FriendOf)).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('preserves subtype on both entities and relations', async () => {
|
||||||
|
const a = await brain.add({ data: 'Doc', type: NounType.Document, subtype: 'invoice' })
|
||||||
|
const b = await brain.add({ data: 'Doc2', type: NounType.Document, subtype: 'receipt' })
|
||||||
|
await brain.relate({ from: a, to: b, type: VerbType.RelatedTo, subtype: 'supersedes' })
|
||||||
|
|
||||||
|
const backup = await brain.data().then((d) => d.export())
|
||||||
|
|
||||||
|
const ea = backup.entities.find((e) => e.id === a)
|
||||||
|
expect(ea?.subtype).toBe('invoice')
|
||||||
|
expect(backup.relations[0].subtype).toBe('supersedes')
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('selectors', () => {
|
||||||
|
it('ids — exports exactly the requested entities', async () => {
|
||||||
|
const a = await brain.add({ data: 'A', type: NounType.Thing })
|
||||||
|
const b = await brain.add({ data: 'B', type: NounType.Thing })
|
||||||
|
await brain.add({ data: 'C', type: NounType.Thing })
|
||||||
|
|
||||||
|
const backup = await brain.data().then((d) => d.export({ ids: [a, b] }))
|
||||||
|
expect(backup.entities.map((e) => e.id).sort()).toEqual([a, b].sort())
|
||||||
|
})
|
||||||
|
|
||||||
|
it('collection — exports the collection plus its transitive Contains members', async () => {
|
||||||
|
const root = await brain.add({ data: 'Folder', type: NounType.Collection })
|
||||||
|
const child1 = await brain.add({ data: 'Child1', type: NounType.Document })
|
||||||
|
const child2 = await brain.add({ data: 'Child2', type: NounType.Document })
|
||||||
|
const grandchild = await brain.add({ data: 'Grandchild', type: NounType.Document })
|
||||||
|
await brain.add({ data: 'Outside', type: NounType.Document })
|
||||||
|
await brain.relate({ from: root, to: child1, type: VerbType.Contains })
|
||||||
|
await brain.relate({ from: root, to: child2, type: VerbType.Contains })
|
||||||
|
await brain.relate({ from: child1, to: grandchild, type: VerbType.Contains })
|
||||||
|
|
||||||
|
const backup = await brain.data().then((d) => d.export({ collection: root }))
|
||||||
|
expect(backup.entities.map((e) => e.id).sort()).toEqual(
|
||||||
|
[root, child1, child2, grandchild].sort()
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('connected — exports an N-hop neighbourhood', async () => {
|
||||||
|
const a = await brain.add({ data: 'A', type: NounType.Thing })
|
||||||
|
const b = await brain.add({ data: 'B', type: NounType.Thing })
|
||||||
|
const c = await brain.add({ data: 'C', type: NounType.Thing })
|
||||||
|
await brain.relate({ from: a, to: b, type: VerbType.RelatedTo })
|
||||||
|
await brain.relate({ from: b, to: c, type: VerbType.RelatedTo })
|
||||||
|
|
||||||
|
const depth1 = await brain.data().then((d) => d.export({ connected: { from: a, depth: 1 } }))
|
||||||
|
expect(depth1.entities.map((e) => e.id).sort()).toEqual([a, b].sort())
|
||||||
|
|
||||||
|
const depth2 = await brain.data().then((d) => d.export({ connected: { from: a, depth: 2 } }))
|
||||||
|
expect(depth2.entities.map((e) => e.id).sort()).toEqual([a, b, c].sort())
|
||||||
|
})
|
||||||
|
|
||||||
|
it('predicate — filters by type', async () => {
|
||||||
|
await brain.add({ data: 'P', type: NounType.Person })
|
||||||
|
await brain.add({ data: 'D', type: NounType.Document })
|
||||||
|
const backup = await brain.data().then((d) => d.export({ type: NounType.Person }))
|
||||||
|
expect(backup.entities.every((e) => e.type === NounType.Person)).toBe(true)
|
||||||
|
expect(backup.entities).toHaveLength(1)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('compose — structural selector + predicate filter', async () => {
|
||||||
|
const root = await brain.add({ data: 'Folder', type: NounType.Collection })
|
||||||
|
const open = await brain.add({
|
||||||
|
data: 'Open',
|
||||||
|
type: NounType.Document,
|
||||||
|
metadata: { status: 'open' }
|
||||||
|
})
|
||||||
|
const closed = await brain.add({
|
||||||
|
data: 'Closed',
|
||||||
|
type: NounType.Document,
|
||||||
|
metadata: { status: 'closed' }
|
||||||
|
})
|
||||||
|
await brain.relate({ from: root, to: open, type: VerbType.Contains })
|
||||||
|
await brain.relate({ from: root, to: closed, type: VerbType.Contains })
|
||||||
|
|
||||||
|
const backup = await brain
|
||||||
|
.data()
|
||||||
|
.then((d) => d.export({ collection: root, where: { status: 'open' } }))
|
||||||
|
expect(backup.entities.map((e) => e.id)).toContain(open)
|
||||||
|
expect(backup.entities.map((e) => e.id)).not.toContain(closed)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('edge policy', () => {
|
||||||
|
let a: string
|
||||||
|
let b: string
|
||||||
|
let c: string
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
a = await brain.add({ data: 'A', type: NounType.Thing })
|
||||||
|
b = await brain.add({ data: 'B', type: NounType.Thing })
|
||||||
|
c = await brain.add({ data: 'C', type: NounType.Thing })
|
||||||
|
await brain.relate({ from: a, to: b, type: VerbType.RelatedTo }) // both in {a,b}
|
||||||
|
await brain.relate({ from: b, to: c, type: VerbType.RelatedTo }) // dangles to c
|
||||||
|
})
|
||||||
|
|
||||||
|
it('induced (default) — only edges with both endpoints in the set', async () => {
|
||||||
|
const backup = await brain.data().then((d) => d.export({ ids: [a, b] }))
|
||||||
|
expect(backup.relations).toHaveLength(1)
|
||||||
|
expect(backup.relations[0].from).toBe(a)
|
||||||
|
expect(backup.danglingIds).toBeUndefined()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('incident — includes dangling edges + records danglingIds', async () => {
|
||||||
|
const backup = await brain.data().then((d) => d.export({ ids: [a, b] }, { edges: 'incident' }))
|
||||||
|
expect(backup.relations).toHaveLength(2)
|
||||||
|
expect(backup.danglingIds).toContain(c)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('none — nodes only', async () => {
|
||||||
|
const backup = await brain.data().then((d) => d.export({ ids: [a, b] }, { edges: 'none' }))
|
||||||
|
expect(backup.relations).toHaveLength(0)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('vectors + re-embedding', () => {
|
||||||
|
it('omits vectors by default and re-embeds on import', async () => {
|
||||||
|
const a = await brain.add({ data: 'Re-embed me', type: NounType.Thing })
|
||||||
|
const backup = await brain.data().then((d) => d.export({ ids: [a] }))
|
||||||
|
expect(backup.entities[0].vector).toBeUndefined()
|
||||||
|
|
||||||
|
await brain.data().then((d) => d.clear())
|
||||||
|
const result = await brain.data().then((d) => d.import(backup))
|
||||||
|
expect(result.reembedded).toBe(1)
|
||||||
|
|
||||||
|
const restored = await brain.get(a, { includeVectors: true })
|
||||||
|
expect(restored?.vector?.length).toBeGreaterThan(0)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('carries vectors verbatim with includeVectors (no re-embed)', async () => {
|
||||||
|
const a = await brain.add({ data: 'Keep my vector', type: NounType.Thing })
|
||||||
|
const original = await brain.get(a, { includeVectors: true })
|
||||||
|
|
||||||
|
const backup = await brain.data().then((d) => d.export({ ids: [a] }, { includeVectors: true }))
|
||||||
|
expect(backup.entities[0].vector?.length).toBe(original?.vector?.length)
|
||||||
|
|
||||||
|
await brain.data().then((d) => d.clear())
|
||||||
|
const result = await brain.data().then((d) => d.import(backup))
|
||||||
|
expect(result.reembedded).toBe(0)
|
||||||
|
|
||||||
|
const restored = await brain.get(a, { includeVectors: true })
|
||||||
|
expect(restored?.vector).toEqual(original?.vector)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('reembed:never records an error when no vector is carried', async () => {
|
||||||
|
const a = await brain.add({ data: 'No vector', type: NounType.Thing })
|
||||||
|
const backup = await brain.data().then((d) => d.export({ ids: [a] })) // no vectors
|
||||||
|
await brain.data().then((d) => d.clear())
|
||||||
|
|
||||||
|
const result = await brain.data().then((d) => d.import(backup, { reembed: 'never' }))
|
||||||
|
expect(result.imported).toBe(0)
|
||||||
|
expect(result.errors).toHaveLength(1)
|
||||||
|
expect(result.errors[0].error).toMatch(/reembed:never/)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('conflict handling', () => {
|
||||||
|
it('merge (default) — updates existing entities in place', async () => {
|
||||||
|
const a = await brain.add({ data: 'V1', type: NounType.Thing, metadata: { n: 1 } })
|
||||||
|
const backup = await brain.data().then((d) => d.export({ ids: [a] }, { includeVectors: true }))
|
||||||
|
|
||||||
|
await brain.update({ id: a, metadata: { n: 99 }, merge: false })
|
||||||
|
const result = await brain.data().then((d) => d.import(backup, { onConflict: 'merge' }))
|
||||||
|
expect(result.merged).toBe(1)
|
||||||
|
expect(result.imported).toBe(0)
|
||||||
|
|
||||||
|
const restored = await brain.get(a)
|
||||||
|
expect(restored?.metadata?.n).toBe(1)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('skip — leaves existing entities untouched', async () => {
|
||||||
|
const a = await brain.add({ data: 'Orig', type: NounType.Thing, metadata: { n: 1 } })
|
||||||
|
const backup = await brain.data().then((d) => d.export({ ids: [a] }))
|
||||||
|
await brain.update({ id: a, metadata: { n: 2 }, merge: false })
|
||||||
|
|
||||||
|
const result = await brain.data().then((d) => d.import(backup, { onConflict: 'skip' }))
|
||||||
|
expect(result.skipped).toBe(1)
|
||||||
|
const restored = await brain.get(a)
|
||||||
|
expect(restored?.metadata?.n).toBe(2)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('id remapping (clone)', () => {
|
||||||
|
it('imports a subgraph under fresh ids', async () => {
|
||||||
|
const a = await brain.add({ data: 'A', type: NounType.Thing })
|
||||||
|
const b = await brain.add({ data: 'B', type: NounType.Thing })
|
||||||
|
await brain.relate({ from: a, to: b, type: VerbType.RelatedTo })
|
||||||
|
const backup = await brain.data().then((d) => d.export({ ids: [a, b] }, { includeVectors: true }))
|
||||||
|
|
||||||
|
const remap = new Map<string, string>([
|
||||||
|
[a, randomUUID()],
|
||||||
|
[b, randomUUID()]
|
||||||
|
])
|
||||||
|
const result = await brain
|
||||||
|
.data()
|
||||||
|
.then((d) => d.import(backup, { remapIds: (id) => remap.get(id) ?? id }))
|
||||||
|
expect(result.imported).toBe(2)
|
||||||
|
|
||||||
|
const clonedA = await brain.get(remap.get(a)!)
|
||||||
|
expect(clonedA?.id).toBe(remap.get(a))
|
||||||
|
// The original entities still exist (a clone, not a move).
|
||||||
|
expect((await brain.get(a))?.id).toBe(a)
|
||||||
|
const clonedRels = await brain.getRelations({ from: remap.get(a)! })
|
||||||
|
expect(clonedRels.some((r) => r.to === remap.get(b))).toBe(true)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('system entities', () => {
|
||||||
|
it('excludes the VFS root from a whole-brain export by default', async () => {
|
||||||
|
await brain.add({ data: 'User entity', type: NounType.Thing })
|
||||||
|
const backup = await brain.data().then((d) => d.export())
|
||||||
|
expect(backup.entities.map((e) => e.id)).not.toContain(VFS_ROOT_ID)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('import validation', () => {
|
||||||
|
it('rejects a non-BackupData payload', async () => {
|
||||||
|
const d = await brain.data()
|
||||||
|
await expect(d.import({ entities: [] } as any)).rejects.toThrow(/BackupData/)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('rejects a newer formatVersion', async () => {
|
||||||
|
const d = await brain.data()
|
||||||
|
const future: BackupData = {
|
||||||
|
format: 'brainy-backup',
|
||||||
|
formatVersion: 999,
|
||||||
|
brainyVersion: 'x',
|
||||||
|
createdAt: new Date().toISOString(),
|
||||||
|
embedding: { model: 'm', dimensions: 384 },
|
||||||
|
entities: [],
|
||||||
|
relations: [],
|
||||||
|
stats: { entityCount: 0, relationCount: 0, blobCount: 0 }
|
||||||
|
}
|
||||||
|
await expect(d.import(future)).rejects.toThrow(/formatVersion/)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
})
|
||||||
Loading…
Add table
Add a link
Reference in a new issue