feat: portable graph export()/import() (BackupData v1) on brain.data()
brain.data() now serializes part or all of a brain to a versioned, portable
JSON document (BackupData) and restores it — the path for partial backups,
cross-environment moves, and 7.x→8.0 migration.
- export(selector?, options?): select by ids, collection (+transitive Contains),
connected neighbourhood, vfsPath subtree, predicate, or whole brain; structural
and predicate selectors compose. Options: includeVectors, includeContent (VFS
blobs), includeSystem, edges ('induced'|'incident'|'none').
- import(backup, options?): onConflict 'merge' (dedup-by-id) | 'replace' | 'skip';
reembed 'auto' (re-embed from data when no vector carried) | 'never'; remapIds
to clone a subgraph under fresh ids.
- BackupData v1: format/formatVersion/brainyVersion/createdAt/embedding/entities/
relations/blobs?/danglingIds?/stats. Reserved fields (subtype, data, confidence,
weight, service) top-level; metadata custom-only. Current-state, no generations.
- Replaces the prior flat-entity export() (dropped relations) with a graph-complete
document. Distinct from brain.import(file) ingestion, which is unchanged.
- Export BackupData/BackupEntity/BackupRelation/ExportSelector/ExportOptions/
ImportOptions/ImportResult/DataAPI from the package root. CLI `brainy export`
now writes a BackupData document.
Guide: docs/guides/backup-and-export.md. Tests: tests/unit/api/data-backup.test.ts.
This commit is contained in:
parent
89c6d043ba
commit
a408d3799d
11 changed files with 1393 additions and 382 deletions
47
RELEASES.md
47
RELEASES.md
|
|
@ -10,6 +10,53 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so
|
|||
|
||||
---
|
||||
|
||||
## v7.32.0 — 2026-06-16
|
||||
|
||||
**Affected products:** anyone who needs a **portable graph backup**, a partial export, or a
|
||||
**7.x → 8.0 migration path**. Purely additive — no API removed, no data change. Drop-in.
|
||||
|
||||
### New — portable graph `export()` / `import()` on `brain.data()`
|
||||
|
||||
`brain.data()` gains a portable backup/restore pair that serializes part or all of a brain
|
||||
to a single versioned JSON document (`BackupData`) and restores it. This is the format that
|
||||
embeds inside portable artifacts (e.g. a serialized subgraph carried by a higher-level file
|
||||
type) and the recommended way to move a graph between environments or **upgrade an existing
|
||||
7.x brain into an 8.0 brain**.
|
||||
|
||||
```typescript
|
||||
const data = await brain.data()
|
||||
|
||||
const backup = await data.export() // whole brain → BackupData
|
||||
const subset = await data.export({ ids }, { includeVectors: true })
|
||||
await otherBrain.data().then(d => d.import(subset, { onConflict: 'merge' }))
|
||||
```
|
||||
|
||||
- **Selectors** — `export(selector?, options?)` picks the node set: `{ ids }`, `{ collection }`
|
||||
(alias `memberOf`) + transitive `Contains` members, `{ connected: { from, depth, verbs?, direction? } }`,
|
||||
`{ vfsPath, recursive? }`, a predicate (`{ type, subtype, where, service }`), or the whole brain
|
||||
(omit). Structural + predicate selectors **compose**.
|
||||
- **Options** — `includeVectors` (default off → re-embed on import), `includeContent` (VFS file
|
||||
bytes in `blobs`), `includeSystem` (include the VFS root etc.), `edges` (`'induced'` default /
|
||||
`'incident'` / `'none'`).
|
||||
- **Import** — `import(backup, options?)` → `{ imported, merged, skipped, reembedded, blobsWritten, errors }`.
|
||||
`onConflict` defaults to `'merge'` (dedup-by-id, so you can assemble many documents into one
|
||||
graph); `reembed` defaults to `'auto'` (re-embed from `data` when no vector is carried);
|
||||
`remapIds` clones a subgraph under fresh ids.
|
||||
- **Format** — `{ format:'brainy-backup', formatVersion:1, brainyVersion, createdAt, embedding,
|
||||
entities, relations, blobs?, danglingIds?, stats }`. Entities carry `subtype` and the standard
|
||||
reserved fields at the top level; `metadata` is custom-only. **Current-state** (no generation
|
||||
history). `formatVersion` gates cross-version import (a 7.x document imports into 8.0).
|
||||
- **Types** — `BackupData`, `BackupEntity`, `BackupRelation`, `ExportSelector`, `ExportOptions`,
|
||||
`ImportOptions`, `ImportResult`, and the `DataAPI` class are now exported from the package root.
|
||||
|
||||
This replaces the previous `brain.data().export()` flat-entity array (which dropped relations)
|
||||
with a graph-complete document. Distinct from `brain.import(file)` (CSV/PDF/Excel/JSON ingestion),
|
||||
which is unchanged. The CLI `brainy export` now writes a `BackupData` document (json/jsonl/csv).
|
||||
|
||||
Guide: `docs/guides/export-and-import.md`. Tested in `tests/unit/api/data-backup.test.ts`.
|
||||
|
||||
---
|
||||
|
||||
## v7.31.8 — 2026-06-16
|
||||
|
||||
**Affected products:** every consumer on a bare VM or `mmap-filesystem` storage — **upgrade recommended** (Memory, Workshop on 7.31.x; Venue on 7.28.x). Two platform-wide production fixes. Drop-in; no API or data changes.
|
||||
|
|
|
|||
|
|
@ -1765,20 +1765,33 @@ await brain.import('https://api.example.com/data.json')
|
|||
|
||||
---
|
||||
|
||||
### Export & Snapshots
|
||||
### Export & Import (portable backup)
|
||||
|
||||
`brain.data()` exposes a portable graph backup/restore API. `export(selector?, options?)`
|
||||
serializes part or all of the graph to a versioned, portable `BackupData` document;
|
||||
`import(backup, options?)` restores it (dedup-by-id merge by default, re-embedding when
|
||||
vectors are absent).
|
||||
|
||||
```typescript
|
||||
// Export to file
|
||||
await brain.export('/path/to/backup.brainy')
|
||||
const data = await brain.data()
|
||||
|
||||
// Create instant snapshot using COW fork
|
||||
await brain.fork('backup-2025-01-19')
|
||||
// Whole brain → a portable BackupData document
|
||||
const backup = await data.export()
|
||||
|
||||
// Time-travel to specific commit
|
||||
const snapshot = await brain.asOf(commitId)
|
||||
const entities = await snapshot.find({ limit: 100 })
|
||||
// Just one workbench's members, with vectors, then restore elsewhere (merge by id)
|
||||
const subset = await data.export({ ids }, { includeVectors: true })
|
||||
await otherBrain.data().then(d => d.import(subset, { onConflict: 'merge' }))
|
||||
|
||||
// A collection + its children; a connected neighbourhood; a VFS subtree (+ bytes)
|
||||
await data.export({ collection: collectionId })
|
||||
await data.export({ connected: { from: id, depth: 2 } })
|
||||
await data.export({ vfsPath: '/docs' }, { includeContent: true })
|
||||
```
|
||||
|
||||
The format is versioned (`formatVersion`) and current-state (no generation history) — see
|
||||
the **[Export & Import guide →](../guides/export-and-import.md)**. This is distinct from
|
||||
`brain.import(file)` (CSV/PDF/Excel/JSON ingestion).
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
|
|
|||
|
|
@ -457,23 +457,19 @@ await brain.storage.withLock('resource-id', async () => {
|
|||
|
||||
### Export Data
|
||||
```typescript
|
||||
// Export entire database
|
||||
const backup = await brain.export({
|
||||
format: 'json',
|
||||
includeVectors: true,
|
||||
includeIndexes: false
|
||||
})
|
||||
// Export the whole brain to a portable BackupData document
|
||||
const backup = await brain.data().then(d => d.export(undefined, { includeVectors: true }))
|
||||
```
|
||||
|
||||
### Import Data
|
||||
```typescript
|
||||
// Import from backup
|
||||
await brain.import(backup, {
|
||||
mode: 'merge', // or 'replace'
|
||||
validateSchema: true
|
||||
})
|
||||
// Restore a BackupData document (dedup-by-id merge by default)
|
||||
await brain.data().then(d => d.import(backup, { onConflict: 'merge' }))
|
||||
```
|
||||
|
||||
See the [Export & Import guide](../guides/export-and-import.md) for partial exports
|
||||
(by id, collection, connected neighbourhood, VFS subtree, or predicate).
|
||||
|
||||
### Storage Migration
|
||||
```typescript
|
||||
// Migrate between storage types
|
||||
|
|
@ -483,9 +479,9 @@ const newBrain = new Brainy({ storage: { type: 's3' } })
|
|||
await oldBrain.init()
|
||||
await newBrain.init()
|
||||
|
||||
// Transfer all data
|
||||
const data = await oldBrain.export()
|
||||
await newBrain.import(data)
|
||||
// Transfer all data via a portable backup
|
||||
const data = await oldBrain.data().then(d => d.export(undefined, { includeVectors: true }))
|
||||
await newBrain.data().then(d => d.import(data))
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
|
|
|||
|
|
@ -546,11 +546,9 @@ const appComponent = await brain.findOne({ type: 'ReactComponent', name: 'App' }
|
|||
const imports = await brain.getRelated(appComponent.id, 'Imports')
|
||||
console.log(`App component imports:`, imports.map(c => c.name))
|
||||
|
||||
// Export diagram
|
||||
const diagram = await brain.export({
|
||||
format: 'react-diagram'
|
||||
})
|
||||
console.log(diagram.diagram) // Mermaid diagram
|
||||
// Export the analyzed component graph as a portable BackupData document
|
||||
const graph = await brain.data().then(d => d.export({ type: 'ReactComponent' }))
|
||||
console.log(`Exported ${graph.stats.entityCount} components, ${graph.stats.relationCount} edges`)
|
||||
```
|
||||
|
||||
### Step 4: Premium Licensing (Optional)
|
||||
|
|
|
|||
173
docs/guides/export-and-import.md
Normal file
173
docs/guides/export-and-import.md
Normal file
|
|
@ -0,0 +1,173 @@
|
|||
---
|
||||
title: Export & Import (portable graph)
|
||||
slug: guides/export-and-import
|
||||
public: true
|
||||
category: guides
|
||||
template: guide
|
||||
order: 9
|
||||
description: Export part or all of a brain to a portable, versioned BackupData document and import it back — by id, collection, connected neighbourhood, VFS subtree, predicate, or the whole brain. Covers vectors, edge policy, conflict handling, and id remapping.
|
||||
next:
|
||||
- guides/subtypes-and-facets
|
||||
- api/README
|
||||
---
|
||||
|
||||
# Export & Import (portable graph)
|
||||
|
||||
`brain.data()` exposes a **portable graph export/import** API. One method serializes a
|
||||
graph — an item, a collection, a connected neighbourhood, a VFS subtree, a predicate
|
||||
match, or the whole brain — into a single versioned JSON document (`BackupData`); the
|
||||
inverse restores it.
|
||||
|
||||
```typescript
|
||||
const data = await brain.data()
|
||||
|
||||
const backup = await data.export() // whole brain → BackupData
|
||||
await data.import(backup) // restore (merge by id, re-embed if no vectors)
|
||||
```
|
||||
|
||||
It is **portable** (human-readable JSON), **versioned** (`formatVersion`, so a 7.x export
|
||||
imports cleanly into 8.0), and **current-state** (the entities and edges as they are now —
|
||||
no generation history). Use it for portable artifacts, partial backups, cross-environment
|
||||
moves, and version upgrades.
|
||||
|
||||
## When to use which
|
||||
|
||||
| You want… | Use |
|
||||
|-----------|-----|
|
||||
| A portable, partial-or-whole, cross-version graph document | **`brain.data().export()` / `import()`** (this guide) |
|
||||
| To ingest a CSV / PDF / Excel / JSON **file** as new entities | `brain.import(file)` — see [Import Anything](./import-anything.md) |
|
||||
|
||||
The two are different operations that happen to share a verb: `import(file)` parses a
|
||||
foreign document into new entities; `data().import(backup)` restores a graph this API
|
||||
exported.
|
||||
|
||||
## Exporting
|
||||
|
||||
```typescript
|
||||
export(selector?, options?): Promise<BackupData>
|
||||
```
|
||||
|
||||
### Selectors — *what* to export
|
||||
|
||||
Omit the selector to export the whole brain. Otherwise pick a node set:
|
||||
|
||||
| Scenario | Selector |
|
||||
|----------|----------|
|
||||
| Just an item (or items) | `{ ids: ['a', 'b'] }` |
|
||||
| A collection + its children | `{ collection: collectionId }` (alias `memberOf`) |
|
||||
| A connected neighbourhood | `{ connected: { from: id, depth: 2, verbs?, direction? } }` |
|
||||
| A VFS directory / file (+ subtree) | `{ vfsPath: '/docs', recursive?: true }` |
|
||||
| Everything matching a predicate | `{ type, subtype, where, service }` |
|
||||
| The whole brain | *(omit)* |
|
||||
|
||||
The selector reuses `find()`'s grammar — *"export what `find()` would match, minus ranking
|
||||
and limit."* Structural and predicate selectors **compose** — a structural selector picks
|
||||
the nodes, and predicate keys then filter them:
|
||||
|
||||
```typescript
|
||||
// Members of a collection whose status is "open"
|
||||
await data.export({ collection: collectionId, where: { status: 'open' } })
|
||||
```
|
||||
|
||||
If you already have results from `find()`, export exactly those with the `ids` selector:
|
||||
|
||||
```typescript
|
||||
const hits = await brain.find({ type: NounType.Document })
|
||||
await data.export({ ids: hits.map(r => r.id) })
|
||||
```
|
||||
|
||||
### Options — *how* to serialize
|
||||
|
||||
| Option | Default | Effect |
|
||||
|--------|---------|--------|
|
||||
| `includeVectors` | `false` | Carry embedding vectors verbatim. Off ⇒ `import()` re-embeds from `data`. |
|
||||
| `includeContent` | `false` | Include VFS file bytes in `blobs` so files round-trip byte-identically. |
|
||||
| `includeSystem` | `false` | Include `system` entities such as the VFS root. |
|
||||
| `edges` | `'induced'` | `'induced'` (both endpoints in the set), `'incident'` (also dangling edges, recorded in `danglingIds`), or `'none'` (nodes only). |
|
||||
|
||||
```typescript
|
||||
// A self-contained subgraph with vectors, ready to restore elsewhere
|
||||
const subset = await data.export({ ids }, { includeVectors: true })
|
||||
|
||||
// A VFS subtree including file bytes
|
||||
const tree = await data.export({ vfsPath: '/docs' }, { includeContent: true })
|
||||
```
|
||||
|
||||
## Importing
|
||||
|
||||
```typescript
|
||||
import(backup, options?): Promise<ImportResult>
|
||||
```
|
||||
|
||||
```typescript
|
||||
const result = await brain.data().then(d => d.import(backup, { onConflict: 'merge' }))
|
||||
// → { imported, merged, skipped, reembedded, blobsWritten, errors }
|
||||
```
|
||||
|
||||
| Option | Default | Effect |
|
||||
|--------|---------|--------|
|
||||
| `onConflict` | `'merge'` | `'merge'` (update existing id in place — assemble many backups), `'replace'` (delete + recreate), or `'skip'`. |
|
||||
| `reembed` | `'auto'` | `'auto'` (use the carried vector, else re-embed from `data`) or `'never'` (require a carried vector; record an error if absent). |
|
||||
| `remapIds` | — | Rewrite every id on the way in, e.g. to clone a template subgraph under fresh ids. |
|
||||
|
||||
The default `onConflict: 'merge'` is what lets you assemble one working graph from many
|
||||
exported documents that share entity ids — re-importing an id merges rather than duplicates.
|
||||
|
||||
```typescript
|
||||
// Clone a subgraph under fresh ids (a copy, not a move)
|
||||
const remap = new Map(backup.entities.map(e => [e.id, crypto.randomUUID()]))
|
||||
await data.import(backup, { remapIds: id => remap.get(id) ?? id })
|
||||
```
|
||||
|
||||
## The `BackupData` format
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"format": "brainy-backup",
|
||||
"formatVersion": 1, // import gates on this (cross-version migration)
|
||||
"brainyVersion": "7.32.0",
|
||||
"createdAt": "2026-06-16T…Z",
|
||||
"embedding": { "model": "all-MiniLM-L6-v2", "dimensions": 384 },
|
||||
"selector": { … }, // echoes what was exported (provenance)
|
||||
"entities": [
|
||||
{
|
||||
"id": "…", "type": "Document", "subtype": "invoice",
|
||||
"data": "…", // the embedding source
|
||||
"confidence": 1, "weight": 1, "service": "…",
|
||||
"vector": [ … ], // only with includeVectors
|
||||
"metadata": { … } // custom fields only (reserved fields are top-level)
|
||||
}
|
||||
],
|
||||
"relations": [
|
||||
{ "id":"…", "from":"…", "to":"…", "type":"Contains", "subtype":"…",
|
||||
"weight":1, "confidence":1, "metadata": { … } }
|
||||
],
|
||||
"blobs": { "<sha256>": "<base64>" }, // only with includeContent
|
||||
"danglingIds": [ "…" ], // only with edges:'incident'
|
||||
"stats": { "entityCount": 0, "relationCount": 0, "blobCount": 0, "vectorDimensions": 384 }
|
||||
}
|
||||
```
|
||||
|
||||
Standard fields (`subtype`, `data`, `confidence`, `weight`, `service`) sit at the top level
|
||||
of each entity; `metadata` holds **only** custom user fields — mirroring the in-memory
|
||||
`Entity` shape, so `import()` maps each field to its dedicated parameter.
|
||||
|
||||
## Cross-version (7.x → 8.0)
|
||||
|
||||
Because the document is shared and versioned, a backup written by 7.x imports into 8.0:
|
||||
`formatVersion` is read forward, `subtype` is carried so 8.0 re-types correctly, and the
|
||||
same 384-dimension model on both lines means `includeVectors:false` re-embeds identically
|
||||
(or `true` carries vectors verbatim). The format is **current-state** — if you need a
|
||||
whole-brain snapshot *with* generation history, that is a separate native facility
|
||||
(`db.persist()` / `Brainy.load()` on 8.0).
|
||||
|
||||
## VFS
|
||||
|
||||
VFS directories are `Collection` entities and files are entities linked by `Contains`, so
|
||||
the whole filesystem (or any subtree) exports through the `vfsPath` selector:
|
||||
|
||||
```typescript
|
||||
await data.export({ vfsPath: '/' }, { includeContent: true }) // all VFS + bytes
|
||||
await data.export({ vfsPath: '/docs' }, { includeContent: true }) // one directory
|
||||
await data.export({ vfsPath: '/a/b.txt' }, { includeContent: true }) // one file
|
||||
```
|
||||
|
|
@ -540,11 +540,11 @@ export async function generateStaticProps() {
|
|||
})
|
||||
await brain.init()
|
||||
|
||||
// Build search index
|
||||
const allContent = await brain.export()
|
||||
// Build a search index — export the whole brain as a portable BackupData document
|
||||
const backup = await brain.data().then(d => d.export())
|
||||
|
||||
return {
|
||||
props: { searchIndex: allContent }
|
||||
props: { searchIndex: backup.entities }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
|
|
|||
1048
src/api/DataAPI.ts
1048
src/api/DataAPI.ts
File diff suppressed because it is too large
Load diff
|
|
@ -5770,12 +5770,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
*/
|
||||
async data() {
|
||||
const { DataAPI } = await import('./api/DataAPI.js')
|
||||
return new DataAPI(
|
||||
this.storage,
|
||||
(id: string) => this.get(id),
|
||||
undefined, // No getRelation method yet
|
||||
this
|
||||
)
|
||||
return new DataAPI(this.storage, this)
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -876,49 +876,52 @@ export const coreCommands = {
|
|||
const brain = getBrainy()
|
||||
const format = options.format || 'json'
|
||||
|
||||
// Export all data
|
||||
// Export the whole brain as a portable BackupData document (vectors + VFS file bytes).
|
||||
const dataApi = await brain.data()
|
||||
const data = await dataApi.export({ format: 'json' })
|
||||
const backup = await dataApi.export(undefined, {
|
||||
includeVectors: true,
|
||||
includeContent: true
|
||||
})
|
||||
let output = ''
|
||||
|
||||
const csvCell = (v: any): string => {
|
||||
if (v === undefined || v === null) return ''
|
||||
const s = typeof v === 'object' ? JSON.stringify(v) : String(v)
|
||||
return /[",\n]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s
|
||||
}
|
||||
|
||||
switch (format) {
|
||||
case 'json':
|
||||
output = options.pretty
|
||||
? JSON.stringify(data, null, 2)
|
||||
: JSON.stringify(data)
|
||||
? JSON.stringify(backup, null, 2)
|
||||
: JSON.stringify(backup)
|
||||
break
|
||||
|
||||
case 'jsonl':
|
||||
if (Array.isArray(data)) {
|
||||
output = data.map(item => JSON.stringify(item)).join('\n')
|
||||
} else {
|
||||
output = JSON.stringify(data)
|
||||
case 'jsonl': {
|
||||
// NDJSON: a header line, then one line per entity, then one line per relation.
|
||||
const { entities, relations, ...header } = backup
|
||||
const lines: string[] = [JSON.stringify({ kind: 'header', ...header })]
|
||||
for (const e of entities) lines.push(JSON.stringify({ kind: 'entity', ...e }))
|
||||
for (const r of relations) lines.push(JSON.stringify({ kind: 'relation', ...r }))
|
||||
output = lines.join('\n')
|
||||
break
|
||||
}
|
||||
break
|
||||
|
||||
case 'csv':
|
||||
if (Array.isArray(data) && data.length > 0) {
|
||||
// Get all unique keys for headers
|
||||
case 'csv': {
|
||||
// CSV represents entities only — a graph's edges and blobs can't be tabular.
|
||||
const entities = backup.entities
|
||||
if (entities.length > 0) {
|
||||
const headers = new Set<string>()
|
||||
data.forEach(item => {
|
||||
Object.keys(item).forEach(key => headers.add(key))
|
||||
})
|
||||
entities.forEach((e: Record<string, any>) => Object.keys(e).forEach(k => headers.add(k)))
|
||||
const headerArray = Array.from(headers)
|
||||
|
||||
// Create CSV
|
||||
output = headerArray.join(',') + '\n'
|
||||
output += data.map(item => {
|
||||
return headerArray.map(h => {
|
||||
const value = item[h]
|
||||
if (typeof value === 'object') {
|
||||
return JSON.stringify(value)
|
||||
}
|
||||
return value || ''
|
||||
}).join(',')
|
||||
}).join('\n')
|
||||
output += entities
|
||||
.map((e: Record<string, any>) => headerArray.map(h => csvCell(e[h])).join(','))
|
||||
.join('\n')
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if (file) {
|
||||
writeFileSync(file, output)
|
||||
|
|
@ -927,9 +930,12 @@ export const coreCommands = {
|
|||
if (!options.json) {
|
||||
console.log(chalk.green(`✓ Successfully exported database to ${file}`))
|
||||
console.log(chalk.dim(` Format: ${format}`))
|
||||
console.log(chalk.dim(` Items: ${Array.isArray(data) ? data.length : 1}`))
|
||||
console.log(chalk.dim(` Entities: ${backup.stats.entityCount} Relations: ${backup.stats.relationCount}`))
|
||||
} else {
|
||||
formatOutput({ file, format, count: Array.isArray(data) ? data.length : 1 }, options)
|
||||
formatOutput(
|
||||
{ file, format, entityCount: backup.stats.entityCount, relationCount: backup.stats.relationCount },
|
||||
options
|
||||
)
|
||||
}
|
||||
} else {
|
||||
spinner.succeed('Export complete')
|
||||
|
|
|
|||
12
src/index.ts
12
src/index.ts
|
|
@ -116,6 +116,18 @@ export {
|
|||
// Export version utilities
|
||||
export { getBrainyVersion } from './utils/version.js'
|
||||
|
||||
// Export portable graph backup/restore API (brain.data())
|
||||
export { DataAPI } from './api/DataAPI.js'
|
||||
export type {
|
||||
BackupData,
|
||||
BackupEntity,
|
||||
BackupRelation,
|
||||
ExportSelector,
|
||||
ExportOptions,
|
||||
ImportOptions,
|
||||
ImportResult
|
||||
} from './api/DataAPI.js'
|
||||
|
||||
// Export plugin system
|
||||
export type { BrainyPlugin, BrainyPluginContext, StorageAdapterFactory } from './plugin.js'
|
||||
export { PluginRegistry } from './plugin.js'
|
||||
|
|
|
|||
311
tests/unit/api/data-backup.test.ts
Normal file
311
tests/unit/api/data-backup.test.ts
Normal file
|
|
@ -0,0 +1,311 @@
|
|||
/**
|
||||
* Unit tests for the portable graph backup/restore API (brain.data()).
|
||||
*
|
||||
* Exercises BackupData v1 export()/import(): real round-trips against in-memory
|
||||
* storage (no mocks of the API under test) — selectors, edge policy, vectors,
|
||||
* conflict handling, id remapping, subtype fidelity, and cross-version guards.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { Brainy } from '../../../src/brainy'
|
||||
import { createTestConfig } from '../../helpers/test-factory'
|
||||
import { NounType, VerbType } from '../../../src/types/graphTypes'
|
||||
import type { BackupData } from '../../../src/api/DataAPI'
|
||||
|
||||
const VFS_ROOT_ID = '00000000-0000-0000-0000-000000000000'
|
||||
|
||||
describe('DataAPI — portable graph backup/restore (BackupData v1)', () => {
|
||||
let brain: Brainy
|
||||
|
||||
beforeEach(async () => {
|
||||
brain = new Brainy(createTestConfig())
|
||||
await brain.init()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await brain.close()
|
||||
})
|
||||
|
||||
describe('format + whole-brain round-trip', () => {
|
||||
it('exports a self-describing, versioned BackupData document', async () => {
|
||||
const a = await brain.add({ data: 'Alice', type: NounType.Person, subtype: 'employee' })
|
||||
const b = await brain.add({ data: 'Acme', type: NounType.Organization })
|
||||
await brain.relate({ from: a, to: b, type: VerbType.WorksWith, subtype: 'full-time' })
|
||||
|
||||
const backup = await brain.data().then((d) => d.export())
|
||||
|
||||
expect(backup.format).toBe('brainy-backup')
|
||||
expect(backup.formatVersion).toBe(1)
|
||||
expect(typeof backup.brainyVersion).toBe('string')
|
||||
expect(backup.brainyVersion.length).toBeGreaterThan(0)
|
||||
expect(typeof backup.createdAt).toBe('string')
|
||||
expect(backup.embedding.dimensions).toBeGreaterThan(0)
|
||||
expect(backup.stats.entityCount).toBe(backup.entities.length)
|
||||
expect(backup.stats.relationCount).toBe(backup.relations.length)
|
||||
// Two user entities, one relation (VFS root excluded by default).
|
||||
expect(backup.entities.map((e) => e.id).sort()).toEqual([a, b].sort())
|
||||
expect(backup.relations).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('round-trips entities and relations through clear() + import()', async () => {
|
||||
const a = await brain.add({ data: 'Alice', type: NounType.Person })
|
||||
const b = await brain.add({ data: 'Bob', type: NounType.Person })
|
||||
const c = await brain.add({ data: 'Carol', type: NounType.Person })
|
||||
await brain.relate({ from: a, to: b, type: VerbType.FriendOf })
|
||||
await brain.relate({ from: b, to: c, type: VerbType.FriendOf })
|
||||
|
||||
const backup = await brain.data().then((d) => d.export({}, { includeVectors: true }))
|
||||
|
||||
await brain.data().then((d) => d.clear())
|
||||
const afterClear = await brain.find({ limit: 1000 })
|
||||
expect(afterClear.filter((r) => [a, b, c].includes(r.id))).toHaveLength(0)
|
||||
|
||||
const result = await brain.data().then((d) => d.import(backup))
|
||||
expect(result.imported).toBe(3)
|
||||
expect(result.errors).toHaveLength(0)
|
||||
|
||||
const ra = await brain.get(a)
|
||||
expect(ra?.id).toBe(a)
|
||||
const rels = await brain.getRelations({ from: a })
|
||||
expect(rels.some((r) => r.to === b && r.type === VerbType.FriendOf)).toBe(true)
|
||||
})
|
||||
|
||||
it('preserves subtype on both entities and relations', async () => {
|
||||
const a = await brain.add({ data: 'Doc', type: NounType.Document, subtype: 'invoice' })
|
||||
const b = await brain.add({ data: 'Doc2', type: NounType.Document, subtype: 'receipt' })
|
||||
await brain.relate({ from: a, to: b, type: VerbType.RelatedTo, subtype: 'supersedes' })
|
||||
|
||||
const backup = await brain.data().then((d) => d.export())
|
||||
|
||||
const ea = backup.entities.find((e) => e.id === a)
|
||||
expect(ea?.subtype).toBe('invoice')
|
||||
expect(backup.relations[0].subtype).toBe('supersedes')
|
||||
})
|
||||
})
|
||||
|
||||
describe('selectors', () => {
|
||||
it('ids — exports exactly the requested entities', async () => {
|
||||
const a = await brain.add({ data: 'A', type: NounType.Thing })
|
||||
const b = await brain.add({ data: 'B', type: NounType.Thing })
|
||||
await brain.add({ data: 'C', type: NounType.Thing })
|
||||
|
||||
const backup = await brain.data().then((d) => d.export({ ids: [a, b] }))
|
||||
expect(backup.entities.map((e) => e.id).sort()).toEqual([a, b].sort())
|
||||
})
|
||||
|
||||
it('collection — exports the collection plus its transitive Contains members', async () => {
|
||||
const root = await brain.add({ data: 'Folder', type: NounType.Collection })
|
||||
const child1 = await brain.add({ data: 'Child1', type: NounType.Document })
|
||||
const child2 = await brain.add({ data: 'Child2', type: NounType.Document })
|
||||
const grandchild = await brain.add({ data: 'Grandchild', type: NounType.Document })
|
||||
await brain.add({ data: 'Outside', type: NounType.Document })
|
||||
await brain.relate({ from: root, to: child1, type: VerbType.Contains })
|
||||
await brain.relate({ from: root, to: child2, type: VerbType.Contains })
|
||||
await brain.relate({ from: child1, to: grandchild, type: VerbType.Contains })
|
||||
|
||||
const backup = await brain.data().then((d) => d.export({ collection: root }))
|
||||
expect(backup.entities.map((e) => e.id).sort()).toEqual(
|
||||
[root, child1, child2, grandchild].sort()
|
||||
)
|
||||
})
|
||||
|
||||
it('connected — exports an N-hop neighbourhood', async () => {
|
||||
const a = await brain.add({ data: 'A', type: NounType.Thing })
|
||||
const b = await brain.add({ data: 'B', type: NounType.Thing })
|
||||
const c = await brain.add({ data: 'C', type: NounType.Thing })
|
||||
await brain.relate({ from: a, to: b, type: VerbType.RelatedTo })
|
||||
await brain.relate({ from: b, to: c, type: VerbType.RelatedTo })
|
||||
|
||||
const depth1 = await brain.data().then((d) => d.export({ connected: { from: a, depth: 1 } }))
|
||||
expect(depth1.entities.map((e) => e.id).sort()).toEqual([a, b].sort())
|
||||
|
||||
const depth2 = await brain.data().then((d) => d.export({ connected: { from: a, depth: 2 } }))
|
||||
expect(depth2.entities.map((e) => e.id).sort()).toEqual([a, b, c].sort())
|
||||
})
|
||||
|
||||
it('predicate — filters by type', async () => {
|
||||
await brain.add({ data: 'P', type: NounType.Person })
|
||||
await brain.add({ data: 'D', type: NounType.Document })
|
||||
const backup = await brain.data().then((d) => d.export({ type: NounType.Person }))
|
||||
expect(backup.entities.every((e) => e.type === NounType.Person)).toBe(true)
|
||||
expect(backup.entities).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('compose — structural selector + predicate filter', async () => {
|
||||
const root = await brain.add({ data: 'Folder', type: NounType.Collection })
|
||||
const open = await brain.add({
|
||||
data: 'Open',
|
||||
type: NounType.Document,
|
||||
metadata: { status: 'open' }
|
||||
})
|
||||
const closed = await brain.add({
|
||||
data: 'Closed',
|
||||
type: NounType.Document,
|
||||
metadata: { status: 'closed' }
|
||||
})
|
||||
await brain.relate({ from: root, to: open, type: VerbType.Contains })
|
||||
await brain.relate({ from: root, to: closed, type: VerbType.Contains })
|
||||
|
||||
const backup = await brain
|
||||
.data()
|
||||
.then((d) => d.export({ collection: root, where: { status: 'open' } }))
|
||||
expect(backup.entities.map((e) => e.id)).toContain(open)
|
||||
expect(backup.entities.map((e) => e.id)).not.toContain(closed)
|
||||
})
|
||||
})
|
||||
|
||||
describe('edge policy', () => {
|
||||
let a: string
|
||||
let b: string
|
||||
let c: string
|
||||
|
||||
beforeEach(async () => {
|
||||
a = await brain.add({ data: 'A', type: NounType.Thing })
|
||||
b = await brain.add({ data: 'B', type: NounType.Thing })
|
||||
c = await brain.add({ data: 'C', type: NounType.Thing })
|
||||
await brain.relate({ from: a, to: b, type: VerbType.RelatedTo }) // both in {a,b}
|
||||
await brain.relate({ from: b, to: c, type: VerbType.RelatedTo }) // dangles to c
|
||||
})
|
||||
|
||||
it('induced (default) — only edges with both endpoints in the set', async () => {
|
||||
const backup = await brain.data().then((d) => d.export({ ids: [a, b] }))
|
||||
expect(backup.relations).toHaveLength(1)
|
||||
expect(backup.relations[0].from).toBe(a)
|
||||
expect(backup.danglingIds).toBeUndefined()
|
||||
})
|
||||
|
||||
it('incident — includes dangling edges + records danglingIds', async () => {
|
||||
const backup = await brain.data().then((d) => d.export({ ids: [a, b] }, { edges: 'incident' }))
|
||||
expect(backup.relations).toHaveLength(2)
|
||||
expect(backup.danglingIds).toContain(c)
|
||||
})
|
||||
|
||||
it('none — nodes only', async () => {
|
||||
const backup = await brain.data().then((d) => d.export({ ids: [a, b] }, { edges: 'none' }))
|
||||
expect(backup.relations).toHaveLength(0)
|
||||
})
|
||||
})
|
||||
|
||||
describe('vectors + re-embedding', () => {
|
||||
it('omits vectors by default and re-embeds on import', async () => {
|
||||
const a = await brain.add({ data: 'Re-embed me', type: NounType.Thing })
|
||||
const backup = await brain.data().then((d) => d.export({ ids: [a] }))
|
||||
expect(backup.entities[0].vector).toBeUndefined()
|
||||
|
||||
await brain.data().then((d) => d.clear())
|
||||
const result = await brain.data().then((d) => d.import(backup))
|
||||
expect(result.reembedded).toBe(1)
|
||||
|
||||
const restored = await brain.get(a, { includeVectors: true })
|
||||
expect(restored?.vector?.length).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
it('carries vectors verbatim with includeVectors (no re-embed)', async () => {
|
||||
const a = await brain.add({ data: 'Keep my vector', type: NounType.Thing })
|
||||
const original = await brain.get(a, { includeVectors: true })
|
||||
|
||||
const backup = await brain.data().then((d) => d.export({ ids: [a] }, { includeVectors: true }))
|
||||
expect(backup.entities[0].vector?.length).toBe(original?.vector?.length)
|
||||
|
||||
await brain.data().then((d) => d.clear())
|
||||
const result = await brain.data().then((d) => d.import(backup))
|
||||
expect(result.reembedded).toBe(0)
|
||||
|
||||
const restored = await brain.get(a, { includeVectors: true })
|
||||
expect(restored?.vector).toEqual(original?.vector)
|
||||
})
|
||||
|
||||
it('reembed:never records an error when no vector is carried', async () => {
|
||||
const a = await brain.add({ data: 'No vector', type: NounType.Thing })
|
||||
const backup = await brain.data().then((d) => d.export({ ids: [a] })) // no vectors
|
||||
await brain.data().then((d) => d.clear())
|
||||
|
||||
const result = await brain.data().then((d) => d.import(backup, { reembed: 'never' }))
|
||||
expect(result.imported).toBe(0)
|
||||
expect(result.errors).toHaveLength(1)
|
||||
expect(result.errors[0].error).toMatch(/reembed:never/)
|
||||
})
|
||||
})
|
||||
|
||||
describe('conflict handling', () => {
|
||||
it('merge (default) — updates existing entities in place', async () => {
|
||||
const a = await brain.add({ data: 'V1', type: NounType.Thing, metadata: { n: 1 } })
|
||||
const backup = await brain.data().then((d) => d.export({ ids: [a] }, { includeVectors: true }))
|
||||
|
||||
await brain.update({ id: a, metadata: { n: 99 }, merge: false })
|
||||
const result = await brain.data().then((d) => d.import(backup, { onConflict: 'merge' }))
|
||||
expect(result.merged).toBe(1)
|
||||
expect(result.imported).toBe(0)
|
||||
|
||||
const restored = await brain.get(a)
|
||||
expect(restored?.metadata?.n).toBe(1)
|
||||
})
|
||||
|
||||
it('skip — leaves existing entities untouched', async () => {
|
||||
const a = await brain.add({ data: 'Orig', type: NounType.Thing, metadata: { n: 1 } })
|
||||
const backup = await brain.data().then((d) => d.export({ ids: [a] }))
|
||||
await brain.update({ id: a, metadata: { n: 2 }, merge: false })
|
||||
|
||||
const result = await brain.data().then((d) => d.import(backup, { onConflict: 'skip' }))
|
||||
expect(result.skipped).toBe(1)
|
||||
const restored = await brain.get(a)
|
||||
expect(restored?.metadata?.n).toBe(2)
|
||||
})
|
||||
})
|
||||
|
||||
describe('id remapping (clone)', () => {
|
||||
it('imports a subgraph under fresh ids', async () => {
|
||||
const a = await brain.add({ data: 'A', type: NounType.Thing })
|
||||
const b = await brain.add({ data: 'B', type: NounType.Thing })
|
||||
await brain.relate({ from: a, to: b, type: VerbType.RelatedTo })
|
||||
const backup = await brain.data().then((d) => d.export({ ids: [a, b] }, { includeVectors: true }))
|
||||
|
||||
const remap = new Map<string, string>([
|
||||
[a, randomUUID()],
|
||||
[b, randomUUID()]
|
||||
])
|
||||
const result = await brain
|
||||
.data()
|
||||
.then((d) => d.import(backup, { remapIds: (id) => remap.get(id) ?? id }))
|
||||
expect(result.imported).toBe(2)
|
||||
|
||||
const clonedA = await brain.get(remap.get(a)!)
|
||||
expect(clonedA?.id).toBe(remap.get(a))
|
||||
// The original entities still exist (a clone, not a move).
|
||||
expect((await brain.get(a))?.id).toBe(a)
|
||||
const clonedRels = await brain.getRelations({ from: remap.get(a)! })
|
||||
expect(clonedRels.some((r) => r.to === remap.get(b))).toBe(true)
|
||||
})
|
||||
})
|
||||
|
||||
describe('system entities', () => {
|
||||
it('excludes the VFS root from a whole-brain export by default', async () => {
|
||||
await brain.add({ data: 'User entity', type: NounType.Thing })
|
||||
const backup = await brain.data().then((d) => d.export())
|
||||
expect(backup.entities.map((e) => e.id)).not.toContain(VFS_ROOT_ID)
|
||||
})
|
||||
})
|
||||
|
||||
describe('import validation', () => {
|
||||
it('rejects a non-BackupData payload', async () => {
|
||||
const d = await brain.data()
|
||||
await expect(d.import({ entities: [] } as any)).rejects.toThrow(/BackupData/)
|
||||
})
|
||||
|
||||
it('rejects a newer formatVersion', async () => {
|
||||
const d = await brain.data()
|
||||
const future: BackupData = {
|
||||
format: 'brainy-backup',
|
||||
formatVersion: 999,
|
||||
brainyVersion: 'x',
|
||||
createdAt: new Date().toISOString(),
|
||||
embedding: { model: 'm', dimensions: 384 },
|
||||
entities: [],
|
||||
relations: [],
|
||||
stats: { entityCount: 0, relationCount: 0, blobCount: 0 }
|
||||
}
|
||||
await expect(d.import(future)).rejects.toThrow(/formatVersion/)
|
||||
})
|
||||
})
|
||||
})
|
||||
Loading…
Add table
Add a link
Reference in a new issue