diff --git a/RELEASES.md b/RELEASES.md index 81ef906a..ea16c8d2 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -10,6 +10,53 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so --- +## v7.32.0 — 2026-06-16 + +**Affected products:** anyone who needs a **portable graph backup**, a partial export, or a +**7.x → 8.0 migration path**. Purely additive — no API removed, no data change. Drop-in. + +### New — portable graph `export()` / `import()` on `brain.data()` + +`brain.data()` gains a portable backup/restore pair that serializes part or all of a brain +to a single versioned JSON document (`BackupData`) and restores it. This is the format that +embeds inside portable artifacts (e.g. a serialized subgraph carried by a higher-level file +type) and the recommended way to move a graph between environments or **upgrade an existing +7.x brain into an 8.0 brain**. + +```typescript +const data = await brain.data() + +const backup = await data.export() // whole brain → BackupData +const subset = await data.export({ ids }, { includeVectors: true }) +await otherBrain.data().then(d => d.import(subset, { onConflict: 'merge' })) +``` + +- **Selectors** — `export(selector?, options?)` picks the node set: `{ ids }`, `{ collection }` + (alias `memberOf`) + transitive `Contains` members, `{ connected: { from, depth, verbs?, direction? } }`, + `{ vfsPath, recursive? }`, a predicate (`{ type, subtype, where, service }`), or the whole brain + (omit). Structural + predicate selectors **compose**. +- **Options** — `includeVectors` (default off → re-embed on import), `includeContent` (VFS file + bytes in `blobs`), `includeSystem` (include the VFS root etc.), `edges` (`'induced'` default / + `'incident'` / `'none'`). +- **Import** — `import(backup, options?)` → `{ imported, merged, skipped, reembedded, blobsWritten, errors }`. + `onConflict` defaults to `'merge'` (dedup-by-id, so you can assemble many documents into one + graph); `reembed` defaults to `'auto'` (re-embed from `data` when no vector is carried); + `remapIds` clones a subgraph under fresh ids. +- **Format** — `{ format:'brainy-backup', formatVersion:1, brainyVersion, createdAt, embedding, + entities, relations, blobs?, danglingIds?, stats }`. Entities carry `subtype` and the standard + reserved fields at the top level; `metadata` is custom-only. **Current-state** (no generation + history). `formatVersion` gates cross-version import (a 7.x document imports into 8.0). +- **Types** — `BackupData`, `BackupEntity`, `BackupRelation`, `ExportSelector`, `ExportOptions`, + `ImportOptions`, `ImportResult`, and the `DataAPI` class are now exported from the package root. + +This replaces the previous `brain.data().export()` flat-entity array (which dropped relations) +with a graph-complete document. Distinct from `brain.import(file)` (CSV/PDF/Excel/JSON ingestion), +which is unchanged. The CLI `brainy export` now writes a `BackupData` document (json/jsonl/csv). + +Guide: `docs/guides/export-and-import.md`. Tested in `tests/unit/api/data-backup.test.ts`. + +--- + ## v7.31.8 — 2026-06-16 **Affected products:** every consumer on a bare VM or `mmap-filesystem` storage — **upgrade recommended** (Memory, Workshop on 7.31.x; Venue on 7.28.x). Two platform-wide production fixes. Drop-in; no API or data changes. diff --git a/docs/api/README.md b/docs/api/README.md index 69df13fb..595a3bbd 100644 --- a/docs/api/README.md +++ b/docs/api/README.md @@ -1765,20 +1765,33 @@ await brain.import('https://api.example.com/data.json') --- -### Export & Snapshots +### Export & Import (portable backup) + +`brain.data()` exposes a portable graph backup/restore API. `export(selector?, options?)` +serializes part or all of the graph to a versioned, portable `BackupData` document; +`import(backup, options?)` restores it (dedup-by-id merge by default, re-embedding when +vectors are absent). ```typescript -// Export to file -await brain.export('/path/to/backup.brainy') +const data = await brain.data() -// Create instant snapshot using COW fork -await brain.fork('backup-2025-01-19') +// Whole brain → a portable BackupData document +const backup = await data.export() -// Time-travel to specific commit -const snapshot = await brain.asOf(commitId) -const entities = await snapshot.find({ limit: 100 }) +// Just one workbench's members, with vectors, then restore elsewhere (merge by id) +const subset = await data.export({ ids }, { includeVectors: true }) +await otherBrain.data().then(d => d.import(subset, { onConflict: 'merge' })) + +// A collection + its children; a connected neighbourhood; a VFS subtree (+ bytes) +await data.export({ collection: collectionId }) +await data.export({ connected: { from: id, depth: 2 } }) +await data.export({ vfsPath: '/docs' }, { includeContent: true }) ``` +The format is versioned (`formatVersion`) and current-state (no generation history) — see +the **[Export & Import guide →](../guides/export-and-import.md)**. This is distinct from +`brain.import(file)` (CSV/PDF/Excel/JSON ingestion). + --- ## Configuration diff --git a/docs/architecture/storage-architecture.md b/docs/architecture/storage-architecture.md index 5c71d50c..fb9c0025 100644 --- a/docs/architecture/storage-architecture.md +++ b/docs/architecture/storage-architecture.md @@ -457,23 +457,19 @@ await brain.storage.withLock('resource-id', async () => { ### Export Data ```typescript -// Export entire database -const backup = await brain.export({ - format: 'json', - includeVectors: true, - includeIndexes: false -}) +// Export the whole brain to a portable BackupData document +const backup = await brain.data().then(d => d.export(undefined, { includeVectors: true })) ``` ### Import Data ```typescript -// Import from backup -await brain.import(backup, { - mode: 'merge', // or 'replace' - validateSchema: true -}) +// Restore a BackupData document (dedup-by-id merge by default) +await brain.data().then(d => d.import(backup, { onConflict: 'merge' })) ``` +See the [Export & Import guide](../guides/export-and-import.md) for partial exports +(by id, collection, connected neighbourhood, VFS subtree, or predicate). + ### Storage Migration ```typescript // Migrate between storage types @@ -483,9 +479,9 @@ const newBrain = new Brainy({ storage: { type: 's3' } }) await oldBrain.init() await newBrain.init() -// Transfer all data -const data = await oldBrain.export() -await newBrain.import(data) +// Transfer all data via a portable backup +const data = await oldBrain.data().then(d => d.export(undefined, { includeVectors: true })) +await newBrain.data().then(d => d.import(data)) ``` ## Performance Tuning diff --git a/docs/augmentations/EXAMPLES.md b/docs/augmentations/EXAMPLES.md index 142f68fd..808139ba 100644 --- a/docs/augmentations/EXAMPLES.md +++ b/docs/augmentations/EXAMPLES.md @@ -546,11 +546,9 @@ const appComponent = await brain.findOne({ type: 'ReactComponent', name: 'App' } const imports = await brain.getRelated(appComponent.id, 'Imports') console.log(`App component imports:`, imports.map(c => c.name)) -// Export diagram -const diagram = await brain.export({ - format: 'react-diagram' -}) -console.log(diagram.diagram) // Mermaid diagram +// Export the analyzed component graph as a portable BackupData document +const graph = await brain.data().then(d => d.export({ type: 'ReactComponent' })) +console.log(`Exported ${graph.stats.entityCount} components, ${graph.stats.relationCount} edges`) ``` ### Step 4: Premium Licensing (Optional) diff --git a/docs/guides/export-and-import.md b/docs/guides/export-and-import.md new file mode 100644 index 00000000..3f367d33 --- /dev/null +++ b/docs/guides/export-and-import.md @@ -0,0 +1,173 @@ +--- +title: Export & Import (portable graph) +slug: guides/export-and-import +public: true +category: guides +template: guide +order: 9 +description: Export part or all of a brain to a portable, versioned BackupData document and import it back — by id, collection, connected neighbourhood, VFS subtree, predicate, or the whole brain. Covers vectors, edge policy, conflict handling, and id remapping. +next: + - guides/subtypes-and-facets + - api/README +--- + +# Export & Import (portable graph) + +`brain.data()` exposes a **portable graph export/import** API. One method serializes a +graph — an item, a collection, a connected neighbourhood, a VFS subtree, a predicate +match, or the whole brain — into a single versioned JSON document (`BackupData`); the +inverse restores it. + +```typescript +const data = await brain.data() + +const backup = await data.export() // whole brain → BackupData +await data.import(backup) // restore (merge by id, re-embed if no vectors) +``` + +It is **portable** (human-readable JSON), **versioned** (`formatVersion`, so a 7.x export +imports cleanly into 8.0), and **current-state** (the entities and edges as they are now — +no generation history). Use it for portable artifacts, partial backups, cross-environment +moves, and version upgrades. + +## When to use which + +| You want… | Use | +|-----------|-----| +| A portable, partial-or-whole, cross-version graph document | **`brain.data().export()` / `import()`** (this guide) | +| To ingest a CSV / PDF / Excel / JSON **file** as new entities | `brain.import(file)` — see [Import Anything](./import-anything.md) | + +The two are different operations that happen to share a verb: `import(file)` parses a +foreign document into new entities; `data().import(backup)` restores a graph this API +exported. + +## Exporting + +```typescript +export(selector?, options?): Promise +``` + +### Selectors — *what* to export + +Omit the selector to export the whole brain. Otherwise pick a node set: + +| Scenario | Selector | +|----------|----------| +| Just an item (or items) | `{ ids: ['a', 'b'] }` | +| A collection + its children | `{ collection: collectionId }` (alias `memberOf`) | +| A connected neighbourhood | `{ connected: { from: id, depth: 2, verbs?, direction? } }` | +| A VFS directory / file (+ subtree) | `{ vfsPath: '/docs', recursive?: true }` | +| Everything matching a predicate | `{ type, subtype, where, service }` | +| The whole brain | *(omit)* | + +The selector reuses `find()`'s grammar — *"export what `find()` would match, minus ranking +and limit."* Structural and predicate selectors **compose** — a structural selector picks +the nodes, and predicate keys then filter them: + +```typescript +// Members of a collection whose status is "open" +await data.export({ collection: collectionId, where: { status: 'open' } }) +``` + +If you already have results from `find()`, export exactly those with the `ids` selector: + +```typescript +const hits = await brain.find({ type: NounType.Document }) +await data.export({ ids: hits.map(r => r.id) }) +``` + +### Options — *how* to serialize + +| Option | Default | Effect | +|--------|---------|--------| +| `includeVectors` | `false` | Carry embedding vectors verbatim. Off ⇒ `import()` re-embeds from `data`. | +| `includeContent` | `false` | Include VFS file bytes in `blobs` so files round-trip byte-identically. | +| `includeSystem` | `false` | Include `system` entities such as the VFS root. | +| `edges` | `'induced'` | `'induced'` (both endpoints in the set), `'incident'` (also dangling edges, recorded in `danglingIds`), or `'none'` (nodes only). | + +```typescript +// A self-contained subgraph with vectors, ready to restore elsewhere +const subset = await data.export({ ids }, { includeVectors: true }) + +// A VFS subtree including file bytes +const tree = await data.export({ vfsPath: '/docs' }, { includeContent: true }) +``` + +## Importing + +```typescript +import(backup, options?): Promise +``` + +```typescript +const result = await brain.data().then(d => d.import(backup, { onConflict: 'merge' })) +// → { imported, merged, skipped, reembedded, blobsWritten, errors } +``` + +| Option | Default | Effect | +|--------|---------|--------| +| `onConflict` | `'merge'` | `'merge'` (update existing id in place — assemble many backups), `'replace'` (delete + recreate), or `'skip'`. | +| `reembed` | `'auto'` | `'auto'` (use the carried vector, else re-embed from `data`) or `'never'` (require a carried vector; record an error if absent). | +| `remapIds` | — | Rewrite every id on the way in, e.g. to clone a template subgraph under fresh ids. | + +The default `onConflict: 'merge'` is what lets you assemble one working graph from many +exported documents that share entity ids — re-importing an id merges rather than duplicates. + +```typescript +// Clone a subgraph under fresh ids (a copy, not a move) +const remap = new Map(backup.entities.map(e => [e.id, crypto.randomUUID()])) +await data.import(backup, { remapIds: id => remap.get(id) ?? id }) +``` + +## The `BackupData` format + +```jsonc +{ + "format": "brainy-backup", + "formatVersion": 1, // import gates on this (cross-version migration) + "brainyVersion": "7.32.0", + "createdAt": "2026-06-16T…Z", + "embedding": { "model": "all-MiniLM-L6-v2", "dimensions": 384 }, + "selector": { … }, // echoes what was exported (provenance) + "entities": [ + { + "id": "…", "type": "Document", "subtype": "invoice", + "data": "…", // the embedding source + "confidence": 1, "weight": 1, "service": "…", + "vector": [ … ], // only with includeVectors + "metadata": { … } // custom fields only (reserved fields are top-level) + } + ], + "relations": [ + { "id":"…", "from":"…", "to":"…", "type":"Contains", "subtype":"…", + "weight":1, "confidence":1, "metadata": { … } } + ], + "blobs": { "": "" }, // only with includeContent + "danglingIds": [ "…" ], // only with edges:'incident' + "stats": { "entityCount": 0, "relationCount": 0, "blobCount": 0, "vectorDimensions": 384 } +} +``` + +Standard fields (`subtype`, `data`, `confidence`, `weight`, `service`) sit at the top level +of each entity; `metadata` holds **only** custom user fields — mirroring the in-memory +`Entity` shape, so `import()` maps each field to its dedicated parameter. + +## Cross-version (7.x → 8.0) + +Because the document is shared and versioned, a backup written by 7.x imports into 8.0: +`formatVersion` is read forward, `subtype` is carried so 8.0 re-types correctly, and the +same 384-dimension model on both lines means `includeVectors:false` re-embeds identically +(or `true` carries vectors verbatim). The format is **current-state** — if you need a +whole-brain snapshot *with* generation history, that is a separate native facility +(`db.persist()` / `Brainy.load()` on 8.0). + +## VFS + +VFS directories are `Collection` entities and files are entities linked by `Contains`, so +the whole filesystem (or any subtree) exports through the `vfsPath` selector: + +```typescript +await data.export({ vfsPath: '/' }, { includeContent: true }) // all VFS + bytes +await data.export({ vfsPath: '/docs' }, { includeContent: true }) // one directory +await data.export({ vfsPath: '/a/b.txt' }, { includeContent: true }) // one file +``` diff --git a/docs/guides/framework-integration.md b/docs/guides/framework-integration.md index 23364178..fab81011 100644 --- a/docs/guides/framework-integration.md +++ b/docs/guides/framework-integration.md @@ -540,11 +540,11 @@ export async function generateStaticProps() { }) await brain.init() - // Build search index - const allContent = await brain.export() + // Build a search index — export the whole brain as a portable BackupData document + const backup = await brain.data().then(d => d.export()) return { - props: { searchIndex: allContent } + props: { searchIndex: backup.entities } } } ``` diff --git a/src/api/DataAPI.ts b/src/api/DataAPI.ts index 8ba1a49f..b873a9ed 100644 --- a/src/api/DataAPI.ts +++ b/src/api/DataAPI.ts @@ -1,302 +1,495 @@ /** - * Data Management API for Brainy 3.0 - * Provides backup, restore, import, export, and data management + * @module DataAPI + * @description Portable graph backup/restore for Brainy — the `BackupData v1` + * export/import format plus `clear()`/`getStats()` data-management helpers. + * + * Accessed via `brain.data()`. The two headline methods are: + * + * - **`export(selector?, options?) → BackupData`** — serialize a graph (an item, a + * collection + children, a connected neighbourhood, a VFS subtree, a predicate + * match, or the whole brain) into ONE versioned, portable JSON document. + * - **`import(backup, options?) → ImportResult`** — restore a `BackupData` into the + * brain (dedup-by-id merge by default), re-embedding from `data` when vectors are + * absent. + * + * This is the **portable** round-trip: human-readable, partial-or-whole, and + * cross-version (a `formatVersion: 1` document written by 7.x imports cleanly into + * 8.0). It is distinct from `brain.import()` (file ingestion — CSV/PDF/Excel/JSON) + * and, on 8.0, from `db.persist()`/`Brainy.load()` (the native whole-brain snapshot, + * which preserves generation history but is neither portable JSON nor cross-version). + * + * **Design decision (reserved-field split):** entities carry Brainy's standard fields + * (`subtype`, `data`, `confidence`, `weight`, `service`, `createdBy`, `createdAt`) at + * the TOP LEVEL of each `BackupEntity`, and `metadata` holds ONLY custom user fields — + * mirroring the in-memory `Entity` shape, so `import()` maps each field to its dedicated + * `add()`/`relate()` parameter rather than dumping everything into the metadata bag. */ -import { StorageAdapter, HNSWNoun, GraphVerb } from '../coreTypes.js' +import { StorageAdapter } from '../coreTypes.js' import { Entity, Relation } from '../types/brainy.types.js' import { NounType, VerbType } from '../types/graphTypes.js' +import { getBrainyVersion } from '../utils/version.js' -export interface BackupOptions { - includeVectors?: boolean - compress?: boolean - format?: 'json' | 'binary' -} +/** The fixed entity id of the VFS root collection (excluded from exports unless `includeSystem`). */ +const VFS_ROOT_ID = '00000000-0000-0000-0000-000000000000' +/** Magic string identifying a Brainy portable backup document. */ +const BACKUP_FORMAT = 'brainy-backup' +/** Current portable-format version. Import gates on this for cross-version migration. */ +const BACKUP_FORMAT_VERSION = 1 +/** Default embedding model label (informational; `dimensions` is the real compat gate). */ +const DEFAULT_EMBED_MODEL = 'all-MiniLM-L6-v2' +/** Storage-layer fetch ceiling for enumerations (well above any single-brain entity count). */ +const ENUMERATION_LIMIT = 1_000_000 +/** Per-node relation fetch ceiling (filtered by source/target, so no unfiltered-scan warning). */ +const RELATION_FETCH_LIMIT = 100_000 -export interface RestoreOptions { - merge?: boolean - overwrite?: boolean - validate?: boolean -} - -export interface ImportOptions { - format: 'json' | 'csv' - mapping?: Record - batchSize?: number - validate?: boolean -} - -export interface ExportOptions { - format?: 'json' | 'csv' - filter?: { - type?: NounType | NounType[] - where?: Record - service?: string - } - includeVectors?: boolean -} - -export interface BackupData { - version: string - timestamp: number - entities: Array<{ - id: string - vector?: number[] - type: string - metadata: any - service?: string - }> - relations: Array<{ - id: string +/** + * @description Selects WHICH part of the graph to export. Reuses `find()`'s grammar + * (export "what find would match, minus ranking/limit") plus export-specific selectors. + * Omit the selector entirely (or pass `{}`) to export the whole brain. + * + * Structural selectors (`ids` / `collection` / `connected` / `vfsPath`) and predicate + * selectors (`type` / `subtype` / `where` / `service` / `visibility`) **compose**: a + * structural selector picks the node set, and any predicate keys then filter it + * (e.g. `{ collection: id, where: { status: 'open' } }` = members matching the predicate). + */ +export interface ExportSelector { + /** Exactly these entity ids. */ + ids?: string[] + /** A collection id → the collection + its transitive `Contains` members. */ + collection?: string + /** Alias for `collection`. */ + memberOf?: string + /** An entity + its N-hop neighbourhood (reuses graph traversal). */ + connected?: { + /** Start entity id. */ from: string - to: string - type: string - weight: number - metadata?: any - }> - config?: Record + /** Hops to traverse (default: 1). */ + depth?: number + /** Restrict to these verb types (default: all). */ + verbs?: VerbType[] + /** Edge direction to follow (default: `'out'`). */ + direction?: 'out' | 'in' | 'both' + } + /** A VFS path → the directory/file + (for a directory) its `Contains` subtree. */ + vfsPath?: string + /** For `vfsPath` directories: include the whole subtree (default: true). */ + recursive?: boolean + /** For `collection`/`vfsPath`: cap traversal depth (default: unbounded). */ + depth?: number + /** Predicate: entity type(s). */ + type?: NounType | NounType[] + /** Predicate: entity subtype(s). */ + subtype?: string | string[] + /** Predicate: exact-match metadata fields. */ + where?: Record + /** Predicate: multi-tenancy service id. */ + service?: string + /** Predicate: visibility (`'public'` matches entities with no explicit visibility). */ + visibility?: string +} + +/** + * @description Controls HOW the selected graph is serialized. + */ +export interface ExportOptions { + /** Include embedding vectors verbatim (default: false → `import()` re-embeds from `data`). */ + includeVectors?: boolean + /** Include VFS file bytes in `blobs` so files round-trip byte-identically (default: false). */ + includeContent?: boolean + /** Include `visibility:'system'` entities (e.g. the VFS root) (default: false). */ + includeSystem?: boolean + /** + * Which edges to include (default: `'induced'`): + * - `'induced'` — only edges whose BOTH endpoints are in the node set (self-contained subgraph). + * - `'incident'` — also edges that dangle to outside ids (recorded in `danglingIds`). + * - `'none'` — nodes only. + */ + edges?: 'induced' | 'incident' | 'none' +} + +/** + * @description Controls how a `BackupData` is applied to the brain on `import()`. + */ +export interface ImportOptions { + /** + * Conflict policy when an entity id already exists (default: `'merge'`): + * - `'merge'` — update in place (dedup-by-id; the default that lets you assemble many backups). + * - `'replace'` — delete then re-create. + * - `'skip'` — leave the existing entity untouched. + */ + onConflict?: 'merge' | 'replace' | 'skip' + /** + * Vector policy (default: `'auto'`): + * - `'auto'` — use the carried vector when present, otherwise re-embed from `data`. + * - `'never'` — use the carried vector when present, otherwise record an error (no re-embed). + */ + reembed?: 'auto' | 'never' + /** Rewrite every id on the way in (e.g. to clone a template subgraph under fresh ids). */ + remapIds?: (id: string) => string +} + +/** One entity in a `BackupData`. Standard fields top-level; `metadata` is custom-only. */ +export interface BackupEntity { + id: string + /** NounType value. */ + type: string + /** Per-product sub-classification. */ + subtype?: string + /** Visibility (omitted ⇒ `'public'`; never `'system'` unless `includeSystem`). */ + visibility?: string + /** Opaque content payload (the embedding source). */ + data?: any + /** Type-classification confidence (0–1). */ + confidence?: number + /** Entity importance/salience (0–1). */ + weight?: number + /** Multi-tenancy service id. */ + service?: string + /** Provenance: what created this entity. */ + createdBy?: any + /** Original creation timestamp (informational; the target brain assigns its own on import). */ + createdAt?: number + /** Embedding vector — present only when exported with `includeVectors`. */ + vector?: number[] + /** Custom user fields only (reserved fields live at the top level). */ + metadata?: any +} + +/** One relation (edge) in a `BackupData`. */ +export interface BackupRelation { + id: string + from: string + to: string + /** VerbType value. */ + type: string + /** Per-product edge sub-classification. */ + subtype?: string + /** Visibility (omitted ⇒ `'public'`). */ + visibility?: string + /** Connection strength (0–1). */ + weight?: number + /** Relationship certainty (0–1). */ + confidence?: number + /** Custom user fields on the edge. */ + metadata?: any +} + +/** + * @description A self-describing, versioned, portable graph document. The same shape + * is produced/consumed on 7.x and 8.0; `formatVersion` gates cross-version migration. + */ +export interface BackupData { + /** Always `'brainy-backup'` — identifies the document type. */ + format: typeof BACKUP_FORMAT + /** Integer format version (import gates on this). */ + formatVersion: number + /** The Brainy version that produced the document (informational). */ + brainyVersion: string + /** ISO-8601 creation time. */ + createdAt: string + /** Embedding manifest — `import()` verifies dimension compatibility before re-embedding. */ + embedding: { model: string; dimensions: number } + /** Echo of the selector that produced this document (provenance). */ + selector?: ExportSelector + /** Exported entities. */ + entities: BackupEntity[] + /** Exported relations. */ + relations: BackupRelation[] + /** VFS file bytes keyed by sha256 — present only with `includeContent`. */ + blobs?: Record + /** Endpoints referenced by `edges:'incident'` that fell outside the node set. */ + danglingIds?: string[] + /** Summary counts. */ stats: { entityCount: number relationCount: number + blobCount: number vectorDimensions?: number } } +/** Outcome of an `import()`. */ export interface ImportResult { - successful: number - failed: number - errors: Array<{ item: any; error: string }> - duration: number + /** New entities created. */ + imported: number + /** Existing entities merged (dedup-by-id). */ + merged: number + /** Existing entities left untouched (`onConflict:'skip'`). */ + skipped: number + /** Entities re-embedded from `data` because no vector was carried. */ + reembedded: number + /** VFS blob bytes written. */ + blobsWritten: number + /** Per-record failures (id + message); the import continues past individual errors. */ + errors: Array<{ id: string; error: string }> } +/** + * @description Data-management API for a Brainy instance: portable graph + * `export()`/`import()` plus `clear()`/`getStats()`. Constructed by `brain.data()`. + */ export class DataAPI { - private brain: any // Reference to Brainy instance for neural import - + /** + * @param storage - The brain's storage adapter (used for blob bytes + bulk enumeration). + * @param brain - The owning Brainy instance (drives export/import via its public API). + */ constructor( private storage: StorageAdapter, - private getEntity: (id: string) => Promise, - private getRelation?: (id: string) => Promise, - brain?: any - ) { - this.brain = brain + private brain: any + ) {} + + // ============================================================================ + // EXPORT + // ============================================================================ + + /** + * @description Serialize part or all of the graph into a portable `BackupData`. + * @param selector - WHAT to export (omit for the whole brain). See {@link ExportSelector}. + * @param options - HOW to export (vectors, file bytes, edge policy). See {@link ExportOptions}. + * @returns A versioned, portable `BackupData` document. + * @example + * // A single workbench's members (exact id set), with vectors: + * const backup = await brain.data().export({ ids }, { includeVectors: true }) + * @example + * // A collection and everything under it: + * const backup = await brain.data().export({ collection: collectionId }) + * @example + * // A VFS subtree including file bytes: + * const backup = await brain.data().export({ vfsPath: '/docs' }, { includeContent: true }) + * @example + * // The whole brain: + * const backup = await brain.data().export() + */ + async export(selector: ExportSelector = {}, options: ExportOptions = {}): Promise { + if (!this.brain) { + throw new Error('DataAPI.export() requires a Brainy instance (use brain.data().export()).') + } + const { + includeVectors = false, + includeContent = false, + includeSystem = false, + edges = 'induced' + } = options + + // 1. Resolve the candidate node set (structural selector, or all ids for whole/predicate). + let candidateIds = await this.resolveCandidates(selector, includeSystem) + + // 2. Read canonical entities (reserved fields top-level, metadata custom-only). + const entityMap = await this.brain.batchGet(candidateIds, { includeVectors }) + + // 3. Apply predicate filtering on the canonical entities (handles predicate-only + compose). + let idSet: Set + if (this.hasPredicate(selector)) { + idSet = new Set() + for (const id of candidateIds) { + const e = entityMap.get(id) + if (e && this.matchesPredicate(e, selector)) idSet.add(id) + } + } else { + idSet = new Set(candidateIds.filter((id: string) => entityMap.has(id))) + } + + // 4. Build entity records. + const entities: BackupEntity[] = [] + for (const id of idSet) { + const e = entityMap.get(id) + if (e) entities.push(this.toBackupEntity(e, includeVectors)) + } + + // 5. Collect edges per policy. + const { relations, danglingIds } = await this.collectEdges(idSet, edges) + + // 6. Collect VFS blob bytes (only when requested). + let blobs: Record | undefined + if (includeContent) { + blobs = await this.collectBlobs(entities, entityMap) + } + + // 7. Resolve the embedding dimension (from a carried vector, else the brain's dimension). + const dimensions = + entities.find((e) => e.vector && e.vector.length)?.vector?.length ?? + this.detectDimensions() ?? + 384 + + const blobCount = blobs ? Object.keys(blobs).length : 0 + + return { + format: BACKUP_FORMAT, + formatVersion: BACKUP_FORMAT_VERSION, + brainyVersion: getBrainyVersion(), + createdAt: new Date().toISOString(), + embedding: { model: this.detectModel(), dimensions }, + selector, + entities, + relations, + ...(blobs && blobCount > 0 ? { blobs } : {}), + ...(danglingIds && danglingIds.length > 0 ? { danglingIds } : {}), + stats: { + entityCount: entities.length, + relationCount: relations.length, + blobCount, + vectorDimensions: dimensions + } + } } - async clear(params: { - entities?: boolean - relations?: boolean - config?: boolean - } = {}): Promise { - const { entities = true, relations = true, config = false } = params + // ============================================================================ + // IMPORT + // ============================================================================ + + /** + * @description Restore a `BackupData` into the brain. Dedup-by-id merge by default, so + * assembling many backups that share entity ids merges rather than duplicates. Vectors + * are re-embedded from `data` when absent (`reembed:'auto'`). + * @param data - A `BackupData` document (must have `format:'brainy-backup'`). + * @param options - Conflict/vector/id-remap policy. See {@link ImportOptions}. + * @returns Counts of imported/merged/skipped/re-embedded entities + any per-record errors. + * @throws If `data` is not a `BackupData`, or its `formatVersion` is newer than supported. + * @example + * const result = await brain.data().import(backup, { onConflict: 'merge' }) + */ + async import(data: BackupData, options: ImportOptions = {}): Promise { + if (!this.brain) { + throw new Error('DataAPI.import() requires a Brainy instance (use brain.data().import()).') + } + if (!data || (data as any).format !== BACKUP_FORMAT) { + throw new Error( + `DataAPI.import() expects a BackupData document (format:'${BACKUP_FORMAT}'). ` + + `For file ingestion (CSV/PDF/Excel/JSON), use brain.import() instead.` + ) + } + if (typeof data.formatVersion === 'number' && data.formatVersion > BACKUP_FORMAT_VERSION) { + throw new Error( + `Backup formatVersion ${data.formatVersion} is newer than this Brainy supports ` + + `(max ${BACKUP_FORMAT_VERSION}). Upgrade Brainy to import this document.` + ) + } + + const { onConflict = 'merge', reembed = 'auto', remapIds } = options + const result: ImportResult = { + imported: 0, + merged: 0, + skipped: 0, + reembedded: 0, + blobsWritten: 0, + errors: [] + } + const mapId = (id: string) => (remapIds ? remapIds(id) : id) + + // 1. Write blob bytes first so file entities resolve their content. + if (data.blobs && Object.keys(data.blobs).length > 0) { + const blobStorage = (this.storage as any).blobStorage + if (blobStorage?.write) { + for (const [hash, b64] of Object.entries(data.blobs)) { + try { + await blobStorage.write(Buffer.from(b64, 'base64')) + result.blobsWritten++ + } catch (e) { + result.errors.push({ id: hash, error: `blob: ${(e as Error).message}` }) + } + } + } else { + for (const hash of Object.keys(data.blobs)) { + result.errors.push({ id: hash, error: 'blob: storage does not support binary blobs' }) + } + } + } + + // 2. Entities (so relation endpoints exist before edges are created). + for (const be of data.entities || []) { + const id = mapId(be.id) + try { + const exists = await this.entityExists(id) + if (exists) { + if (onConflict === 'skip') { + result.skipped++ + continue + } + if (onConflict === 'merge') { + await this.brain.update({ id, ...this.entityUpdateFields(be), merge: true }) + result.merged++ + continue + } + await this.brain.delete(id) // 'replace' + } + + const useVector = Array.isArray(be.vector) && be.vector.length > 0 + if (!useVector && reembed === 'never') { + result.errors.push({ id, error: 'no vector carried and reembed:never' }) + continue + } + await this.brain.add({ + id, + ...this.entityAddFields(be), + ...(useVector ? { vector: be.vector } : {}) + }) + result.imported++ + if (!useVector) result.reembedded++ + } catch (e) { + result.errors.push({ id, error: (e as Error).message }) + } + } + + // 3. Relations. + for (const br of data.relations || []) { + const from = mapId(br.from) + const to = mapId(br.to) + try { + await this.brain.relate({ + from, + to, + type: br.type as VerbType, + ...(br.subtype !== undefined ? { subtype: br.subtype } : {}), + ...(br.weight !== undefined ? { weight: br.weight } : {}), + ...(br.confidence !== undefined ? { confidence: br.confidence } : {}), + ...(br.metadata !== undefined ? { metadata: br.metadata } : {}) + }) + } catch (e) { + result.errors.push({ id: br.id, error: `relation: ${(e as Error).message}` }) + } + } + + return result + } + + // ============================================================================ + // CLEAR / STATS + // ============================================================================ + + /** + * @description Delete data from the brain. + * @param params - Which categories to clear (`entities`/`relations` default true; `config` false). + */ + async clear( + params: { entities?: boolean; relations?: boolean; config?: boolean } = {} + ): Promise { + const { entities = true, relations = true } = params if (entities) { - // Clear all entities - const nounsResult = await this.storage.getNouns({ - pagination: { limit: 1000000 } - }) - + const nounsResult = await this.storage.getNouns({ pagination: { limit: ENUMERATION_LIMIT } }) for (const noun of nounsResult.items) { await this.storage.deleteNoun(noun.id) } - - // Also clear the HNSW index if available if (this.brain?.index?.clear) { this.brain.index.clear() } - - // Clear metadata index if available if (this.brain?.metadataIndex) { - await this.brain.metadataIndex.rebuild() // Rebuild empty index + await this.brain.metadataIndex.rebuild() } } if (relations) { - // Clear all relations - const verbsResult = await this.storage.getVerbs({ - pagination: { limit: 1000000 } - }) - + const verbsResult = await this.storage.getVerbs({ pagination: { limit: ENUMERATION_LIMIT } }) for (const verb of verbsResult.items) { await this.storage.deleteVerb(verb.id) } } - - if (config) { - // Clear configuration would be handled by ConfigAPI - // For now, skip this - } } /** - * Import data from various formats - */ - async import(params: ImportOptions & { data: any }): Promise { - const { - data, - format, - mapping = {}, - batchSize = 100, - validate = true - } = params - - const result: ImportResult = { - successful: 0, - failed: 0, - errors: [], - duration: 0 - } - - const startTime = Date.now() - - try { - // ALWAYS use neural import for proper type matching - const { UniversalImportAPI } = await import('./UniversalImportAPI.js') - const universalImport = new UniversalImportAPI(this.brain) - await universalImport.init() - - // Convert to ImportSource format - const neuralResult = await universalImport.import({ - type: 'object', - data, - format: format || 'json', - metadata: { mapping, batchSize, validate } - }) - - // Convert neural result to ImportResult format - result.successful = neuralResult.stats.entitiesCreated - result.failed = 0 // Neural import always succeeds with best match - result.duration = neuralResult.stats.processingTimeMs - - // Log relationships created - if (neuralResult.stats.relationshipsCreated > 0) { - console.log(`Neural import also created ${neuralResult.stats.relationshipsCreated} relationships`) - } - - return result - } catch (error) { - // Fallback to legacy import ONLY if neural import fails to load - console.warn('Neural import failed, using legacy import:', error) - - let items: any[] = [] - - // Parse data based on format - switch (format) { - case 'json': - items = Array.isArray(data) ? data : [data] - break - - case 'csv': - // CSV parsing would go here - // For now, assume data is already parsed - items = data - break - - // Parquet format removed - not implemented - - default: - throw new Error(`Unsupported format: ${format}`) - } - - // Process items in batches - for (let i = 0; i < items.length; i += batchSize) { - const batch = items.slice(i, i + batchSize) - - for (const item of batch) { - try { - // Apply field mapping - const mapped = this.applyMapping(item, mapping) - - // Validate if requested - if (validate) { - this.validateImportItem(mapped) - } - - // Save entity - separate vector and metadata - const id = mapped.id || this.generateId() - const noun: HNSWNoun = { - id, - vector: mapped.vector || new Array(384).fill(0), - connections: new Map(), - level: 0 - } - - await this.storage.saveNoun(noun) - await this.storage.saveNounMetadata(id, { ...mapped, createdAt: Date.now() }) - result.successful++ - } catch (error) { - result.failed++ - result.errors.push({ - item, - error: (error as Error).message - }) - } - } - } - - result.duration = Date.now() - startTime - return result - } - } - - /** - * Export data to various formats - */ - async export(params: ExportOptions = {}): Promise { - const { - format = 'json', - filter = {}, - includeVectors = false - } = params - - // Get filtered entities - const nounsResult = await this.storage.getNouns({ - pagination: { limit: 1000000 } - }) - - let entities = nounsResult.items - - // Apply filters - if (filter.type) { - const types = Array.isArray(filter.type) ? filter.type : [filter.type] - entities = entities.filter(e => - types.includes(e.metadata?.noun as NounType) - ) - } - - if (filter.service) { - entities = entities.filter(e => - e.metadata?.service === filter.service - ) - } - - if (filter.where) { - entities = entities.filter(e => - this.matchesFilter(e.metadata, filter.where!) - ) - } - - // Format data based on export format - switch (format) { - case 'json': - return entities.map(e => ({ - id: e.id, - vector: includeVectors ? e.vector : undefined, - ...e.metadata - })) - - case 'csv': - // Convert to CSV format - // For now, return simplified format - return this.convertToCSV(entities) - - // Parquet format removed - not implemented - - default: - throw new Error(`Unsupported export format: ${format}`) - } - } - - /** - * Get storage statistics + * @description Summary counts for the brain. + * @returns Entity/relation totals and the vector dimensionality. */ async getStats(): Promise<{ entities: number @@ -304,14 +497,9 @@ export class DataAPI { storageSize?: number vectorDimensions?: number }> { - const nounsResult = await this.storage.getNouns({ - pagination: { limit: 1 } - }) - const verbsResult = await this.storage.getVerbs({ - pagination: { limit: 1 } - }) - - const firstNoun = nounsResult.items[0] + const nounsResult = await this.storage.getNouns({ pagination: { limit: 1 } }) + const verbsResult = await this.storage.getVerbs({ pagination: { limit: 1 } }) + const firstNoun = nounsResult.items[0] as any return { entities: nounsResult.totalCount || nounsResult.items.length, @@ -320,70 +508,342 @@ export class DataAPI { } } - // Helper methods + // ============================================================================ + // SELECTOR RESOLUTION (private) + // ============================================================================ - private applyMapping(item: any, mapping: Record): any { - const mapped: any = {} - - for (const [key, value] of Object.entries(item)) { - const mappedKey = mapping[key] || key - mapped[mappedKey] = value - } - - return mapped + /** True if the selector names a structural node set. */ + private hasStructural(s: ExportSelector): boolean { + return !!(s.ids || s.collection || s.memberOf || s.connected || s.vfsPath) } - private validateImportItem(item: any): void { - // Basic validation - if (!item || typeof item !== 'object') { - throw new Error('Invalid item: must be an object') - } - - // Could add more validation here + /** True if the selector carries predicate (filter) keys. */ + private hasPredicate(s: ExportSelector): boolean { + return ( + s.type !== undefined || + s.subtype !== undefined || + s.where !== undefined || + s.service !== undefined || + s.visibility !== undefined + ) } - private matchesFilter(metadata: any, filter: Record): boolean { - for (const [key, value] of Object.entries(filter)) { - if (metadata[key] !== value) { - return false + /** + * Resolve the candidate id list: the structural node set when a structural selector is + * present, otherwise every entity id (for predicate-only and whole-brain exports). + * Predicate filtering is applied later, on the canonical entities. + */ + private async resolveCandidates(s: ExportSelector, includeSystem: boolean): Promise { + let idSet: Set + + if (s.ids && s.ids.length) { + idSet = new Set(s.ids) + } else if (s.collection ?? s.memberOf) { + idSet = await this.resolveCollectionSubtree((s.collection ?? s.memberOf)!, s.depth) + } else if (s.connected) { + idSet = await this.resolveConnected(s.connected) + } else if (s.vfsPath) { + idSet = await this.resolveVfsPath(s.vfsPath, s.recursive ?? true, s.depth) + } else { + idSet = await this.allEntityIds() + } + + if (!includeSystem) idSet.delete(VFS_ROOT_ID) + return Array.from(idSet) + } + + /** Every entity id in the brain (storage-layer enumeration; no query-limit enforcement). */ + private async allEntityIds(): Promise> { + const result = await this.storage.getNouns({ pagination: { limit: ENUMERATION_LIMIT } }) + return new Set(result.items.map((n: any) => n.id)) + } + + /** A collection id + its transitive `Contains` members (BFS, optionally depth-capped). */ + private async resolveCollectionSubtree(rootId: string, depth?: number): Promise> { + const set = new Set([rootId]) + const maxDepth = depth ?? Infinity + let frontier = [rootId] + let d = 0 + while (frontier.length && d < maxDepth) { + const next: string[] = [] + for (const id of frontier) { + const rels = await this.brain.getRelations({ + from: id, + type: VerbType.Contains, + limit: RELATION_FETCH_LIMIT + }) + for (const r of rels as Relation[]) { + if (!set.has(r.to)) { + set.add(r.to) + next.push(r.to) + } + } } + frontier = next + d++ + } + return set + } + + /** An entity + its N-hop neighbourhood, following `verbs`/`direction`. */ + private async resolveConnected(c: NonNullable): Promise> { + const { from, depth = 1, verbs, direction = 'out' } = c + const set = new Set([from]) + let frontier = [from] + for (let d = 0; d < depth; d++) { + const next: string[] = [] + for (const id of frontier) { + const neighbours = await this.neighboursOf(id, direction, verbs) + for (const n of neighbours) { + if (!set.has(n)) { + set.add(n) + next.push(n) + } + } + } + frontier = next + if (!next.length) break + } + return set + } + + /** Neighbour ids of an entity in the requested direction, optionally verb-filtered. */ + private async neighboursOf( + id: string, + direction: 'out' | 'in' | 'both', + verbs?: VerbType[] + ): Promise { + const out: string[] = [] + if (direction === 'out' || direction === 'both') { + const rels = (await this.brain.getRelations({ from: id, limit: RELATION_FETCH_LIMIT })) as Relation[] + for (const r of rels) if (!verbs || verbs.includes(r.type)) out.push(r.to) + } + if (direction === 'in' || direction === 'both') { + const rels = (await this.brain.getRelations({ to: id, limit: RELATION_FETCH_LIMIT })) as Relation[] + for (const r of rels) if (!verbs || verbs.includes(r.type)) out.push(r.from) + } + return out + } + + /** A VFS path → the resolved entity, plus (for a directory) its subtree. */ + private async resolveVfsPath(path: string, recursive: boolean, depth?: number): Promise> { + const dirId = await this.resolveVfsPathToId(path) + if (!dirId) return new Set() + if (!recursive) { + const set = new Set([dirId]) + const rels = (await this.brain.getRelations({ + from: dirId, + type: VerbType.Contains, + limit: RELATION_FETCH_LIMIT + })) as Relation[] + for (const r of rels) set.add(r.to) + return set + } + return this.resolveCollectionSubtree(dirId, depth) + } + + /** Walk `Contains` from the VFS root, matching each path segment against `metadata.name`. */ + private async resolveVfsPathToId(path: string): Promise { + const segments = path.split('/').filter(Boolean) + let currentId = VFS_ROOT_ID + for (const seg of segments) { + const rels = (await this.brain.getRelations({ + from: currentId, + type: VerbType.Contains, + limit: RELATION_FETCH_LIMIT + })) as Relation[] + let found: string | null = null + for (const r of rels) { + const child = await this.brain.get(r.to) + if (child?.metadata?.name === seg) { + found = r.to + break + } + } + if (!found) return null + currentId = found + } + return currentId + } + + /** True if an entity satisfies the selector's predicate keys. */ + private matchesPredicate(e: Entity, s: ExportSelector): boolean { + if (s.type !== undefined) { + const types = Array.isArray(s.type) ? s.type : [s.type] + if (!types.includes(e.type)) return false + } + if (s.subtype !== undefined) { + const subs = Array.isArray(s.subtype) ? s.subtype : [s.subtype] + if (e.subtype === undefined || !subs.includes(e.subtype)) return false + } + if (s.service !== undefined && e.service !== s.service) return false + if (s.visibility !== undefined) { + const vis = (e as any).visibility ?? 'public' + if (vis !== s.visibility) return false + } + if (s.where && !this.matchesWhere(e.metadata, s.where)) return false + return true + } + + /** Exact-match metadata predicate. */ + private matchesWhere(metadata: any, where: Record): boolean { + if (!metadata) return false + for (const [key, value] of Object.entries(where)) { + if (metadata[key] !== value) return false } return true } - private convertToCSV(entities: Array<{ id: string; metadata?: any }>): string { - if (entities.length === 0) return '' + // ============================================================================ + // SERIALIZATION HELPERS (private) + // ============================================================================ - // Get all unique keys from metadata - const keys = new Set() - for (const entity of entities) { - if (entity.metadata) { - Object.keys(entity.metadata).forEach(k => keys.add(k)) + /** Map a canonical `Entity` to a `BackupEntity` (reserved fields top-level). */ + private toBackupEntity(e: Entity, includeVectors: boolean): BackupEntity { + const be: BackupEntity = { id: e.id, type: e.type as string } + if (e.subtype !== undefined) be.subtype = e.subtype + const vis = (e as any).visibility + if (vis !== undefined && vis !== 'public') be.visibility = vis + if (e.data !== undefined) be.data = e.data + if (e.confidence !== undefined) be.confidence = e.confidence + if (e.weight !== undefined) be.weight = e.weight + if (e.service !== undefined) be.service = e.service + if (e.createdBy !== undefined) be.createdBy = e.createdBy + if (e.createdAt !== undefined) be.createdAt = e.createdAt + if (includeVectors && e.vector && e.vector.length) be.vector = e.vector + if (e.metadata && Object.keys(e.metadata).length) be.metadata = e.metadata + return be + } + + /** Map a canonical `Relation` to a `BackupRelation`. */ + private toBackupRelation(r: Relation): BackupRelation { + const br: BackupRelation = { id: r.id, from: r.from, to: r.to, type: r.type as string } + if (r.subtype !== undefined) br.subtype = r.subtype + const vis = (r as any).visibility + if (vis !== undefined && vis !== 'public') br.visibility = vis + if (r.weight !== undefined) br.weight = r.weight + if (r.confidence !== undefined) br.confidence = r.confidence + if (r.metadata && Object.keys(r.metadata).length) br.metadata = r.metadata + return br + } + + /** Collect edges among the node set per the edge policy. */ + private async collectEdges( + idSet: Set, + edges: 'induced' | 'incident' | 'none' + ): Promise<{ relations: BackupRelation[]; danglingIds?: string[] }> { + if (edges === 'none') return { relations: [] } + + const relations: BackupRelation[] = [] + const dangling = new Set() + const seen = new Set() + + // Outgoing edges from each in-set node (captures every induced edge exactly once). + for (const id of idSet) { + const rels = (await this.brain.getRelations({ from: id, limit: RELATION_FETCH_LIMIT })) as Relation[] + for (const r of rels) { + if (seen.has(r.id)) continue + const toIn = idSet.has(r.to) + if (edges === 'induced' && !toIn) continue + if (!toIn) dangling.add(r.to) + seen.add(r.id) + relations.push(this.toBackupRelation(r)) } } - // Create CSV header - const headers = ['id', ...Array.from(keys)] - const rows = [headers.join(',')] - - // Add data rows - for (const entity of entities) { - const row = [entity.id] - for (const key of keys) { - const value = entity.metadata?.[key] || '' - // Escape values that contain commas - const escaped = String(value).includes(',') - ? `"${String(value).replace(/"/g, '""')}"` - : String(value) - row.push(escaped) + // For 'incident', also capture edges arriving from outside the set. + if (edges === 'incident') { + for (const id of idSet) { + const rels = (await this.brain.getRelations({ to: id, limit: RELATION_FETCH_LIMIT })) as Relation[] + for (const r of rels) { + if (seen.has(r.id)) continue + if (!idSet.has(r.from)) { + dangling.add(r.from) + seen.add(r.id) + relations.push(this.toBackupRelation(r)) + } + } } - rows.push(row.join(',')) } - return rows.join('\n') + return dangling.size > 0 ? { relations, danglingIds: Array.from(dangling) } : { relations } } - private generateId(): string { - return `import_${Date.now()}_${Math.random().toString(36).substr(2, 9)}` + /** Read VFS file bytes (base64) for file entities, keyed by content hash. */ + private async collectBlobs( + entities: BackupEntity[], + entityMap: Map + ): Promise> { + const blobs: Record = {} + const blobStorage = (this.storage as any).blobStorage + if (!blobStorage?.read) return blobs + + for (const be of entities) { + const e = entityMap.get(be.id) as any + const storageMeta = e?.metadata?.storage + const hash = storageMeta?.hash + if (storageMeta?.type === 'blob' && hash && !blobs[hash]) { + try { + const buf = await blobStorage.read(hash) + blobs[hash] = Buffer.from(buf).toString('base64') + } catch { + // Referenced blob bytes are unreadable (storage drift): skip — the file entity's + // structure still travels, and stats.blobCount reflects what was actually captured. + } + } + } + return blobs } -} \ No newline at end of file + + /** Best-effort embedding model label from the brain's configuration. */ + private detectModel(): string { + return this.brain?.config?.embedding?.model || DEFAULT_EMBED_MODEL + } + + /** Embedding dimensionality from the brain's configuration, if exposed. */ + private detectDimensions(): number | undefined { + const dim = this.brain?.config?.dimensions ?? this.brain?.dimensions + return typeof dim === 'number' ? dim : undefined + } + + // ============================================================================ + // IMPORT HELPERS (private) + // ============================================================================ + + /** Existence check that never throws on id-format quirks (storage-layer read). */ + private async entityExists(id: string): Promise { + try { + const meta = await this.storage.getNounMetadata(id) + return !!meta + } catch { + return false + } + } + + /** `add()` params from a `BackupEntity` (excludes brain-managed fields like `createdAt`). */ + private entityAddFields(be: BackupEntity): Record { + const fields: Record = { + data: be.data, + type: be.type as NounType + } + if (be.subtype !== undefined) fields.subtype = be.subtype + if (be.service !== undefined) fields.service = be.service + if (be.confidence !== undefined) fields.confidence = be.confidence + if (be.weight !== undefined) fields.weight = be.weight + if (be.metadata !== undefined) fields.metadata = be.metadata + return fields + } + + /** `update()` params from a `BackupEntity` (for `onConflict:'merge'`). */ + private entityUpdateFields(be: BackupEntity): Record { + const fields: Record = {} + if (be.data !== undefined) fields.data = be.data + if (be.type !== undefined) fields.type = be.type as NounType + if (be.subtype !== undefined) fields.subtype = be.subtype + if (be.confidence !== undefined) fields.confidence = be.confidence + if (be.weight !== undefined) fields.weight = be.weight + if (be.metadata !== undefined) fields.metadata = be.metadata + if (Array.isArray(be.vector) && be.vector.length) fields.vector = be.vector + return fields + } +} diff --git a/src/brainy.ts b/src/brainy.ts index a8c6e8e4..33bcb91d 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -5770,12 +5770,7 @@ export class Brainy implements BrainyInterface { */ async data() { const { DataAPI } = await import('./api/DataAPI.js') - return new DataAPI( - this.storage, - (id: string) => this.get(id), - undefined, // No getRelation method yet - this - ) + return new DataAPI(this.storage, this) } /** diff --git a/src/cli/commands/core.ts b/src/cli/commands/core.ts index c8509f5e..e0322bfe 100644 --- a/src/cli/commands/core.ts +++ b/src/cli/commands/core.ts @@ -876,60 +876,66 @@ export const coreCommands = { const brain = getBrainy() const format = options.format || 'json' - // Export all data + // Export the whole brain as a portable BackupData document (vectors + VFS file bytes). const dataApi = await brain.data() - const data = await dataApi.export({ format: 'json' }) + const backup = await dataApi.export(undefined, { + includeVectors: true, + includeContent: true + }) let output = '' - + + const csvCell = (v: any): string => { + if (v === undefined || v === null) return '' + const s = typeof v === 'object' ? JSON.stringify(v) : String(v) + return /[",\n]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s + } + switch (format) { case 'json': - output = options.pretty - ? JSON.stringify(data, null, 2) - : JSON.stringify(data) + output = options.pretty + ? JSON.stringify(backup, null, 2) + : JSON.stringify(backup) break - - case 'jsonl': - if (Array.isArray(data)) { - output = data.map(item => JSON.stringify(item)).join('\n') - } else { - output = JSON.stringify(data) - } + + case 'jsonl': { + // NDJSON: a header line, then one line per entity, then one line per relation. + const { entities, relations, ...header } = backup + const lines: string[] = [JSON.stringify({ kind: 'header', ...header })] + for (const e of entities) lines.push(JSON.stringify({ kind: 'entity', ...e })) + for (const r of relations) lines.push(JSON.stringify({ kind: 'relation', ...r })) + output = lines.join('\n') break - - case 'csv': - if (Array.isArray(data) && data.length > 0) { - // Get all unique keys for headers + } + + case 'csv': { + // CSV represents entities only — a graph's edges and blobs can't be tabular. + const entities = backup.entities + if (entities.length > 0) { const headers = new Set() - data.forEach(item => { - Object.keys(item).forEach(key => headers.add(key)) - }) + entities.forEach((e: Record) => Object.keys(e).forEach(k => headers.add(k))) const headerArray = Array.from(headers) - - // Create CSV output = headerArray.join(',') + '\n' - output += data.map(item => { - return headerArray.map(h => { - const value = item[h] - if (typeof value === 'object') { - return JSON.stringify(value) - } - return value || '' - }).join(',') - }).join('\n') + output += entities + .map((e: Record) => headerArray.map(h => csvCell(e[h])).join(',')) + .join('\n') } break + } } - + if (file) { writeFileSync(file, output) spinner.succeed(`Exported to ${file}`) - + if (!options.json) { console.log(chalk.green(`✓ Successfully exported database to ${file}`)) console.log(chalk.dim(` Format: ${format}`)) - console.log(chalk.dim(` Items: ${Array.isArray(data) ? data.length : 1}`)) + console.log(chalk.dim(` Entities: ${backup.stats.entityCount} Relations: ${backup.stats.relationCount}`)) } else { - formatOutput({ file, format, count: Array.isArray(data) ? data.length : 1 }, options) + formatOutput( + { file, format, entityCount: backup.stats.entityCount, relationCount: backup.stats.relationCount }, + options + ) } } else { spinner.succeed('Export complete') diff --git a/src/index.ts b/src/index.ts index bacd8c5b..47e1b30b 100644 --- a/src/index.ts +++ b/src/index.ts @@ -116,6 +116,18 @@ export { // Export version utilities export { getBrainyVersion } from './utils/version.js' +// Export portable graph backup/restore API (brain.data()) +export { DataAPI } from './api/DataAPI.js' +export type { + BackupData, + BackupEntity, + BackupRelation, + ExportSelector, + ExportOptions, + ImportOptions, + ImportResult +} from './api/DataAPI.js' + // Export plugin system export type { BrainyPlugin, BrainyPluginContext, StorageAdapterFactory } from './plugin.js' export { PluginRegistry } from './plugin.js' diff --git a/tests/unit/api/data-backup.test.ts b/tests/unit/api/data-backup.test.ts new file mode 100644 index 00000000..2d3e88d1 --- /dev/null +++ b/tests/unit/api/data-backup.test.ts @@ -0,0 +1,311 @@ +/** + * Unit tests for the portable graph backup/restore API (brain.data()). + * + * Exercises BackupData v1 export()/import(): real round-trips against in-memory + * storage (no mocks of the API under test) — selectors, edge policy, vectors, + * conflict handling, id remapping, subtype fidelity, and cross-version guards. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { randomUUID } from 'node:crypto' +import { Brainy } from '../../../src/brainy' +import { createTestConfig } from '../../helpers/test-factory' +import { NounType, VerbType } from '../../../src/types/graphTypes' +import type { BackupData } from '../../../src/api/DataAPI' + +const VFS_ROOT_ID = '00000000-0000-0000-0000-000000000000' + +describe('DataAPI — portable graph backup/restore (BackupData v1)', () => { + let brain: Brainy + + beforeEach(async () => { + brain = new Brainy(createTestConfig()) + await brain.init() + }) + + afterEach(async () => { + await brain.close() + }) + + describe('format + whole-brain round-trip', () => { + it('exports a self-describing, versioned BackupData document', async () => { + const a = await brain.add({ data: 'Alice', type: NounType.Person, subtype: 'employee' }) + const b = await brain.add({ data: 'Acme', type: NounType.Organization }) + await brain.relate({ from: a, to: b, type: VerbType.WorksWith, subtype: 'full-time' }) + + const backup = await brain.data().then((d) => d.export()) + + expect(backup.format).toBe('brainy-backup') + expect(backup.formatVersion).toBe(1) + expect(typeof backup.brainyVersion).toBe('string') + expect(backup.brainyVersion.length).toBeGreaterThan(0) + expect(typeof backup.createdAt).toBe('string') + expect(backup.embedding.dimensions).toBeGreaterThan(0) + expect(backup.stats.entityCount).toBe(backup.entities.length) + expect(backup.stats.relationCount).toBe(backup.relations.length) + // Two user entities, one relation (VFS root excluded by default). + expect(backup.entities.map((e) => e.id).sort()).toEqual([a, b].sort()) + expect(backup.relations).toHaveLength(1) + }) + + it('round-trips entities and relations through clear() + import()', async () => { + const a = await brain.add({ data: 'Alice', type: NounType.Person }) + const b = await brain.add({ data: 'Bob', type: NounType.Person }) + const c = await brain.add({ data: 'Carol', type: NounType.Person }) + await brain.relate({ from: a, to: b, type: VerbType.FriendOf }) + await brain.relate({ from: b, to: c, type: VerbType.FriendOf }) + + const backup = await brain.data().then((d) => d.export({}, { includeVectors: true })) + + await brain.data().then((d) => d.clear()) + const afterClear = await brain.find({ limit: 1000 }) + expect(afterClear.filter((r) => [a, b, c].includes(r.id))).toHaveLength(0) + + const result = await brain.data().then((d) => d.import(backup)) + expect(result.imported).toBe(3) + expect(result.errors).toHaveLength(0) + + const ra = await brain.get(a) + expect(ra?.id).toBe(a) + const rels = await brain.getRelations({ from: a }) + expect(rels.some((r) => r.to === b && r.type === VerbType.FriendOf)).toBe(true) + }) + + it('preserves subtype on both entities and relations', async () => { + const a = await brain.add({ data: 'Doc', type: NounType.Document, subtype: 'invoice' }) + const b = await brain.add({ data: 'Doc2', type: NounType.Document, subtype: 'receipt' }) + await brain.relate({ from: a, to: b, type: VerbType.RelatedTo, subtype: 'supersedes' }) + + const backup = await brain.data().then((d) => d.export()) + + const ea = backup.entities.find((e) => e.id === a) + expect(ea?.subtype).toBe('invoice') + expect(backup.relations[0].subtype).toBe('supersedes') + }) + }) + + describe('selectors', () => { + it('ids — exports exactly the requested entities', async () => { + const a = await brain.add({ data: 'A', type: NounType.Thing }) + const b = await brain.add({ data: 'B', type: NounType.Thing }) + await brain.add({ data: 'C', type: NounType.Thing }) + + const backup = await brain.data().then((d) => d.export({ ids: [a, b] })) + expect(backup.entities.map((e) => e.id).sort()).toEqual([a, b].sort()) + }) + + it('collection — exports the collection plus its transitive Contains members', async () => { + const root = await brain.add({ data: 'Folder', type: NounType.Collection }) + const child1 = await brain.add({ data: 'Child1', type: NounType.Document }) + const child2 = await brain.add({ data: 'Child2', type: NounType.Document }) + const grandchild = await brain.add({ data: 'Grandchild', type: NounType.Document }) + await brain.add({ data: 'Outside', type: NounType.Document }) + await brain.relate({ from: root, to: child1, type: VerbType.Contains }) + await brain.relate({ from: root, to: child2, type: VerbType.Contains }) + await brain.relate({ from: child1, to: grandchild, type: VerbType.Contains }) + + const backup = await brain.data().then((d) => d.export({ collection: root })) + expect(backup.entities.map((e) => e.id).sort()).toEqual( + [root, child1, child2, grandchild].sort() + ) + }) + + it('connected — exports an N-hop neighbourhood', async () => { + const a = await brain.add({ data: 'A', type: NounType.Thing }) + const b = await brain.add({ data: 'B', type: NounType.Thing }) + const c = await brain.add({ data: 'C', type: NounType.Thing }) + await brain.relate({ from: a, to: b, type: VerbType.RelatedTo }) + await brain.relate({ from: b, to: c, type: VerbType.RelatedTo }) + + const depth1 = await brain.data().then((d) => d.export({ connected: { from: a, depth: 1 } })) + expect(depth1.entities.map((e) => e.id).sort()).toEqual([a, b].sort()) + + const depth2 = await brain.data().then((d) => d.export({ connected: { from: a, depth: 2 } })) + expect(depth2.entities.map((e) => e.id).sort()).toEqual([a, b, c].sort()) + }) + + it('predicate — filters by type', async () => { + await brain.add({ data: 'P', type: NounType.Person }) + await brain.add({ data: 'D', type: NounType.Document }) + const backup = await brain.data().then((d) => d.export({ type: NounType.Person })) + expect(backup.entities.every((e) => e.type === NounType.Person)).toBe(true) + expect(backup.entities).toHaveLength(1) + }) + + it('compose — structural selector + predicate filter', async () => { + const root = await brain.add({ data: 'Folder', type: NounType.Collection }) + const open = await brain.add({ + data: 'Open', + type: NounType.Document, + metadata: { status: 'open' } + }) + const closed = await brain.add({ + data: 'Closed', + type: NounType.Document, + metadata: { status: 'closed' } + }) + await brain.relate({ from: root, to: open, type: VerbType.Contains }) + await brain.relate({ from: root, to: closed, type: VerbType.Contains }) + + const backup = await brain + .data() + .then((d) => d.export({ collection: root, where: { status: 'open' } })) + expect(backup.entities.map((e) => e.id)).toContain(open) + expect(backup.entities.map((e) => e.id)).not.toContain(closed) + }) + }) + + describe('edge policy', () => { + let a: string + let b: string + let c: string + + beforeEach(async () => { + a = await brain.add({ data: 'A', type: NounType.Thing }) + b = await brain.add({ data: 'B', type: NounType.Thing }) + c = await brain.add({ data: 'C', type: NounType.Thing }) + await brain.relate({ from: a, to: b, type: VerbType.RelatedTo }) // both in {a,b} + await brain.relate({ from: b, to: c, type: VerbType.RelatedTo }) // dangles to c + }) + + it('induced (default) — only edges with both endpoints in the set', async () => { + const backup = await brain.data().then((d) => d.export({ ids: [a, b] })) + expect(backup.relations).toHaveLength(1) + expect(backup.relations[0].from).toBe(a) + expect(backup.danglingIds).toBeUndefined() + }) + + it('incident — includes dangling edges + records danglingIds', async () => { + const backup = await brain.data().then((d) => d.export({ ids: [a, b] }, { edges: 'incident' })) + expect(backup.relations).toHaveLength(2) + expect(backup.danglingIds).toContain(c) + }) + + it('none — nodes only', async () => { + const backup = await brain.data().then((d) => d.export({ ids: [a, b] }, { edges: 'none' })) + expect(backup.relations).toHaveLength(0) + }) + }) + + describe('vectors + re-embedding', () => { + it('omits vectors by default and re-embeds on import', async () => { + const a = await brain.add({ data: 'Re-embed me', type: NounType.Thing }) + const backup = await brain.data().then((d) => d.export({ ids: [a] })) + expect(backup.entities[0].vector).toBeUndefined() + + await brain.data().then((d) => d.clear()) + const result = await brain.data().then((d) => d.import(backup)) + expect(result.reembedded).toBe(1) + + const restored = await brain.get(a, { includeVectors: true }) + expect(restored?.vector?.length).toBeGreaterThan(0) + }) + + it('carries vectors verbatim with includeVectors (no re-embed)', async () => { + const a = await brain.add({ data: 'Keep my vector', type: NounType.Thing }) + const original = await brain.get(a, { includeVectors: true }) + + const backup = await brain.data().then((d) => d.export({ ids: [a] }, { includeVectors: true })) + expect(backup.entities[0].vector?.length).toBe(original?.vector?.length) + + await brain.data().then((d) => d.clear()) + const result = await brain.data().then((d) => d.import(backup)) + expect(result.reembedded).toBe(0) + + const restored = await brain.get(a, { includeVectors: true }) + expect(restored?.vector).toEqual(original?.vector) + }) + + it('reembed:never records an error when no vector is carried', async () => { + const a = await brain.add({ data: 'No vector', type: NounType.Thing }) + const backup = await brain.data().then((d) => d.export({ ids: [a] })) // no vectors + await brain.data().then((d) => d.clear()) + + const result = await brain.data().then((d) => d.import(backup, { reembed: 'never' })) + expect(result.imported).toBe(0) + expect(result.errors).toHaveLength(1) + expect(result.errors[0].error).toMatch(/reembed:never/) + }) + }) + + describe('conflict handling', () => { + it('merge (default) — updates existing entities in place', async () => { + const a = await brain.add({ data: 'V1', type: NounType.Thing, metadata: { n: 1 } }) + const backup = await brain.data().then((d) => d.export({ ids: [a] }, { includeVectors: true })) + + await brain.update({ id: a, metadata: { n: 99 }, merge: false }) + const result = await brain.data().then((d) => d.import(backup, { onConflict: 'merge' })) + expect(result.merged).toBe(1) + expect(result.imported).toBe(0) + + const restored = await brain.get(a) + expect(restored?.metadata?.n).toBe(1) + }) + + it('skip — leaves existing entities untouched', async () => { + const a = await brain.add({ data: 'Orig', type: NounType.Thing, metadata: { n: 1 } }) + const backup = await brain.data().then((d) => d.export({ ids: [a] })) + await brain.update({ id: a, metadata: { n: 2 }, merge: false }) + + const result = await brain.data().then((d) => d.import(backup, { onConflict: 'skip' })) + expect(result.skipped).toBe(1) + const restored = await brain.get(a) + expect(restored?.metadata?.n).toBe(2) + }) + }) + + describe('id remapping (clone)', () => { + it('imports a subgraph under fresh ids', async () => { + const a = await brain.add({ data: 'A', type: NounType.Thing }) + const b = await brain.add({ data: 'B', type: NounType.Thing }) + await brain.relate({ from: a, to: b, type: VerbType.RelatedTo }) + const backup = await brain.data().then((d) => d.export({ ids: [a, b] }, { includeVectors: true })) + + const remap = new Map([ + [a, randomUUID()], + [b, randomUUID()] + ]) + const result = await brain + .data() + .then((d) => d.import(backup, { remapIds: (id) => remap.get(id) ?? id })) + expect(result.imported).toBe(2) + + const clonedA = await brain.get(remap.get(a)!) + expect(clonedA?.id).toBe(remap.get(a)) + // The original entities still exist (a clone, not a move). + expect((await brain.get(a))?.id).toBe(a) + const clonedRels = await brain.getRelations({ from: remap.get(a)! }) + expect(clonedRels.some((r) => r.to === remap.get(b))).toBe(true) + }) + }) + + describe('system entities', () => { + it('excludes the VFS root from a whole-brain export by default', async () => { + await brain.add({ data: 'User entity', type: NounType.Thing }) + const backup = await brain.data().then((d) => d.export()) + expect(backup.entities.map((e) => e.id)).not.toContain(VFS_ROOT_ID) + }) + }) + + describe('import validation', () => { + it('rejects a non-BackupData payload', async () => { + const d = await brain.data() + await expect(d.import({ entities: [] } as any)).rejects.toThrow(/BackupData/) + }) + + it('rejects a newer formatVersion', async () => { + const d = await brain.data() + const future: BackupData = { + format: 'brainy-backup', + formatVersion: 999, + brainyVersion: 'x', + createdAt: new Date().toISOString(), + embedding: { model: 'm', dimensions: 384 }, + entities: [], + relations: [], + stats: { entityCount: 0, relationCount: 0, blobCount: 0 } + } + await expect(d.import(future)).rejects.toThrow(/formatVersion/) + }) + }) +})