refactor(8.0)!: remove orphaned zero-config subsystem + dead cloud/progressive-init storage vestige
The old config-generation subsystem (src/config/ + autoConfiguration.ts) was
superseded during the 8.0 rework and never wired into init(): it emitted settings
for a partitioning subsystem that no longer exists and probed deleted cloud env
vars. The live zero-config path is inline — recall preset → HNSW knobs, storage
auto-detect, auto persistMode, container-memory-aware cache sizing.
The storage progressive-init / cloud-detection cluster was equally dead after the
cloud adapters were dropped: isCloudStorage() is permanently false (no overriders),
scheduleBackgroundInit/runBackgroundInit were never called (the latter an empty
body), initMode was never assigned, and Brainy.isFullyInitialized()/
awaitBackgroundInit() were always-trivial with zero callers. scheduleCountPersist()
collapses to its only-ever-taken immediate write-through path.
Removed:
- src/config/{index,zeroConfig,storageAutoConfig,modelAutoConfig,sharedConfigManager}.ts
- src/utils/autoConfiguration.ts + the inert BrainyZeroConfig export
- Brainy.isFullyInitialized()/awaitBackgroundInit() (+ BrainyInterface decls)
- InitMode type, isCloudStorage/detectCloudEnvironment/resolveInitMode,
scheduleBackgroundInit/runBackgroundInit/ensureValidatedForWrite and their state
- Dead cloud env-var probes (K_SERVICE/K_REVISION/AWS_LAMBDA_FUNCTION_NAME/
FUNCTIONS_TARGET/AZURE_FUNCTIONS_ENVIRONMENT)
Kept (verified live): production-detection logging (environment.ts), container-
memory cache sizing (memoryDetection/paramValidation), on-disk hash bucketing
(sharding.ts).
Docs: scrubbed deleted-subsystem references (JS quantization knobs, cloud/OPFS
adapters, partitioning, old zero-config API) across 14 files; deleted two wholly-
obsolete feature docs (complete-feature-list, v3-features); rewrote
architecture/zero-config for 8.0.
~3,700 LOC removed. Build clean; 1392 unit + 24 db-mvcc green.
This commit is contained in:
parent
00d3203d68
commit
35b9d7ef43
28 changed files with 596 additions and 3752 deletions
|
|
@ -428,8 +428,7 @@ brainy inspect health /data/brain
|
||||||
|
|
||||||
See [the multi-process model](docs/concepts/multi-process.md) and the
|
See [the multi-process model](docs/concepts/multi-process.md) and the
|
||||||
[inspection guide](docs/guides/inspection.md) for the full story, including
|
[inspection guide](docs/guides/inspection.md) for the full story, including
|
||||||
stale-lock detection, the cross-process flush RPC, and what's not yet
|
stale-lock detection and the cross-process flush RPC.
|
||||||
enforced on cloud storage backends.
|
|
||||||
|
|
||||||
## Contributing
|
## Contributing
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -391,13 +391,12 @@ await brain.init() // Instant (0-10ms)
|
||||||
|
|
||||||
### Vector Index Tuning Knobs
|
### Vector Index Tuning Knobs
|
||||||
|
|
||||||
Brainy 8.0 exposes exactly three knobs on `config.vector`:
|
Brainy 8.0 exposes two knobs on `config.vector`:
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
const brain = new Brainy({
|
const brain = new Brainy({
|
||||||
vector: {
|
vector: {
|
||||||
recall: 'fast', // 'fast' | 'balanced' | 'accurate'
|
recall: 'fast', // 'fast' | 'balanced' | 'accurate'
|
||||||
quantization: { bits: 8 }, // 4 | 8 (SQ4 / SQ8)
|
|
||||||
persistMode: 'deferred' // 'immediate' | 'deferred'
|
persistMode: 'deferred' // 'immediate' | 'deferred'
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
|
||||||
|
|
@ -35,10 +35,9 @@ Brainy 8.0 is a **single-node library**. There is no cluster, no peer discovery,
|
||||||
The three knobs that matter most:
|
The three knobs that matter most:
|
||||||
|
|
||||||
1. **`config.vector.recall`** — `'fast'`, `'balanced'`, or `'accurate'` (default `'balanced'`)
|
1. **`config.vector.recall`** — `'fast'`, `'balanced'`, or `'accurate'` (default `'balanced'`)
|
||||||
2. **`config.vector.quantization`** — `{ bits: 4 | 8 }` for memory savings on the open-core JS vector index
|
2. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput
|
||||||
3. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput
|
|
||||||
|
|
||||||
The native vector provider (via the optional `@soulcraft/cortex` package) extends this with a higher-performing index when installed.
|
The native vector provider (via the optional `@soulcraft/cortex` package) extends this with a higher-performing index — and its own at-scale acceleration such as on-disk compressed indexing — when installed.
|
||||||
|
|
||||||
## Storage Configurations
|
## Storage Configurations
|
||||||
|
|
||||||
|
|
@ -95,7 +94,6 @@ const brain = new Brainy({
|
||||||
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
|
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
|
||||||
vector: {
|
vector: {
|
||||||
recall: 'fast', // Trade recall for latency
|
recall: 'fast', // Trade recall for latency
|
||||||
quantization: { bits: 8 }, // SQ8 quantization
|
|
||||||
persistMode: 'deferred' // Batch persistence
|
persistMode: 'deferred' // Batch persistence
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
@ -142,8 +140,7 @@ Your service layer handles routing and isolation; Brainy stays simple.
|
||||||
const brain = new Brainy({
|
const brain = new Brainy({
|
||||||
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
|
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
|
||||||
vector: {
|
vector: {
|
||||||
recall: 'accurate',
|
recall: 'accurate'
|
||||||
quantization: { bits: 8 }
|
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
```
|
```
|
||||||
|
|
@ -153,7 +150,6 @@ const brain = new Brainy({
|
||||||
| Setting | Values | When to change |
|
| Setting | Values | When to change |
|
||||||
|---------|--------|----------------|
|
|---------|--------|----------------|
|
||||||
| `vector.recall` | `'fast'` / `'balanced'` / `'accurate'` | Trade recall for latency |
|
| `vector.recall` | `'fast'` / `'balanced'` / `'accurate'` | Trade recall for latency |
|
||||||
| `vector.quantization.bits` | `4` / `8` | Smaller index, lower memory |
|
|
||||||
| `vector.persistMode` | `'immediate'` / `'deferred'` | Throughput vs. durability |
|
| `vector.persistMode` | `'immediate'` / `'deferred'` | Throughput vs. durability |
|
||||||
| `storage.cache.maxSize` | integer | Hot-path read cache size |
|
| `storage.cache.maxSize` | integer | Hot-path read cache size |
|
||||||
| `storage.cache.ttl` | ms | Cache freshness |
|
| `storage.cache.ttl` | ms | Cache freshness |
|
||||||
|
|
@ -174,14 +170,13 @@ const stats = await brain.stats()
|
||||||
|
|
||||||
### Issue: Slow queries
|
### Issue: Slow queries
|
||||||
1. Switch to `vector.recall: 'fast'`
|
1. Switch to `vector.recall: 'fast'`
|
||||||
2. Enable SQ8 quantization
|
2. Increase the read cache (`storage.cache.maxSize`)
|
||||||
3. Increase the read cache (`storage.cache.maxSize`)
|
3. Consider the optional native vector provider via `@soulcraft/cortex`
|
||||||
4. Consider the optional native vector provider via `@soulcraft/cortex`
|
|
||||||
|
|
||||||
### Issue: Memory pressure
|
### Issue: Memory pressure
|
||||||
1. Enable `vector.quantization: { bits: 4 }` or `{ bits: 8 }`
|
1. Reduce `storage.cache.maxSize`
|
||||||
2. Reduce `storage.cache.maxSize`
|
2. Move to `vector.persistMode: 'deferred'` to batch writes
|
||||||
3. Move to `vector.persistMode: 'deferred'` to batch writes
|
3. Consider the optional native vector provider via `@soulcraft/cortex` for at-scale index acceleration
|
||||||
|
|
||||||
### Issue: Slow startup after a crash
|
### Issue: Slow startup after a crash
|
||||||
1. Use `vector.persistMode: 'immediate'` so the index file stays in sync with storage
|
1. Use `vector.persistMode: 'immediate'` so the index file stays in sync with storage
|
||||||
|
|
@ -198,5 +193,5 @@ const stats = await brain.stats()
|
||||||
|
|
||||||
- Brainy 8.0 is a **library**, not a cluster
|
- Brainy 8.0 is a **library**, not a cluster
|
||||||
- Storage adapters: `filesystem`, `memory`, `auto`
|
- Storage adapters: `filesystem`, `memory`, `auto`
|
||||||
- Vector tuning: `recall`, `quantization`, `persistMode`
|
- Vector tuning: `recall`, `persistMode`
|
||||||
- Backup is an operator-layer concern — snapshot `rootDirectory`
|
- Backup is an operator-layer concern — snapshot `rootDirectory`
|
||||||
|
|
|
||||||
|
|
@ -1441,10 +1441,9 @@ const brain = new Brainy({
|
||||||
rootDirectory: './brainy-data'
|
rootDirectory: './brainy-data'
|
||||||
},
|
},
|
||||||
|
|
||||||
// Vector index configuration (3 knobs)
|
// Vector index configuration (2 knobs)
|
||||||
vector: {
|
vector: {
|
||||||
recall: 'balanced', // 'fast' | 'balanced' | 'accurate'
|
recall: 'balanced', // 'fast' | 'balanced' | 'accurate'
|
||||||
quantization: { bits: 8 }, // 4 | 8 (SQ4 / SQ8)
|
|
||||||
persistMode: 'immediate' // 'immediate' | 'deferred'
|
persistMode: 'immediate' // 'immediate' | 'deferred'
|
||||||
},
|
},
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -47,16 +47,13 @@ await visualizationAugmentation.graphRelationships(authors)
|
||||||
|
|
||||||
#### 2. **Data Portability**
|
#### 2. **Data Portability**
|
||||||
```typescript
|
```typescript
|
||||||
// Export from one Brainy instance
|
// Snapshot from one Brainy instance, restore into another —
|
||||||
const data = await brain1.export()
|
// types are universally understood
|
||||||
|
const pin = brain1.now()
|
||||||
|
await pin.persist('/snapshots/brain1')
|
||||||
|
await pin.release()
|
||||||
|
|
||||||
// Import to another—types are universally understood
|
const brain2 = await Brainy.load('/snapshots/brain1') // Types match perfectly
|
||||||
await brain2.import(data)
|
|
||||||
|
|
||||||
// Or sync between different storage backends
|
|
||||||
const cloudBrain = new Brainy({ storage: 's3' })
|
|
||||||
const localBrain = new Brainy({ storage: 'filesystem' })
|
|
||||||
await cloudBrain.sync(localBrain) // Types match perfectly
|
|
||||||
```
|
```
|
||||||
|
|
||||||
#### 3. **AI Model Compatibility**
|
#### 3. **AI Model Compatibility**
|
||||||
|
|
|
||||||
|
|
@ -45,7 +45,7 @@ brainy-data/
|
||||||
### Vector Index
|
### Vector Index
|
||||||
Pluggable vector index (`VectorIndexProvider`) for efficient nearest-neighbor search. The default JS implementation, `JsHnswVectorIndex`, uses a hierarchical graph:
|
Pluggable vector index (`VectorIndexProvider`) for efficient nearest-neighbor search. The default JS implementation, `JsHnswVectorIndex`, uses a hierarchical graph:
|
||||||
- **Performance**: O(log n) search complexity
|
- **Performance**: O(log n) search complexity
|
||||||
- **Memory Efficient**: SQ4/SQ8 scalar quantization support
|
- **Configurable recall**: `fast` / `balanced` / `accurate` presets trade recall for latency
|
||||||
- **Scalable**: Handles millions of vectors per process
|
- **Scalable**: Handles millions of vectors per process
|
||||||
- **Persistent**: Serializable to storage
|
- **Persistent**: Serializable to storage
|
||||||
- **Swappable**: Replace with a native implementation (such as `@soulcraft/cortex`) via the plugin system without changing application code
|
- **Swappable**: Replace with a native implementation (such as `@soulcraft/cortex`) via the plugin system without changing application code
|
||||||
|
|
|
||||||
|
|
@ -13,776 +13,145 @@ next:
|
||||||
|
|
||||||
# Zero Configuration & Auto-Adaptation
|
# Zero Configuration & Auto-Adaptation
|
||||||
|
|
||||||
> **Status (8.0):** This document predates Brainy 8.0 and needs a rewrite before
|
> **"Zero config by default, fully tunable when you need it."** Construct a
|
||||||
> republication. Large parts describe storage backends and environments that 8.0
|
> `Brainy()` with no options and it picks sensible, environment-aware defaults.
|
||||||
> removed (browser/OPFS/IndexedDB, edge KV, S3) or features that were never built
|
> Every default below is overridable through the constructor — see the
|
||||||
> (model auto-selection, workload detection). Brainy 8.0 is server-only (Node.js/Bun)
|
> [API Reference](../api/README.md#configuration).
|
||||||
> with two storage adapters: `memory` and `filesystem` — see
|
|
||||||
> [Storage Adapters](../concepts/storage-adapters.md) for the accurate story.
|
|
||||||
> Basic zero-config (`new Brainy()` with auto-selected storage) works as described.
|
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
Brainy is designed with **"Zero Config by Default, Infinite Tunability"** philosophy. It automatically detects your environment, adapts to available resources, learns from usage patterns, and optimizes itself for your specific workload—all without any configuration.
|
Brainy 8.0 is server-only (Node.js 22+ / Bun). With no configuration it:
|
||||||
|
|
||||||
## Zero Configuration Magic
|
- selects a storage adapter from the runtime,
|
||||||
|
- initializes the embedding model (all-MiniLM-L6-v2, 384 dimensions),
|
||||||
|
- builds and maintains the metadata, graph, and vector indexes,
|
||||||
|
- sizes its caches and write buffers to the detected memory budget,
|
||||||
|
- chooses a persistence mode that matches the storage backend, and
|
||||||
|
- quiets its own logging when it detects a production environment.
|
||||||
|
|
||||||
### Instant Start
|
There is no public config-generation function — adaptation happens inside the
|
||||||
|
constructor and `init()`.
|
||||||
|
|
||||||
|
## Instant Start
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
import { Brainy } from 'brainy'
|
import { Brainy } from '@soulcraft/brainy'
|
||||||
|
|
||||||
// That's it. No config needed.
|
// That's it. No config needed.
|
||||||
const brain = new Brainy()
|
const brain = new Brainy()
|
||||||
await brain.init()
|
await brain.init()
|
||||||
|
|
||||||
// Brainy automatically:
|
await brain.add({ data: 'First entity', type: 'concept' })
|
||||||
// ✓ Detects environment (Node.js, Browser, Edge, Deno)
|
const results = await brain.find('first')
|
||||||
// ✓ Chooses optimal storage (FileSystem, OPFS, Memory)
|
|
||||||
// ✓ Downloads required models (if needed)
|
|
||||||
// ✓ Configures vector dimensions (384 optimal)
|
|
||||||
// ✓ Sets up indexing strategies
|
|
||||||
// ✓ Enables appropriate augmentations
|
|
||||||
// ✓ Configures caching layers
|
|
||||||
// ✓ Optimizes for your hardware
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Environment Detection ✅ Available
|
## What Auto-Adaptation Covers
|
||||||
|
|
||||||
Brainy automatically detects and adapts to your runtime:
|
### 1. Storage auto-detection
|
||||||
|
|
||||||
|
With no `storage` option, Brainy uses `type: 'auto'`:
|
||||||
|
|
||||||
|
- **Filesystem** when running on a runtime with a writable Node filesystem and a
|
||||||
|
resolvable root directory. This is the default for typical Node/Bun servers and
|
||||||
|
persists across restarts.
|
||||||
|
- **In-memory** otherwise (no filesystem access, or an explicit memory request).
|
||||||
|
Fast, zero I/O, discarded on process exit — ideal for tests and ephemeral
|
||||||
|
caches.
|
||||||
|
|
||||||
|
8.0 ships exactly two storage adapters — `memory` and `filesystem` — plus the
|
||||||
|
`auto` selector that resolves to one of them. See
|
||||||
|
[Storage Adapters](../concepts/storage-adapters.md) for the full contract.
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// Brainy's environment detection
|
// Explicit override when you want a specific root
|
||||||
const environment = {
|
const brain = new Brainy({
|
||||||
// Runtime detection
|
storage: { type: 'filesystem', rootDirectory: './brainy-data' }
|
||||||
isNode: typeof process !== 'undefined',
|
|
||||||
isBrowser: typeof window !== 'undefined',
|
|
||||||
isDeno: typeof Deno !== 'undefined',
|
|
||||||
isEdge: typeof EdgeRuntime !== 'undefined',
|
|
||||||
isWebWorker: typeof WorkerGlobalScope !== 'undefined',
|
|
||||||
|
|
||||||
// Capability detection
|
|
||||||
hasFileSystem: /* auto-detected */,
|
|
||||||
hasIndexedDB: /* auto-detected */,
|
|
||||||
hasOPFS: /* auto-detected */,
|
|
||||||
hasWebGPU: /* auto-detected */,
|
|
||||||
hasWASM: /* auto-detected */,
|
|
||||||
|
|
||||||
// Resource detection
|
|
||||||
cpuCores: /* auto-detected */,
|
|
||||||
memory: /* auto-detected */,
|
|
||||||
storage: /* auto-detected */
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Auto-Adaptive Storage ✅ Available
|
|
||||||
|
|
||||||
> **Current**: Brainy automatically selects the best storage adapter for your environment.
|
|
||||||
|
|
||||||
### Storage Selection Logic
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Brainy's intelligent storage selection
|
|
||||||
async function autoSelectStorage() {
|
|
||||||
// Server environments
|
|
||||||
if (environment.isNode) {
|
|
||||||
if (await hasWritePermission('./data')) {
|
|
||||||
return 'filesystem' // Best for servers
|
|
||||||
} else if (process.env.S3_BUCKET) {
|
|
||||||
return 's3' // Cloud deployment
|
|
||||||
} else {
|
|
||||||
return 'memory' // Fallback for restricted environments
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Browser environments
|
|
||||||
if (environment.isBrowser) {
|
|
||||||
if (await navigator.storage.estimate() > 1GB) {
|
|
||||||
return 'opfs' // Best for modern browsers
|
|
||||||
} else if (indexedDB) {
|
|
||||||
return 'indexeddb' // Fallback for older browsers
|
|
||||||
} else {
|
|
||||||
return 'memory' // In-memory for restricted contexts
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Edge environments
|
|
||||||
if (environment.isEdge) {
|
|
||||||
return 'kv' // Use edge KV stores (Cloudflare, Vercel)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Storage Migration
|
|
||||||
|
|
||||||
Brainy seamlessly migrates between storage types:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Start with memory storage (development)
|
|
||||||
const brain = new Brainy() // Auto-selects memory
|
|
||||||
|
|
||||||
// Later, migrate to production storage
|
|
||||||
await brain.migrate({
|
|
||||||
to: 'filesystem',
|
|
||||||
path: './production-data'
|
|
||||||
})
|
})
|
||||||
// All data seamlessly transferred
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Learning & Optimization 🚧 Coming Soon
|
### 2. HNSW quality from the `recall` preset
|
||||||
|
|
||||||
> **Note**: These features are planned for Q2 2025. Currently, Brainy uses static optimizations.
|
Vector-index quality comes from a single preset rather than hand-tuned graph
|
||||||
|
parameters. `config.vector.recall` accepts `'fast'`, `'balanced'`, or
|
||||||
### Query Pattern Learning 🚧 Planned
|
`'accurate'` and defaults to `'balanced'`. The preset maps internally to the
|
||||||
|
HNSW construction and search parameters (`M` / `efConstruction` / `efSearch`),
|
||||||
Brainy learns from your query patterns and optimizes accordingly:
|
so you trade recall against latency with one knob instead of three.
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// Brainy observes query patterns
|
const brain = new Brainy({
|
||||||
class QueryPatternLearner {
|
vector: { recall: 'fast' } // favor latency over recall
|
||||||
analyze(queries: Query[]) {
|
})
|
||||||
return {
|
|
||||||
// Frequency analysis
|
|
||||||
mostCommonFields: this.getTopFields(queries),
|
|
||||||
avgResultSize: this.getAvgSize(queries),
|
|
||||||
temporalPatterns: this.getTimePatterns(queries),
|
|
||||||
|
|
||||||
// Relationship analysis
|
|
||||||
commonTraversals: this.getGraphPatterns(queries),
|
|
||||||
typicalDepth: this.getAvgDepth(queries),
|
|
||||||
|
|
||||||
// Performance analysis
|
|
||||||
slowQueries: this.getSlowQueries(queries),
|
|
||||||
cacheability: this.getCacheability(queries)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Automatic optimizations based on learning:
|
|
||||||
// - Creates indexes for frequently queried fields
|
|
||||||
// - Pre-computes common graph traversals
|
|
||||||
// - Adjusts cache sizes based on working set
|
|
||||||
// - Optimizes vector search parameters
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Auto-Indexing 🚧 Planned
|
The default JS index is `JsHnswVectorIndex`. An optional native acceleration
|
||||||
|
provider (the `@soulcraft/cortex` package) can replace it with a
|
||||||
|
higher-performing implementation; the public knobs stay the same. Quantization
|
||||||
|
and other index-internal acceleration are the native provider's concern, not a
|
||||||
|
Brainy configuration option.
|
||||||
|
|
||||||
Brainy automatically creates indexes based on usage:
|
### 3. Persistence mode follows the backend
|
||||||
|
|
||||||
|
`config.vector.persistMode` accepts `'immediate'` or `'deferred'`. Left unset,
|
||||||
|
Brainy chooses for you:
|
||||||
|
|
||||||
|
- **Immediate** on filesystem storage, so the index file stays in lock-step with
|
||||||
|
the data and survives a crash.
|
||||||
|
- **Deferred** on in-memory storage, where there is nothing durable to sync to,
|
||||||
|
so writes are batched for throughput.
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// No manual index configuration needed
|
const brain = new Brainy({
|
||||||
await brain.find({ where: { category: "tech" } }) // First query
|
vector: { persistMode: 'deferred' } // batch persistence for write-heavy loads
|
||||||
// Brainy notices 'category' field usage
|
})
|
||||||
|
|
||||||
await brain.find({ where: { category: "science" } }) // Second query
|
|
||||||
// Pattern detected - auto-creates category index
|
|
||||||
|
|
||||||
await brain.find({ where: { category: "tech" } }) // Third query
|
|
||||||
// Now using index - 100x faster!
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Adaptive Caching 🚧 Planned
|
### 4. Memory-aware cache and buffer sizing
|
||||||
|
|
||||||
Cache strategies adapt to your access patterns:
|
Brainy reads the container's memory budget — `CLOUD_RUN_MEMORY`, `MEMORY_LIMIT`,
|
||||||
|
or the cgroup memory limit when running in a container — and sizes its read
|
||||||
|
caches and write buffers to fit. On a small instance it stays conservative; on a
|
||||||
|
large one it uses more of the available headroom. Query-result limits are capped
|
||||||
|
against the same budget (roughly 25 KB per result) to keep a single oversized
|
||||||
|
query from exhausting memory.
|
||||||
|
|
||||||
|
You can pin the cache explicitly:
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
class AdaptiveCache {
|
const brain = new Brainy({
|
||||||
async adapt(metrics: AccessMetrics) {
|
cache: { maxSize: 10000, ttl: 3_600_000 }
|
||||||
if (metrics.hitRate < 0.3) {
|
})
|
||||||
// Low hit rate - switch strategy
|
|
||||||
this.strategy = 'lfu' // Least Frequently Used
|
|
||||||
} else if (metrics.workingSet > this.size) {
|
|
||||||
// Working set too large - increase size
|
|
||||||
this.size = Math.min(metrics.workingSet * 1.5, maxMemory)
|
|
||||||
} else if (metrics.temporalLocality > 0.8) {
|
|
||||||
// High temporal locality - use time-based eviction
|
|
||||||
this.strategy = 'ttl'
|
|
||||||
this.ttl = metrics.avgAccessInterval * 2
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Performance Auto-Scaling 🚧 Coming Soon
|
### 5. Logging quiets in production
|
||||||
|
|
||||||
### Dynamic Batch Sizing
|
Brainy detects production-style environments (for example `NODE_ENV` set to a
|
||||||
|
non-development value) and reduces its own log verbosity automatically. This is
|
||||||
Brainy adjusts batch sizes based on system load:
|
logging-only behavior — it does not change indexing, storage, or query results.
|
||||||
|
|
||||||
```typescript
|
|
||||||
class DynamicBatcher {
|
|
||||||
calculateOptimalBatch() {
|
|
||||||
const cpuUsage = process.cpuUsage()
|
|
||||||
const memoryUsage = process.memoryUsage()
|
|
||||||
|
|
||||||
if (cpuUsage < 30 && memoryUsage < 50) {
|
|
||||||
return 1000 // System idle - large batches
|
|
||||||
} else if (cpuUsage < 60 && memoryUsage < 70) {
|
|
||||||
return 100 // Moderate load - medium batches
|
|
||||||
} else {
|
|
||||||
return 10 // High load - small batches
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Automatically applied during bulk operations
|
|
||||||
for (const item of millionItems) {
|
|
||||||
await brain.add(item) // Internally batched optimally
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Memory Management
|
|
||||||
|
|
||||||
Automatic memory pressure handling:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
class MemoryManager {
|
|
||||||
async handlePressure() {
|
|
||||||
const usage = process.memoryUsage()
|
|
||||||
const available = os.freemem()
|
|
||||||
|
|
||||||
if (available < 100 * 1024 * 1024) { // Less than 100MB free
|
|
||||||
// Emergency mode
|
|
||||||
await this.flushCaches()
|
|
||||||
await this.compactIndexes()
|
|
||||||
await this.offloadToDisk()
|
|
||||||
} else if (usage.heapUsed / usage.heapTotal > 0.9) {
|
|
||||||
// Preventive mode
|
|
||||||
await this.reduceCacheSizes()
|
|
||||||
await this.pauseBackgroundTasks()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Connection Pooling
|
|
||||||
|
|
||||||
Automatic connection management for storage backends:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
class ConnectionPool {
|
|
||||||
async getOptimalPoolSize() {
|
|
||||||
// Adapts based on workload
|
|
||||||
const metrics = await this.getMetrics()
|
|
||||||
|
|
||||||
if (metrics.waitTime > 100) {
|
|
||||||
// Queries waiting - increase pool
|
|
||||||
this.size = Math.min(this.size * 1.5, this.maxSize)
|
|
||||||
} else if (metrics.idleConnections > this.size * 0.5) {
|
|
||||||
// Too many idle - decrease pool
|
|
||||||
this.size = Math.max(this.size * 0.7, this.minSize)
|
|
||||||
}
|
|
||||||
|
|
||||||
return this.size
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Model Auto-Selection
|
|
||||||
|
|
||||||
### Embedding Model Selection
|
|
||||||
|
|
||||||
Brainy chooses the best embedding model for your use case:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
async function autoSelectModel(data: Sample[]) {
|
|
||||||
const analysis = {
|
|
||||||
languages: detectLanguages(data),
|
|
||||||
domainSpecific: detectDomain(data),
|
|
||||||
averageLength: getAvgLength(data),
|
|
||||||
requiresMultilingual: languages.length > 1
|
|
||||||
}
|
|
||||||
|
|
||||||
if (analysis.requiresMultilingual) {
|
|
||||||
return 'multilingual-e5-base' // Handles 100+ languages
|
|
||||||
} else if (analysis.domainSpecific === 'code') {
|
|
||||||
return 'codebert-base' // Optimized for code
|
|
||||||
} else if (analysis.averageLength > 512) {
|
|
||||||
return 'all-mpnet-base-v2' // Better for long text
|
|
||||||
} else {
|
|
||||||
return 'all-MiniLM-L6-v2' // Fast and efficient default
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Model Downloading
|
|
||||||
|
|
||||||
Models are automatically downloaded when needed:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// First use - model auto-downloads
|
|
||||||
const brain = new Brainy()
|
|
||||||
await brain.init() // Downloads model if not cached
|
|
||||||
|
|
||||||
// Intelligent model caching
|
|
||||||
const modelCache = {
|
|
||||||
location: process.env.MODEL_CACHE || '~/.brainy/models',
|
|
||||||
maxSize: 5 * 1024 * 1024 * 1024, // 5GB max
|
|
||||||
strategy: 'lru', // Least recently used eviction
|
|
||||||
|
|
||||||
// CDN selection based on location
|
|
||||||
cdn: await selectFastestCDN([
|
|
||||||
'https://cdn.brainy.io',
|
|
||||||
'https://brainy.b-cdn.net',
|
|
||||||
'https://models.huggingface.co'
|
|
||||||
])
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Workload Detection
|
|
||||||
|
|
||||||
### Pattern Recognition
|
|
||||||
|
|
||||||
Brainy identifies your workload type and optimizes:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
enum WorkloadType {
|
|
||||||
OLTP = 'oltp', // Many small transactions
|
|
||||||
OLAP = 'olap', // Analytical queries
|
|
||||||
STREAMING = 'streaming', // Real-time ingestion
|
|
||||||
BATCH = 'batch', // Bulk processing
|
|
||||||
HYBRID = 'hybrid' // Mixed workload
|
|
||||||
}
|
|
||||||
|
|
||||||
class WorkloadDetector {
|
|
||||||
detect(metrics: OperationMetrics): WorkloadType {
|
|
||||||
if (metrics.writesPerSecond > 1000 && metrics.avgWriteSize < 1024) {
|
|
||||||
return WorkloadType.STREAMING
|
|
||||||
} else if (metrics.avgQueryComplexity > 0.8 && metrics.avgResultSize > 10000) {
|
|
||||||
return WorkloadType.OLAP
|
|
||||||
} else if (metrics.batchOperations > metrics.singleOperations) {
|
|
||||||
return WorkloadType.BATCH
|
|
||||||
} else if (metrics.writeReadRatio > 0.3 && metrics.writeReadRatio < 0.7) {
|
|
||||||
return WorkloadType.HYBRID
|
|
||||||
} else {
|
|
||||||
return WorkloadType.OLTP
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Optimization Strategies
|
|
||||||
|
|
||||||
Different optimizations for different workloads:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
class WorkloadOptimizer {
|
|
||||||
optimize(workload: WorkloadType) {
|
|
||||||
switch (workload) {
|
|
||||||
case WorkloadType.STREAMING:
|
|
||||||
return {
|
|
||||||
entityRegistry: true, // Deduplication
|
|
||||||
batchSize: 1000,
|
|
||||||
walEnabled: true,
|
|
||||||
cacheSize: 'small',
|
|
||||||
indexStrategy: 'lazy'
|
|
||||||
}
|
|
||||||
|
|
||||||
case WorkloadType.OLAP:
|
|
||||||
return {
|
|
||||||
entityRegistry: false,
|
|
||||||
batchSize: 10000,
|
|
||||||
walEnabled: false,
|
|
||||||
cacheSize: 'large',
|
|
||||||
indexStrategy: 'eager',
|
|
||||||
parallelQueries: true
|
|
||||||
}
|
|
||||||
|
|
||||||
case WorkloadType.BATCH:
|
|
||||||
return {
|
|
||||||
entityRegistry: false,
|
|
||||||
batchSize: 50000,
|
|
||||||
walEnabled: false,
|
|
||||||
cacheSize: 'minimal',
|
|
||||||
indexStrategy: 'deferred'
|
|
||||||
}
|
|
||||||
|
|
||||||
default:
|
|
||||||
return this.defaultConfig
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Hardware Adaptation 🚧 Coming Soon
|
|
||||||
|
|
||||||
> **Note**: GPU acceleration and hardware optimization planned for Q3 2025.
|
|
||||||
|
|
||||||
### CPU Optimization
|
|
||||||
|
|
||||||
Adapts to available CPU resources:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
class CPUAdapter {
|
|
||||||
async optimize() {
|
|
||||||
const cores = os.cpus().length
|
|
||||||
const type = os.cpus()[0].model
|
|
||||||
|
|
||||||
// Parallel processing based on cores
|
|
||||||
this.parallelism = Math.max(1, cores - 1) // Leave one core free
|
|
||||||
|
|
||||||
// SIMD detection for vector operations
|
|
||||||
if (type.includes('Intel') || type.includes('AMD')) {
|
|
||||||
this.enableSIMD = await checkSIMDSupport()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Thread pool sizing
|
|
||||||
this.threadPoolSize = cores * 2 // Optimal for I/O bound
|
|
||||||
|
|
||||||
// Vector search optimization
|
|
||||||
if (cores >= 8) {
|
|
||||||
this.hnswConstruction = 200 // Higher quality index
|
|
||||||
this.hnswSearch = 100 // More accurate search
|
|
||||||
} else {
|
|
||||||
this.hnswConstruction = 100 // Balanced
|
|
||||||
this.hnswSearch = 50 // Faster search
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Memory Adaptation
|
|
||||||
|
|
||||||
Intelligent memory allocation:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
class MemoryAdapter {
|
|
||||||
async configure() {
|
|
||||||
const totalMemory = os.totalmem()
|
|
||||||
const availableMemory = os.freemem()
|
|
||||||
|
|
||||||
// Allocate based on available memory
|
|
||||||
const allocation = {
|
|
||||||
cache: Math.min(availableMemory * 0.25, 2 * GB),
|
|
||||||
vectors: Math.min(availableMemory * 0.30, 4 * GB),
|
|
||||||
indexes: Math.min(availableMemory * 0.20, 2 * GB),
|
|
||||||
working: Math.min(availableMemory * 0.25, 2 * GB)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Adjust for low memory systems
|
|
||||||
if (totalMemory < 4 * GB) {
|
|
||||||
allocation.cache *= 0.5
|
|
||||||
allocation.vectors *= 0.7
|
|
||||||
this.enableSwapping = true
|
|
||||||
}
|
|
||||||
|
|
||||||
return allocation
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### GPU Acceleration
|
|
||||||
|
|
||||||
Automatic GPU detection and utilization:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
class GPUAdapter {
|
|
||||||
async detect() {
|
|
||||||
// WebGPU in browsers
|
|
||||||
if (navigator?.gpu) {
|
|
||||||
const adapter = await navigator.gpu.requestAdapter()
|
|
||||||
return {
|
|
||||||
available: true,
|
|
||||||
type: 'webgpu',
|
|
||||||
memory: adapter.limits.maxBufferSize,
|
|
||||||
compute: adapter.limits.maxComputeWorkgroupsPerDimension
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// CUDA in Node.js
|
|
||||||
if (process.platform === 'linux' || process.platform === 'win32') {
|
|
||||||
const hasCuda = await checkCudaSupport()
|
|
||||||
if (hasCuda) {
|
|
||||||
return {
|
|
||||||
available: true,
|
|
||||||
type: 'cuda',
|
|
||||||
memory: await getCudaMemory(),
|
|
||||||
compute: await getCudaCores()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return { available: false }
|
|
||||||
}
|
|
||||||
|
|
||||||
async optimize(gpu: GPUInfo) {
|
|
||||||
if (gpu.available) {
|
|
||||||
// Offload vector operations to GPU
|
|
||||||
this.vectorOps = 'gpu'
|
|
||||||
this.embeddingGeneration = 'gpu'
|
|
||||||
this.matrixMultiplication = 'gpu'
|
|
||||||
|
|
||||||
// Larger batch sizes for GPU
|
|
||||||
this.batchSize = gpu.memory > 8 * GB ? 10000 : 1000
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Network Adaptation
|
|
||||||
|
|
||||||
### Bandwidth Detection
|
|
||||||
|
|
||||||
Optimizes for available network bandwidth:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
class NetworkAdapter {
|
|
||||||
async measureBandwidth() {
|
|
||||||
const testSize = 1 * MB
|
|
||||||
const start = Date.now()
|
|
||||||
await this.transfer(testSize)
|
|
||||||
const duration = Date.now() - start
|
|
||||||
|
|
||||||
const bandwidth = (testSize / duration) * 1000 // bytes/sec
|
|
||||||
|
|
||||||
if (bandwidth < 1 * MB) {
|
|
||||||
// Low bandwidth - optimize
|
|
||||||
this.compression = 'aggressive'
|
|
||||||
this.batchTransfers = true
|
|
||||||
this.cacheRemote = true
|
|
||||||
} else if (bandwidth > 100 * MB) {
|
|
||||||
// High bandwidth
|
|
||||||
this.compression = 'minimal'
|
|
||||||
this.parallelTransfers = true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Latency Optimization
|
|
||||||
|
|
||||||
Adapts to network latency:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
class LatencyOptimizer {
|
|
||||||
async optimize() {
|
|
||||||
const latency = await this.measureLatency()
|
|
||||||
|
|
||||||
if (latency > 100) { // High latency
|
|
||||||
// Batch operations
|
|
||||||
this.minBatchSize = 100
|
|
||||||
|
|
||||||
// Aggressive prefetching
|
|
||||||
this.prefetchDepth = 3
|
|
||||||
|
|
||||||
// Local caching
|
|
||||||
this.cacheStrategy = 'aggressive'
|
|
||||||
|
|
||||||
// Connection pooling
|
|
||||||
this.connectionPool = Math.min(latency / 10, 50)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Cloud Provider Detection 🚧 Coming Soon
|
|
||||||
|
|
||||||
> **Note**: Cloud provider auto-detection planned for Q3 2025.
|
|
||||||
|
|
||||||
### Automatic Cloud Optimization
|
|
||||||
|
|
||||||
Detects and optimizes for cloud providers:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
class CloudDetector {
|
|
||||||
async detect() {
|
|
||||||
// AWS Detection
|
|
||||||
if (process.env.AWS_REGION || await canReachMetadata('169.254.169.254')) {
|
|
||||||
return {
|
|
||||||
provider: 'aws',
|
|
||||||
instance: await getEC2InstanceType(),
|
|
||||||
region: process.env.AWS_REGION,
|
|
||||||
services: {
|
|
||||||
storage: 's3',
|
|
||||||
cache: 'elasticache',
|
|
||||||
compute: 'lambda'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Google Cloud Detection
|
|
||||||
if (process.env.GOOGLE_CLOUD_PROJECT || await canReachMetadata('metadata.google.internal')) {
|
|
||||||
return {
|
|
||||||
provider: 'gcp',
|
|
||||||
instance: await getGCEInstanceType(),
|
|
||||||
region: process.env.GOOGLE_CLOUD_REGION,
|
|
||||||
services: {
|
|
||||||
storage: 'gcs',
|
|
||||||
cache: 'memorystore',
|
|
||||||
compute: 'cloud-run'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Vercel Edge Detection
|
|
||||||
if (process.env.VERCEL) {
|
|
||||||
return {
|
|
||||||
provider: 'vercel',
|
|
||||||
region: process.env.VERCEL_REGION,
|
|
||||||
services: {
|
|
||||||
storage: 'vercel-kv',
|
|
||||||
cache: 'edge-config',
|
|
||||||
compute: 'edge-runtime'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Development vs Production
|
|
||||||
|
|
||||||
### Automatic Environment Detection
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
class EnvironmentDetector {
|
|
||||||
detect() {
|
|
||||||
const indicators = {
|
|
||||||
// Development indicators
|
|
||||||
isDevelopment:
|
|
||||||
process.env.NODE_ENV === 'development' ||
|
|
||||||
process.env.DEBUG ||
|
|
||||||
process.argv.includes('--dev') ||
|
|
||||||
isLocalhost() ||
|
|
||||||
hasDevTools(),
|
|
||||||
|
|
||||||
// Test indicators
|
|
||||||
isTest:
|
|
||||||
process.env.NODE_ENV === 'test' ||
|
|
||||||
process.env.CI ||
|
|
||||||
isTestRunner(),
|
|
||||||
|
|
||||||
// Production indicators
|
|
||||||
isProduction:
|
|
||||||
process.env.NODE_ENV === 'production' ||
|
|
||||||
process.env.VERCEL ||
|
|
||||||
process.env.NETLIFY ||
|
|
||||||
!isLocalhost()
|
|
||||||
}
|
|
||||||
|
|
||||||
return indicators
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Different defaults for different environments
|
|
||||||
const config = environment.isProduction ? {
|
|
||||||
storage: 'filesystem',
|
|
||||||
wal: true,
|
|
||||||
monitoring: true,
|
|
||||||
compression: true,
|
|
||||||
caching: 'aggressive'
|
|
||||||
} : {
|
|
||||||
storage: 'memory',
|
|
||||||
wal: false,
|
|
||||||
monitoring: false,
|
|
||||||
compression: false,
|
|
||||||
caching: 'minimal'
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Error Recovery
|
|
||||||
|
|
||||||
### Automatic Fallbacks
|
|
||||||
|
|
||||||
Brainy automatically recovers from errors:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
class AutoRecovery {
|
|
||||||
async handleStorageFailure() {
|
|
||||||
try {
|
|
||||||
await this.primaryStorage.write(data)
|
|
||||||
} catch (error) {
|
|
||||||
console.warn('Primary storage failed, trying fallback')
|
|
||||||
|
|
||||||
// Try secondary storage
|
|
||||||
if (this.secondaryStorage) {
|
|
||||||
await this.secondaryStorage.write(data)
|
|
||||||
} else {
|
|
||||||
// Fall back to memory
|
|
||||||
await this.memoryStorage.write(data)
|
|
||||||
|
|
||||||
// Schedule retry
|
|
||||||
this.scheduleRetry(data)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async handleModelFailure() {
|
|
||||||
try {
|
|
||||||
return await this.primaryModel.embed(text)
|
|
||||||
} catch (error) {
|
|
||||||
// Fall back to simpler model
|
|
||||||
return await this.fallbackModel.embed(text)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Configuration Override
|
## Configuration Override
|
||||||
|
|
||||||
While zero-config is default, you can override when needed:
|
Zero-config is the default, not a ceiling. Every adaptive decision above has an
|
||||||
|
explicit constructor option:
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// Explicit configuration when needed
|
|
||||||
const brain = new Brainy({
|
const brain = new Brainy({
|
||||||
// Override auto-detection
|
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
|
||||||
storage: {
|
vector: {
|
||||||
type: 'filesystem',
|
recall: 'accurate',
|
||||||
path: '/custom/path'
|
persistMode: 'immediate'
|
||||||
},
|
},
|
||||||
|
cache: { maxSize: 50000, ttl: 600_000 }
|
||||||
// Override auto-optimization
|
|
||||||
optimization: {
|
|
||||||
autoIndex: false,
|
|
||||||
autoCache: false,
|
|
||||||
autoBatch: false
|
|
||||||
},
|
|
||||||
|
|
||||||
// Override auto-scaling
|
|
||||||
scaling: {
|
|
||||||
maxMemory: 2 * GB,
|
|
||||||
maxConnections: 100,
|
|
||||||
maxBatchSize: 1000
|
|
||||||
}
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
## Monitoring Auto-Adaptation
|
|
||||||
|
|
||||||
Brainy provides visibility into its auto-adaptation:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
brain.on('adaptation', (event) => {
|
|
||||||
console.log(`Brainy adapted: ${event.type}`)
|
|
||||||
console.log(`Reason: ${event.reason}`)
|
|
||||||
console.log(`Before: ${JSON.stringify(event.before)}`)
|
|
||||||
console.log(`After: ${JSON.stringify(event.after)}`)
|
|
||||||
})
|
})
|
||||||
|
|
||||||
// Example events:
|
await brain.init()
|
||||||
// - Index created for frequently queried field
|
|
||||||
// - Cache strategy changed due to low hit rate
|
|
||||||
// - Batch size increased due to high throughput
|
|
||||||
// - Storage migrated due to space constraints
|
|
||||||
// - Model switched due to multilingual content
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Conclusion
|
See the [API Reference](../api/README.md#configuration) for the complete option
|
||||||
|
list.
|
||||||
Brainy's zero-configuration and auto-adaptation capabilities mean you can focus on your application logic while Brainy handles:
|
|
||||||
|
|
||||||
- Environment detection and optimization
|
|
||||||
- Storage selection and migration
|
|
||||||
- Performance tuning and scaling
|
|
||||||
- Resource management
|
|
||||||
- Error recovery
|
|
||||||
- Workload optimization
|
|
||||||
|
|
||||||
Just create a Brainy instance and start using it. Brainy will learn, adapt, and optimize itself for your specific use case—no configuration required.
|
|
||||||
|
|
||||||
## See Also
|
## See Also
|
||||||
|
|
||||||
- [Architecture Overview](./overview.md)
|
- [Architecture Overview](./overview.md)
|
||||||
- [Storage Architecture](./storage.md)
|
- [Storage Adapters](../concepts/storage-adapters.md)
|
||||||
- [Performance Guide](../guides/performance.md)
|
- [Scaling Guide](../SCALING.md)
|
||||||
- [Augmentations System](./augmentations.md)
|
- [API Reference](../api/README.md)
|
||||||
|
|
|
||||||
|
|
@ -135,11 +135,12 @@ The CLI `brainy inspect` subcommands all do this for you by default
|
||||||
|
|
||||||
## What's not enforced (yet)
|
## What's not enforced (yet)
|
||||||
|
|
||||||
- **Cloud storage backends** (S3, GCS, R2, Azure) do not currently enforce
|
- **Non-filesystem backends** are out of scope in 8.0, which ships only the
|
||||||
multi-process locking. Two processes pointing at the same bucket can both
|
filesystem and memory adapters. A custom `BaseStorage` subclass that is not
|
||||||
succeed at `init()` in writer mode and clobber each other's writes. A
|
filesystem-backed does not enforce multi-process locking by default: two
|
||||||
best-effort warning is logged in writer mode against a non-filesystem
|
processes can both succeed at `init()` in writer mode and clobber each
|
||||||
backend. Lock semantics for cloud backends will land in a future release.
|
other's writes. A best-effort warning is logged in writer mode against a
|
||||||
|
non-filesystem backend.
|
||||||
- **Long-running readers** do not automatically pick up new Cortex segments
|
- **Long-running readers** do not automatically pick up new Cortex segments
|
||||||
the writer publishes. One-shot inspector calls re-open the store and see
|
the writer publishes. One-shot inspector calls re-open the store and see
|
||||||
fresh segments; a reader that stays open for hours sees its column store
|
fresh segments; a reader that stays open for hours sees its column store
|
||||||
|
|
|
||||||
|
|
@ -75,7 +75,7 @@ That's the full ceremony for inheriting multi-process safety.
|
||||||
|
|
||||||
## When NOT to extend FileSystemStorage
|
## When NOT to extend FileSystemStorage
|
||||||
|
|
||||||
If your storage is **not filesystem-backed** (S3, GCS, R2, Azure, a custom
|
If your storage is **not filesystem-backed** (a custom
|
||||||
network backend), extend `BaseStorage` directly:
|
network backend), extend `BaseStorage` directly:
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
|
|
|
||||||
|
|
@ -1,414 +0,0 @@
|
||||||
# 🚀 Brainy 2.0 - Complete Feature List
|
|
||||||
|
|
||||||
> **The Truth**: Brainy is MORE powerful than previously documented! This is the complete list of ALL implemented features.
|
|
||||||
|
|
||||||
## 🧠 Core Intelligence Engine
|
|
||||||
|
|
||||||
### Triple Intelligence System ✅
|
|
||||||
Unified query system that automatically combines:
|
|
||||||
- **Vector Search**: HNSW-indexed semantic similarity (O(log n) performance)
|
|
||||||
- **Graph Traversal**: Relationship-based discovery
|
|
||||||
- **Field Filtering**: Metadata and attribute queries
|
|
||||||
- **Auto-optimization**: Queries are automatically optimized based on data patterns
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// All three intelligences work together automatically
|
|
||||||
const results = await brain.find({
|
|
||||||
like: 'AI research', // Vector search
|
|
||||||
where: { year: 2024 }, // Metadata filtering
|
|
||||||
connected: { to: authorId } // Graph traversal
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
### Neural Query Understanding ✅
|
|
||||||
- **220+ embedded patterns** for query intent detection
|
|
||||||
- Natural language query processing
|
|
||||||
- Automatic query type detection
|
|
||||||
- Query rewriting and optimization
|
|
||||||
|
|
||||||
## 🔧 12+ Production Augmentations
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Full crash recovery, checkpointing, replay
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Entity Registry ✅
|
|
||||||
```typescript
|
|
||||||
import { EntityRegistryAugmentation } from 'brainy'
|
|
||||||
// Bloom filter-based deduplication for streaming data
|
|
||||||
// Handles millions of entities with minimal memory
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Auto-Register Entities ✅
|
|
||||||
```typescript
|
|
||||||
import { AutoRegisterEntitiesAugmentation } from 'brainy'
|
|
||||||
// Automatically extracts and registers entities from text
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. Intelligent Verb Scoring ✅
|
|
||||||
```typescript
|
|
||||||
import { IntelligentVerbScoringAugmentation } from 'brainy'
|
|
||||||
// Multi-factor relationship strength:
|
|
||||||
// - Semantic similarity
|
|
||||||
// - Temporal decay
|
|
||||||
// - Frequency amplification
|
|
||||||
// - Context awareness
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5. Batch Processing ✅
|
|
||||||
```typescript
|
|
||||||
import { BatchProcessingAugmentation } from 'brainy'
|
|
||||||
// Adaptive batching with backpressure
|
|
||||||
// Dynamically adjusts batch size based on load
|
|
||||||
```
|
|
||||||
|
|
||||||
### 6. Connection Pool ✅
|
|
||||||
```typescript
|
|
||||||
import { ConnectionPoolAugmentation } from 'brainy'
|
|
||||||
// Auto-scaling connection management
|
|
||||||
// Optimized for high-concurrency workloads
|
|
||||||
```
|
|
||||||
|
|
||||||
### 7. Request Deduplicator ✅
|
|
||||||
```typescript
|
|
||||||
import { RequestDeduplicatorAugmentation } from 'brainy'
|
|
||||||
// In-flight request deduplication
|
|
||||||
// 3x performance boost for concurrent operations
|
|
||||||
```
|
|
||||||
|
|
||||||
### 8. WebSocket Conduit ✅
|
|
||||||
```typescript
|
|
||||||
import { WebSocketConduitAugmentation } from 'brainy'
|
|
||||||
// Real-time bidirectional streaming
|
|
||||||
// Auto-reconnection and heartbeat
|
|
||||||
```
|
|
||||||
|
|
||||||
### 9. WebRTC Conduit ✅
|
|
||||||
```typescript
|
|
||||||
import { WebRTCConduitAugmentation } from 'brainy'
|
|
||||||
// Peer-to-peer data channels
|
|
||||||
// Direct browser-to-browser communication
|
|
||||||
```
|
|
||||||
|
|
||||||
### 10. Memory Storage Optimization ✅
|
|
||||||
```typescript
|
|
||||||
import { MemoryStorageAugmentation } from 'brainy'
|
|
||||||
// Memory-specific optimizations
|
|
||||||
// Circular buffers, compression
|
|
||||||
```
|
|
||||||
|
|
||||||
### 11. Server Search Conduit ✅
|
|
||||||
```typescript
|
|
||||||
import { ServerSearchConduitAugmentation } from 'brainy'
|
|
||||||
// Forwards queries to a remote Brainy server
|
|
||||||
```
|
|
||||||
|
|
||||||
### 12. Neural Import ✅
|
|
||||||
```typescript
|
|
||||||
import { NeuralImportAugmentation } from 'brainy'
|
|
||||||
// AI-powered data understanding
|
|
||||||
// Automatic entity detection and classification
|
|
||||||
// Relationship discovery
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🤖 Neural Import Capabilities (FULLY IMPLEMENTED!)
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const neuralImport = new NeuralImport(brain)
|
|
||||||
|
|
||||||
// ALL of these work TODAY:
|
|
||||||
await neuralImport.neuralImport('data.csv')
|
|
||||||
await neuralImport.detectEntitiesWithNeuralAnalysis(data)
|
|
||||||
await neuralImport.detectNounType(entity)
|
|
||||||
await neuralImport.detectRelationships(entities)
|
|
||||||
await neuralImport.generateInsights(data)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Features:
|
|
||||||
- **Auto-detects file format** (CSV, JSON, XML, etc.)
|
|
||||||
- **Identifies entity types** using AI
|
|
||||||
- **Discovers relationships** between entities
|
|
||||||
- **Generates insights** about the data
|
|
||||||
- **Creates optimal graph structure** automatically
|
|
||||||
|
|
||||||
## 🎯 Zero-Config Model Loading Cascade
|
|
||||||
|
|
||||||
Brainy automatically loads models with ZERO configuration required:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const brain = new Brainy() // That's it!
|
|
||||||
await brain.init()
|
|
||||||
// Models load automatically from best available source
|
|
||||||
```
|
|
||||||
|
|
||||||
### Loading Priority:
|
|
||||||
1. **Local Cache** (./models) - Instant, no network
|
|
||||||
2. **CDN** (models.soulcraft.com) - Fast, global [Coming Soon]
|
|
||||||
3. **GitHub Releases** - Reliable backup
|
|
||||||
4. **HuggingFace** - Ultimate fallback
|
|
||||||
|
|
||||||
### Key Features:
|
|
||||||
- **Automatic fallback** if sources fail
|
|
||||||
- **Model verification** with checksums
|
|
||||||
- **Offline support** with bundled models
|
|
||||||
- **No environment variables needed**
|
|
||||||
- **Works in all environments** (Node, Browser, Workers)
|
|
||||||
|
|
||||||
## 🏢 Operation Modes
|
|
||||||
|
|
||||||
Run many reader processes against one shared on-disk store with a single writer.
|
|
||||||
|
|
||||||
### Reader Mode ✅
|
|
||||||
```typescript
|
|
||||||
const brain = new Brainy({ mode: 'reader' })
|
|
||||||
// Optimized for read-heavy workloads
|
|
||||||
// 80% cache ratio, aggressive prefetch
|
|
||||||
// 1 hour TTL, minimal writes
|
|
||||||
```
|
|
||||||
|
|
||||||
### Writer Mode ✅
|
|
||||||
```typescript
|
|
||||||
const brain = new Brainy({ mode: 'writer' })
|
|
||||||
// Optimized for write-heavy workloads
|
|
||||||
// Large write buffers, batch writes
|
|
||||||
// Minimal caching, fast ingestion
|
|
||||||
```
|
|
||||||
|
|
||||||
### Hybrid Mode ✅
|
|
||||||
```typescript
|
|
||||||
const brain = new Brainy({ mode: 'hybrid' })
|
|
||||||
// Balanced for mixed workloads
|
|
||||||
// Adaptive caching and batching
|
|
||||||
```
|
|
||||||
|
|
||||||
## 💾 Advanced Caching System
|
|
||||||
|
|
||||||
### 3-Level Cache Architecture ✅
|
|
||||||
```typescript
|
|
||||||
const cacheConfig = {
|
|
||||||
hotCache: {
|
|
||||||
size: 1000, // L1 - RAM
|
|
||||||
ttl: 60000 // 1 minute
|
|
||||||
},
|
|
||||||
warmCache: {
|
|
||||||
size: 10000, // L2 - Fast storage
|
|
||||||
ttl: 300000 // 5 minutes
|
|
||||||
},
|
|
||||||
coldCache: {
|
|
||||||
size: 100000, // L3 - Persistent
|
|
||||||
ttl: null // No expiry
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Cache Features:
|
|
||||||
- **Automatic promotion/demotion** between levels
|
|
||||||
- **LRU eviction** within each level
|
|
||||||
- **Compression** for cold cache
|
|
||||||
- **Statistics tracking** for optimization
|
|
||||||
|
|
||||||
## 📊 Comprehensive Statistics
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const stats = await brain.getStats()
|
|
||||||
// Returns detailed metrics:
|
|
||||||
{
|
|
||||||
nouns: {
|
|
||||||
count, created, updated, deleted,
|
|
||||||
size, avgSize
|
|
||||||
},
|
|
||||||
verbs: {
|
|
||||||
count, created, types,
|
|
||||||
weights: { min, max, avg }
|
|
||||||
},
|
|
||||||
vectors: {
|
|
||||||
dimensions: 384,
|
|
||||||
indexSize, partitions,
|
|
||||||
avgSearchTime
|
|
||||||
},
|
|
||||||
cache: {
|
|
||||||
hits, misses, evictions,
|
|
||||||
hitRate, sizes
|
|
||||||
},
|
|
||||||
performance: {
|
|
||||||
operations, avgTimes,
|
|
||||||
p95Latency, p99Latency
|
|
||||||
},
|
|
||||||
storage: {
|
|
||||||
used, available,
|
|
||||||
compression, files
|
|
||||||
},
|
|
||||||
throttling: {
|
|
||||||
delays, rateLimited,
|
|
||||||
backoffMs, retries
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🚀 GPU Acceleration Support
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Automatic GPU detection
|
|
||||||
const device = await detectBestDevice()
|
|
||||||
// Returns: 'cpu' | 'webgpu' | 'cuda'
|
|
||||||
|
|
||||||
// WebGPU in browser (when available)
|
|
||||||
if (device === 'webgpu') {
|
|
||||||
// Transformer models use WebGPU automatically
|
|
||||||
}
|
|
||||||
|
|
||||||
// CUDA in Node.js (future GPU support)
|
|
||||||
if (device === 'cuda') {
|
|
||||||
// Future: GPU acceleration for embeddings
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🔄 Adaptive Systems
|
|
||||||
|
|
||||||
### Adaptive Backpressure ✅
|
|
||||||
```typescript
|
|
||||||
// Automatically adjusts flow based on system load
|
|
||||||
// Prevents OOM and maintains throughput
|
|
||||||
```
|
|
||||||
|
|
||||||
### Adaptive Socket Manager ✅
|
|
||||||
```typescript
|
|
||||||
// Dynamic connection pooling
|
|
||||||
// Scales connections based on traffic patterns
|
|
||||||
```
|
|
||||||
|
|
||||||
### Cache Auto-Configuration ✅
|
|
||||||
```typescript
|
|
||||||
// Sizes cache based on available memory
|
|
||||||
// Adjusts strategies based on usage patterns
|
|
||||||
```
|
|
||||||
|
|
||||||
### S3 Throttling Protection ✅
|
|
||||||
```typescript
|
|
||||||
// Built-in exponential backoff
|
|
||||||
// Rate limit detection and adaptation
|
|
||||||
// Automatic retry with jitter
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🛠️ Storage Adapters
|
|
||||||
|
|
||||||
All included, auto-selected based on environment:
|
|
||||||
|
|
||||||
### FileSystem Storage ✅
|
|
||||||
- Default for Node.js
|
|
||||||
- Efficient file-based storage
|
|
||||||
- Automatic directory management
|
|
||||||
|
|
||||||
### Memory Storage ✅
|
|
||||||
- Ultra-fast in-memory operations
|
|
||||||
- Perfect for testing and temporary data
|
|
||||||
- Circular buffer support
|
|
||||||
|
|
||||||
### OPFS Storage ✅
|
|
||||||
- Browser persistent storage
|
|
||||||
- Survives page refreshes
|
|
||||||
- Quota management
|
|
||||||
|
|
||||||
### S3 Storage ✅
|
|
||||||
- AWS S3 compatible
|
|
||||||
- Automatic multipart uploads
|
|
||||||
- Throttling protection
|
|
||||||
- Batch operations
|
|
||||||
|
|
||||||
## 🎨 Natural Language Processing
|
|
||||||
|
|
||||||
### Built-in Patterns (220+)
|
|
||||||
- Question types (what, why, how, when, where)
|
|
||||||
- Temporal queries (yesterday, last week, 2024)
|
|
||||||
- Comparative queries (better than, similar to)
|
|
||||||
- Aggregations (count, sum, average)
|
|
||||||
- Filters (only, except, without)
|
|
||||||
- Relationships (related to, connected with)
|
|
||||||
|
|
||||||
### Coverage: 94-98% of typical queries!
|
|
||||||
|
|
||||||
## 🔐 Security Features
|
|
||||||
|
|
||||||
### Built-in Security ✅
|
|
||||||
- Automatic input sanitization
|
|
||||||
- SQL injection prevention
|
|
||||||
- XSS protection for web contexts
|
|
||||||
- Rate limiting support
|
|
||||||
|
|
||||||
### Encryption Ready ✅
|
|
||||||
```typescript
|
|
||||||
import { crypto } from 'brainy/utils'
|
|
||||||
// AES-256-GCM encryption utilities
|
|
||||||
// Key derivation functions
|
|
||||||
// Secure random generation
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🎯 Key Design Principles
|
|
||||||
|
|
||||||
### 1. Zero Configuration
|
|
||||||
```typescript
|
|
||||||
const brain = new Brainy()
|
|
||||||
await brain.init()
|
|
||||||
// Everything else is automatic!
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Fixed Dimensions (384)
|
|
||||||
- **ALWAYS** uses all-MiniLM-L6-v2 model
|
|
||||||
- **ALWAYS** 384 dimensions
|
|
||||||
- **NOT** configurable (by design)
|
|
||||||
- Ensures everything works together
|
|
||||||
|
|
||||||
### 3. Progressive Enhancement
|
|
||||||
- Starts simple, scales automatically
|
|
||||||
- Adapts to workload patterns
|
|
||||||
- Optimizes based on usage
|
|
||||||
|
|
||||||
### 4. Universal Compatibility
|
|
||||||
- Works in Node.js 18+
|
|
||||||
- Works in modern browsers
|
|
||||||
- Works in Web Workers
|
|
||||||
- Works in Edge environments
|
|
||||||
|
|
||||||
## 📦 What Ships in Core (MIT Licensed)
|
|
||||||
|
|
||||||
**EVERYTHING** is included in the core package:
|
|
||||||
- ✅ All engines (vector, graph, field, neural)
|
|
||||||
- ✅ All augmentations (12+)
|
|
||||||
- ✅ All storage adapters
|
|
||||||
- ✅ Reader / writer / hybrid operation modes
|
|
||||||
- ✅ Complete statistics
|
|
||||||
- ✅ GPU support
|
|
||||||
- ✅ No feature limitations
|
|
||||||
- ✅ No premium tiers
|
|
||||||
- ✅ 100% MIT licensed
|
|
||||||
|
|
||||||
## 🚀 Quick Start
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import { Brainy } from 'brainy'
|
|
||||||
|
|
||||||
// Zero config required!
|
|
||||||
const brain = new Brainy()
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Add data (auto-detects type)
|
|
||||||
await brain.add('Content here')
|
|
||||||
|
|
||||||
// Search with natural language
|
|
||||||
const results = await brain.find('related content from last week')
|
|
||||||
|
|
||||||
// Everything else is automatic!
|
|
||||||
```
|
|
||||||
|
|
||||||
## 📈 Performance Characteristics
|
|
||||||
|
|
||||||
- **Vector Search**: O(log n) with HNSW indexing
|
|
||||||
- **Graph Traversal**: O(k) for k-hop queries
|
|
||||||
- **Field Filtering**: O(1) with metadata index
|
|
||||||
- **Memory Usage**: ~100MB base + data
|
|
||||||
- **Embedding Speed**: ~100ms for batch of 10
|
|
||||||
- **Query Speed**: <10ms for most queries
|
|
||||||
|
|
||||||
## 🎉 Summary
|
|
||||||
|
|
||||||
Brainy 2.0 is a **complete**, **production-ready** AI database that requires **ZERO configuration**. Every feature listed here is **implemented and working** today. No configuration, no setup, no complexity - just powerful AI capabilities that work out of the box!
|
|
||||||
|
|
@ -1,242 +0,0 @@
|
||||||
# 🚀 Brainy - Production-Ready Features
|
|
||||||
|
|
||||||
> **Status**: All features listed here are IMPLEMENTED and TESTED
|
|
||||||
|
|
||||||
## 📊 Performance Metrics
|
|
||||||
- **Search Latency**: <10ms for 10,000+ items
|
|
||||||
- **Write Throughput**: 10,000+ ops/sec
|
|
||||||
- **Memory Efficiency**: <500MB for 10K items
|
|
||||||
- **Concurrent Operations**: 100+ simultaneous operations
|
|
||||||
|
|
||||||
## 🧠 Core Intelligence Features
|
|
||||||
|
|
||||||
### Triple Intelligence System ✅
|
|
||||||
Unified query system combining three types of intelligence:
|
|
||||||
```typescript
|
|
||||||
const results = await brain.find({
|
|
||||||
like: 'AI research', // Vector similarity search
|
|
||||||
where: { year: 2024 }, // Metadata filtering
|
|
||||||
connected: { to: authorId } // Graph relationships
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
### Intelligent Type Mapping ✅
|
|
||||||
Prevents semantic degradation by intelligently inferring types:
|
|
||||||
```typescript
|
|
||||||
// Automatically infers 'person' from email field
|
|
||||||
brain.add({ name: "John", email: "john@example.com" }, 'entity')
|
|
||||||
// → Stored as type: 'person', not generic 'entity'
|
|
||||||
```
|
|
||||||
|
|
||||||
### Neural Query Understanding ✅
|
|
||||||
- 220+ embedded patterns for intent detection
|
|
||||||
- Natural language query processing
|
|
||||||
- Automatic query optimization
|
|
||||||
- Pattern-based query rewriting
|
|
||||||
|
|
||||||
## 🔐 Security & Compliance
|
|
||||||
|
|
||||||
### Rate Limiting ✅
|
|
||||||
Per-operation configurable limits:
|
|
||||||
```typescript
|
|
||||||
const rateLimiter = createRateLimitAugmentation({
|
|
||||||
limits: {
|
|
||||||
searches: 1000, // per minute
|
|
||||||
writes: 100,
|
|
||||||
reads: 5000,
|
|
||||||
deletes: 50
|
|
||||||
}
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
### Audit Logging ✅
|
|
||||||
Comprehensive operation tracking:
|
|
||||||
```typescript
|
|
||||||
const auditLogger = createAuditLogAugmentation({
|
|
||||||
logLevel: 'detailed',
|
|
||||||
retention: 90, // days
|
|
||||||
includeMetadata: true
|
|
||||||
})
|
|
||||||
|
|
||||||
// Query audit logs
|
|
||||||
const logs = auditLogger.queryLogs({
|
|
||||||
operation: 'add',
|
|
||||||
startTime: Date.now() - 3600000
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
## 📦 Storage & Persistence
|
|
||||||
|
|
||||||
Full crash recovery and replay:
|
|
||||||
```typescript
|
|
||||||
enabled: true,
|
|
||||||
checkpointInterval: 1000,
|
|
||||||
maxLogSize: 100 * 1024 * 1024 // 100MB
|
|
||||||
}))
|
|
||||||
```
|
|
||||||
|
|
||||||
### Multi-Tenancy ✅
|
|
||||||
Service-based data isolation:
|
|
||||||
```typescript
|
|
||||||
// Isolated data per service
|
|
||||||
await brain.add(data, 'document', { service: 'tenant-1' })
|
|
||||||
await brain.find('query', { service: 'tenant-1' })
|
|
||||||
```
|
|
||||||
|
|
||||||
### Write-Only Mode ✅
|
|
||||||
For dedicated write nodes:
|
|
||||||
```typescript
|
|
||||||
const brain = new Brainy({
|
|
||||||
mode: 'write-only',
|
|
||||||
storage: 's3://bucket/path'
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🚀 Performance Features
|
|
||||||
|
|
||||||
### Batch Operations ✅
|
|
||||||
Optimized bulk processing:
|
|
||||||
```typescript
|
|
||||||
// Parallel processing with automatic batching
|
|
||||||
await brain.addMany(items) // <10ms per item
|
|
||||||
await brain.updateMany(updates)
|
|
||||||
await brain.removeMany(filters)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Request Deduplication ✅
|
|
||||||
Automatic duplicate request handling:
|
|
||||||
```typescript
|
|
||||||
brain.use(new RequestDeduplicatorAugmentation())
|
|
||||||
// Identical concurrent requests return same result
|
|
||||||
```
|
|
||||||
|
|
||||||
### Smart Caching ✅
|
|
||||||
Intelligent search result caching:
|
|
||||||
```typescript
|
|
||||||
brain.use(new CacheAugmentation({
|
|
||||||
maxSize: 10000,
|
|
||||||
ttl: 300000, // 5 minutes
|
|
||||||
invalidateOnWrite: true
|
|
||||||
}))
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🔄 Data Processing
|
|
||||||
|
|
||||||
### Entity Registry ✅
|
|
||||||
Bloom filter-based deduplication:
|
|
||||||
```typescript
|
|
||||||
brain.use(new EntityRegistryAugmentation())
|
|
||||||
// Handles millions of entities with minimal memory
|
|
||||||
```
|
|
||||||
|
|
||||||
### Neural Import ✅
|
|
||||||
Intelligent data import with type inference:
|
|
||||||
```typescript
|
|
||||||
await brain.import({
|
|
||||||
source: 'data.json',
|
|
||||||
autoDetectTypes: true,
|
|
||||||
batchSize: 1000
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
### Streaming Pipeline ✅
|
|
||||||
Real-time data processing:
|
|
||||||
```typescript
|
|
||||||
brain.stream()
|
|
||||||
.pipe(transform)
|
|
||||||
.pipe(enrich)
|
|
||||||
.pipe(brain.writer())
|
|
||||||
```
|
|
||||||
|
|
||||||
## 📊 Analytics & Monitoring
|
|
||||||
|
|
||||||
### Metrics Collection ✅
|
|
||||||
Built-in performance metrics:
|
|
||||||
```typescript
|
|
||||||
const metrics = brain.getMetrics()
|
|
||||||
// {
|
|
||||||
// operations: { add: 1000, find: 5000 },
|
|
||||||
// performance: { p95: 8, p99: 12 },
|
|
||||||
// cache: { hits: 4500, misses: 500 }
|
|
||||||
// }
|
|
||||||
```
|
|
||||||
|
|
||||||
### Health Monitoring ✅
|
|
||||||
Automatic health checks:
|
|
||||||
```typescript
|
|
||||||
const health = brain.getHealth()
|
|
||||||
// {
|
|
||||||
// status: 'healthy',
|
|
||||||
// storage: 'connected',
|
|
||||||
// memory: { used: 245, limit: 512 }
|
|
||||||
// }
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🛠️ Developer Experience
|
|
||||||
|
|
||||||
### Zero Configuration ✅
|
|
||||||
Works out of the box:
|
|
||||||
```typescript
|
|
||||||
import Brainy from '@soulcraft/brainy'
|
|
||||||
const brain = new Brainy() // Auto-configures everything
|
|
||||||
```
|
|
||||||
|
|
||||||
### TypeScript First ✅
|
|
||||||
Full type safety and inference:
|
|
||||||
```typescript
|
|
||||||
// Types are automatically inferred
|
|
||||||
const results = await brain.find<MyType>('query')
|
|
||||||
```
|
|
||||||
|
|
||||||
### Augmentation System ✅
|
|
||||||
Extensible plugin architecture:
|
|
||||||
```typescript
|
|
||||||
class CustomAugmentation extends BaseAugmentation {
|
|
||||||
execute(operation, params, next) {
|
|
||||||
// Your custom logic
|
|
||||||
return next()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🔧 Operational Features
|
|
||||||
|
|
||||||
### Graceful Shutdown ✅
|
|
||||||
Clean shutdown with data persistence:
|
|
||||||
```typescript
|
|
||||||
process.on('SIGTERM', async () => {
|
|
||||||
await brain.shutdown() // Saves all pending data
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
### Hot Reload ✅
|
|
||||||
Configuration updates without restart:
|
|
||||||
```typescript
|
|
||||||
brain.updateConfig({
|
|
||||||
cache: { enabled: false }
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
### Backup & Restore ✅
|
|
||||||
Full data backup capabilities:
|
|
||||||
```typescript
|
|
||||||
await brain.backup('backup.bin')
|
|
||||||
await brain.restore('backup.bin')
|
|
||||||
```
|
|
||||||
|
|
||||||
## 📈 Proven at Scale
|
|
||||||
|
|
||||||
- **10,000+ items**: Sub-10ms search
|
|
||||||
- **1M+ operations**: Stable memory usage
|
|
||||||
- **100+ concurrent users**: No performance degradation
|
|
||||||
|
|
||||||
## 🚫 NOT Implemented (Planned)
|
|
||||||
|
|
||||||
These features are documented but NOT yet implemented:
|
|
||||||
- GraphQL API (use REST API instead)
|
|
||||||
- Kubernetes operators (use Docker)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
*Last Updated: Latest Version*
|
|
||||||
*All features listed above are production-ready and tested*
|
|
||||||
|
|
@ -1,15 +1,17 @@
|
||||||
# Framework Integration Guide
|
# Framework Integration Guide
|
||||||
|
|
||||||
Brainy 3.0 is **framework-first** - designed from the ground up to work seamlessly with modern JavaScript frameworks. This guide shows you how to integrate Brainy into any framework.
|
Brainy is **framework-friendly** - designed to drop into the server side of any modern JavaScript framework. This guide shows you how to integrate Brainy into framework-based apps.
|
||||||
|
|
||||||
## 🎯 Why Framework-First?
|
> **Runtime**: Brainy 8.0 runs on Node.js 22+ and Bun (server-side only). It is not a browser library. In framework apps, run Brainy in API routes, server components, server actions, loaders, or a dedicated backend service - never in client-side bundles.
|
||||||
|
|
||||||
Traditional AI databases require complex browser polyfills and bundler configurations. Brainy 3.0 trusts your framework to handle this:
|
## 🎯 Why Server-Side?
|
||||||
|
|
||||||
|
Brainy embeds an HNSW vector index, a graph engine, and a filesystem-backed persistence layer. These belong on the server:
|
||||||
|
|
||||||
- **Zero configuration**: Just `import { Brainy } from '@soulcraft/brainy'`
|
- **Zero configuration**: Just `import { Brainy } from '@soulcraft/brainy'`
|
||||||
- **Framework responsibility**: Let Next.js, Vite, Webpack handle Node.js polyfills
|
- **Auto storage detection**: `new Brainy()` auto-selects filesystem persistence on Node
|
||||||
- **Cleaner code**: No browser-specific entry points or conditional imports
|
- **Cleaner code**: No browser polyfills, no conditional client/server imports
|
||||||
- **Better DX**: Same API everywhere - browser, server, edge
|
- **Better DX**: One instance shared across your server routes
|
||||||
|
|
||||||
## 🚀 Quick Start
|
## 🚀 Quick Start
|
||||||
|
|
||||||
|
|
@ -24,7 +26,8 @@ npm install @soulcraft/brainy
|
||||||
```javascript
|
```javascript
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
import { Brainy } from '@soulcraft/brainy'
|
||||||
|
|
||||||
// Works in any framework!
|
// Run on the server (API route, server component, backend service)
|
||||||
|
// new Brainy() auto-detects filesystem persistence on Node
|
||||||
const brain = new Brainy()
|
const brain = new Brainy()
|
||||||
await brain.init()
|
await brain.init()
|
||||||
|
|
||||||
|
|
@ -41,52 +44,48 @@ const results = await brain.find("framework integration")
|
||||||
|
|
||||||
## ⚛️ React Integration
|
## ⚛️ React Integration
|
||||||
|
|
||||||
|
Brainy runs on the server, so a React client component talks to it through an API endpoint (see the Next.js API route below). The hook fetches results; it never instantiates Brainy in the browser.
|
||||||
|
|
||||||
### Basic Hook Pattern
|
### Basic Hook Pattern
|
||||||
|
|
||||||
```jsx
|
```jsx
|
||||||
import { useState, useEffect, useCallback } from 'react'
|
import { useState, useCallback } from 'react'
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
|
||||||
|
|
||||||
function useBrainy() {
|
function useBrainySearch(endpoint = '/api/search') {
|
||||||
const [brain, setBrain] = useState(null)
|
const [results, setResults] = useState([])
|
||||||
const [isReady, setIsReady] = useState(false)
|
const [loading, setLoading] = useState(false)
|
||||||
|
|
||||||
useEffect(() => {
|
const search = useCallback(async (query) => {
|
||||||
const initBrain = async () => {
|
if (!query) return
|
||||||
const newBrain = new Brainy({
|
setLoading(true)
|
||||||
storage: { type: 'opfs' } // Browser storage
|
try {
|
||||||
|
const res = await fetch(endpoint, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ query })
|
||||||
})
|
})
|
||||||
await newBrain.init()
|
const { results } = await res.json()
|
||||||
setBrain(newBrain)
|
setResults(results)
|
||||||
setIsReady(true)
|
} finally {
|
||||||
|
setLoading(false)
|
||||||
}
|
}
|
||||||
|
}, [endpoint])
|
||||||
|
|
||||||
initBrain()
|
return { results, loading, search }
|
||||||
}, [])
|
|
||||||
|
|
||||||
return { brain, isReady }
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Usage in component
|
// Usage in component
|
||||||
function SearchComponent() {
|
function SearchComponent() {
|
||||||
const { brain, isReady } = useBrainy()
|
const { results, loading, search } = useBrainySearch()
|
||||||
const [results, setResults] = useState([])
|
|
||||||
|
|
||||||
const handleSearch = useCallback(async (query) => {
|
|
||||||
if (!isReady) return
|
|
||||||
const searchResults = await brain.find(query)
|
|
||||||
setResults(searchResults)
|
|
||||||
}, [brain, isReady])
|
|
||||||
|
|
||||||
if (!isReady) return <div>Loading AI...</div>
|
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div>
|
<div>
|
||||||
<input
|
<input
|
||||||
type="text"
|
type="text"
|
||||||
placeholder="Search..."
|
placeholder="Search..."
|
||||||
onChange={(e) => handleSearch(e.target.value)}
|
onChange={(e) => search(e.target.value)}
|
||||||
/>
|
/>
|
||||||
|
{loading && <div>Searching...</div>}
|
||||||
<div>
|
<div>
|
||||||
{results.map(result => (
|
{results.map(result => (
|
||||||
<div key={result.id}>
|
<div key={result.id}>
|
||||||
|
|
@ -100,48 +99,34 @@ function SearchComponent() {
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
### React Context Pattern
|
### Shared Server Instance
|
||||||
|
|
||||||
```jsx
|
On the server, create one Brainy instance and reuse it across requests. This module is imported only by server code (API routes, server components), never by client components:
|
||||||
import React, { createContext, useContext, useEffect, useState } from 'react'
|
|
||||||
|
```javascript
|
||||||
|
// lib/brain.server.js
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
import { Brainy } from '@soulcraft/brainy'
|
||||||
|
|
||||||
const BrainyContext = createContext()
|
let brainPromise
|
||||||
|
|
||||||
export function BrainyProvider({ children }) {
|
export function getBrain() {
|
||||||
const [brain, setBrain] = useState(null)
|
if (!brainPromise) {
|
||||||
const [isReady, setIsReady] = useState(false)
|
brainPromise = (async () => {
|
||||||
|
// new Brainy() auto-detects filesystem persistence on Node
|
||||||
useEffect(() => {
|
const brain = new Brainy()
|
||||||
const initBrain = async () => {
|
await brain.init()
|
||||||
const newBrain = new Brainy()
|
return brain
|
||||||
await newBrain.init()
|
})()
|
||||||
setBrain(newBrain)
|
|
||||||
setIsReady(true)
|
|
||||||
}
|
}
|
||||||
|
return brainPromise
|
||||||
initBrain()
|
|
||||||
}, [])
|
|
||||||
|
|
||||||
return (
|
|
||||||
<BrainyContext.Provider value={{ brain, isReady }}>
|
|
||||||
{children}
|
|
||||||
</BrainyContext.Provider>
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
export function useBrainContext() {
|
|
||||||
const context = useContext(BrainyContext)
|
|
||||||
if (!context) {
|
|
||||||
throw new Error('useBrainContext must be used within BrainyProvider')
|
|
||||||
}
|
|
||||||
return context
|
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🟢 Vue.js Integration
|
## 🟢 Vue.js Integration
|
||||||
|
|
||||||
### Composition API
|
Vue components call a server endpoint (see the Nuxt server route in the [Vue.js Integration Guide](vue-integration.md)); Brainy itself runs on the server.
|
||||||
|
|
||||||
|
### Composition API (client component)
|
||||||
|
|
||||||
```vue
|
```vue
|
||||||
<template>
|
<template>
|
||||||
|
|
@ -155,100 +140,70 @@ export function useBrainContext() {
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { ref, onMounted } from 'vue'
|
import { ref } from 'vue'
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
|
||||||
|
|
||||||
const brain = ref(null)
|
|
||||||
const isReady = ref(false)
|
|
||||||
const query = ref('')
|
const query = ref('')
|
||||||
const results = ref([])
|
const results = ref([])
|
||||||
|
|
||||||
onMounted(async () => {
|
|
||||||
brain.value = new Brainy({
|
|
||||||
storage: { type: 'opfs' }
|
|
||||||
})
|
|
||||||
await brain.value.init()
|
|
||||||
isReady.value = true
|
|
||||||
})
|
|
||||||
|
|
||||||
const search = async () => {
|
const search = async () => {
|
||||||
if (!isReady.value || !query.value) return
|
if (!query.value) return
|
||||||
results.value = await brain.value.find(query.value)
|
const res = await fetch('/api/search', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ query: query.value })
|
||||||
|
})
|
||||||
|
results.value = (await res.json()).results
|
||||||
}
|
}
|
||||||
</script>
|
</script>
|
||||||
```
|
```
|
||||||
|
|
||||||
### Vue 3 Plugin
|
### Shared Server Instance
|
||||||
|
|
||||||
|
On the server, create one Brainy instance and reuse it across requests:
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// plugins/brainy.js
|
// server/brain.js (server-only module)
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
import { Brainy } from '@soulcraft/brainy'
|
||||||
|
|
||||||
export default {
|
let brainPromise
|
||||||
install(app, options) {
|
|
||||||
const brain = new Brainy(options)
|
|
||||||
|
|
||||||
app.config.globalProperties.$brain = brain
|
export function getBrain() {
|
||||||
app.provide('brain', brain)
|
if (!brainPromise) {
|
||||||
|
brainPromise = (async () => {
|
||||||
// Initialize on app mount
|
// new Brainy() auto-detects filesystem persistence on Node
|
||||||
brain.init()
|
const brain = new Brainy()
|
||||||
|
await brain.init()
|
||||||
|
return brain
|
||||||
|
})()
|
||||||
}
|
}
|
||||||
|
return brainPromise
|
||||||
}
|
}
|
||||||
|
|
||||||
// main.js
|
|
||||||
import { createApp } from 'vue'
|
|
||||||
import BrainyPlugin from './plugins/brainy'
|
|
||||||
|
|
||||||
const app = createApp(App)
|
|
||||||
app.use(BrainyPlugin, {
|
|
||||||
storage: { type: 'opfs' }
|
|
||||||
})
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🅰️ Angular Integration
|
## 🅰️ Angular Integration
|
||||||
|
|
||||||
### Service Pattern
|
The Angular service calls your backend over HTTP; Brainy lives in that backend, not in the browser.
|
||||||
|
|
||||||
|
### Service Pattern (calls the backend)
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// brainy.service.ts
|
// brainy.service.ts
|
||||||
import { Injectable } from '@angular/core'
|
import { Injectable } from '@angular/core'
|
||||||
import { BehaviorSubject, Observable } from 'rxjs'
|
import { HttpClient } from '@angular/common/http'
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
import { Observable } from 'rxjs'
|
||||||
|
|
||||||
@Injectable({
|
@Injectable({
|
||||||
providedIn: 'root'
|
providedIn: 'root'
|
||||||
})
|
})
|
||||||
export class BrainyService {
|
export class BrainyService {
|
||||||
private brain: Brainy
|
constructor(private http: HttpClient) {}
|
||||||
private readySubject = new BehaviorSubject<boolean>(false)
|
|
||||||
|
|
||||||
ready$: Observable<boolean> = this.readySubject.asObservable()
|
search(query: string): Observable<{ results: any[] }> {
|
||||||
|
return this.http.post<{ results: any[] }>('/api/search', { query })
|
||||||
constructor() {
|
|
||||||
this.initBrain()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private async initBrain() {
|
add(data: any, type: string, metadata?: any): Observable<{ id: string }> {
|
||||||
this.brain = new Brainy({
|
return this.http.post<{ id: string }>('/api/add', { data, type, metadata })
|
||||||
storage: { type: 'opfs' }
|
|
||||||
})
|
|
||||||
await this.brain.init()
|
|
||||||
this.readySubject.next(true)
|
|
||||||
}
|
|
||||||
|
|
||||||
async search(query: string): Promise<any[]> {
|
|
||||||
if (!this.readySubject.value) {
|
|
||||||
throw new Error('Brain not ready')
|
|
||||||
}
|
|
||||||
return await this.brain.find(query)
|
|
||||||
}
|
|
||||||
|
|
||||||
async add(data: any, type: string, metadata?: any): Promise<string> {
|
|
||||||
if (!this.readySubject.value) {
|
|
||||||
throw new Error('Brain not ready')
|
|
||||||
}
|
|
||||||
return await this.brain.add({ data, type, metadata })
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
@ -280,64 +235,52 @@ export class SearchComponent {
|
||||||
|
|
||||||
constructor(private brainyService: BrainyService) {}
|
constructor(private brainyService: BrainyService) {}
|
||||||
|
|
||||||
async search() {
|
search() {
|
||||||
if (!this.query) return
|
if (!this.query) return
|
||||||
this.results = await this.brainyService.search(this.query)
|
this.brainyService.search(this.query).subscribe(({ results }) => {
|
||||||
|
this.results = results
|
||||||
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The matching backend endpoint uses Brainy directly (Node/Bun):
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// server: api/search
|
||||||
|
import { Brainy } from '@soulcraft/brainy'
|
||||||
|
|
||||||
|
const brain = new Brainy() // auto-detects filesystem persistence on Node
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
|
export async function handleSearch(query: string) {
|
||||||
|
return await brain.find(query)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
## 🚀 Next.js Integration
|
## 🚀 Next.js Integration
|
||||||
|
|
||||||
### App Router (Next.js 13+)
|
In Next.js, Brainy lives in server code only: API routes, server components, or server actions. Create one shared instance in a server-only module.
|
||||||
|
|
||||||
```jsx
|
### Shared Server Instance
|
||||||
// app/providers.jsx
|
|
||||||
'use client'
|
```javascript
|
||||||
import { createContext, useContext, useEffect, useState } from 'react'
|
// lib/brain.server.js (imported only by server code)
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
import { Brainy } from '@soulcraft/brainy'
|
||||||
|
|
||||||
const BrainyContext = createContext()
|
let brainPromise
|
||||||
|
|
||||||
export function BrainyProvider({ children }) {
|
export function getBrain() {
|
||||||
const [brain, setBrain] = useState(null)
|
if (!brainPromise) {
|
||||||
const [isReady, setIsReady] = useState(false)
|
brainPromise = (async () => {
|
||||||
|
const brain = new Brainy({
|
||||||
useEffect(() => {
|
storage: { type: 'filesystem', rootDirectory: './data' }
|
||||||
const initBrain = async () => {
|
})
|
||||||
const newBrain = new Brainy()
|
await brain.init()
|
||||||
await newBrain.init()
|
return brain
|
||||||
setBrain(newBrain)
|
})()
|
||||||
setIsReady(true)
|
|
||||||
}
|
}
|
||||||
|
return brainPromise
|
||||||
initBrain()
|
|
||||||
}, [])
|
|
||||||
|
|
||||||
return (
|
|
||||||
<BrainyContext.Provider value={{ brain, isReady }}>
|
|
||||||
{children}
|
|
||||||
</BrainyContext.Provider>
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
export const useBrainy = () => useContext(BrainyContext)
|
|
||||||
```
|
|
||||||
|
|
||||||
```jsx
|
|
||||||
// app/layout.jsx
|
|
||||||
import { BrainyProvider } from './providers'
|
|
||||||
|
|
||||||
export default function RootLayout({ children }) {
|
|
||||||
return (
|
|
||||||
<html>
|
|
||||||
<body>
|
|
||||||
<BrainyProvider>
|
|
||||||
{children}
|
|
||||||
</BrainyProvider>
|
|
||||||
</body>
|
|
||||||
</html>
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -345,45 +288,78 @@ export default function RootLayout({ children }) {
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// app/api/search/route.js
|
// app/api/search/route.js
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
import { getBrain } from '@/lib/brain.server'
|
||||||
|
|
||||||
const brain = new Brainy({
|
|
||||||
storage: { type: 'filesystem', rootDirectory: './data' }
|
|
||||||
})
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
export async function POST(request) {
|
export async function POST(request) {
|
||||||
const { query } = await request.json()
|
const { query } = await request.json()
|
||||||
|
const brain = await getBrain()
|
||||||
const results = await brain.find(query)
|
const results = await brain.find(query)
|
||||||
|
|
||||||
return Response.json({ results })
|
return Response.json({ results })
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🔷 Svelte Integration
|
### Server Action
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// app/actions.js
|
||||||
|
'use server'
|
||||||
|
import { getBrain } from '@/lib/brain.server'
|
||||||
|
|
||||||
|
export async function search(query) {
|
||||||
|
const brain = await getBrain()
|
||||||
|
return await brain.find(query)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🔷 SvelteKit Integration
|
||||||
|
|
||||||
|
Brainy runs in a server-only module (`*.server.js`); the component fetches results from an endpoint.
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// src/lib/server/brain.js (server-only — note the .server suffix)
|
||||||
|
import { Brainy } from '@soulcraft/brainy'
|
||||||
|
|
||||||
|
let brainPromise
|
||||||
|
|
||||||
|
export function getBrain() {
|
||||||
|
if (!brainPromise) {
|
||||||
|
brainPromise = (async () => {
|
||||||
|
const brain = new Brainy() // auto-detects filesystem persistence
|
||||||
|
await brain.init()
|
||||||
|
return brain
|
||||||
|
})()
|
||||||
|
}
|
||||||
|
return brainPromise
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// src/routes/api/search/+server.js
|
||||||
|
import { json } from '@sveltejs/kit'
|
||||||
|
import { getBrain } from '$lib/server/brain'
|
||||||
|
|
||||||
|
export async function POST({ request }) {
|
||||||
|
const { query } = await request.json()
|
||||||
|
const brain = await getBrain()
|
||||||
|
return json({ results: await brain.find(query) })
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
```svelte
|
```svelte
|
||||||
<!-- SearchComponent.svelte -->
|
<!-- SearchComponent.svelte -->
|
||||||
<script>
|
<script>
|
||||||
import { onMount } from 'svelte'
|
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
|
||||||
|
|
||||||
let brain = null
|
|
||||||
let isReady = false
|
|
||||||
let query = ''
|
let query = ''
|
||||||
let results = []
|
let results = []
|
||||||
|
|
||||||
onMount(async () => {
|
|
||||||
brain = new Brainy({
|
|
||||||
storage: { type: 'opfs' }
|
|
||||||
})
|
|
||||||
await brain.init()
|
|
||||||
isReady = true
|
|
||||||
})
|
|
||||||
|
|
||||||
async function search() {
|
async function search() {
|
||||||
if (!isReady || !query) return
|
if (!query) return
|
||||||
results = await brain.find(query)
|
const res = await fetch('/api/search', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ query })
|
||||||
|
})
|
||||||
|
results = (await res.json()).results
|
||||||
}
|
}
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
|
|
@ -401,27 +377,23 @@ export async function POST(request) {
|
||||||
|
|
||||||
## 🌟 Solid.js Integration
|
## 🌟 Solid.js Integration
|
||||||
|
|
||||||
|
The component calls a server endpoint (use SolidStart server routes, or any backend, to host Brainy):
|
||||||
|
|
||||||
```jsx
|
```jsx
|
||||||
import { createSignal, onMount } from 'solid-js'
|
import { createSignal } from 'solid-js'
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
|
||||||
|
|
||||||
function SearchComponent() {
|
function SearchComponent() {
|
||||||
const [brain, setBrain] = createSignal(null)
|
|
||||||
const [isReady, setIsReady] = createSignal(false)
|
|
||||||
const [query, setQuery] = createSignal('')
|
const [query, setQuery] = createSignal('')
|
||||||
const [results, setResults] = createSignal([])
|
const [results, setResults] = createSignal([])
|
||||||
|
|
||||||
onMount(async () => {
|
|
||||||
const newBrain = new Brainy()
|
|
||||||
await newBrain.init()
|
|
||||||
setBrain(newBrain)
|
|
||||||
setIsReady(true)
|
|
||||||
})
|
|
||||||
|
|
||||||
const search = async () => {
|
const search = async () => {
|
||||||
if (!isReady() || !query()) return
|
if (!query()) return
|
||||||
const searchResults = await brain().find(query())
|
const res = await fetch('/api/search', {
|
||||||
setResults(searchResults)
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ query: query() })
|
||||||
|
})
|
||||||
|
setResults((await res.json()).results)
|
||||||
}
|
}
|
||||||
|
|
||||||
return (
|
return (
|
||||||
|
|
@ -450,57 +422,25 @@ function SearchComponent() {
|
||||||
|
|
||||||
## 📦 Bundler Configuration
|
## 📦 Bundler Configuration
|
||||||
|
|
||||||
### Vite (Recommended)
|
Brainy is a server-side dependency, so keep it out of client bundles. Import it only from server-only modules (`*.server.js`, API routes, server components, server actions). If your bundler ever tries to pull Brainy into a client bundle, that's a sign it's being imported from a client component — move the import to a server module.
|
||||||
|
|
||||||
|
For server builds, mark Brainy as external so the bundler doesn't inline it:
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// vite.config.js
|
// vite.config.js (SSR build)
|
||||||
import { defineConfig } from 'vite'
|
import { defineConfig } from 'vite'
|
||||||
|
|
||||||
export default defineConfig({
|
export default defineConfig({
|
||||||
define: {
|
ssr: {
|
||||||
global: 'globalThis'
|
external: ['@soulcraft/brainy']
|
||||||
},
|
|
||||||
optimizeDeps: {
|
|
||||||
include: ['@soulcraft/brainy']
|
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
```
|
```
|
||||||
|
|
||||||
### Webpack
|
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// webpack.config.js
|
// rollup.config.js (server bundle)
|
||||||
module.exports = {
|
|
||||||
resolve: {
|
|
||||||
fallback: {
|
|
||||||
"fs": false,
|
|
||||||
"path": require.resolve("path-browserify"),
|
|
||||||
"crypto": require.resolve("crypto-browserify")
|
|
||||||
}
|
|
||||||
},
|
|
||||||
plugins: [
|
|
||||||
new webpack.ProvidePlugin({
|
|
||||||
global: 'global'
|
|
||||||
})
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Rollup
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// rollup.config.js
|
|
||||||
import { nodeResolve } from '@rollup/plugin-node-resolve'
|
|
||||||
import commonjs from '@rollup/plugin-commonjs'
|
|
||||||
|
|
||||||
export default {
|
export default {
|
||||||
plugins: [
|
external: ['@soulcraft/brainy', 'node:fs', 'node:path', 'node:crypto']
|
||||||
nodeResolve({
|
|
||||||
browser: true,
|
|
||||||
preferBuiltins: false
|
|
||||||
}),
|
|
||||||
commonjs()
|
|
||||||
]
|
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -508,30 +448,24 @@ export default {
|
||||||
|
|
||||||
### Server-Side Rendering
|
### Server-Side Rendering
|
||||||
|
|
||||||
|
Instantiate Brainy on the server and feed its results into the rendered page. Never construct it in client code.
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// Check if running in browser
|
// Server-side data loading (framework loader / getServerSideProps / load fn)
|
||||||
if (typeof window !== 'undefined') {
|
import { getBrain } from './brain.server'
|
||||||
// Browser-only code
|
|
||||||
const brain = new Brainy({
|
|
||||||
storage: { type: 'opfs' }
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// Or use dynamic imports
|
export async function load({ url }) {
|
||||||
const initBrainForBrowser = async () => {
|
const brain = await getBrain()
|
||||||
if (typeof window === 'undefined') return null
|
const query = url.searchParams.get('q') ?? ''
|
||||||
|
const results = query ? await brain.find(query) : []
|
||||||
const { Brainy } = await import('@soulcraft/brainy')
|
return { results }
|
||||||
const brain = new Brainy()
|
|
||||||
await brain.init()
|
|
||||||
return brain
|
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
### Static Site Generation
|
### Static Site Generation
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// For build-time usage
|
// For build-time usage (runs in Node during the build)
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
import { Brainy } from '@soulcraft/brainy'
|
||||||
|
|
||||||
export async function generateStaticProps() {
|
export async function generateStaticProps() {
|
||||||
|
|
@ -552,67 +486,46 @@ export async function generateStaticProps() {
|
||||||
## 🔧 Framework-Specific Tips
|
## 🔧 Framework-Specific Tips
|
||||||
|
|
||||||
### React
|
### React
|
||||||
- Use `useCallback` for search functions to prevent re-renders
|
- Keep components client-side and call a Brainy-backed API route
|
||||||
- Consider `useMemo` for expensive brain operations
|
- Use `useCallback` for fetch handlers to prevent re-renders
|
||||||
- Implement cleanup in `useEffect` for proper memory management
|
- Debounce keystroke-driven searches before hitting the endpoint
|
||||||
|
|
||||||
### Vue
|
### Vue
|
||||||
- Use `shallowRef` for the brain instance (it's not reactive data)
|
- Components call an endpoint; the shared instance lives in a server module
|
||||||
- Consider Pinia for global brain state management
|
- Consider Pinia for caching results client-side
|
||||||
- Use `watchEffect` for reactive search queries
|
- Debounce reactive search queries
|
||||||
|
|
||||||
### Angular
|
### Angular
|
||||||
- Implement proper dependency injection with services
|
- Use `HttpClient` and RxJS to call the backend
|
||||||
- Use RxJS observables for reactive search
|
- Hold the shared Brainy instance in your Node backend, not the app
|
||||||
- Consider lazy loading brain in feature modules
|
- Consider lazy loading search features in feature modules
|
||||||
|
|
||||||
### Next.js
|
### Next.js
|
||||||
- Use dynamic imports for client-side only features
|
- Put Brainy in server-only modules (`*.server.js`), API routes, or server actions
|
||||||
- Consider API routes for server-side brain operations
|
- Reuse one shared instance across requests
|
||||||
- Implement proper error boundaries
|
- Implement proper error boundaries for failed fetches
|
||||||
|
|
||||||
## 🚨 Common Issues & Solutions
|
## 🚨 Common Issues & Solutions
|
||||||
|
|
||||||
### Issue: "crypto is not defined"
|
### Issue: "fs module not found" / "crypto is not defined" in the browser
|
||||||
**Solution**: Your framework should handle this automatically. If not:
|
**Cause**: Brainy was imported into a client bundle. Brainy 8.0 is a server-side library (Node 22+/Bun) and uses Node built-ins like `fs` and `crypto`.
|
||||||
```javascript
|
**Solution**: Import Brainy only from server code — server-only modules (`*.server.js`), API routes, server components, or server actions. From client components, call those endpoints instead.
|
||||||
// Add to your bundle config
|
|
||||||
define: {
|
|
||||||
global: 'globalThis'
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Issue: "fs module not found"
|
### Issue: Large client bundle size
|
||||||
**Solution**: This is expected in browsers. Use browser-compatible storage:
|
**Cause**: A client module is pulling in Brainy.
|
||||||
```javascript
|
**Solution**: Move the `import { Brainy } from '@soulcraft/brainy'` into a server-only module so it never reaches the browser bundle.
|
||||||
const brain = new Brainy({
|
|
||||||
storage: { type: 'opfs' } // Or 'memory' for development
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
### Issue: Large bundle size
|
|
||||||
**Solution**: Use dynamic imports for optional features:
|
|
||||||
```javascript
|
|
||||||
const brain = await import('@soulcraft/brainy').then(m => new m.Brainy())
|
|
||||||
```
|
|
||||||
|
|
||||||
### Issue: SSR hydration mismatch
|
### Issue: SSR hydration mismatch
|
||||||
**Solution**: Initialize brain only on client:
|
**Solution**: Run the search on the server (loader / server action / API route) and pass the results down as props, so server and client render the same markup.
|
||||||
```javascript
|
|
||||||
useEffect(() => {
|
|
||||||
// Browser-only initialization
|
|
||||||
initBrain()
|
|
||||||
}, [])
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🎯 Best Practices
|
## 🎯 Best Practices
|
||||||
|
|
||||||
1. **Initialize Once**: Create brain instance at app level, not component level
|
1. **Initialize Once**: Create one shared Brainy instance per server process, not per request
|
||||||
2. **Use Context**: Share brain instance across components with context/providers
|
2. **Server-Only**: Import Brainy only from server modules — never from client components
|
||||||
3. **Handle Loading**: Always show loading states during brain initialization
|
3. **Endpoint Boundary**: Expose search/add through API routes or server actions
|
||||||
4. **Error Boundaries**: Implement proper error handling for brain operations
|
4. **Handle Loading**: Show loading states in the client while the fetch is in flight
|
||||||
5. **Memory Management**: Clean up brain instances on unmount
|
5. **Error Handling**: Catch and surface failed endpoint calls gracefully
|
||||||
6. **Storage Strategy**: Choose appropriate storage for your deployment target
|
6. **Storage**: Use `filesystem` for persistence (the default on Node) or `memory` for ephemeral/tests
|
||||||
|
|
||||||
## 📚 Next Steps
|
## 📚 Next Steps
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1743,7 +1743,7 @@ For off-site backup, snapshot `rootDirectory` from your scheduler with `gsutil r
|
||||||
- HNSW Index: O(log n) search (1B entities = ~30 hops)
|
- HNSW Index: O(log n) search (1B entities = ~30 hops)
|
||||||
- Metadata Index: O(1) filtering
|
- Metadata Index: O(1) filtering
|
||||||
- Graph Adjacency: O(1) relationship lookups
|
- Graph Adjacency: O(1) relationship lookups
|
||||||
- Storage: Unlimited (cloud buckets)
|
- Storage: Bounded by the filesystem volume backing `rootDirectory`
|
||||||
|
|
||||||
### Optimization Tips
|
### Optimization Tips
|
||||||
|
|
||||||
|
|
@ -1887,7 +1887,7 @@ groupBy: 'type'
|
||||||
- ✅ O(1) metadata filtering
|
- ✅ O(1) metadata filtering
|
||||||
- ✅ O(1) relationship traversal
|
- ✅ O(1) relationship traversal
|
||||||
- ✅ Human-readable VFS structure
|
- ✅ Human-readable VFS structure
|
||||||
- ✅ Cloud storage support (GCS/S3/R2)
|
- ✅ Filesystem-backed persistence (snapshot/sync the directory for off-site backup)
|
||||||
- ✅ Billion-scale performance
|
- ✅ Billion-scale performance
|
||||||
- ✅ Zero mocks, production-ready!
|
- ✅ Zero mocks, production-ready!
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -45,7 +45,7 @@ console.log('Brainy ready.')
|
||||||
|
|
||||||
## Native Acceleration (Optional)
|
## Native Acceleration (Optional)
|
||||||
|
|
||||||
For production workloads, add Cortex for Rust-accelerated SIMD distance calculations, vector quantization, and native embeddings:
|
For production workloads, add Cortex for Rust-accelerated SIMD distance calculations and native embeddings:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm install @soulcraft/cortex
|
npm install @soulcraft/cortex
|
||||||
|
|
|
||||||
|
|
@ -108,4 +108,4 @@ await brain.find({ query: 'people who work at Anthropic' })
|
||||||
- [Triple Intelligence](/docs/concepts/triple-intelligence) — understand how the query engine works
|
- [Triple Intelligence](/docs/concepts/triple-intelligence) — understand how the query engine works
|
||||||
- [The Find System](/docs/guides/find-system) — advanced queries, operators, and graph traversal
|
- [The Find System](/docs/guides/find-system) — advanced queries, operators, and graph traversal
|
||||||
- [API Reference](/docs/api/reference) — complete method documentation
|
- [API Reference](/docs/api/reference) — complete method documentation
|
||||||
- [Storage Adapters](/docs/guides/storage-adapters) — S3, GCS, Azure, filesystem, OPFS
|
- [Storage Adapters](/docs/guides/storage-adapters) — filesystem, memory
|
||||||
|
|
|
||||||
|
|
@ -271,7 +271,7 @@ Progress is reported after each batch (batch size is determined by the storage a
|
||||||
|
|
||||||
## Storage Backend Compatibility
|
## Storage Backend Compatibility
|
||||||
|
|
||||||
Migrations work identically across all storage backends (Memory, FileSystem, S3, R2, GCS, OPFS). The system uses `BaseStorage` methods (`getNouns`, `saveNounMetadata`, `getVerbs`, `saveVerbMetadata`) which are implemented by every adapter.
|
Migrations work identically across all storage backends (Memory, FileSystem). The system uses `BaseStorage` methods (`getNouns`, `saveNounMetadata`, `getVerbs`, `saveVerbMetadata`) which are implemented by every adapter.
|
||||||
|
|
||||||
Batch size and rate limiting are automatically configured per adapter — no tuning required.
|
Batch size and rate limiting are automatically configured per adapter — no tuning required.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -2,6 +2,8 @@
|
||||||
|
|
||||||
Complete guide to integrating Brainy with Vue.js applications, covering Vue 3, Nuxt.js, Composition API, Options API, and advanced patterns.
|
Complete guide to integrating Brainy with Vue.js applications, covering Vue 3, Nuxt.js, Composition API, Options API, and advanced patterns.
|
||||||
|
|
||||||
|
> **Runtime**: Brainy 8.0 runs on Node.js 22+ and Bun, server-side only — it is not a browser library. In Vue apps, Brainy lives behind an HTTP endpoint (a Nuxt server route, or any Node/Bun backend), and your components call that endpoint. The composables, plugin, and store below are wrappers around `fetch`; the single Brainy instance is created once on the server (see [Nuxt Server Route](#nuxt-server-route)).
|
||||||
|
|
||||||
## 🚀 Quick Start
|
## 🚀 Quick Start
|
||||||
|
|
||||||
### Installation
|
### Installation
|
||||||
|
|
@ -15,34 +17,47 @@ npm install @soulcraft/brainy
|
||||||
|
|
||||||
### Basic Setup
|
### Basic Setup
|
||||||
|
|
||||||
|
The composable talks to a Brainy-backed endpoint (see [Nuxt Server Route](#nuxt-server-route)). It is always "ready" because there is no client-side initialization — the server owns the Brainy instance.
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// src/composables/useBrainy.js
|
// src/composables/useBrainy.js
|
||||||
import { ref, onMounted } from 'vue'
|
import { ref } from 'vue'
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
|
||||||
|
|
||||||
const brain = ref(null)
|
export function useBrainy(endpoint = '/api/brain') {
|
||||||
const isReady = ref(false)
|
const error = ref(null)
|
||||||
const error = ref(null)
|
|
||||||
|
|
||||||
export function useBrainy() {
|
const search = async (query, options = {}) => {
|
||||||
onMounted(async () => {
|
error.value = null
|
||||||
try {
|
try {
|
||||||
brain.value = new Brainy({
|
const res = await fetch(`${endpoint}/search`, {
|
||||||
storage: { type: 'opfs' } // Browser storage
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ query, options })
|
||||||
})
|
})
|
||||||
await brain.value.init()
|
if (!res.ok) throw new Error(`Search failed: ${res.status}`)
|
||||||
isReady.value = true
|
return (await res.json()).results
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
error.value = err.message
|
error.value = err.message
|
||||||
console.error('Brainy initialization failed:', err)
|
console.error('Brainy search failed:', err)
|
||||||
|
return []
|
||||||
|
}
|
||||||
}
|
}
|
||||||
})
|
|
||||||
|
|
||||||
return {
|
const add = async (data, type, metadata) => {
|
||||||
brain: readonly(brain),
|
const res = await fetch(`${endpoint}/add`, {
|
||||||
isReady: readonly(isReady),
|
method: 'POST',
|
||||||
error: readonly(error)
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ data, type, metadata })
|
||||||
|
})
|
||||||
|
return (await res.json()).id
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const stats = async () => {
|
||||||
|
const res = await fetch(`${endpoint}/stats`)
|
||||||
|
return await res.json()
|
||||||
|
}
|
||||||
|
|
||||||
|
return { search, add, stats, error: readonly(error) }
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -54,17 +69,11 @@ export function useBrainy() {
|
||||||
<!-- src/components/Search.vue -->
|
<!-- src/components/Search.vue -->
|
||||||
<template>
|
<template>
|
||||||
<div class="search-container">
|
<div class="search-container">
|
||||||
<div v-if="!isReady" class="loading">
|
|
||||||
<div class="spinner"></div>
|
|
||||||
<span>Initializing AI...</span>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div v-else>
|
|
||||||
<div class="search-input">
|
<div class="search-input">
|
||||||
<input
|
<input
|
||||||
v-model="query"
|
v-model="query"
|
||||||
@input="handleSearch"
|
@input="handleSearch"
|
||||||
placeholder="Search with AI..."
|
placeholder="Search..."
|
||||||
class="input"
|
class="input"
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
|
|
@ -93,7 +102,6 @@ export function useBrainy() {
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
|
|
@ -101,22 +109,21 @@ import { ref, watch } from 'vue'
|
||||||
import { useBrainy } from '../composables/useBrainy'
|
import { useBrainy } from '../composables/useBrainy'
|
||||||
import { debounce } from '../utils/debounce'
|
import { debounce } from '../utils/debounce'
|
||||||
|
|
||||||
const { brain, isReady } = useBrainy()
|
const { search: searchBrain } = useBrainy()
|
||||||
|
|
||||||
const query = ref('')
|
const query = ref('')
|
||||||
const results = ref([])
|
const results = ref([])
|
||||||
const loading = ref(false)
|
const loading = ref(false)
|
||||||
|
|
||||||
const search = async (searchQuery) => {
|
const search = async (searchQuery) => {
|
||||||
if (!isReady.value || !searchQuery.trim()) {
|
if (!searchQuery.trim()) {
|
||||||
results.value = []
|
results.value = []
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
loading.value = true
|
loading.value = true
|
||||||
try {
|
try {
|
||||||
const searchResults = await brain.value.find(searchQuery)
|
results.value = await searchBrain(searchQuery)
|
||||||
results.value = searchResults
|
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
console.error('Search error:', error)
|
console.error('Search error:', error)
|
||||||
results.value = []
|
results.value = []
|
||||||
|
|
@ -228,11 +235,7 @@ watch(query, (newQuery) => {
|
||||||
<div class="data-manager">
|
<div class="data-manager">
|
||||||
<h2>Data Management</h2>
|
<h2>Data Management</h2>
|
||||||
|
|
||||||
<div v-if="!isReady" class="loading">
|
<div>
|
||||||
Initializing...
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div v-else>
|
|
||||||
<!-- Add Data Form -->
|
<!-- Add Data Form -->
|
||||||
<form @submit.prevent="addData" class="add-form">
|
<form @submit.prevent="addData" class="add-form">
|
||||||
<h3>Add New Data</h3>
|
<h3>Add New Data</h3>
|
||||||
|
|
@ -289,7 +292,7 @@ watch(query, (newQuery) => {
|
||||||
import { ref, reactive, onMounted } from 'vue'
|
import { ref, reactive, onMounted } from 'vue'
|
||||||
import { useBrainy } from '../composables/useBrainy'
|
import { useBrainy } from '../composables/useBrainy'
|
||||||
|
|
||||||
const { brain, isReady } = useBrainy()
|
const { add, stats: fetchStats } = useBrainy()
|
||||||
|
|
||||||
const newItem = reactive({
|
const newItem = reactive({
|
||||||
data: '',
|
data: '',
|
||||||
|
|
@ -299,18 +302,7 @@ const newItem = reactive({
|
||||||
|
|
||||||
const stats = ref(null)
|
const stats = ref(null)
|
||||||
|
|
||||||
onMounted(async () => {
|
onMounted(loadStats)
|
||||||
if (isReady.value) {
|
|
||||||
await loadStats()
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
// Watch for brain readiness
|
|
||||||
watch(isReady, async (ready) => {
|
|
||||||
if (ready) {
|
|
||||||
await loadStats()
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
const addData = async () => {
|
const addData = async () => {
|
||||||
try {
|
try {
|
||||||
|
|
@ -319,11 +311,7 @@ const addData = async () => {
|
||||||
tags: newItem.tags.split(',').map(tag => tag.trim()).filter(Boolean)
|
tags: newItem.tags.split(',').map(tag => tag.trim()).filter(Boolean)
|
||||||
}
|
}
|
||||||
|
|
||||||
await brain.value.add({
|
await add(newItem.data, newItem.type, metadata)
|
||||||
data: newItem.data,
|
|
||||||
type: newItem.type,
|
|
||||||
metadata
|
|
||||||
})
|
|
||||||
|
|
||||||
// Reset form
|
// Reset form
|
||||||
newItem.data = ''
|
newItem.data = ''
|
||||||
|
|
@ -339,7 +327,7 @@ const addData = async () => {
|
||||||
|
|
||||||
const loadStats = async () => {
|
const loadStats = async () => {
|
||||||
try {
|
try {
|
||||||
stats.value = await brain.value.stats()
|
stats.value = await fetchStats()
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
console.error('Failed to load stats:', error)
|
console.error('Failed to load stats:', error)
|
||||||
}
|
}
|
||||||
|
|
@ -435,11 +423,6 @@ button:disabled {
|
||||||
<!-- src/components/SearchOptions.vue -->
|
<!-- src/components/SearchOptions.vue -->
|
||||||
<template>
|
<template>
|
||||||
<div class="search-container">
|
<div class="search-container">
|
||||||
<div v-if="!isReady" class="loading">
|
|
||||||
Initializing AI...
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div v-else>
|
|
||||||
<input
|
<input
|
||||||
v-model="query"
|
v-model="query"
|
||||||
@input="handleSearch"
|
@input="handleSearch"
|
||||||
|
|
@ -460,48 +443,35 @@ button:disabled {
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script>
|
<script>
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
|
||||||
import { debounce } from '../utils/debounce'
|
import { debounce } from '../utils/debounce'
|
||||||
|
|
||||||
export default {
|
export default {
|
||||||
name: 'SearchOptions',
|
name: 'SearchOptions',
|
||||||
data() {
|
data() {
|
||||||
return {
|
return {
|
||||||
brain: null,
|
|
||||||
isReady: false,
|
|
||||||
query: '',
|
query: '',
|
||||||
results: [],
|
results: [],
|
||||||
loading: false
|
loading: false
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
async mounted() {
|
|
||||||
await this.initBrain()
|
|
||||||
},
|
|
||||||
methods: {
|
methods: {
|
||||||
async initBrain() {
|
|
||||||
try {
|
|
||||||
this.brain = new Brainy({
|
|
||||||
storage: { type: 'opfs' }
|
|
||||||
})
|
|
||||||
await this.brain.init()
|
|
||||||
this.isReady = true
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Brain initialization failed:', error)
|
|
||||||
}
|
|
||||||
},
|
|
||||||
async search(query) {
|
async search(query) {
|
||||||
if (!this.isReady || !query.trim()) {
|
if (!query.trim()) {
|
||||||
this.results = []
|
this.results = []
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
this.loading = true
|
this.loading = true
|
||||||
try {
|
try {
|
||||||
this.results = await this.brain.find(query)
|
const res = await fetch('/api/brain/search', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ query })
|
||||||
|
})
|
||||||
|
this.results = (await res.json()).results
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
console.error('Search error:', error)
|
console.error('Search error:', error)
|
||||||
this.results = []
|
this.results = []
|
||||||
|
|
@ -520,10 +490,6 @@ export default {
|
||||||
this.loading = false
|
this.loading = false
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
},
|
|
||||||
beforeUnmount() {
|
|
||||||
// Cleanup if needed
|
|
||||||
this.brain = null
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
</script>
|
</script>
|
||||||
|
|
@ -533,31 +499,22 @@ export default {
|
||||||
|
|
||||||
### Global Brainy Plugin
|
### Global Brainy Plugin
|
||||||
|
|
||||||
|
The plugin exposes a global `$searchBrain` helper that calls your Brainy-backed endpoint. Brainy itself runs on the server (see [Nuxt Server Route](#nuxt-server-route)).
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// src/plugins/brainy.js
|
// src/plugins/brainy.js
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
|
||||||
|
|
||||||
export default {
|
export default {
|
||||||
install(app, options = {}) {
|
install(app, options = {}) {
|
||||||
const defaultOptions = {
|
const endpoint = options.endpoint ?? '/api/brain'
|
||||||
storage: { type: 'opfs' },
|
|
||||||
...options
|
|
||||||
}
|
|
||||||
|
|
||||||
const brain = new Brainy(defaultOptions)
|
// Add global search method (calls the server endpoint)
|
||||||
|
app.config.globalProperties.$searchBrain = async (query, searchOptions) => {
|
||||||
// Make brain available globally
|
const res = await fetch(`${endpoint}/search`, {
|
||||||
app.config.globalProperties.$brain = brain
|
method: 'POST',
|
||||||
app.provide('brain', brain)
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ query, options: searchOptions })
|
||||||
// Auto-initialize
|
|
||||||
brain.init().catch(error => {
|
|
||||||
console.error('Brainy plugin initialization failed:', error)
|
|
||||||
})
|
})
|
||||||
|
return (await res.json()).results
|
||||||
// Add global method
|
|
||||||
app.config.globalProperties.$searchBrain = async (query, options) => {
|
|
||||||
return await brain.find(query, options)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -574,7 +531,7 @@ import BrainyPlugin from './plugins/brainy'
|
||||||
const app = createApp(App)
|
const app = createApp(App)
|
||||||
|
|
||||||
app.use(BrainyPlugin, {
|
app.use(BrainyPlugin, {
|
||||||
storage: { type: 'opfs' }
|
endpoint: '/api/brain'
|
||||||
})
|
})
|
||||||
|
|
||||||
app.mount('#app')
|
app.mount('#app')
|
||||||
|
|
@ -611,24 +568,58 @@ export default {
|
||||||
|
|
||||||
## 🏰 Nuxt.js Integration
|
## 🏰 Nuxt.js Integration
|
||||||
|
|
||||||
### Nuxt Plugin
|
Nuxt's server engine (Nitro) is the natural home for Brainy: it runs on Node/Bun, so the Brainy instance lives in `server/`, and pages/components call its routes.
|
||||||
|
|
||||||
|
### Nuxt Server Route
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// plugins/brainy.client.js
|
// server/utils/brain.js (server-only — Nitro never bundles this into the client)
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
import { Brainy } from '@soulcraft/brainy'
|
||||||
|
|
||||||
export default defineNuxtPlugin(async () => {
|
let brainPromise
|
||||||
const brain = new Brainy({
|
|
||||||
storage: { type: 'opfs' }
|
|
||||||
})
|
|
||||||
|
|
||||||
|
export function getBrain() {
|
||||||
|
if (!brainPromise) {
|
||||||
|
brainPromise = (async () => {
|
||||||
|
// new Brainy() auto-detects filesystem persistence on Node
|
||||||
|
const brain = new Brainy()
|
||||||
await brain.init()
|
await brain.init()
|
||||||
|
return brain
|
||||||
|
})()
|
||||||
|
}
|
||||||
|
return brainPromise
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
return {
|
```javascript
|
||||||
provide: {
|
// server/api/brain/search.post.js
|
||||||
brain
|
import { getBrain } from '../../utils/brain'
|
||||||
}
|
|
||||||
}
|
export default defineEventHandler(async (event) => {
|
||||||
|
const { query, options } = await readBody(event)
|
||||||
|
const brain = await getBrain()
|
||||||
|
return { results: await brain.find(query, options) }
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// server/api/brain/add.post.js
|
||||||
|
import { getBrain } from '../../utils/brain'
|
||||||
|
|
||||||
|
export default defineEventHandler(async (event) => {
|
||||||
|
const { data, type, metadata } = await readBody(event)
|
||||||
|
const brain = await getBrain()
|
||||||
|
return { id: await brain.add({ data, type, metadata }) }
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// server/api/brain/stats.get.js
|
||||||
|
import { getBrain } from '../../utils/brain'
|
||||||
|
|
||||||
|
export default defineEventHandler(async () => {
|
||||||
|
const brain = await getBrain()
|
||||||
|
return await brain.stats()
|
||||||
})
|
})
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -637,26 +628,25 @@ export default defineNuxtPlugin(async () => {
|
||||||
```javascript
|
```javascript
|
||||||
// composables/useBrainy.js
|
// composables/useBrainy.js
|
||||||
export const useBrainy = () => {
|
export const useBrainy = () => {
|
||||||
const { $brain } = useNuxtApp()
|
|
||||||
|
|
||||||
const search = async (query, options = {}) => {
|
const search = async (query, options = {}) => {
|
||||||
return await $brain.find(query, options)
|
return (await $fetch('/api/brain/search', {
|
||||||
|
method: 'POST',
|
||||||
|
body: { query, options }
|
||||||
|
})).results
|
||||||
}
|
}
|
||||||
|
|
||||||
const add = async (data, type, metadata) => {
|
const add = async (data, type, metadata) => {
|
||||||
return await $brain.add({ data, type, metadata })
|
return (await $fetch('/api/brain/add', {
|
||||||
|
method: 'POST',
|
||||||
|
body: { data, type, metadata }
|
||||||
|
})).id
|
||||||
}
|
}
|
||||||
|
|
||||||
const stats = async () => {
|
const stats = async () => {
|
||||||
return await $brain.stats()
|
return await $fetch('/api/brain/stats')
|
||||||
}
|
}
|
||||||
|
|
||||||
return {
|
return { search, add, stats }
|
||||||
brain: $brain,
|
|
||||||
search,
|
|
||||||
add,
|
|
||||||
stats
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -666,13 +656,8 @@ export const useBrainy = () => {
|
||||||
<!-- pages/search.vue -->
|
<!-- pages/search.vue -->
|
||||||
<template>
|
<template>
|
||||||
<div>
|
<div>
|
||||||
<h1>AI Search</h1>
|
<h1>Search</h1>
|
||||||
|
|
||||||
<div v-if="pending" class="loading">
|
|
||||||
Initializing AI...
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div v-else>
|
|
||||||
<input
|
<input
|
||||||
v-model="query"
|
v-model="query"
|
||||||
@input="handleSearch"
|
@input="handleSearch"
|
||||||
|
|
@ -688,7 +673,6 @@ export const useBrainy = () => {
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
|
|
@ -697,12 +681,6 @@ const { search } = useBrainy()
|
||||||
const query = ref('')
|
const query = ref('')
|
||||||
const results = ref([])
|
const results = ref([])
|
||||||
const searching = ref(false)
|
const searching = ref(false)
|
||||||
const pending = ref(true)
|
|
||||||
|
|
||||||
// Initialize
|
|
||||||
onMounted(() => {
|
|
||||||
pending.value = false
|
|
||||||
})
|
|
||||||
|
|
||||||
const handleSearch = debounce(async () => {
|
const handleSearch = debounce(async () => {
|
||||||
if (!query.value.trim()) {
|
if (!query.value.trim()) {
|
||||||
|
|
@ -722,82 +700,58 @@ const handleSearch = debounce(async () => {
|
||||||
</script>
|
</script>
|
||||||
```
|
```
|
||||||
|
|
||||||
### Nuxt Configuration
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// nuxt.config.ts
|
|
||||||
export default defineNuxtConfig({
|
|
||||||
ssr: false, // Disable SSR for browser-only features
|
|
||||||
|
|
||||||
// Or use client-side hydration
|
|
||||||
nitro: {
|
|
||||||
experimental: {
|
|
||||||
wasm: true
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
vite: {
|
|
||||||
define: {
|
|
||||||
global: 'globalThis'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🛠️ Advanced Patterns
|
## 🛠️ Advanced Patterns
|
||||||
|
|
||||||
### Global State Management with Pinia
|
### Global State Management with Pinia
|
||||||
|
|
||||||
|
The store caches results and stats client-side; all Brainy work happens behind the server endpoints.
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// stores/brainy.js
|
// stores/brainy.js
|
||||||
import { defineStore } from 'pinia'
|
import { defineStore } from 'pinia'
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
|
||||||
|
|
||||||
export const useBrainyStore = defineStore('brainy', () => {
|
export const useBrainyStore = defineStore('brainy', () => {
|
||||||
const brain = ref(null)
|
|
||||||
const isReady = ref(false)
|
|
||||||
const error = ref(null)
|
const error = ref(null)
|
||||||
const stats = ref(null)
|
const stats = ref(null)
|
||||||
|
|
||||||
const init = async (options = {}) => {
|
const search = async (query, options = {}) => {
|
||||||
|
error.value = null
|
||||||
try {
|
try {
|
||||||
brain.value = new Brainy(options)
|
const res = await fetch('/api/brain/search', {
|
||||||
await brain.value.init()
|
method: 'POST',
|
||||||
isReady.value = true
|
headers: { 'Content-Type': 'application/json' },
|
||||||
await loadStats()
|
body: JSON.stringify({ query, options })
|
||||||
|
})
|
||||||
|
return (await res.json()).results
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
error.value = err.message
|
error.value = err.message
|
||||||
console.error('Brainy initialization failed:', err)
|
throw err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const search = async (query, options = {}) => {
|
|
||||||
if (!isReady.value) throw new Error('Brain not ready')
|
|
||||||
return await brain.value.find(query, options)
|
|
||||||
}
|
|
||||||
|
|
||||||
const add = async (data, type, metadata) => {
|
const add = async (data, type, metadata) => {
|
||||||
if (!isReady.value) throw new Error('Brain not ready')
|
const res = await fetch('/api/brain/add', {
|
||||||
const id = await brain.value.add({ data, type, metadata })
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ data, type, metadata })
|
||||||
|
})
|
||||||
|
const { id } = await res.json()
|
||||||
await loadStats() // Refresh stats
|
await loadStats() // Refresh stats
|
||||||
return id
|
return id
|
||||||
}
|
}
|
||||||
|
|
||||||
const loadStats = async () => {
|
const loadStats = async () => {
|
||||||
if (!isReady.value) return
|
|
||||||
try {
|
try {
|
||||||
stats.value = await brain.value.stats()
|
const res = await fetch('/api/brain/stats')
|
||||||
|
stats.value = await res.json()
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
console.error('Failed to load stats:', err)
|
console.error('Failed to load stats:', err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
brain: readonly(brain),
|
|
||||||
isReady: readonly(isReady),
|
|
||||||
error: readonly(error),
|
error: readonly(error),
|
||||||
stats: readonly(stats),
|
stats: readonly(stats),
|
||||||
init,
|
|
||||||
search,
|
search,
|
||||||
add,
|
add,
|
||||||
loadStats
|
loadStats
|
||||||
|
|
@ -867,7 +821,7 @@ const showResults = ref(false)
|
||||||
const selectedIndex = ref(-1)
|
const selectedIndex = ref(-1)
|
||||||
|
|
||||||
const search = async (searchQuery) => {
|
const search = async (searchQuery) => {
|
||||||
if (!brainyStore.isReady || !searchQuery.trim()) {
|
if (!searchQuery.trim()) {
|
||||||
results.value = []
|
results.value = []
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
@ -1166,21 +1120,23 @@ onMounted(() => {
|
||||||
|
|
||||||
### Component Testing with Vitest
|
### Component Testing with Vitest
|
||||||
|
|
||||||
|
The component talks to the server over `fetch`, so mock the endpoint, not Brainy itself.
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// tests/components/Search.test.js
|
// tests/components/Search.test.js
|
||||||
import { mount } from '@vue/test-utils'
|
import { mount } from '@vue/test-utils'
|
||||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||||
import Search from '../src/components/Search.vue'
|
import Search from '../src/components/Search.vue'
|
||||||
|
|
||||||
// Mock Brainy
|
// Mock the server endpoint
|
||||||
vi.mock('@soulcraft/brainy', () => ({
|
beforeEach(() => {
|
||||||
Brainy: vi.fn().mockImplementation(() => ({
|
global.fetch = vi.fn().mockResolvedValue({
|
||||||
init: vi.fn().mockResolvedValue(undefined),
|
ok: true,
|
||||||
find: vi.fn().mockResolvedValue([
|
json: async () => ({
|
||||||
{ id: '1', data: 'Test result', score: 0.9 }
|
results: [{ id: '1', data: 'Test result', score: 0.9 }]
|
||||||
])
|
})
|
||||||
}))
|
})
|
||||||
}))
|
})
|
||||||
|
|
||||||
describe('Search Component', () => {
|
describe('Search Component', () => {
|
||||||
let wrapper
|
let wrapper
|
||||||
|
|
@ -1193,14 +1149,7 @@ describe('Search Component', () => {
|
||||||
expect(wrapper.find('input').exists()).toBe(true)
|
expect(wrapper.find('input').exists()).toBe(true)
|
||||||
})
|
})
|
||||||
|
|
||||||
it('shows loading state initially', () => {
|
|
||||||
expect(wrapper.text()).toContain('Initializing AI')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('performs search when input changes', async () => {
|
it('performs search when input changes', async () => {
|
||||||
// Wait for brain to initialize
|
|
||||||
await wrapper.vm.$nextTick()
|
|
||||||
|
|
||||||
const input = wrapper.find('input')
|
const input = wrapper.find('input')
|
||||||
await input.setValue('test query')
|
await input.setValue('test query')
|
||||||
await input.trigger('input')
|
await input.trigger('input')
|
||||||
|
|
@ -1208,6 +1157,7 @@ describe('Search Component', () => {
|
||||||
// Wait for debounced search
|
// Wait for debounced search
|
||||||
await new Promise(resolve => setTimeout(resolve, 350))
|
await new Promise(resolve => setTimeout(resolve, 350))
|
||||||
|
|
||||||
|
expect(global.fetch).toHaveBeenCalled()
|
||||||
expect(wrapper.text()).toContain('Test result')
|
expect(wrapper.text()).toContain('Test result')
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
@ -1241,6 +1191,8 @@ test('search functionality works', async ({ page }) => {
|
||||||
|
|
||||||
### Bundle Optimization
|
### Bundle Optimization
|
||||||
|
|
||||||
|
Keep Brainy out of the client bundle — it belongs to the server build only. Mark it external for SSR so the bundler resolves it at runtime instead of inlining it.
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// vite.config.js
|
// vite.config.js
|
||||||
import { defineConfig } from 'vite'
|
import { defineConfig } from 'vite'
|
||||||
|
|
@ -1248,63 +1200,46 @@ import vue from '@vitejs/plugin-vue'
|
||||||
|
|
||||||
export default defineConfig({
|
export default defineConfig({
|
||||||
plugins: [vue()],
|
plugins: [vue()],
|
||||||
define: {
|
ssr: {
|
||||||
global: 'globalThis'
|
external: ['@soulcraft/brainy']
|
||||||
},
|
|
||||||
optimizeDeps: {
|
|
||||||
include: ['@soulcraft/brainy']
|
|
||||||
},
|
|
||||||
build: {
|
|
||||||
rollupOptions: {
|
|
||||||
output: {
|
|
||||||
manualChunks: {
|
|
||||||
'brainy': ['@soulcraft/brainy']
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
```
|
```
|
||||||
|
|
||||||
### Error Handling
|
### Error Handling
|
||||||
|
|
||||||
|
Wrap the endpoint call with retry/backoff so transient network failures don't surface to the user.
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// src/composables/useBrainyWithErrorHandling.js
|
// src/composables/useBrainyWithErrorHandling.js
|
||||||
import { ref, onMounted } from 'vue'
|
import { ref } from 'vue'
|
||||||
import { Brainy } from '@soulcraft/brainy'
|
|
||||||
|
|
||||||
export function useBrainyWithErrorHandling() {
|
export function useBrainyWithErrorHandling(endpoint = '/api/brain') {
|
||||||
const brain = ref(null)
|
|
||||||
const isReady = ref(false)
|
|
||||||
const error = ref(null)
|
const error = ref(null)
|
||||||
const retryCount = ref(0)
|
|
||||||
|
|
||||||
const init = async () => {
|
const search = async (query, options = {}, maxRetries = 3) => {
|
||||||
try {
|
|
||||||
brain.value = new Brainy()
|
|
||||||
await brain.value.init()
|
|
||||||
isReady.value = true
|
|
||||||
error.value = null
|
error.value = null
|
||||||
retryCount.value = 0
|
for (let attempt = 0; attempt <= maxRetries; attempt++) {
|
||||||
|
try {
|
||||||
|
const res = await fetch(`${endpoint}/search`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ query, options })
|
||||||
|
})
|
||||||
|
if (!res.ok) throw new Error(`Search failed: ${res.status}`)
|
||||||
|
return (await res.json()).results
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
error.value = err.message
|
error.value = err.message
|
||||||
console.error('Brainy initialization failed:', err)
|
if (attempt === maxRetries) throw err
|
||||||
|
// Exponential backoff before retrying
|
||||||
// Retry logic
|
await new Promise(resolve => setTimeout(resolve, 2000 * (attempt + 1)))
|
||||||
if (retryCount.value < 3) {
|
|
||||||
retryCount.value++
|
|
||||||
setTimeout(init, 2000 * retryCount.value)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
onMounted(init)
|
|
||||||
|
|
||||||
return {
|
return {
|
||||||
brain: readonly(brain),
|
|
||||||
isReady: readonly(isReady),
|
|
||||||
error: readonly(error),
|
error: readonly(error),
|
||||||
retry: init
|
search
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
@ -1315,6 +1250,13 @@ Here's a complete Vue 3 application structure:
|
||||||
|
|
||||||
```
|
```
|
||||||
my-brainy-vue-app/
|
my-brainy-vue-app/
|
||||||
|
├── server/ # Server-side (Node/Bun) — hosts Brainy
|
||||||
|
│ ├── utils/
|
||||||
|
│ │ └── brain.js # Shared Brainy instance (getBrain)
|
||||||
|
│ └── api/brain/
|
||||||
|
│ ├── search.post.js
|
||||||
|
│ ├── add.post.js
|
||||||
|
│ └── stats.get.js
|
||||||
├── src/
|
├── src/
|
||||||
│ ├── components/
|
│ ├── components/
|
||||||
│ │ ├── Search.vue
|
│ │ ├── Search.vue
|
||||||
|
|
@ -1336,7 +1278,7 @@ my-brainy-vue-app/
|
||||||
└── package.json
|
└── package.json
|
||||||
```
|
```
|
||||||
|
|
||||||
This provides a complete, production-ready Vue.js application with comprehensive Brainy integration.
|
This provides a complete, production-ready Vue.js application: Brainy runs on the server, and the client talks to it over HTTP.
|
||||||
|
|
||||||
## 📚 Next Steps
|
## 📚 Next Steps
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -386,8 +386,7 @@ npm install @soulcraft/brainy
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
- Node.js 18+ (for server/desktop)
|
- Node.js 22+ or Bun (server-only)
|
||||||
- Modern browser (for web apps)
|
|
||||||
- Brainy 3.0+
|
- Brainy 3.0+
|
||||||
|
|
||||||
## API Reference
|
## API Reference
|
||||||
|
|
|
||||||
|
|
@ -172,19 +172,6 @@ interface MetadataIndexOptionalHooks {
|
||||||
close?: () => Promise<void> | void
|
close?: () => Promise<void> | void
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Optional progressive-initialization surface implemented by cloud storage
|
|
||||||
* adapters (plugin-provided in 8.0 — the built-in filesystem/memory adapters
|
|
||||||
* complete their full init synchronously inside `init()` and don't expose
|
|
||||||
* these). Feature-detected with `typeof x === 'function'` before invoking.
|
|
||||||
*/
|
|
||||||
interface StorageBackgroundInitHooks {
|
|
||||||
/** Whether all background init tasks (bucket validation, count sync) finished. */
|
|
||||||
isBackgroundInitComplete?: () => boolean
|
|
||||||
/** Resolves when all background init tasks complete. */
|
|
||||||
awaitBackgroundInit?: () => Promise<void>
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Brainy's internal resolved configuration. `normalizeConfig()` defaults every
|
* Brainy's internal resolved configuration. `normalizeConfig()` defaults every
|
||||||
* field at construction except the genuinely-optional ones listed in the
|
* field at construction except the genuinely-optional ones listed in the
|
||||||
|
|
@ -1075,13 +1062,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
* This Promise is created in the constructor and resolves when init() completes.
|
* This Promise is created in the constructor and resolves when init() completes.
|
||||||
* It can be awaited multiple times safely - the result is cached.
|
* It can be awaited multiple times safely - the result is cached.
|
||||||
*
|
*
|
||||||
* This enables reliable readiness detection for consumers,
|
* This enables reliable readiness detection for consumers — await it
|
||||||
* especially in cloud environments where progressive initialization means
|
* before issuing queries when init() was fired without awaiting.
|
||||||
* init() returns quickly but background tasks may still be running.
|
|
||||||
*
|
*
|
||||||
* @example Waiting for readiness before API calls
|
* @example Waiting for readiness before API calls
|
||||||
* ```typescript
|
* ```typescript
|
||||||
* const brain = new Brainy({ storage: { type: 'gcs', ... } })
|
* const brain = new Brainy()
|
||||||
* brain.init() // Fire and forget
|
* brain.init() // Fire and forget
|
||||||
*
|
*
|
||||||
* // Elsewhere in your code (e.g., API handler)
|
* // Elsewhere in your code (e.g., API handler)
|
||||||
|
|
@ -1115,82 +1101,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
return this._readyPromise
|
return this._readyPromise
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Check if Brainy is fully initialized including all background tasks
|
|
||||||
*
|
|
||||||
* This checks both:
|
|
||||||
* 1. Basic initialization complete (init() returned)
|
|
||||||
* 2. Storage background tasks complete (bucket validation, count sync)
|
|
||||||
*
|
|
||||||
* Useful for determining if all lazy/progressive initialization is done.
|
|
||||||
*
|
|
||||||
* @returns true if all initialization including background tasks is complete
|
|
||||||
*
|
|
||||||
* @example Health check with background status
|
|
||||||
* ```typescript
|
|
||||||
* app.get('/health', (req, res) => {
|
|
||||||
* res.json({
|
|
||||||
* ready: brain.isInitialized,
|
|
||||||
* fullyInitialized: brain.isFullyInitialized(),
|
|
||||||
* status: brain.isFullyInitialized() ? 'ready' : 'warming'
|
|
||||||
* })
|
|
||||||
* })
|
|
||||||
* ```
|
|
||||||
*/
|
|
||||||
isFullyInitialized(): boolean {
|
|
||||||
if (!this.initialized) return false
|
|
||||||
|
|
||||||
// Check if storage has background init methods (cloud storage adapters)
|
|
||||||
const storage = this.storage as BaseStorage & StorageBackgroundInitHooks
|
|
||||||
if (typeof storage?.isBackgroundInitComplete === 'function') {
|
|
||||||
return storage.isBackgroundInitComplete()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Non-cloud storage adapters are fully initialized after init()
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Wait for all background initialization tasks to complete
|
|
||||||
*
|
|
||||||
* For cloud storage adapters with progressive initialization,
|
|
||||||
* this waits for:
|
|
||||||
* - Bucket/container validation
|
|
||||||
* - Count synchronization
|
|
||||||
* - Any other background tasks
|
|
||||||
*
|
|
||||||
* For non-cloud storage, this resolves immediately.
|
|
||||||
*
|
|
||||||
* **Use Case**: Call this when you need guaranteed consistency, such as:
|
|
||||||
* - Before running batch operations
|
|
||||||
* - Before reporting full system health
|
|
||||||
* - When transitioning from "initializing" to "ready" status
|
|
||||||
*
|
|
||||||
* @returns Promise that resolves when all background tasks complete
|
|
||||||
*
|
|
||||||
* @example Ensuring full initialization
|
|
||||||
* ```typescript
|
|
||||||
* const brain = new Brainy({ storage: { type: 'gcs', ... } })
|
|
||||||
* await brain.init() // Fast return in cloud (<200ms)
|
|
||||||
*
|
|
||||||
* // Optional: wait for background tasks if needed
|
|
||||||
* await brain.awaitBackgroundInit()
|
|
||||||
* console.log('All background tasks complete')
|
|
||||||
* ```
|
|
||||||
*/
|
|
||||||
async awaitBackgroundInit(): Promise<void> {
|
|
||||||
// Must be initialized first
|
|
||||||
await this.ready
|
|
||||||
|
|
||||||
// Check if storage has background init methods (cloud storage adapters)
|
|
||||||
const storage = this.storage as BaseStorage & StorageBackgroundInitHooks
|
|
||||||
if (typeof storage?.awaitBackgroundInit === 'function') {
|
|
||||||
await storage.awaitBackgroundInit()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Non-cloud storage: no background init to wait for
|
|
||||||
}
|
|
||||||
|
|
||||||
// ============= CORE CRUD OPERATIONS =============
|
// ============= CORE CRUD OPERATIONS =============
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
|
||||||
|
|
@ -1,54 +0,0 @@
|
||||||
/**
|
|
||||||
* Zero-Configuration System
|
|
||||||
* Main entry point for all auto-configuration features
|
|
||||||
*/
|
|
||||||
|
|
||||||
// Model configuration (simplified - always Q8 WASM)
|
|
||||||
export {
|
|
||||||
getModelPrecision,
|
|
||||||
shouldAutoDownloadModels,
|
|
||||||
getModelPath
|
|
||||||
} from './modelAutoConfig.js'
|
|
||||||
|
|
||||||
// Storage configuration
|
|
||||||
export {
|
|
||||||
autoDetectStorage,
|
|
||||||
StorageType,
|
|
||||||
StoragePreset,
|
|
||||||
StorageConfigResult,
|
|
||||||
logStorageConfig,
|
|
||||||
// Backward compatibility types
|
|
||||||
type StorageTypeString,
|
|
||||||
type StoragePresetString
|
|
||||||
} from './storageAutoConfig.js'
|
|
||||||
|
|
||||||
// Shared configuration for multi-instance
|
|
||||||
export {
|
|
||||||
SharedConfig,
|
|
||||||
SharedConfigManager
|
|
||||||
} from './sharedConfigManager.js'
|
|
||||||
|
|
||||||
// Main zero-config processor
|
|
||||||
export {
|
|
||||||
BrainyZeroConfig,
|
|
||||||
processZeroConfig,
|
|
||||||
createEmbeddingFunctionWithPrecision
|
|
||||||
} from './zeroConfig.js'
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Main zero-config processor
|
|
||||||
* This is what Brainy will call
|
|
||||||
*/
|
|
||||||
export async function applyZeroConfig(input?: string | any): Promise<any> {
|
|
||||||
// Handle legacy config (full object) by detecting known legacy properties
|
|
||||||
if (input && typeof input === 'object' &&
|
|
||||||
(input.storage?.forceMemoryStorage || input.storage?.forceFileSystemStorage || input.storage?.s3Storage)) {
|
|
||||||
// This is a legacy config object - pass through unchanged
|
|
||||||
console.log('📦 Using legacy configuration format')
|
|
||||||
return input
|
|
||||||
}
|
|
||||||
|
|
||||||
// Process as zero-config (includes new object format)
|
|
||||||
const { processZeroConfig } = await import('./zeroConfig.js')
|
|
||||||
return processZeroConfig(input)
|
|
||||||
}
|
|
||||||
|
|
@ -1,71 +0,0 @@
|
||||||
/**
|
|
||||||
* Model Configuration
|
|
||||||
* Brainy uses Q8 WASM embeddings - no configuration needed (zero-config)
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { isNode } from '../utils/environment.js'
|
|
||||||
|
|
||||||
interface ModelConfigResult {
|
|
||||||
precision: 'q8'
|
|
||||||
reason: string
|
|
||||||
autoSelected: boolean
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get model precision configuration
|
|
||||||
* Always returns Q8 - the optimal balance of size and accuracy
|
|
||||||
*/
|
|
||||||
export function getModelPrecision(): ModelConfigResult {
|
|
||||||
return {
|
|
||||||
precision: 'q8',
|
|
||||||
reason: 'Q8 WASM (23MB bundled, no downloads)',
|
|
||||||
autoSelected: true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Check if running in a serverless environment
|
|
||||||
*/
|
|
||||||
function isServerlessEnvironment(): boolean {
|
|
||||||
if (!isNode()) return false
|
|
||||||
|
|
||||||
return !!(
|
|
||||||
process.env.AWS_LAMBDA_FUNCTION_NAME ||
|
|
||||||
process.env.VERCEL ||
|
|
||||||
process.env.NETLIFY ||
|
|
||||||
process.env.CLOUDFLARE_WORKERS ||
|
|
||||||
process.env.FUNCTIONS_WORKER_RUNTIME ||
|
|
||||||
process.env.K_SERVICE // Google Cloud Run
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Check if models need to be downloaded
|
|
||||||
* With bundled WASM model, this is rarely needed
|
|
||||||
*/
|
|
||||||
export function shouldAutoDownloadModels(): boolean {
|
|
||||||
// Model is bundled - no downloads needed in normal operation
|
|
||||||
// This flag exists for edge cases only
|
|
||||||
const explicitlyDisabled = process.env.BRAINY_ALLOW_REMOTE_MODELS === 'false'
|
|
||||||
return !explicitlyDisabled
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get the model path
|
|
||||||
* With bundled WASM model, this points to the package assets
|
|
||||||
*/
|
|
||||||
export function getModelPath(): string {
|
|
||||||
// Check if user explicitly set a path (for advanced users)
|
|
||||||
if (process.env.BRAINY_MODELS_PATH) {
|
|
||||||
return process.env.BRAINY_MODELS_PATH
|
|
||||||
}
|
|
||||||
|
|
||||||
// Serverless - use /tmp for ephemeral storage
|
|
||||||
if (isServerlessEnvironment()) {
|
|
||||||
return '/tmp/.brainy/models'
|
|
||||||
}
|
|
||||||
|
|
||||||
// Node-like runtime - use home directory for persistent storage
|
|
||||||
const homeDir = process.env.HOME || process.env.USERPROFILE || '~'
|
|
||||||
return `${homeDir}/.brainy/models`
|
|
||||||
}
|
|
||||||
|
|
@ -1,266 +0,0 @@
|
||||||
/**
|
|
||||||
* Shared Configuration Manager
|
|
||||||
* Ensures configuration consistency across multiple instances using shared storage
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { StorageType } from './storageAutoConfig.js'
|
|
||||||
import { getBrainyVersion } from '../utils/version.js'
|
|
||||||
|
|
||||||
export interface SharedConfig {
|
|
||||||
// Critical parameters that MUST match across instances
|
|
||||||
version: string
|
|
||||||
precision: 'q8'
|
|
||||||
dimensions: number
|
|
||||||
hnswM: number
|
|
||||||
hnswEfConstruction: number
|
|
||||||
distanceFunction: string
|
|
||||||
createdAt: string
|
|
||||||
lastUpdated: string
|
|
||||||
|
|
||||||
// Informational (can differ between instances)
|
|
||||||
instanceCount?: number
|
|
||||||
lastAccessedBy?: string
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Manages configuration consistency for shared storage
|
|
||||||
*/
|
|
||||||
export class SharedConfigManager {
|
|
||||||
private static CONFIG_FILE = '.brainy/config.json'
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Load or create shared configuration
|
|
||||||
* When connecting to existing data, this OVERRIDES auto-configuration!
|
|
||||||
*/
|
|
||||||
static async loadOrCreateSharedConfig(
|
|
||||||
storage: any,
|
|
||||||
localConfig: any
|
|
||||||
): Promise<{ config: any; warnings: string[] }> {
|
|
||||||
const warnings: string[] = []
|
|
||||||
|
|
||||||
try {
|
|
||||||
// Check if we're using shared storage
|
|
||||||
if (!this.isSharedStorage(localConfig.storageType)) {
|
|
||||||
// Local storage - use local config
|
|
||||||
return { config: localConfig, warnings: [] }
|
|
||||||
}
|
|
||||||
|
|
||||||
// Try to load existing configuration from shared storage
|
|
||||||
const existingConfig = await this.loadConfigFromStorage(storage)
|
|
||||||
|
|
||||||
if (existingConfig) {
|
|
||||||
// EXISTING SHARED DATA - Must use its configuration!
|
|
||||||
console.log('📁 Found existing shared data configuration')
|
|
||||||
|
|
||||||
// Check for critical mismatches
|
|
||||||
const mismatches = this.checkCriticalMismatches(localConfig, existingConfig)
|
|
||||||
|
|
||||||
if (mismatches.length > 0) {
|
|
||||||
console.warn('⚠️ Configuration override required for shared storage:')
|
|
||||||
mismatches.forEach(m => {
|
|
||||||
console.warn(` - ${m.param}: ${m.local} → ${m.shared} (using shared)`)
|
|
||||||
warnings.push(`${m.param} overridden: ${m.local} → ${m.shared}`)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// Override critical parameters with shared values
|
|
||||||
const mergedConfig = this.mergeWithSharedConfig(localConfig, existingConfig)
|
|
||||||
|
|
||||||
// Update last accessed
|
|
||||||
await this.updateAccessInfo(storage, existingConfig)
|
|
||||||
|
|
||||||
return { config: mergedConfig, warnings }
|
|
||||||
|
|
||||||
} else {
|
|
||||||
// NEW SHARED STORAGE - Save our configuration
|
|
||||||
console.log('📝 Initializing new shared storage with configuration')
|
|
||||||
|
|
||||||
const sharedConfig = this.createSharedConfig(localConfig)
|
|
||||||
await this.saveConfigToStorage(storage, sharedConfig)
|
|
||||||
|
|
||||||
return { config: localConfig, warnings: [] }
|
|
||||||
}
|
|
||||||
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Failed to manage shared configuration:', error)
|
|
||||||
warnings.push('Could not verify shared configuration - proceeding with caution')
|
|
||||||
return { config: localConfig, warnings }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Check if storage type is shared (multi-instance)
|
|
||||||
*/
|
|
||||||
private static isSharedStorage(storageType: StorageType): boolean {
|
|
||||||
return ['s3', 'gcs', 'r2'].includes(storageType)
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Load configuration from shared storage
|
|
||||||
*/
|
|
||||||
private static async loadConfigFromStorage(storage: any): Promise<SharedConfig | null> {
|
|
||||||
try {
|
|
||||||
const configData = await storage.get(this.CONFIG_FILE)
|
|
||||||
if (!configData) return null
|
|
||||||
|
|
||||||
const config = JSON.parse(configData)
|
|
||||||
return config as SharedConfig
|
|
||||||
} catch (error) {
|
|
||||||
// Config doesn't exist yet
|
|
||||||
return null
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Save configuration to shared storage
|
|
||||||
*/
|
|
||||||
private static async saveConfigToStorage(storage: any, config: SharedConfig): Promise<void> {
|
|
||||||
try {
|
|
||||||
await storage.set(this.CONFIG_FILE, JSON.stringify(config, null, 2))
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Failed to save shared configuration:', error)
|
|
||||||
throw new Error('Cannot initialize shared storage without saving configuration')
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Create shared configuration from local config
|
|
||||||
*/
|
|
||||||
private static createSharedConfig(localConfig: any): SharedConfig {
|
|
||||||
return {
|
|
||||||
version: getBrainyVersion(), // Brainy version
|
|
||||||
precision: localConfig.embeddingOptions?.precision || 'fp32',
|
|
||||||
dimensions: 384, // Fixed for all-MiniLM-L6-v2
|
|
||||||
hnswM: localConfig.hnsw?.M || 16,
|
|
||||||
hnswEfConstruction: localConfig.hnsw?.efConstruction || 200,
|
|
||||||
distanceFunction: localConfig.distanceFunction || 'cosine',
|
|
||||||
createdAt: new Date().toISOString(),
|
|
||||||
lastUpdated: new Date().toISOString(),
|
|
||||||
instanceCount: 1,
|
|
||||||
lastAccessedBy: this.getInstanceIdentifier()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Check for critical configuration mismatches
|
|
||||||
*/
|
|
||||||
private static checkCriticalMismatches(
|
|
||||||
localConfig: any,
|
|
||||||
sharedConfig: SharedConfig
|
|
||||||
): Array<{ param: string; local: any; shared: any }> {
|
|
||||||
const mismatches: Array<{ param: string; local: any; shared: any }> = []
|
|
||||||
|
|
||||||
// Model precision - CRITICAL!
|
|
||||||
const localPrecision = localConfig.embeddingOptions?.precision || 'fp32'
|
|
||||||
if (localPrecision !== sharedConfig.precision) {
|
|
||||||
mismatches.push({
|
|
||||||
param: 'Model Precision',
|
|
||||||
local: localPrecision,
|
|
||||||
shared: sharedConfig.precision
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// HNSW parameters - Important for index consistency
|
|
||||||
const localM = localConfig.hnsw?.M || 16
|
|
||||||
if (localM !== sharedConfig.hnswM) {
|
|
||||||
mismatches.push({
|
|
||||||
param: 'HNSW M',
|
|
||||||
local: localM,
|
|
||||||
shared: sharedConfig.hnswM
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
const localEf = localConfig.hnsw?.efConstruction || 200
|
|
||||||
if (localEf !== sharedConfig.hnswEfConstruction) {
|
|
||||||
mismatches.push({
|
|
||||||
param: 'HNSW efConstruction',
|
|
||||||
local: localEf,
|
|
||||||
shared: sharedConfig.hnswEfConstruction
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// Distance function
|
|
||||||
const localDistance = localConfig.distanceFunction || 'cosine'
|
|
||||||
if (localDistance !== sharedConfig.distanceFunction) {
|
|
||||||
mismatches.push({
|
|
||||||
param: 'Distance Function',
|
|
||||||
local: localDistance,
|
|
||||||
shared: sharedConfig.distanceFunction
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
return mismatches
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Merge local config with shared config (shared takes precedence for critical params)
|
|
||||||
*/
|
|
||||||
private static mergeWithSharedConfig(localConfig: any, sharedConfig: SharedConfig): any {
|
|
||||||
return {
|
|
||||||
...localConfig,
|
|
||||||
|
|
||||||
// Override critical parameters with shared values
|
|
||||||
embeddingOptions: {
|
|
||||||
...localConfig.embeddingOptions,
|
|
||||||
precision: sharedConfig.precision // MUST use shared precision!
|
|
||||||
},
|
|
||||||
|
|
||||||
hnsw: {
|
|
||||||
...localConfig.hnsw,
|
|
||||||
M: sharedConfig.hnswM,
|
|
||||||
efConstruction: sharedConfig.hnswEfConstruction
|
|
||||||
},
|
|
||||||
|
|
||||||
distanceFunction: sharedConfig.distanceFunction,
|
|
||||||
|
|
||||||
// Add metadata about shared configuration
|
|
||||||
_sharedConfig: {
|
|
||||||
loaded: true,
|
|
||||||
version: sharedConfig.version,
|
|
||||||
createdAt: sharedConfig.createdAt,
|
|
||||||
precision: sharedConfig.precision
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Update access information in shared config
|
|
||||||
*/
|
|
||||||
private static async updateAccessInfo(storage: any, config: SharedConfig): Promise<void> {
|
|
||||||
try {
|
|
||||||
config.lastUpdated = new Date().toISOString()
|
|
||||||
config.instanceCount = (config.instanceCount || 0) + 1
|
|
||||||
config.lastAccessedBy = this.getInstanceIdentifier()
|
|
||||||
|
|
||||||
await this.saveConfigToStorage(storage, config)
|
|
||||||
} catch {
|
|
||||||
// Non-critical - don't fail if we can't update access info
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get unique identifier for this instance
|
|
||||||
*/
|
|
||||||
private static getInstanceIdentifier(): string {
|
|
||||||
if (process.env.HOSTNAME) return process.env.HOSTNAME
|
|
||||||
if (process.env.CONTAINER_ID) return process.env.CONTAINER_ID
|
|
||||||
if (process.env.K_SERVICE) return process.env.K_SERVICE
|
|
||||||
if (process.env.AWS_LAMBDA_FUNCTION_NAME) return process.env.AWS_LAMBDA_FUNCTION_NAME
|
|
||||||
|
|
||||||
// Generate a random identifier
|
|
||||||
return `instance-${Date.now()}-${Math.random().toString(36).substr(2, 9)}`
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Validate that a configuration is compatible with shared data
|
|
||||||
*/
|
|
||||||
static validateCompatibility(config1: SharedConfig, config2: SharedConfig): boolean {
|
|
||||||
return (
|
|
||||||
config1.precision === config2.precision &&
|
|
||||||
config1.dimensions === config2.dimensions &&
|
|
||||||
config1.hnswM === config2.hnswM &&
|
|
||||||
config1.hnswEfConstruction === config2.hnswEfConstruction &&
|
|
||||||
config1.distanceFunction === config2.distanceFunction
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,113 +0,0 @@
|
||||||
/**
|
|
||||||
* @module config/storageAutoConfig
|
|
||||||
* @description Storage configuration auto-detection for Brainy 8.0.
|
|
||||||
*
|
|
||||||
* Brainy 8.0 ships **two storage adapters**:
|
|
||||||
* - `FILESYSTEM` — persistent on-disk storage (Node, Bun, Deno).
|
|
||||||
* - `MEMORY` — in-memory, ephemeral.
|
|
||||||
*
|
|
||||||
* Cloud storage adapters (GCS / S3 / R2 / Azure) and the browser-only OPFS
|
|
||||||
* adapter were removed. Cloud backup is now an operator concern handled by
|
|
||||||
* `gsutil` / `aws s3 cp` / `rclone` / `azcopy` against the on-disk artefact.
|
|
||||||
*
|
|
||||||
* This module is kept thin: a two-value enum, a tiny preset enum that mirrors
|
|
||||||
* it, and an `autoDetectStorage()` that picks filesystem when available and
|
|
||||||
* memory otherwise.
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { isNode } from '../utils/environment.js'
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Storage backend types shipped in Brainy 8.0.
|
|
||||||
*/
|
|
||||||
export enum StorageType {
|
|
||||||
MEMORY = 'memory',
|
|
||||||
FILESYSTEM = 'filesystem',
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* High-level storage presets. `AUTO` picks filesystem when available, memory
|
|
||||||
* otherwise; `MEMORY` and `DISK` are explicit overrides.
|
|
||||||
*/
|
|
||||||
export enum StoragePreset {
|
|
||||||
AUTO = 'auto',
|
|
||||||
MEMORY = 'memory',
|
|
||||||
DISK = 'disk',
|
|
||||||
}
|
|
||||||
|
|
||||||
export type StorageTypeString = 'memory' | 'filesystem'
|
|
||||||
export type StoragePresetString = 'auto' | 'memory' | 'disk'
|
|
||||||
|
|
||||||
export interface StorageConfigResult {
|
|
||||||
type: StorageType | StorageTypeString
|
|
||||||
config: { type: StorageType | StorageTypeString; rootDirectory?: string }
|
|
||||||
reason: string
|
|
||||||
autoSelected: boolean
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Resolve a storage configuration. Accepts a {@link StorageType},
|
|
||||||
* {@link StoragePreset}, plain string equivalent, or a free-form storage
|
|
||||||
* config object. Returns a normalized result with the picked type, the
|
|
||||||
* underlying config payload, a human-readable reason, and whether the
|
|
||||||
* pick was auto-selected.
|
|
||||||
*/
|
|
||||||
export async function autoDetectStorage(
|
|
||||||
override?: StorageType | StoragePreset | StorageTypeString | StoragePresetString | any
|
|
||||||
): Promise<StorageConfigResult> {
|
|
||||||
if (override && typeof override === 'object') {
|
|
||||||
return {
|
|
||||||
type: (override.type as StorageType) ?? StorageType.MEMORY,
|
|
||||||
config: override,
|
|
||||||
reason: 'Manually configured storage',
|
|
||||||
autoSelected: false,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (override === StorageType.MEMORY || override === StoragePreset.MEMORY || override === 'memory') {
|
|
||||||
return {
|
|
||||||
type: StorageType.MEMORY,
|
|
||||||
config: { type: StorageType.MEMORY },
|
|
||||||
reason: 'Explicit memory storage',
|
|
||||||
autoSelected: false,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (override === StorageType.FILESYSTEM || override === StoragePreset.DISK || override === 'filesystem' || override === 'disk') {
|
|
||||||
return {
|
|
||||||
type: StorageType.FILESYSTEM,
|
|
||||||
config: { type: StorageType.FILESYSTEM, rootDirectory: './brainy-data' },
|
|
||||||
reason: 'Explicit filesystem storage',
|
|
||||||
autoSelected: false,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Default: filesystem on Node-like runtimes, memory in browsers.
|
|
||||||
if (isNode()) {
|
|
||||||
return {
|
|
||||||
type: StorageType.FILESYSTEM,
|
|
||||||
config: { type: StorageType.FILESYSTEM, rootDirectory: './brainy-data' },
|
|
||||||
reason: 'Auto-selected: filesystem (Node-like runtime detected)',
|
|
||||||
autoSelected: true,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return {
|
|
||||||
type: StorageType.MEMORY,
|
|
||||||
config: { type: StorageType.MEMORY },
|
|
||||||
reason: 'Auto-selected: memory (no Node-like runtime detected)',
|
|
||||||
autoSelected: true,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Log a human-readable summary of a resolved storage config. Used by the
|
|
||||||
* zero-config init path for transparency.
|
|
||||||
*/
|
|
||||||
export function logStorageConfig(config: StorageConfigResult, verbose: boolean = false): void {
|
|
||||||
const prefix = config.autoSelected ? '[brainy] auto-selected' : '[brainy] using'
|
|
||||||
console.log(`${prefix} storage: ${config.type} — ${config.reason}`)
|
|
||||||
if (verbose) {
|
|
||||||
console.log('[brainy] storage config:', JSON.stringify(config.config))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,354 +0,0 @@
|
||||||
/**
|
|
||||||
* Zero-Configuration System for Brainy
|
|
||||||
* Provides intelligent defaults while preserving full control
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { getModelPrecision, getModelPath, shouldAutoDownloadModels } from './modelAutoConfig.js'
|
|
||||||
import { autoDetectStorage, StorageType, StoragePreset } from './storageAutoConfig.js'
|
|
||||||
import { AutoConfiguration } from '../utils/autoConfiguration.js'
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Simplified configuration interface
|
|
||||||
* Everything is optional - zero config by default!
|
|
||||||
*/
|
|
||||||
export interface BrainyZeroConfig {
|
|
||||||
/**
|
|
||||||
* Configuration preset for common scenarios
|
|
||||||
* - 'production': Optimized for production (disk storage, auto model, default features)
|
|
||||||
* - 'development': Optimized for development (memory storage, fp32, verbose logging)
|
|
||||||
* - 'minimal': Minimal footprint (memory storage, q8, minimal features)
|
|
||||||
* - 'zero': True zero config (all auto-detected)
|
|
||||||
* - 'writer': Writer process (read + write) — the single mutating process in
|
|
||||||
* Brainy's multi-process model
|
|
||||||
* - 'reader': Reader process (read-only, all mutations throw) — any number of
|
|
||||||
* these may run against the same on-disk store as the writer
|
|
||||||
*/
|
|
||||||
mode?: 'production' | 'development' | 'minimal' | 'zero' | 'writer' | 'reader'
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Storage configuration
|
|
||||||
* - 'memory': In-memory only (no persistence)
|
|
||||||
* - 'disk': Local disk (filesystem or OPFS)
|
|
||||||
* - 'cloud': Cloud storage (S3/GCS/R2 if configured)
|
|
||||||
* - 'auto': Auto-detect best option (default)
|
|
||||||
* - Object: Custom storage configuration
|
|
||||||
*/
|
|
||||||
storage?: StorageType | StoragePreset | any
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Feature set configuration
|
|
||||||
* - 'minimal': Core features only (fastest startup)
|
|
||||||
* - 'default': Standard features (balanced)
|
|
||||||
* - 'full': All features enabled (most capable)
|
|
||||||
* - Array: Specific features to enable
|
|
||||||
*/
|
|
||||||
features?: 'minimal' | 'default' | 'full' | string[]
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Logging verbosity
|
|
||||||
* - true: Show configuration decisions and progress
|
|
||||||
* - false: Silent operation (default in production)
|
|
||||||
*/
|
|
||||||
verbose?: boolean
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Advanced configuration (escape hatch for power users)
|
|
||||||
* Any additional configuration can be passed here
|
|
||||||
*/
|
|
||||||
advanced?: any
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Configuration presets for common scenarios
|
|
||||||
*/
|
|
||||||
const PRESETS = {
|
|
||||||
production: {
|
|
||||||
storage: 'disk' as const,
|
|
||||||
features: 'default' as const,
|
|
||||||
verbose: false
|
|
||||||
},
|
|
||||||
development: {
|
|
||||||
storage: 'memory' as const,
|
|
||||||
features: 'full' as const,
|
|
||||||
verbose: true
|
|
||||||
},
|
|
||||||
minimal: {
|
|
||||||
storage: 'memory' as const,
|
|
||||||
features: 'minimal' as const,
|
|
||||||
verbose: false
|
|
||||||
},
|
|
||||||
zero: {
|
|
||||||
storage: 'auto' as const,
|
|
||||||
features: 'default' as const,
|
|
||||||
verbose: false
|
|
||||||
},
|
|
||||||
writer: {
|
|
||||||
storage: 'auto' as const,
|
|
||||||
features: 'minimal' as const,
|
|
||||||
verbose: false,
|
|
||||||
// The single mutating process: HybridMode (read + write).
|
|
||||||
mode: 'writer' as const
|
|
||||||
},
|
|
||||||
reader: {
|
|
||||||
storage: 'auto' as const,
|
|
||||||
features: 'default' as const,
|
|
||||||
verbose: false,
|
|
||||||
// Read-only process: ReaderMode — every mutation throws.
|
|
||||||
mode: 'reader' as const
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Feature sets configuration
|
|
||||||
*/
|
|
||||||
const FEATURE_SETS = {
|
|
||||||
minimal: [
|
|
||||||
'core',
|
|
||||||
'search',
|
|
||||||
'storage'
|
|
||||||
],
|
|
||||||
default: [
|
|
||||||
'core',
|
|
||||||
'search',
|
|
||||||
'storage',
|
|
||||||
'cache',
|
|
||||||
'metadata-index',
|
|
||||||
'batch-processing',
|
|
||||||
'entity-registry',
|
|
||||||
'request-deduplicator'
|
|
||||||
],
|
|
||||||
full: [
|
|
||||||
'core',
|
|
||||||
'search',
|
|
||||||
'storage',
|
|
||||||
'cache',
|
|
||||||
'metadata-index',
|
|
||||||
'batch-processing',
|
|
||||||
'entity-registry',
|
|
||||||
'request-deduplicator',
|
|
||||||
'connection-pool',
|
|
||||||
'wal',
|
|
||||||
'monitoring',
|
|
||||||
'metrics',
|
|
||||||
'intelligent-verb-scoring',
|
|
||||||
'triple-intelligence',
|
|
||||||
'neural-api'
|
|
||||||
]
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Process zero-config input into full configuration
|
|
||||||
*/
|
|
||||||
export async function processZeroConfig(input?: string | BrainyZeroConfig): Promise<any> {
|
|
||||||
let config: BrainyZeroConfig = {}
|
|
||||||
|
|
||||||
// Handle string shorthand (preset name)
|
|
||||||
if (typeof input === 'string') {
|
|
||||||
if (input in PRESETS) {
|
|
||||||
// The `in` guard above proved the string is a preset key, which is
|
|
||||||
// exactly the BrainyZeroConfig mode union.
|
|
||||||
config = { mode: input as keyof typeof PRESETS }
|
|
||||||
} else {
|
|
||||||
throw new Error(`Unknown preset: ${input}. Valid presets: ${Object.keys(PRESETS).join(', ')}`)
|
|
||||||
}
|
|
||||||
} else if (input) {
|
|
||||||
config = input
|
|
||||||
}
|
|
||||||
|
|
||||||
// Apply preset if specified
|
|
||||||
if (config.mode && config.mode in PRESETS) {
|
|
||||||
const preset = PRESETS[config.mode]
|
|
||||||
config = {
|
|
||||||
...preset,
|
|
||||||
...config,
|
|
||||||
// Preserve explicit overrides
|
|
||||||
storage: config.storage ?? preset.storage,
|
|
||||||
features: config.features ?? preset.features,
|
|
||||||
verbose: config.verbose ?? preset.verbose
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Auto-detect environment if not in preset mode
|
|
||||||
const environment = detectEnvironmentMode()
|
|
||||||
|
|
||||||
// Get model configuration (always Q8 WASM)
|
|
||||||
const modelConfig = getModelPrecision()
|
|
||||||
|
|
||||||
// Process storage configuration
|
|
||||||
const storageConfig = await autoDetectStorage(config.storage)
|
|
||||||
|
|
||||||
// Process features configuration
|
|
||||||
const features = processFeatures(config.features)
|
|
||||||
|
|
||||||
// Get auto-configuration recommendations. Brainy 8.0 ships filesystem +
|
|
||||||
// memory only; there is no cloud-storage tier.
|
|
||||||
const autoConfig = await AutoConfiguration.getInstance().detectAndConfigure({
|
|
||||||
expectedDataSize: estimateDataSize(environment),
|
|
||||||
s3Available: false,
|
|
||||||
memoryBudget: undefined // Let it auto-detect
|
|
||||||
})
|
|
||||||
|
|
||||||
// Determine verbosity
|
|
||||||
const verbose = config.verbose ?? (process.env.NODE_ENV === 'development')
|
|
||||||
|
|
||||||
// Log configuration decisions if verbose
|
|
||||||
if (verbose) {
|
|
||||||
logConfigurationSummary({
|
|
||||||
mode: config.mode || 'auto',
|
|
||||||
model: modelConfig,
|
|
||||||
storage: storageConfig,
|
|
||||||
features: features,
|
|
||||||
environment: environment,
|
|
||||||
autoConfig: autoConfig
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// Build final configuration
|
|
||||||
const finalConfig: any = {
|
|
||||||
// Model configuration
|
|
||||||
embeddingFunction: undefined, // Will be created with correct precision
|
|
||||||
embeddingOptions: {
|
|
||||||
precision: modelConfig.precision,
|
|
||||||
modelPath: getModelPath(),
|
|
||||||
allowRemoteDownload: shouldAutoDownloadModels()
|
|
||||||
},
|
|
||||||
|
|
||||||
// Storage configuration
|
|
||||||
storage: storageConfig.config,
|
|
||||||
storageType: storageConfig.type,
|
|
||||||
|
|
||||||
// HNSW configuration from auto-config
|
|
||||||
hnsw: {
|
|
||||||
M: autoConfig.recommendedConfig.enablePartitioning ? 32 : 16,
|
|
||||||
efConstruction: autoConfig.recommendedConfig.enablePartitioning ? 400 : 200,
|
|
||||||
maxDatasetSize: autoConfig.recommendedConfig.expectedDatasetSize,
|
|
||||||
partitioning: autoConfig.recommendedConfig.enablePartitioning,
|
|
||||||
maxNodesPerPartition: autoConfig.recommendedConfig.maxNodesPerPartition
|
|
||||||
},
|
|
||||||
|
|
||||||
// Cache configuration from auto-config
|
|
||||||
cache: {
|
|
||||||
autoTune: true,
|
|
||||||
hotCacheMaxSize: Math.floor(autoConfig.recommendedConfig.maxMemoryUsage / (1024 * 1024 * 10)), // 10% of memory budget
|
|
||||||
batchSize: autoConfig.recommendedConfig.enablePartitioning ? 100 : 50
|
|
||||||
},
|
|
||||||
|
|
||||||
// Features configuration
|
|
||||||
enabledFeatures: features,
|
|
||||||
|
|
||||||
// Metadata index configuration
|
|
||||||
metadataIndex: features.includes('metadata-index') ? {
|
|
||||||
enabled: true,
|
|
||||||
autoRebuild: true
|
|
||||||
} : undefined,
|
|
||||||
|
|
||||||
// Intelligent verb scoring
|
|
||||||
intelligentVerbScoring: features.includes('intelligent-verb-scoring') ? {
|
|
||||||
enabled: true
|
|
||||||
} : undefined,
|
|
||||||
|
|
||||||
// Logging configuration
|
|
||||||
logging: {
|
|
||||||
verbose: verbose
|
|
||||||
},
|
|
||||||
|
|
||||||
// Performance flags from auto-config
|
|
||||||
optimizations: autoConfig.optimizationFlags,
|
|
||||||
|
|
||||||
// Advanced overrides (if any)
|
|
||||||
...config.advanced
|
|
||||||
}
|
|
||||||
|
|
||||||
// Multi-process role: writer (read + write) or reader (read-only). This is the
|
|
||||||
// only signal Brainy needs — `mode` drives the operational mode (HybridMode vs
|
|
||||||
// ReaderMode) that gates every mutation path.
|
|
||||||
if (config.mode === 'writer' || config.mode === 'reader') {
|
|
||||||
finalConfig.mode = config.mode
|
|
||||||
|
|
||||||
if (verbose) {
|
|
||||||
console.log(`📡 Process role: ${config.mode.toUpperCase()}`)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return finalConfig
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Detect environment mode if not specified
|
|
||||||
*/
|
|
||||||
function detectEnvironmentMode(): 'production' | 'development' | 'unknown' {
|
|
||||||
if (process.env.NODE_ENV === 'production') return 'production'
|
|
||||||
if (process.env.NODE_ENV === 'development') return 'development'
|
|
||||||
if (process.env.NODE_ENV === 'test') return 'development'
|
|
||||||
|
|
||||||
// Check for CI environments
|
|
||||||
if (process.env.CI || process.env.GITHUB_ACTIONS) return 'production'
|
|
||||||
|
|
||||||
// Check for production indicators
|
|
||||||
if (process.env.VERCEL_ENV === 'production' ||
|
|
||||||
process.env.NETLIFY_ENV === 'production' ||
|
|
||||||
process.env.RAILWAY_ENVIRONMENT === 'production') {
|
|
||||||
return 'production'
|
|
||||||
}
|
|
||||||
|
|
||||||
return 'unknown'
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Process features configuration
|
|
||||||
*/
|
|
||||||
function processFeatures(features?: 'minimal' | 'default' | 'full' | string[]): string[] {
|
|
||||||
if (Array.isArray(features)) {
|
|
||||||
return features
|
|
||||||
}
|
|
||||||
|
|
||||||
if (features && features in FEATURE_SETS) {
|
|
||||||
return FEATURE_SETS[features]
|
|
||||||
}
|
|
||||||
|
|
||||||
// Default based on environment
|
|
||||||
const env = detectEnvironmentMode()
|
|
||||||
if (env === 'production') return FEATURE_SETS.default
|
|
||||||
if (env === 'development') return FEATURE_SETS.full
|
|
||||||
return FEATURE_SETS.default
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Estimate dataset size based on environment
|
|
||||||
*/
|
|
||||||
function estimateDataSize(environment: string): number {
|
|
||||||
switch (environment) {
|
|
||||||
case 'production': return 100000
|
|
||||||
case 'development': return 10000
|
|
||||||
default: return 50000
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Log configuration summary
|
|
||||||
*/
|
|
||||||
function logConfigurationSummary(config: any): void {
|
|
||||||
console.log('\n🧠 Brainy Zero-Config Summary')
|
|
||||||
console.log('================================')
|
|
||||||
console.log(`Mode: ${config.mode}`)
|
|
||||||
console.log(`Environment: ${config.environment}`)
|
|
||||||
console.log(`Model: ${config.model.precision.toUpperCase()} (${config.model.reason})`)
|
|
||||||
console.log(`Storage: ${config.storage.type.toUpperCase()} (${config.storage.reason})`)
|
|
||||||
console.log(`Features: ${config.features.length} enabled`)
|
|
||||||
console.log(`Memory Budget: ${Math.floor(config.autoConfig.recommendedConfig.maxMemoryUsage / (1024 * 1024))}MB`)
|
|
||||||
console.log(`Expected Dataset: ${config.autoConfig.recommendedConfig.expectedDatasetSize.toLocaleString()} items`)
|
|
||||||
console.log('================================\n')
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Create embedding function (always Q8 WASM)
|
|
||||||
*/
|
|
||||||
export async function createEmbeddingFunctionWithPrecision(): Promise<any> {
|
|
||||||
const { createEmbeddingFunction } = await import('../utils/embedding.js')
|
|
||||||
|
|
||||||
// Create embedding function - always Q8 WASM
|
|
||||||
return createEmbeddingFunction({
|
|
||||||
precision: 'q8',
|
|
||||||
verbose: false // Silent by default in zero-config
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
@ -77,11 +77,6 @@ export type {
|
||||||
// Export Aggregation Engine
|
// Export Aggregation Engine
|
||||||
export { AggregationIndex, AggregateMaterializer, bucketTimestamp, parseBucketRange } from './aggregation/index.js'
|
export { AggregationIndex, AggregateMaterializer, bucketTimestamp, parseBucketRange } from './aggregation/index.js'
|
||||||
|
|
||||||
// Export zero-configuration input type
|
|
||||||
export {
|
|
||||||
BrainyZeroConfig
|
|
||||||
} from './config/index.js'
|
|
||||||
|
|
||||||
// Export Neural Import (AI data understanding)
|
// Export Neural Import (AI data understanding)
|
||||||
export { NeuralImport } from './neural/neuralImport.js'
|
export { NeuralImport } from './neural/neuralImport.js'
|
||||||
export type {
|
export type {
|
||||||
|
|
|
||||||
|
|
@ -18,44 +18,6 @@ import { StorageBatchConfig } from '../baseStorage.js'
|
||||||
import { extractFieldNamesFromJson, mapToStandardField } from '../../utils/fieldNameTracking.js'
|
import { extractFieldNamesFromJson, mapToStandardField } from '../../utils/fieldNameTracking.js'
|
||||||
import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
|
import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
|
||||||
|
|
||||||
// =============================================================================
|
|
||||||
// Progressive Initialization Types
|
|
||||||
// =============================================================================
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Initialization mode for cloud storage adapters.
|
|
||||||
*
|
|
||||||
* Controls how storage adapters initialize during startup, enabling
|
|
||||||
* fast cold starts in serverless environments while maintaining strict
|
|
||||||
* validation for local development.
|
|
||||||
*
|
|
||||||
*
|
|
||||||
* | Mode | Description | Use Case |
|
|
||||||
* |------|-------------|----------|
|
|
||||||
* | `'auto'` | Auto-detect environment (progressive in cloud, strict locally) | Default, zero-config |
|
|
||||||
* | `'progressive'` | Fast init (<200ms), validate lazily on first write | Serverless, Cloud Run |
|
|
||||||
* | `'strict'` | Blocking validation during init (traditional behavior) | Local dev, testing |
|
|
||||||
*
|
|
||||||
* @example
|
|
||||||
* ```typescript
|
|
||||||
* // Zero-config: auto-detects Cloud Run, Lambda, etc.
|
|
||||||
* const storage = new FileSystemStorage({ bucket: 'my-bucket' })
|
|
||||||
*
|
|
||||||
* // Force progressive mode for all environments
|
|
||||||
* const storage = new FileSystemStorage({
|
|
||||||
* bucket: 'my-bucket',
|
|
||||||
* initMode: 'progressive'
|
|
||||||
* })
|
|
||||||
*
|
|
||||||
* // Force strict validation (useful for testing)
|
|
||||||
* const storage = new FileSystemStorage({
|
|
||||||
* bucket: 'my-bucket',
|
|
||||||
* initMode: 'strict'
|
|
||||||
* })
|
|
||||||
* ```
|
|
||||||
*/
|
|
||||||
export type InitMode = 'progressive' | 'strict' | 'auto'
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Base class for storage adapters that implements statistics tracking
|
* Base class for storage adapters that implements statistics tracking
|
||||||
*/
|
*/
|
||||||
|
|
@ -372,71 +334,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
||||||
status: 'normal' | 'throttled' | 'recovering'
|
status: 'normal' | 'throttled' | 'recovering'
|
||||||
}> = new Map()
|
}> = new Map()
|
||||||
|
|
||||||
// =============================================
|
|
||||||
// Progressive Initialization State
|
|
||||||
// =============================================
|
|
||||||
// These properties enable fast cold starts in cloud environments
|
|
||||||
// by deferring validation and count loading to background tasks.
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Initialization mode for this storage adapter.
|
|
||||||
*
|
|
||||||
* - `'auto'` (default): Progressive in cloud environments, strict locally
|
|
||||||
* - `'progressive'`: Always use fast init with lazy validation
|
|
||||||
* - `'strict'`: Always validate during init (traditional behavior)
|
|
||||||
*
|
|
||||||
* @protected
|
|
||||||
*/
|
|
||||||
protected initMode: InitMode = 'auto'
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Whether the bucket/container has been validated to exist.
|
|
||||||
*
|
|
||||||
* In progressive mode, this starts as `false` and is set to `true`
|
|
||||||
* after the first successful write operation or background validation.
|
|
||||||
*
|
|
||||||
* @protected
|
|
||||||
*/
|
|
||||||
protected bucketValidated = false
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Error from bucket validation, if any.
|
|
||||||
*
|
|
||||||
* Stored here so subsequent operations can fail fast with the same error.
|
|
||||||
*
|
|
||||||
* @protected
|
|
||||||
*/
|
|
||||||
protected bucketValidationError: Error | null = null
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Whether counts have been loaded from storage.
|
|
||||||
*
|
|
||||||
* In progressive mode, counts start at 0 and are loaded in the background.
|
|
||||||
* Operations work immediately; counts become accurate after background load.
|
|
||||||
*
|
|
||||||
* @protected
|
|
||||||
*/
|
|
||||||
protected countsLoaded = false
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Whether all background initialization tasks have completed.
|
|
||||||
*
|
|
||||||
* Useful for tests and diagnostics to ensure full initialization.
|
|
||||||
*
|
|
||||||
* @protected
|
|
||||||
*/
|
|
||||||
protected backgroundTasksComplete = false
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Promise that resolves when background initialization completes.
|
|
||||||
*
|
|
||||||
* Can be awaited by callers who need to ensure full initialization
|
|
||||||
* before proceeding (e.g., in tests).
|
|
||||||
*
|
|
||||||
* @protected
|
|
||||||
*/
|
|
||||||
protected backgroundInitPromise: Promise<void> | null = null
|
|
||||||
|
|
||||||
// Statistics-specific methods that must be implemented by subclasses
|
// Statistics-specific methods that must be implemented by subclasses
|
||||||
protected abstract saveStatisticsData(
|
protected abstract saveStatisticsData(
|
||||||
statistics: StatisticsData
|
statistics: StatisticsData
|
||||||
|
|
@ -1129,18 +1026,11 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
||||||
protected readonly COUNT_CACHE_TTL = 60000 // 1 minute cache TTL
|
protected readonly COUNT_CACHE_TTL = 60000 // 1 minute cache TTL
|
||||||
|
|
||||||
// =============================================
|
// =============================================
|
||||||
// Smart Count Batching
|
// Count Persistence
|
||||||
// =============================================
|
// =============================================
|
||||||
|
|
||||||
// Count batching state - mirrors statistics batching pattern
|
// Counts changed since the last persist? Drives the write-through flush.
|
||||||
protected pendingCountPersist = false // Counts changed since last persist?
|
protected pendingCountPersist = false
|
||||||
protected lastCountPersistTime = 0 // Timestamp of last persist
|
|
||||||
protected scheduledCountPersistTimeout: NodeJS.Timeout | null = null // Scheduled persist timer
|
|
||||||
protected pendingCountOperations = 0 // Operations since last persist
|
|
||||||
|
|
||||||
// Batching configuration (overridable by subclasses for custom strategies)
|
|
||||||
protected countPersistBatchSize = 10 // Operations before forcing persist (cloud storage)
|
|
||||||
protected countPersistInterval = 5000 // Milliseconds before forcing persist (cloud storage)
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Get total noun count - O(1) operation
|
* Get total noun count - O(1) operation
|
||||||
|
|
@ -1295,230 +1185,23 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
||||||
// =============================================
|
// =============================================
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Detect if this storage adapter uses cloud storage (network I/O)
|
* Persist counts immediately.
|
||||||
* Cloud storage benefits from batching; local storage does not.
|
|
||||||
*
|
*
|
||||||
* Override this method in subclasses for accurate detection.
|
* Filesystem and memory storage have no network latency, so counts are
|
||||||
* Default implementation checks storage type from getStorageStatus().
|
* written through on every change rather than batched.
|
||||||
*
|
|
||||||
* @returns true if cloud storage (GCS, S3, R2), false if local (File, Memory)
|
|
||||||
*/
|
|
||||||
protected isCloudStorage(): boolean {
|
|
||||||
// Default: assume local storage (conservative, prefers reliability over performance)
|
|
||||||
// Subclasses should override this for accurate detection
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
// =============================================
|
|
||||||
// Progressive Initialization Methods
|
|
||||||
// =============================================
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Detect if running in a cloud/serverless environment.
|
|
||||||
*
|
|
||||||
* Cloud environments benefit from progressive initialization to minimize
|
|
||||||
* cold start times. This method checks for common environment variables
|
|
||||||
* set by cloud providers.
|
|
||||||
*
|
|
||||||
* | Provider | Environment Variable |
|
|
||||||
* |----------|---------------------|
|
|
||||||
* | Cloud Run | `K_SERVICE`, `K_REVISION` |
|
|
||||||
* | Lambda | `AWS_LAMBDA_FUNCTION_NAME` |
|
|
||||||
* | Cloud Functions | `FUNCTIONS_TARGET` |
|
|
||||||
* | Azure Functions | `AZURE_FUNCTIONS_ENVIRONMENT` |
|
|
||||||
*
|
|
||||||
* @returns `true` if running in a detected cloud environment
|
|
||||||
* @protected
|
|
||||||
*/
|
|
||||||
protected detectCloudEnvironment(): boolean {
|
|
||||||
return !!(
|
|
||||||
process.env.K_SERVICE || // Google Cloud Run
|
|
||||||
process.env.K_REVISION || // Google Cloud Run (revision)
|
|
||||||
process.env.AWS_LAMBDA_FUNCTION_NAME || // AWS Lambda
|
|
||||||
process.env.FUNCTIONS_TARGET || // Google Cloud Functions
|
|
||||||
process.env.AZURE_FUNCTIONS_ENVIRONMENT // Azure Functions
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Determine the effective initialization mode.
|
|
||||||
*
|
|
||||||
* Resolves `'auto'` to either `'progressive'` or `'strict'` based on
|
|
||||||
* the detected environment.
|
|
||||||
*
|
|
||||||
* @returns The resolved init mode (`'progressive'` or `'strict'`)
|
|
||||||
* @protected
|
|
||||||
*/
|
|
||||||
protected resolveInitMode(): 'progressive' | 'strict' {
|
|
||||||
if (this.initMode === 'auto') {
|
|
||||||
return this.detectCloudEnvironment() ? 'progressive' : 'strict'
|
|
||||||
}
|
|
||||||
return this.initMode
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Schedule background initialization tasks.
|
|
||||||
*
|
|
||||||
* This method is called in progressive mode after the adapter is marked
|
|
||||||
* as initialized. It schedules non-blocking tasks to:
|
|
||||||
* 1. Validate bucket/container existence
|
|
||||||
* 2. Load counts from storage
|
|
||||||
*
|
|
||||||
* Override in subclasses to add storage-specific background tasks.
|
|
||||||
* Always call `super.scheduleBackgroundInit()` first.
|
|
||||||
*
|
|
||||||
* @protected
|
|
||||||
*/
|
|
||||||
protected scheduleBackgroundInit(): void {
|
|
||||||
// Create a promise that tracks all background work
|
|
||||||
this.backgroundInitPromise = this.runBackgroundInit()
|
|
||||||
|
|
||||||
// Don't await - let it run in background
|
|
||||||
this.backgroundInitPromise
|
|
||||||
.then(() => {
|
|
||||||
this.backgroundTasksComplete = true
|
|
||||||
})
|
|
||||||
.catch((error) => {
|
|
||||||
// Log but don't throw - progressive mode is best-effort
|
|
||||||
console.error('[Progressive Init] Background initialization failed:', error)
|
|
||||||
this.backgroundTasksComplete = true // Mark complete even on error
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Run background initialization tasks.
|
|
||||||
*
|
|
||||||
* Override in subclasses to add storage-specific tasks.
|
|
||||||
* The default implementation does nothing (for local storage adapters).
|
|
||||||
*
|
|
||||||
* @protected
|
|
||||||
*/
|
|
||||||
protected async runBackgroundInit(): Promise<void> {
|
|
||||||
// Default implementation: nothing to do for local storage
|
|
||||||
// Cloud storage adapters override this to:
|
|
||||||
// 1. Validate bucket/container
|
|
||||||
// 2. Load counts from storage
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Wait for background initialization to complete.
|
|
||||||
*
|
|
||||||
* Useful in tests or when you need to ensure all background tasks
|
|
||||||
* have finished before proceeding.
|
|
||||||
*
|
|
||||||
* @example
|
|
||||||
* ```typescript
|
|
||||||
* const storage = new FileSystemStorage({ bucket: 'my-bucket' })
|
|
||||||
* await storage.init()
|
|
||||||
*
|
|
||||||
* // Wait for background validation and count loading
|
|
||||||
* await storage.awaitBackgroundInit()
|
|
||||||
*
|
|
||||||
* // Now counts are accurate and bucket is validated
|
|
||||||
* const count = await storage.getNounCount()
|
|
||||||
* ```
|
|
||||||
*
|
|
||||||
* @public
|
|
||||||
*/
|
|
||||||
public async awaitBackgroundInit(): Promise<void> {
|
|
||||||
if (this.backgroundInitPromise) {
|
|
||||||
await this.backgroundInitPromise
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Check if background initialization has completed.
|
|
||||||
*
|
|
||||||
* @returns `true` if all background tasks are complete
|
|
||||||
* @public
|
|
||||||
*/
|
|
||||||
public isBackgroundInitComplete(): boolean {
|
|
||||||
return this.backgroundTasksComplete
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Ensure bucket/container is validated before write operations.
|
|
||||||
*
|
|
||||||
* In progressive mode, bucket validation is deferred until the first
|
|
||||||
* write operation. This method performs lazy validation and caches
|
|
||||||
* the result (or error) for subsequent calls.
|
|
||||||
*
|
|
||||||
* Override in subclasses to implement storage-specific validation.
|
|
||||||
* The base implementation does nothing (for local storage adapters).
|
|
||||||
*
|
|
||||||
* @throws Error if bucket validation fails
|
|
||||||
* @protected
|
|
||||||
*/
|
|
||||||
protected async ensureValidatedForWrite(): Promise<void> {
|
|
||||||
// If already validated, nothing to do
|
|
||||||
if (this.bucketValidated) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
// If we have a cached validation error, throw it
|
|
||||||
if (this.bucketValidationError) {
|
|
||||||
throw this.bucketValidationError
|
|
||||||
}
|
|
||||||
|
|
||||||
// Default implementation: no validation needed (local storage)
|
|
||||||
// Cloud storage adapters override this to validate bucket/container
|
|
||||||
this.bucketValidated = true
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Schedule a smart batched persist operation.
|
|
||||||
*
|
|
||||||
* Strategy:
|
|
||||||
* - Local Storage: Persist immediately (fast, no network latency)
|
|
||||||
* - Cloud Storage: Batch persist (10 ops OR 5 seconds, whichever first)
|
|
||||||
*
|
|
||||||
* This mirrors the statistics batching pattern for consistency.
|
|
||||||
*/
|
*/
|
||||||
protected async scheduleCountPersist(): Promise<void> {
|
protected async scheduleCountPersist(): Promise<void> {
|
||||||
// Mark counts as pending persist
|
|
||||||
this.pendingCountPersist = true
|
this.pendingCountPersist = true
|
||||||
this.pendingCountOperations++
|
|
||||||
|
|
||||||
// Local storage: persist immediately (fast enough, no benefit from batching)
|
|
||||||
if (!this.isCloudStorage()) {
|
|
||||||
await this.flushCounts()
|
await this.flushCounts()
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
// Cloud storage: use smart batching
|
|
||||||
// Persist if we've hit the batch size threshold
|
|
||||||
if (this.pendingCountOperations >= this.countPersistBatchSize) {
|
|
||||||
await this.flushCounts()
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
// Otherwise, schedule a time-based persist if not already scheduled
|
|
||||||
if (!this.scheduledCountPersistTimeout) {
|
|
||||||
this.scheduledCountPersistTimeout = setTimeout(() => {
|
|
||||||
this.flushCounts().catch(error => {
|
|
||||||
console.error('Failed to flush counts on timer:', error)
|
|
||||||
})
|
|
||||||
}, this.countPersistInterval)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Flush counts immediately to storage.
|
* Flush pending counts to storage.
|
||||||
*
|
*
|
||||||
* Used for:
|
* Used for graceful shutdown (SIGTERM handler) and the immediate
|
||||||
* - Graceful shutdown (SIGTERM handler)
|
* write-through path. This is the public API that shutdown hooks can call.
|
||||||
* - Forced persist (batch threshold reached)
|
|
||||||
* - Local storage immediate persist
|
|
||||||
*
|
|
||||||
* This is the public API that shutdown hooks can call.
|
|
||||||
*/
|
*/
|
||||||
async flushCounts(): Promise<void> {
|
async flushCounts(): Promise<void> {
|
||||||
// Clear any scheduled persist
|
|
||||||
if (this.scheduledCountPersistTimeout) {
|
|
||||||
clearTimeout(this.scheduledCountPersistTimeout)
|
|
||||||
this.scheduledCountPersistTimeout = null
|
|
||||||
}
|
|
||||||
|
|
||||||
// Nothing to flush?
|
// Nothing to flush?
|
||||||
if (!this.pendingCountPersist) {
|
if (!this.pendingCountPersist) {
|
||||||
return
|
return
|
||||||
|
|
@ -1527,13 +1210,9 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
||||||
try {
|
try {
|
||||||
// Persist to storage (implemented by subclass)
|
// Persist to storage (implemented by subclass)
|
||||||
await this.persistCounts()
|
await this.persistCounts()
|
||||||
|
|
||||||
// Update state
|
|
||||||
this.lastCountPersistTime = Date.now()
|
|
||||||
this.pendingCountPersist = false
|
this.pendingCountPersist = false
|
||||||
this.pendingCountOperations = 0
|
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
console.error('❌ CRITICAL: Failed to flush counts to storage:', error)
|
console.error('CRITICAL: Failed to flush counts to storage:', error)
|
||||||
// Keep pending flag set so we retry on next operation
|
// Keep pending flag set so we retry on next operation
|
||||||
throw error
|
throw error
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -29,17 +29,6 @@ export interface BrainyInterface<T = unknown> {
|
||||||
*/
|
*/
|
||||||
readonly isInitialized: boolean
|
readonly isInitialized: boolean
|
||||||
|
|
||||||
/**
|
|
||||||
* Check if all initialization including background tasks is complete
|
|
||||||
*/
|
|
||||||
isFullyInitialized(): boolean
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Wait for all background initialization tasks to complete
|
|
||||||
* For cloud storage adapters, this waits for bucket validation and count sync.
|
|
||||||
*/
|
|
||||||
awaitBackgroundInit(): Promise<void>
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Modern add method - unified entity creation
|
* Modern add method - unified entity creation
|
||||||
* @param params Parameters for adding entities
|
* @param params Parameters for adding entities
|
||||||
|
|
|
||||||
|
|
@ -1,428 +0,0 @@
|
||||||
/**
|
|
||||||
* Automatic Configuration System for Brainy Vector Database
|
|
||||||
* Detects environment, resources, and data patterns to provide optimal settings
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { isNode } from './environment.js'
|
|
||||||
|
|
||||||
export interface AutoConfigResult {
|
|
||||||
// Environment details
|
|
||||||
environment: 'nodejs' | 'serverless' | 'unknown'
|
|
||||||
|
|
||||||
// Resource detection
|
|
||||||
availableMemory: number // bytes
|
|
||||||
cpuCores: number
|
|
||||||
|
|
||||||
|
|
||||||
// Storage capabilities
|
|
||||||
persistentStorageAvailable: boolean
|
|
||||||
s3StorageDetected: boolean
|
|
||||||
|
|
||||||
// Recommended configuration
|
|
||||||
recommendedConfig: {
|
|
||||||
expectedDatasetSize: number
|
|
||||||
maxMemoryUsage: number
|
|
||||||
targetSearchLatency: number
|
|
||||||
enablePartitioning: boolean
|
|
||||||
enableCompression: boolean
|
|
||||||
enablePredictiveCaching: boolean
|
|
||||||
partitionStrategy: 'semantic' | 'hash'
|
|
||||||
maxNodesPerPartition: number
|
|
||||||
semanticClusters: number
|
|
||||||
}
|
|
||||||
|
|
||||||
// Performance optimization flags
|
|
||||||
optimizationFlags: {
|
|
||||||
useMemoryMapping: boolean
|
|
||||||
aggressiveCaching: boolean
|
|
||||||
backgroundOptimization: boolean
|
|
||||||
compressionLevel: 'none' | 'light' | 'aggressive'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
export interface DatasetAnalysis {
|
|
||||||
estimatedSize: number
|
|
||||||
vectorDimension?: number
|
|
||||||
growthRate?: number // vectors per second
|
|
||||||
accessPatterns?: 'read-heavy' | 'write-heavy' | 'balanced'
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Automatic configuration system that detects environment and optimizes settings
|
|
||||||
*/
|
|
||||||
export class AutoConfiguration {
|
|
||||||
private static instance: AutoConfiguration
|
|
||||||
private cachedConfig: AutoConfigResult | null = null
|
|
||||||
private datasetStats: DatasetAnalysis = { estimatedSize: 0 }
|
|
||||||
|
|
||||||
private constructor() {}
|
|
||||||
|
|
||||||
public static getInstance(): AutoConfiguration {
|
|
||||||
if (!AutoConfiguration.instance) {
|
|
||||||
AutoConfiguration.instance = new AutoConfiguration()
|
|
||||||
}
|
|
||||||
return AutoConfiguration.instance
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Detect environment and generate optimal configuration
|
|
||||||
*/
|
|
||||||
public async detectAndConfigure(hints?: {
|
|
||||||
expectedDataSize?: number
|
|
||||||
s3Available?: boolean
|
|
||||||
memoryBudget?: number
|
|
||||||
}): Promise<AutoConfigResult> {
|
|
||||||
if (this.cachedConfig && !hints) {
|
|
||||||
return this.cachedConfig
|
|
||||||
}
|
|
||||||
|
|
||||||
const environment = this.detectEnvironment()
|
|
||||||
const resources = await this.detectResources()
|
|
||||||
const storage = await this.detectStorageCapabilities(hints?.s3Available)
|
|
||||||
|
|
||||||
const config: AutoConfigResult = {
|
|
||||||
environment,
|
|
||||||
...resources,
|
|
||||||
...storage,
|
|
||||||
recommendedConfig: this.generateRecommendedConfig(environment, resources, hints),
|
|
||||||
optimizationFlags: this.generateOptimizationFlags(environment, resources)
|
|
||||||
}
|
|
||||||
|
|
||||||
this.cachedConfig = config
|
|
||||||
return config
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Update configuration based on runtime dataset analysis
|
|
||||||
*/
|
|
||||||
public async adaptToDataset(analysis: DatasetAnalysis): Promise<AutoConfigResult> {
|
|
||||||
this.datasetStats = analysis
|
|
||||||
|
|
||||||
// Regenerate configuration with dataset insights
|
|
||||||
const currentConfig = await this.detectAndConfigure()
|
|
||||||
const adaptedConfig = this.adaptConfigurationToData(currentConfig, analysis)
|
|
||||||
|
|
||||||
this.cachedConfig = adaptedConfig
|
|
||||||
return adaptedConfig
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Learn from performance metrics and adjust configuration
|
|
||||||
*/
|
|
||||||
public async learnFromPerformance(metrics: {
|
|
||||||
averageSearchTime: number
|
|
||||||
memoryUsage: number
|
|
||||||
cacheHitRate: number
|
|
||||||
errorRate: number
|
|
||||||
}): Promise<Partial<AutoConfigResult['recommendedConfig']>> {
|
|
||||||
const adjustments: Partial<AutoConfigResult['recommendedConfig']> = {}
|
|
||||||
|
|
||||||
// Learn from search performance
|
|
||||||
if (metrics.averageSearchTime > 200) {
|
|
||||||
// Too slow - optimize for speed
|
|
||||||
adjustments.maxNodesPerPartition = Math.max(10000, (this.cachedConfig?.recommendedConfig.maxNodesPerPartition || 50000) * 0.8)
|
|
||||||
} else if (metrics.averageSearchTime < 50) {
|
|
||||||
// Very fast - can optimize for quality
|
|
||||||
adjustments.maxNodesPerPartition = Math.min(100000, (this.cachedConfig?.recommendedConfig.maxNodesPerPartition || 50000) * 1.2)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Learn from memory usage
|
|
||||||
if (metrics.memoryUsage > (this.cachedConfig?.recommendedConfig.maxMemoryUsage || 0) * 0.9) {
|
|
||||||
// High memory usage - enable compression
|
|
||||||
adjustments.enableCompression = true
|
|
||||||
}
|
|
||||||
|
|
||||||
// Learn from cache performance
|
|
||||||
if (metrics.cacheHitRate < 0.7) {
|
|
||||||
// Poor cache performance - enable predictive caching
|
|
||||||
adjustments.enablePredictiveCaching = true
|
|
||||||
}
|
|
||||||
|
|
||||||
// Update cached config with learned adjustments
|
|
||||||
if (this.cachedConfig) {
|
|
||||||
this.cachedConfig.recommendedConfig = {
|
|
||||||
...this.cachedConfig.recommendedConfig,
|
|
||||||
...adjustments
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return adjustments
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get minimal configuration for quick setup
|
|
||||||
*/
|
|
||||||
public async getQuickSetupConfig(scenario: 'small' | 'medium' | 'large' | 'enterprise'): Promise<{
|
|
||||||
expectedDatasetSize: number
|
|
||||||
maxMemoryUsage: number
|
|
||||||
targetSearchLatency: number
|
|
||||||
s3Required: boolean
|
|
||||||
}> {
|
|
||||||
const environment = this.detectEnvironment()
|
|
||||||
const resources = await this.detectResources()
|
|
||||||
|
|
||||||
switch (scenario) {
|
|
||||||
case 'small':
|
|
||||||
return {
|
|
||||||
expectedDatasetSize: 10000,
|
|
||||||
maxMemoryUsage: Math.min(resources.availableMemory * 0.3, 1024 * 1024 * 1024), // 1GB max
|
|
||||||
targetSearchLatency: 100,
|
|
||||||
s3Required: false
|
|
||||||
}
|
|
||||||
|
|
||||||
case 'medium':
|
|
||||||
return {
|
|
||||||
expectedDatasetSize: 100000,
|
|
||||||
maxMemoryUsage: Math.min(resources.availableMemory * 0.5, 4 * 1024 * 1024 * 1024), // 4GB max
|
|
||||||
targetSearchLatency: 150,
|
|
||||||
s3Required: environment === 'serverless'
|
|
||||||
}
|
|
||||||
|
|
||||||
case 'large':
|
|
||||||
return {
|
|
||||||
expectedDatasetSize: 1000000,
|
|
||||||
maxMemoryUsage: Math.min(resources.availableMemory * 0.7, 8 * 1024 * 1024 * 1024), // 8GB max
|
|
||||||
targetSearchLatency: 200,
|
|
||||||
s3Required: true
|
|
||||||
}
|
|
||||||
|
|
||||||
case 'enterprise':
|
|
||||||
return {
|
|
||||||
expectedDatasetSize: 10000000,
|
|
||||||
maxMemoryUsage: Math.min(resources.availableMemory * 0.8, 32 * 1024 * 1024 * 1024), // 32GB max
|
|
||||||
targetSearchLatency: 300,
|
|
||||||
s3Required: true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Detect the current runtime environment
|
|
||||||
*/
|
|
||||||
private detectEnvironment(): 'nodejs' | 'serverless' | 'unknown' {
|
|
||||||
if (isNode()) {
|
|
||||||
// Check for serverless environment indicators
|
|
||||||
if (process.env.AWS_LAMBDA_FUNCTION_NAME ||
|
|
||||||
process.env.VERCEL ||
|
|
||||||
process.env.NETLIFY ||
|
|
||||||
process.env.CLOUDFLARE_WORKERS) {
|
|
||||||
return 'serverless'
|
|
||||||
}
|
|
||||||
return 'nodejs'
|
|
||||||
}
|
|
||||||
|
|
||||||
return 'unknown'
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Detect available system resources
|
|
||||||
*/
|
|
||||||
private async detectResources(): Promise<{
|
|
||||||
availableMemory: number
|
|
||||||
cpuCores: number
|
|
||||||
}> {
|
|
||||||
let availableMemory = 2 * 1024 * 1024 * 1024 // Default 2GB
|
|
||||||
let cpuCores = 4 // Default 4 cores
|
|
||||||
|
|
||||||
if (isNode()) {
|
|
||||||
try {
|
|
||||||
const os = await import('node:os')
|
|
||||||
availableMemory = os.totalmem() * 0.7 // Use 70% of total memory
|
|
||||||
cpuCores = os.cpus().length
|
|
||||||
} catch {
|
|
||||||
// Fallback to defaults
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return { availableMemory, cpuCores }
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Detect available storage capabilities
|
|
||||||
*/
|
|
||||||
private async detectStorageCapabilities(s3Hint?: boolean): Promise<{
|
|
||||||
persistentStorageAvailable: boolean
|
|
||||||
s3StorageDetected: boolean
|
|
||||||
}> {
|
|
||||||
let persistentStorageAvailable = false
|
|
||||||
let s3StorageDetected = s3Hint || false
|
|
||||||
|
|
||||||
if (isNode()) {
|
|
||||||
persistentStorageAvailable = true
|
|
||||||
// 8.0 dropped cloud storage adapters; the s3StorageDetected flag is
|
|
||||||
// preserved as a hint for operator backup tooling, not adapter wiring.
|
|
||||||
s3StorageDetected = s3Hint ||
|
|
||||||
!!(process.env.AWS_ACCESS_KEY_ID && process.env.AWS_SECRET_ACCESS_KEY) ||
|
|
||||||
!!(process.env.S3_BUCKET_NAME)
|
|
||||||
}
|
|
||||||
|
|
||||||
return {
|
|
||||||
persistentStorageAvailable,
|
|
||||||
s3StorageDetected
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Generate recommended configuration based on detected environment and resources
|
|
||||||
*/
|
|
||||||
private generateRecommendedConfig(
|
|
||||||
environment: string,
|
|
||||||
resources: { availableMemory: number; cpuCores: number },
|
|
||||||
hints?: { expectedDataSize?: number; memoryBudget?: number }
|
|
||||||
): AutoConfigResult['recommendedConfig'] {
|
|
||||||
const datasetSize = hints?.expectedDataSize || this.estimateDatasetSize()
|
|
||||||
const memoryBudget = hints?.memoryBudget || Math.floor(resources.availableMemory * 0.6)
|
|
||||||
|
|
||||||
// Base configuration
|
|
||||||
let config = {
|
|
||||||
expectedDatasetSize: datasetSize,
|
|
||||||
maxMemoryUsage: memoryBudget,
|
|
||||||
targetSearchLatency: 150,
|
|
||||||
enablePartitioning: datasetSize > 25000,
|
|
||||||
enableCompression: memoryBudget < 2 * 1024 * 1024 * 1024,
|
|
||||||
enablePredictiveCaching: true,
|
|
||||||
partitionStrategy: 'semantic' as const,
|
|
||||||
maxNodesPerPartition: 50000,
|
|
||||||
semanticClusters: 8
|
|
||||||
}
|
|
||||||
|
|
||||||
// Environment-specific adjustments
|
|
||||||
switch (environment) {
|
|
||||||
case 'serverless':
|
|
||||||
config = {
|
|
||||||
...config,
|
|
||||||
targetSearchLatency: 500, // Account for cold starts
|
|
||||||
enablePredictiveCaching: false, // Avoid background processes
|
|
||||||
maxNodesPerPartition: 30000 // Moderate partition size
|
|
||||||
}
|
|
||||||
break
|
|
||||||
|
|
||||||
case 'nodejs':
|
|
||||||
config = {
|
|
||||||
...config,
|
|
||||||
targetSearchLatency: 100, // Aggressive for Node.js
|
|
||||||
maxNodesPerPartition: Math.min(100000, Math.floor(datasetSize / 10)), // Larger partitions
|
|
||||||
semanticClusters: Math.min(16, Math.max(4, Math.floor(datasetSize / 50000))) // Scale clusters with data
|
|
||||||
}
|
|
||||||
break
|
|
||||||
}
|
|
||||||
|
|
||||||
// Dataset size adjustments
|
|
||||||
if (datasetSize > 1000000) {
|
|
||||||
config.semanticClusters = Math.min(32, Math.floor(datasetSize / 100000))
|
|
||||||
config.maxNodesPerPartition = 100000
|
|
||||||
} else if (datasetSize < 10000) {
|
|
||||||
config.enablePartitioning = false
|
|
||||||
config.partitionStrategy = 'semantic' // Keep semantic but disable partitioning
|
|
||||||
}
|
|
||||||
|
|
||||||
return config
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Generate optimization flags based on environment and resources
|
|
||||||
*/
|
|
||||||
private generateOptimizationFlags(
|
|
||||||
environment: string,
|
|
||||||
resources: { availableMemory: number; cpuCores: number }
|
|
||||||
): AutoConfigResult['optimizationFlags'] {
|
|
||||||
return {
|
|
||||||
useMemoryMapping: environment === 'nodejs' && resources.availableMemory > 4 * 1024 * 1024 * 1024,
|
|
||||||
aggressiveCaching: resources.availableMemory > 2 * 1024 * 1024 * 1024,
|
|
||||||
backgroundOptimization: environment !== 'serverless' && resources.cpuCores > 2,
|
|
||||||
compressionLevel: resources.availableMemory < 1024 * 1024 * 1024 ? 'aggressive' :
|
|
||||||
resources.availableMemory < 4 * 1024 * 1024 * 1024 ? 'light' : 'none'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Adapt configuration based on actual dataset analysis
|
|
||||||
*/
|
|
||||||
private adaptConfigurationToData(
|
|
||||||
baseConfig: AutoConfigResult,
|
|
||||||
analysis: DatasetAnalysis
|
|
||||||
): AutoConfigResult {
|
|
||||||
const updatedConfig = { ...baseConfig }
|
|
||||||
|
|
||||||
// Adjust based on actual dataset size
|
|
||||||
if (analysis.estimatedSize !== baseConfig.recommendedConfig.expectedDatasetSize) {
|
|
||||||
const sizeRatio = analysis.estimatedSize / baseConfig.recommendedConfig.expectedDatasetSize
|
|
||||||
|
|
||||||
updatedConfig.recommendedConfig.expectedDatasetSize = analysis.estimatedSize
|
|
||||||
|
|
||||||
// Scale partition size with dataset
|
|
||||||
if (sizeRatio > 2) {
|
|
||||||
updatedConfig.recommendedConfig.maxNodesPerPartition = Math.min(
|
|
||||||
100000,
|
|
||||||
Math.floor(updatedConfig.recommendedConfig.maxNodesPerPartition * 1.5)
|
|
||||||
)
|
|
||||||
updatedConfig.recommendedConfig.semanticClusters = Math.min(
|
|
||||||
32,
|
|
||||||
Math.floor(updatedConfig.recommendedConfig.semanticClusters * 1.5)
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Adjust based on vector dimension
|
|
||||||
if (analysis.vectorDimension) {
|
|
||||||
if (analysis.vectorDimension > 1024) {
|
|
||||||
// High-dimensional vectors - optimize for compression
|
|
||||||
updatedConfig.recommendedConfig.enableCompression = true
|
|
||||||
updatedConfig.optimizationFlags.compressionLevel = 'aggressive'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Adjust based on access patterns
|
|
||||||
if (analysis.accessPatterns === 'read-heavy') {
|
|
||||||
updatedConfig.recommendedConfig.enablePredictiveCaching = true
|
|
||||||
updatedConfig.optimizationFlags.aggressiveCaching = true
|
|
||||||
} else if (analysis.accessPatterns === 'write-heavy') {
|
|
||||||
updatedConfig.recommendedConfig.enablePredictiveCaching = false
|
|
||||||
updatedConfig.optimizationFlags.backgroundOptimization = false
|
|
||||||
}
|
|
||||||
|
|
||||||
return updatedConfig
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Estimate dataset size if not provided
|
|
||||||
*/
|
|
||||||
private estimateDatasetSize(): number {
|
|
||||||
// Start with conservative estimate
|
|
||||||
const environment = this.detectEnvironment()
|
|
||||||
|
|
||||||
switch (environment) {
|
|
||||||
case 'serverless': return 50000
|
|
||||||
case 'nodejs': return 100000
|
|
||||||
default: return 25000
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Reset cached configuration (for testing or manual refresh)
|
|
||||||
*/
|
|
||||||
public resetCache(): void {
|
|
||||||
this.cachedConfig = null
|
|
||||||
this.datasetStats = { estimatedSize: 0 }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Convenience function for quick auto-configuration
|
|
||||||
*/
|
|
||||||
export async function autoConfigureBrainy(hints?: {
|
|
||||||
expectedDataSize?: number
|
|
||||||
s3Available?: boolean
|
|
||||||
memoryBudget?: number
|
|
||||||
}): Promise<AutoConfigResult> {
|
|
||||||
const autoConfig = AutoConfiguration.getInstance()
|
|
||||||
return autoConfig.detectAndConfigure(hints)
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get quick setup configuration for common scenarios
|
|
||||||
*/
|
|
||||||
export async function getQuickSetup(scenario: 'small' | 'medium' | 'large' | 'enterprise') {
|
|
||||||
const autoConfig = AutoConfiguration.getInstance()
|
|
||||||
return autoConfig.getQuickSetupConfig(scenario)
|
|
||||||
}
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue